Nmap Development mailing list archives
Re: [NSE] WHOIS - Attempts to queue coroutines to limit the number of whois queries.
From: jah <jah () zadkiel plus com>
Date: Fri, 15 Feb 2008 01:54:53 +0000
On 03/02/2008 22:47, Fyodor wrote:
On Sun, Feb 03, 2008 at 01:46:35AM +0000, jah wrote:I'd like to share the attached whois.nse, which performs whois queries against the five Regional Internet Registries (ARIN, RIPE NCC, APNIC, LACNIC and AFRINIC) in order to return (a small number of) fields from the record pertaining to the range of IP address assignments in which the target IP address resides.It is very common that people scan consecutive ranges. What do you think about caching the resulting net ranges?
Hello folks. I've been experimenting with some ways to achieve a decent method of caching results to limit the number of queries sent to whois services to one, per net range, per whois service (for as many as is needed to find the correct record). The simplest method involves caching the net range and host.ip so that a pointer to host.ip is returned where the current targets host.ip is in the same range. The problem with this very simple method is that the number of queries sent per range depends on the time taken to cache a result for that range. This in turn is mostly dependent upon the responsiveness of the whois services themselves. About one third of the time the services required to find a record respond such that only one instance of action() (I'll refer to this as a coroutine) is running at a time and the result is cached before any more coroutines start. This results in one full result and a pointer for any further targets in that range - which is great. More often, a service will take a few seconds to respond which causes the coroutine to yield for long enough for more coroutines to start, which gives rise to unwanted queries whilst no cached result exists - still not disastrous. On some occasions, a service will take so long to respond that a huge number of queries are generated and this is very undesirable given the possibility of having ones IP address banned from using the service. I've set about trying to find a way to queue coroutines and have them wait in the queue until either a matching cached result was available (->return a pointer) or the coroutine arrives at the front of the queue (->proceed with a query). My problem is; how to make a coroutine wait? I've tried coroutine.yield() which does exactly what it says on the tin, but there's seemingly no way to make a yield() from inside a script resume() again and the coroutine sits there waiting for a callback from NSOCK, which it never gets. I did a little experiment with creating coroutines with the intention of resuming and yielding them under my control, but this hasn't worked either. Whenever NSOCK causes a yield, nmap crashes. nmap.exe!std::list<thread_record,std::allocator<thread_record>
::_Const_iterator<1>::operator*() Line 213
is what's in the stack whilst debugging - not something I understand very well with my current knowledge of C/C++, but I presume that the failure comes from trying to resume a coroutine that is not in the list of coroutines. My rather crude solution to the problem is to use NSOCK to cause a yield by creating a connection to a known filtered port for a fixed time-out period and putting this in a loop that ends when the coroutine arrives at the front of the queue or a when a cached result is found. This very crude hack works remarkably well! Even if a whois service takes 20 seconds to respond and 50 coroutines are started as a result, the goal of only one query per service per range is achieved. In such a scenario, however, hundreds of connect-timeouts are generated and I've seen 60 coroutines waiting in my queue, causing all kinds of havoc (it's a good job the known filtered port is one of mine, or someone might think I was trying to SYN flood them...). Queuing coroutines seems to me to be the best way to limit queries and, naturally, I need a less crude way to make a coroutine wait in the queue. In http://nmap.org/nse/nse-api.html#nse-api-networkio it says: "In NSE you can either program as if you were using a single non blocking socket or you can program as if your connection is blocking." and I took this to mean that if I use a blocking connection I can force coroutines to wait for the socket. I imagined that they would magically wait until an in-use socket was free and so I tried declaring nmap.new_socket() in the scope shared by all the coroutines of a script instance, but was disappointed when instead, I got complaints that the socket was already connected and coroutines ended without returning anything useful. Am I missing something really obvious here or just misunderstanding what's written? I'm going to go and do some reading about how to utilise raw packet network I/O and see if I can get any success there, but I should be most grateful for any predictions as to its usefulness, in this context, in the meantime. If anyone has any thoughts on how I might bend coroutines to my will without having to do something to the API (which I'm not currently qualified to attempt - although I am learning C/C++ and should be so qualified sometime in 2012) then I'd love to hear those thoughts. Finally, on a not entirely unrelated note, I'm using a small chunk of code [todotted()] that converts a dword value into a dotted IP address string - the reverse of ipOps.todword(). For the sake of illustration, the reason is as follows: If a range of addresses is scanned such that the a query results in some message like "Unallocated Resource" it isn't possible to cache a range for subsequent targets to find and it's not feasible to predict what range of addresses will produce the same result. The Unallocated Resource might be any sized block of addresses. So that the script doesn't query for every target address, I make a fairly safe assumption that anything within 32 addresses will produce the same result and cache a 32 address range using: fake_range = host.ip .. " - " .. todotted(ipOps.todword(host.ip) + 32) I wonder if this might be helpful in other situations and whether it might be worth adding to ipOps.lua, in which case I'll submit a patch. jah _______________________________________________ Sent through the nmap-dev mailing list http://cgi.insecure.org/mailman/listinfo/nmap-dev Archived at http://SecLists.Org
Current thread:
- [NSE] WHOIS jah (Feb 02)
- Re: [NSE] WHOIS Fyodor (Feb 03)
- Re: [NSE] WHOIS jah (Feb 04)
- Re: [NSE] WHOIS Diman Todorov (Feb 04)
- Re: [NSE] WHOIS - Attempts to queue coroutines to limit the number of whois queries. jah (Feb 14)
- Re: [NSE] WHOIS jah (Feb 04)
- Re: [NSE] WHOIS Kris Katterjohn (Feb 04)
- <Possible follow-ups>
- Re: [NSE] WHOIS 4N9e Gutek (Feb 03)
- Re: [NSE] WHOIS Fyodor (Feb 03)