Re: [NSE] WHOIS - Attempts to queue coroutines to limit the number of whois queries.

[jah <jah () zadkiel plus com>]

On 03/02/2008 22:47, Fyodor wrote:
> On Sun, Feb 03, 2008 at 01:46:35AM +0000, jah wrote:
>   
> > I'd like to share the attached whois.nse, which performs whois queries 
> > against the five Regional Internet Registries (ARIN, RIPE NCC, APNIC, 
> > LACNIC and AFRINIC) in order to return (a small number of) fields from 
> > the record pertaining to the range of IP address assignments in which 
> > the target IP address resides.
> >     
>
> It is very common that people scan consecutive ranges.  What do you
> think about caching the resulting net ranges? 
Hello folks.

I've been experimenting with some ways to achieve a decent method of
caching results to limit the number of queries sent to whois services to
one, per net range, per whois service (for as many as is needed to find
the correct record).  The simplest method involves caching the net range
and host.ip so that a pointer to host.ip is returned where the current
targets host.ip is in the same range.  The problem with this very simple
method is that the number of queries sent per range depends on the time
taken to cache a result for that range.  This in turn is mostly
dependent upon the responsiveness of the whois services themselves.
About one third of the time the services required to find a record
respond such that only one instance of action() (I'll refer to this as a
coroutine) is running at a time and the result is cached before any more
coroutines start.  This results in one full result and a pointer for any
further targets in that range - which is great.  More often, a service
will take a few seconds to respond which causes the coroutine to yield
for long enough for more coroutines to start, which gives rise to
unwanted queries whilst no cached result exists - still not disastrous. 
On some occasions, a service will take so long to respond that a huge
number of queries are generated and this is very undesirable given the
possibility of having ones IP address banned from using the service.
I've set about trying to find a way to queue coroutines and have them
wait in the queue until either a matching cached result was available
(->return a pointer) or the coroutine arrives at the front of the queue
(->proceed with a query).

My problem is; how to make a coroutine wait?

I've tried coroutine.yield() which does exactly what it says on the tin,
but there's seemingly no way to make a yield() from inside a script
resume() again and the coroutine sits there waiting for a callback from
NSOCK, which it never gets.
I did a little experiment with creating coroutines with the intention of
resuming and yielding them under my control, but this hasn't worked
either.  Whenever NSOCK causes a yield, nmap crashes.
nmap.exe!std::list<thread_record,std::allocator<thread_record>
> ::_Const_iterator<1>::operator*()  Line 213
is what's in the stack whilst debugging - not something I understand
very well with my current knowledge of C/C++, but I presume that the
failure comes from trying to resume a coroutine that is not in the list
of coroutines.

My rather crude solution to the problem is to use NSOCK to cause a yield
by creating a connection to a known filtered port for a fixed time-out
period and putting this in a loop that ends when the coroutine arrives
at the front of the queue or a when a cached result is found.
This very crude hack works remarkably well!  Even if a whois service
takes 20 seconds to respond and 50 coroutines are started as a result,
the goal of only one query per service per range is achieved.  In such a
scenario, however, hundreds of connect-timeouts are generated and I've
seen 60 coroutines waiting in my queue, causing all kinds of havoc (it's
a good job the known filtered port is one of mine, or someone might
think I was trying to SYN flood them...).
Queuing coroutines seems to me to be the best way to limit queries and,
naturally, I need a less crude way to make a coroutine wait in the queue.

In http://nmap.org/nse/nse-api.html#nse-api-networkio it says: "In NSE
you can either program as if you were using a single non blocking socket
or you can program as if your connection is blocking." and I took this
to mean that if I use a blocking connection I can force coroutines to
wait for the socket.  I imagined that they would magically wait until an
in-use socket was free and so I tried declaring nmap.new_socket() in the
scope shared by all the coroutines of a script instance, but was
disappointed when instead, I got complaints that the socket was already
connected and coroutines ended without returning anything useful.
Am I missing something really obvious here or just misunderstanding
what's written?

I'm going to go and do some reading about how to utilise raw packet
network I/O and see if I can get any success there, but I should be most
grateful for any predictions as to its usefulness, in this context, in
the meantime.  If anyone has any thoughts on how I might bend coroutines
to my will without having to do something to the API (which I'm not
currently qualified to attempt - although I am learning C/C++ and should
be so qualified sometime in 2012) then I'd love to hear those thoughts.

Finally, on a not entirely unrelated note, I'm using a small chunk of
code [todotted()] that converts a dword value into a dotted IP address
string - the reverse of ipOps.todword().  For the sake of illustration,
the reason is as follows:  If a range of addresses is scanned such that
the a query results in some message like "Unallocated Resource" it isn't
possible to cache a range for subsequent targets to find and it's not
feasible to predict what range of addresses will produce the same
result.  The Unallocated Resource might be any sized block of
addresses.  So that the script doesn't query for every target address, I
make a fairly safe assumption that anything within 32 addresses will
produce the same result and cache a 32 address range using:
    fake_range = host.ip .. " - " .. todotted(ipOps.todword(host.ip) + 32)

I wonder if this might be helpful in other situations and whether it
might be worth adding to ipOps.lua, in which case I'll submit a patch.

jah



_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org