Re: How is whois.nse coming along?

[jah <jah () zadkiel plus com>]

On 29/06/2008 04:33, Fyodor wrote:
> Hi Jah.  How is whois.nse coming along?  This script was actually the
> prime motivator for the NSE mutex system:
>
> http://nmap.org/book/nse-api.html#nse-mutex
>
> Have you had a chance to try updating whois.nse to use mutex so that
> it solves the query concurrency problem and we can add it to
> nmap/scripts ?  Maybe it could do something like:
>
> 1) Check the registry to see if whois information is already available
>    for a netblock containing the target IP.  If so, either print that,
>    or print a pointer to the IP which was used initially to look up
>    the data.  e.g. "Same as ww.x.yyy.z".  If the data can be expressed
>    in a few lines, it might be best to just repeat it.  If it takes 10
>    lines, we should probably just use the pointer.  If the data isn't
>    available, continue.
>
> 2) Try to lock a whois.nse-specific mutex.
>
> 3) Once you have the mutex locked, you probably need to repeat check
>    #1, as the data might have been added while you were waiting on the
>    mutex.  If not, continue.
>
> 4) Do the whois query, add the information to the registry for the
>    given netblock.
>
> 5) Unlock the mutex.  Return the data.
>
> The process could possibly be optimized a bit.  This is just an
> off-the-cuff proposal.  And you may end up that 64.5.5.10 runs before
> 64.5.5.9, so .9 ends up with a forward pointer to the .10 results.  I
> think that is fine.
I have a working script that requires some more testing due to the
changes made in order to utilise mutex - I've really not had a lot of
time for this lately (or anything else interesting), but it is at the
top of my nmap list of things to do.

The script now uses a locally cached copy of IANA's IPv4 Global Unicast
Address Assignments [1] to determine which service to query by matching
the target against the assignment prefix ( nnn/8 ). The file is
currently being stored alongside nmap-services et al. since the script
uses nmap.fetchfile() to get a directory path.  I wonder if anyone has
any opinions on whether this is a good or bad place to store it - it's
certainly the most convenient and should be satisfactory across all
platforms, but if Kris introduces a data directory for nselib data files
[2], maybe it could go there.

The script utilises a mutex for each whois service defined which allows
both queuing for a given service and concurrent queries at different
services.  This, in addition to the cached assignments data, means that
the number of queries is reduced and that the script finishes as quickly
as is possible.

During testing using Patrick's branch of svn, I found that threads were
no longer starting (or rather, resuming) in the target order defined on
the commandline and so, when scanning a range of targets from a given
assignment, the whois queries were being made for a target that wasn't
necessarily the first one in the range.  This made the presentation of
pointers to the host script result a bit ugly:

    Interesting ports on 116.128.0.1:
    Host script results:
    |_ WHOIS: see the result for 116.128.0.3

    Interesting ports on 116.128.0.2:
    Host script results:
    |_ WHOIS: see the result for 116.128.0.3

    Interesting ports on 116.128.0.3:
    Host script results:
    |  WHOIS: Record found at whois.apnic.net
    |  inetnum: 116.128.0.0 - 116.191.255.255
    |  netname: NGTC
    |  descr: New Guoxin Telecom Corporation
    |_ country: CN

    Interesting ports on 116.128.0.4:
    Host script results:
    |_ WHOIS: see the result for 116.128.0.3

    Interesting ports on 116.128.0.5:
    Host script results:
    |_ WHOIS: see the result for 116.128.0.3

which is more of a pain the more targets scanned.  Since this is really
only an issue of presentation, it was fairly trivial to alter the
procedure for caching results and subsequently checking the cache to
ensure that the first target in range always returns the host script
result.  Strangely enough, since the merge of Patrick's branch, threads
seem to be resuming in order again, so we're now doubly assured of a
nice ordered result!

I'll get the bulk of the testing done and post the script for people to
play with and then I can do any code tidying (I'm sure I can make some
parts a bit more terse) and nsedoc in slowtime.  It'd be good to get the
testing done in time for the next release, but that could be pushing it
given that Nmap 4.69 is probably only days away :)

Regards,

jah


[1] - http://www.iana.org/assignments/ipv4-address-space
[2] - http://seclists.org/nmap-dev/2008/q2/0740.html


_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org