Nmap Development mailing list archives
Re: [NSE] whois.nse
From: Brandon Enright <bmenrigh () ucsd edu>
Date: Wed, 6 Aug 2008 06:54:47 +0000
Sorry for the poor reply quality, I only have my phone right now.Regarding the IPv6 /32 cache, you should probably cache at /48 as that is the size being assinged to organizations. /32s are going to RiRs -- and being chopped into 65536 /48s. Seems like a more logical cache boundary to me.
Brandon Sent from my phone On Aug 6, 2008, at 2:20, jah <jah () zadkiel plus com> wrote:
Hello, Attached is whois.tar.gz which contains whois.nse and whois_dep.nse. The former will work out of the box whereas the latter depends on the version of ipOps.lua posted in [1]. A rundown of what's changed since the last version [2]: * Mutex: Enforces exclusive access to each of the defined whoisservices so that only one connection to each service is allowed ata time. This enables the use of a results cache and reduces the number of queries made when scanning ranges of targets. * IANA Assignments data: Data is downloaded from IANA and cachedlocally. Each thread then performs a lookup against the data, forits target, to determine which service to query. Again, this results in less queries made, especially to ARIN. * Works for IPv4 and 6 targets. Some notes:There's a side-effect to caching results (in order to reduce the numberof queries sent to a whois service) which comes about because of the method of determining whether a result in the cache applies to the current target. When a result is cached, the range of IP addresses to which the result applies (taken from the inetnum/netrange field) is stored in the cache for use as a lookup. Each thread checks whether its Target IP address is within the range stored in any cache entry and if it is, the cachedrecord is returned - rather than perform a query. The effect of this is that the cached entry may be less specific than a record held in a whoisdatabase. For instance, if we cache an entry for target 41.0.0.0 which finds [41.0.0.0-41.255.255.255] then a target IP address of 41.128.1.2 would return the cached entry rather than perform its own query which might have resulted in [41.128.0.0-41.128.255.255] - a more specific record.To mitigate this, a record that applies to an /8 range (or /32 for IPv6) will have a much smaller range stored in the cache for target IP lookupsand thus would not affect all targets in the /8 range. This is a trade-off between accuracy and number of queries made. What if we cache a range of [41.128.0.0-41.128.255.255]? Targets forwhich there is an assignment of [41.128.8.0-41.128.15.255] would returnthe cached record rather than finding their more specific assignmentrecord. We can go on like this right down to the smallest assignment -which for IPv4 is /32. This is clearly not good because, when scanning ranges of targets and caching results, we cannot guarantee the most specific assignment information will be found - which was the basis for the script in the first place! With this in mind, I've implemented "nocache" (--script-args whodb=nocache). This misnomer doesn't stop caching results, but therange of addresses to which any entry in the cache may apply is limitedto a range *no more than* /29 IPv4 (8 hosts) or /32 IPv6 (many hosts). So a target 41.0.0.0 which finds [41.0.0.0-42.255.255.255] will cachethe record, but only targets 41.0.0.0/29 will blindly accept the cached result. 41.0.0.8 would perform its own query, as would .16, .24 and soon. If a query response is the same as one already cached, the thread will still print a pointer to the original full output rather than repeating the same output - helping to keep the host-script results content to a minimum. I chose /29 for IPv4 because after some basic analysis of assignment numbers it looks like roughly 99% of the allocated address space seemsto be within assignments of this size or larger. In practise too, this seems to be the magic number. There isn't the same quantity of data for IPv6, but /32 or larger assignments cover at least 99% of the address space.Obviously this means a lot more queries and the potential for gettingbanned, but helps to discover the most specific assignment information.The host-script results are no longer being manipulated to control whichthread outputs the full result and which show pointers to that result for multiple targets from a single assignment. This was being done purely for aesthetic reasons, but wasn't implemented very well. For instance, if the targets were specified on the command line in ascending order, the first host-script result would show the full output. If the targets were specified in descending output, then the last host-script result would show the full output. If the order specified on the command line wasn't the order in which the scriptthreads executed then a random result would display the full output. If the targets were specified in a random order, well, you get the picture.I decided it wasn't worth the effort to do this properly and that thescript really doesn't need any more complexity. So this means that thefirst thread to cache a result will always be the one that prints the full output. The script is nsedoc documented, but may need altering with any futurechanges to nsedoc that mean I can get rid of the HTML in the comments atthe head of the script. Regards, jah [1] http://seclists.org/nmap-dev/2008/q3/0226.html [2] http://seclists.org/nmap-dev/2008/q2/0148.html <whois.tar.gz> _______________________________________________ Sent through the nmap-dev mailing list http://cgi.insecure.org/mailman/listinfo/nmap-dev Archived at http://SecLists.Org
_______________________________________________ Sent through the nmap-dev mailing list http://cgi.insecure.org/mailman/listinfo/nmap-dev Archived at http://SecLists.Org
Current thread:
- [NSE] whois.nse jah (Aug 05)
- Re: [NSE] whois.nse Brandon Enright (Aug 05)
- Re: [NSE] whois.nse jah (Aug 06)
- Re: [NSE] whois.nse doug (Aug 11)
- Re: [NSE] whois.nse Brandon Enright (Aug 05)