Re: CIDR range capabilities for ipOps (NSE Leftovers - Part 3)

[Brendan Byrd <sineswiper () gmail com>]

On Tue, Sep 18, 2012 at 3:08 PM, David Fifield <david () bamsoftware com> wrote:
> Thank you for doing this. I'm still disinclined to take the patch. I
> don't see the need for it, apart from the SNMP changes which I didn't
> quite understand and which have dropped off my radar. (You can resend
> those if you think they've been unfairly overlooked.)

Like I said earlier, that was only part of that larger patch that I
split up.  Maybe I should send over the main SNMP script that uses all
of this (attached).  My hope is that a script like this would eclipse
"autodiscovery" programs that people spend way too much money on,
while still being better than the brute force method of "ping this
entire /8 block".  NMap can be smarter than that.

Besides, while I know it's a lot of code to take in, I don't see why
the ipOps code wouldn't be useful in other scripts.  Many of the
changes also enhance IPv4 <-> IPv6 interoperability, which will be
pretty critical later on.  (Side note: RIPE just dived into its last
/8 block a few days ago...)

> This patch seems to be working correctly. At least, I tried
>         ipOps.get_ranges_from_ips("0.0.0.1", "255.255.255.254")
> and it seemed to do the right thing.

On a related note, what is your test platform for these?  I had to set
up that "standalone" placeholder to get things to work outside of
NMap, and I don't know where test cases or any testing is done for Lua
or NMap in general.

> Also, I think we may be approaching this the wrong way. I don't think
> that lists of ranges is the best way to represent a set of addresses
> like this. The problem you're trying to solve is not "manage a bunch of
> CIDR ranges," it's "prevent addresses from being scanned twice," which
> to me is really "have a space-efficient data structure representing a
> set of addresses, in which insertion and testing for membership are
> fast, possibly using the assumption that some insertions will be
> contiguous blocks of addresses."

Yeah, and you basically repeated yourself there.  In other words, the
best way to have "a space-efficient data structure representing a set
of addresses" is to manage them as bunches of CIDR ranges.  Also, with
IP blocks being passed in various forms, the goal of "preventing
addresses from being scanned twice" involves making sure that two IP
blocks of slightly different forms don't have duplication problems.
(For example, "192.168.0.0/20" vs. "192.168.1.0/24".  The latter is
swallowed by the former.)

At least, IMHO; I'm open to suggestions.  (One of which could be "C
libraries already do a better job of this"...)

> We already have to do a similar thing with Nmap's --exclude list, and
> for that we use a structure called an addrset from Nbase. It's not quite
> suitable for preventing duplicates because it does set union by adding
> elements to a list. But I think it has a better level of abstraction: I
> want to say
>         remember_i_already_scanned_this("192.168.0.0/24")
> and not think about a list of ranges.

One, Lua doesn't have access to the exclude list, which is a problem.
(I think we talked about this before.)

Two, thinking about the range really is critical.  In your example
above, if I already scanned "192.168.0.0/16", will that command say
true or false?  If it's false, then that's the difference between
re-scanning 253 addresses and not re-scanning 253 addresses (or much
worse as the blocks get bigger).

Three, I would argue that addrset should be closer to the CIDR-mindful
system in this ipOps patch than going the other way around.  And
again, I don't mean to sound alarmist, but IPv4/6 interaction is going
to get _very_ heavy very soon.  Mere string comparisons just aren't
going to cut it any more.  And if you think a re-scan of 253 addresses
is laughable, try re-scanning one of those "billions and billions of
IPs" in a typical IPv6 block :)

> Brandon Enright has criticized the performance of addrset on large
> exclude lists; maybe he has ideas for a better representation that would
> serve the duplicates purpose as well.

Like I said, I'm open to any constructive criticism and new
suggestions on how to implement this.

-- 
Brendan Byrd/SineSwiper <SineSwiper () GMail com>

Attachment: snmp-routing.nse
Description:

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/