Re: GSoC feature creeper ideas

[Fyodor <fyodor () insecure org>]

On Mon, Mar 29, 2010 at 07:14:52PM +0000, Brandon Enright wrote:
> Here is a miscellaneous list of ideas it would be nice to see worked on
> during the GSoC.  Most of these are feature-creeper type projects.

Thanks Brandon!  This sort of thing is very useful, especially coming
from someone who scans even more ports than I do!

> * Currently tcpip.cc:resolve() calls getaddrinfo() or gethostbyname()
>   which are both blocking routines.  This makes forward resolution of
>   targets specified by name exceptionally slow.  Nmap (probably
>   nmap_dns.cc)  should be expanded to handle fast, parallel, forward
>   DNS resolution.

One issue is that getaddrinfo/gethostbyname are generally going to be
more reliable than having Nmap parse host configuration files such as
resolv.conf and sending the raw DNS requests.  After all, if
gethostbyname() generally works on a host or else web browsers and the
like won't work either.

For reverse DNS, we do the raw technique anyway.  But that is far more
performance critical as Nmap does far more rDNS, and the repercussions
of missing a DNS name are much less severe in that case.  If Nmap
can't forward resolve a host, it can't scan it.  But an rDNS failure
just means you don't get the rDNS name printed.

I think most scan targets are of forms like 192.168.0.0/16 or even
scanme.nmap.org/28, but usually not big lists of hostnames.  But I
think performance really only matters much in the
big-list-of-hostnames case.

Of course Nmap is used that way at times, like when we did the favicon
survey.  So I agree that speeding things up would be nice.

One safe option would be to add a command line flag for parallel raw
forward DNS.  But the vast majority of people won't know about the
flag and so it won't get used much.  Another option would be to switch
to parallel forward DNS after say 20 forward DNS queries.  That means
that the vast majority of scans will get the most reliable
gethostbyname() technique, while the scans which really need the speed
most will get the parallel system.  We'd need to have an option to
turn it off for cases where it is problematic.

Another option might be to fallback to gethostbyname() if the parallel
system fails.  But the first time the parallel system finds a host, we
disable the fallback.  And the first time gethostbyname() succeeds
after parallel fails, maybe we disable the parallel system for that
run.

Or we could just implement our forward DNS systems in an extremely
portable and bug-free way, but that might be hard given all the
platforms Nmap needs to support and all the different ways servers
respond.

> * Right now when a host times out all data already collected about that
>   host is discarded.  Oftentimes though, the host has already passed
>   the portscan, service fingerprinting, and OS detection phase and most
>   of the NSE scripts have finished.  A single lingering NSE script will
>   cause all information to be lost.  It would be nice to see hosts that
>   timeout at least have all of the information already collected
>   printed and then a note at the bottom of the scan reporting the the
>   host timed out and the information may be incomplete.

That's a funny one because I independently added this to the TODO
yesterday after seeing a post on pen-test
(http://seclists.org/pen-test/2010/Mar/108).  I talked to David today
and we're going to implement it!

> * Sometimes a host starts out responsive during the portscan phase but
>   stops responding.  It would be nice to allow any phase to be aborted
>   via runtime interaction.  Only information collected would be
>   reported.  This is similar to retaining and reporting information in
>   the case of host timeouts as described above.

It's an interesting idea, especially in combination with the partial
scan reporting item mentioned previously.  If this were implemented,
it would have to be in such a way that someone doesn't accidentally
abort a scan phase by pressing the wrong key.  Maybe you'd have to
type a whole word, or say 'y' to a confirmation prompt or something.

> * NSE scripts should be able to produce XML-formatted output.  Right
>   now they are available in XML like ...<hostscript><script id="nbstat"
>   output="...">...  The normal script output is too free-form and
>   difficult to parse.  Also, output can vary based on verbosity and
>   debugging level which requires a parser for each script's output to
>   handle all sorts of special cases.

I'd love to see more details from you on how this might work (e.g. the
API for scripts to use, maybe a few examples of the output scripts
might use, etc.)  Also, if scripts were able to define their own tag
and attribute names, we might run into validation issues unless we add
them to the DTD.  And that could be a major pain.

> * Sometimes you start Nmap with parameters that are just too slow to
>   get the job done while you're sitting there.  It would be nice to
>   allow adjusting the --min-rate parameter with runtime interaction.
>   Of course this is likely to hurt accuracy but sometimes it is worth
>   it to finish job done in 5 minutes rather than 15 when you think you
>   already have interesting results.

Maybe + could speed things up and - could slow them down in terms of
timing level.  But the thing is that we have so many performance
options it's not clear which one(s) we'd want to let the users
control.  And there are potential issues with changing such things at
runtime.

> * Once in a while a misbehaving host will appear to have tens of
>   thousands of ports open.  This causes the service fingerprinting
>   phase to take forever.  It should be possible to specify limits like
>   --max-fingerprint-ports 200 and Nmap will just pick the first 200
>   ports (in any random order) rather than trying to fingerprint
>   thousands.

Maybe the partial-reporting would allow you to set shorter host
timeouts to deal with this sort of issue.

> I may follow up with more ideas as I think of them.

Thanks!

Cheers,
-F
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/