Re: [nmap-svn] r12765 - nmap/docs

[Patrick Donnelly <batrick () batbytes com>]

On Mon, Mar 30, 2009 at 9:00 PM, Fyodor <fyodor () insecure org> wrote:
> On Mon, Mar 30, 2009 at 08:47:32PM -0600, Patrick Donnelly wrote:
> > > +o Optimize NSE Performance--e.g. measure the current performance and
> > > +  see what can be improved in terms of scheduling scan threads,
> > > +  determining how many to run concurrently, looking at CPU load items,
> > > +  etc.
> >
> > It is my belief that NSE has reached a point where it no longer needs
> > much if any look at performance.
>
> Oh, I think every part of Nmap can due with a regular performance
> review.  David spent at least a month last year looking at port
> scanning (ultra_scan) performance and made some important
> improvements.  And this is despite the fact that ultra_scan was Nmap's
> 3rd generation scanning engine and had more than a decade of history
> (counting the previous engines).

There's no doubt the speed of the scan can be improved, I'm merely
pointing out that NSE consumes _very_ little execution time once it
reaches the stable state. It is the scripts and libraries that must be
improved.

> > The current system will begin by
> > running all threads (which could be thousands) but will finish this
> > process rapidly. Nearly all threads will end up in a waiting queue
>
> I hope these don't use up too much RAM or other resources in the cases
> where there is an enormous number of scripts queued.  Some people use
> giant hostgroups with tons of ports open.  And I expect the number of
> scripts shipped with Nmap to continue to grow.

Each thread consumes roughly 1 Kilobyte of memory. I expect eventually
we will need to take this into account but it hasn't shown to be a
problem yet. Doing the simple math, you could have 10,000 scripts for
every 10 megabytes. I don't think we will often see larger than that
number of scripts.

> > and only --max-parallelism (usually 10) scripts will run at any time. Once
> > NSE reaches this stable state of running only --max-parallelism
> > scripts,
>
> I think deciding on the best --max-parallelism (and whether to vary
> the # of concurrent scripts in certain conditions) is one of the most
> important decisions of the performance review.  Who says we can only
> run 10 at a time?  Why not 20?  Why not 50?  I think it will take
> empirical study to determine the best numbers.

Again, this is a decision made by a library (nsock) not NSE. I believe
we could definitely increase the value as 10 makes large scans take a
while to complete.

> I'm not advocating a detailed NSE performance review because I think
> the performance is bad.  It's because I think NSE is critically
> important and deserves even more performance attention than it has
> received in the past.  The idea is to figure out any clever ways we
> can come up with to help people get the data they want from NSE
> faster.  While still being accurate, reliable, and easy to use and
> maintain.

Ok.

> > the majority of execution time is used by the scripts
> > themselves (Except with the new garbage collection scheme which can be
> > removed once the nsock library is fixed).
> >
> > I believe for NSE performance (that is the length of the scan, not the
> > engine itself) to be improved, we need to look at better communication
> > between scripts that do the same thing (reading a banner in
> > particular).
>
> Could be!  That is the sort of thing that a performance review can
> look at.  Though it can be examined and tested and implemented without
> such a review too.

I think my main point of this e-mail is that NSE, the engine, is not
the bottleneck. Scripts, libraries, and communication need to be
looked at for improvement to decrease scan times. I wanted to be sure
you (and David) knew where the bottlenecks lie.

-- 
-Patrick Donnelly

"One of the lessons of history is that nothing is often a good thing
to do and always a clever thing to say."

-Will Durant

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org