Re: [nmap-svn] r12765 - nmap/docs

[Fyodor <fyodor () insecure org>]

On Mon, Mar 30, 2009 at 08:47:32PM -0600, Patrick Donnelly wrote:
> > +o Optimize NSE Performance--e.g. measure the current performance and
> > +  see what can be improved in terms of scheduling scan threads,
> > +  determining how many to run concurrently, looking at CPU load items,
> > +  etc.
>
> It is my belief that NSE has reached a point where it no longer needs
> much if any look at performance.

Oh, I think every part of Nmap can due with a regular performance
review.  David spent at least a month last year looking at port
scanning (ultra_scan) performance and made some important
improvements.  And this is despite the fact that ultra_scan was Nmap's
3rd generation scanning engine and had more than a decade of history
(counting the previous engines).

> The current system will begin by
> running all threads (which could be thousands) but will finish this
> process rapidly. Nearly all threads will end up in a waiting queue 

I hope these don't use up too much RAM or other resources in the cases
where there is an enormous number of scripts queued.  Some people use
giant hostgroups with tons of ports open.  And I expect the number of
scripts shipped with Nmap to continue to grow.

> and only --max-parallelism (usually 10) scripts will run at any time. Once
> NSE reaches this stable state of running only --max-parallelism
> scripts,

I think deciding on the best --max-parallelism (and whether to vary
the # of concurrent scripts in certain conditions) is one of the most
important decisions of the performance review.  Who says we can only
run 10 at a time?  Why not 20?  Why not 50?  I think it will take
empirical study to determine the best numbers.

I'm not advocating a detailed NSE performance review because I think
the performance is bad.  It's because I think NSE is critically
important and deserves even more performance attention than it has
received in the past.  The idea is to figure out any clever ways we
can come up with to help people get the data they want from NSE
faster.  While still being accurate, reliable, and easy to use and
maintain.

> the majority of execution time is used by the scripts
> themselves (Except with the new garbage collection scheme which can be
> removed once the nsock library is fixed).
>
> I believe for NSE performance (that is the length of the scan, not the
> engine itself) to be improved, we need to look at better communication
> between scripts that do the same thing (reading a banner in
> particular).

Could be!  That is the sort of thing that a performance review can
look at.  Though it can be examined and tested and implemented without
such a review too.

> > +o Consider whether we should include some sort of NSE debugger.  Or we
> > +  could include something simpler.  For example, some developers (such
> > +  as Ron) already make use of Patrick's traceback.nse in their
> > +  experimental trees.
>
> One of the nice things about the new Lua script engine is a developer
> can _very_ easily change the engine to include new debugging output
> without recompiling the nmap. I think there is room for more standard
> debugging info (such as outputting a stack trace when a script
> finishes) to be added throughout the NSE scan but I think we should

Nice.  You probably noticed that this moved up to near the top of the file:

o Merge patrick/nse-lua-merge for easier-to-maintain and simpler
  codebase once David and Patrick are happy with it. [David]

Cheers,
-F

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org