Re: NSE: odd output, testing, etc

["Patrick Donnelly" <batrick.donnelly () gmail com>]

On Sun, Dec 28, 2008 at 6:34 PM, David Fifield <david () bamsoftware com> wrote:
> On Sun, Dec 21, 2008 at 05:10:59AM -0700, Patrick Donnelly wrote:
> > Brandon, Ron, and I have been discussing these problems privately by
> > email to track the causes. Here are the results:
> >
> > > * NSE is overly aggressive with parallelism.  It isn't unusual for NSE
> > >  to report more than 2000 active NSE scripts.  When this happens Lua
> > >  seems to thrash and NSE scanning slows to a crawl.  I think this has
> > >  the ability to trigger the "lock, (null), <int>, tcp, ERROR" errors
> > >  describe below.
> >
> > This shouldn't be a problem as the default of 10 script threads may
> > use network resources, all the other 1990 threads are blocked. Memory
> > is also not a concern as each thread only consumes about 1 KB of
> > memory. NSE shouldn't slow to a crawl. I assume most of the scripts
> > were blocked on a socket. I haven't observed any problems with NSE
> > encountering a deadlock or infinite loop since the problems discussed
> > in [1][2].
>
> So it sounds like the scripts are running and blocking fine. The only
> problem is that they tend to time out before they get a chance to do
> anything. Brandon said that after one round of scanning with more than
> 300 script threads running, in the next round NSE locks immediately.
> Could something about timing out scripts be corrupting the Lua state?
> The problem goes away when r11122 is reverted. That was the change that
> made use of a persistent Lua state.

I need more information than NSE locking up on the second scan to
determine the cause. As far as I know Brandon hasn't had this problem
anymore after I corrected the threads improperly finishing because of
time outs.

I could speculate that some of the scripts may use mutexes with the
same object as the last round of scripts. If those last scripts timed
out with a lock on the needed mutex then there would be no progress. A
garbage collection cycle in between scans may fix the problem to allow
all the mutex functions to be collected (assuming no script thread
saved a persistent reference somewhere).

-- 
-Patrick Donnelly

"One of the lessons of history is that nothing is often a good thing
to do and always a clever thing to say."

-Will Durant

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org