Re: Weird Crash - "WAITING_TO_RUNNING"

[David Fifield <david () bamsoftware com>]

On Mon, Nov 08, 2010 at 05:11:27PM -0500, Patrick Donnelly wrote:
> On Fri, Nov 5, 2010 at 8:02 PM, David Fifield <david () bamsoftware com> wrote:
> > Putting aside the question of where all the open ports are coming from
> > (I don't think that's Nmap's fault), we should do something about Nmap
> > spawning too many scripts at once. I think it is the case that Nmap
> > allocates memory for all script threads at once and runs them all at
> > once.
>
> NSE will create a thread (coroutine) (and reexecute script file
> closure) for each host, for each open port of the host, for each
> script. This can be potentially huge number but many of these are
> garbage collected since the (port|host)rule returns false. For the
> ones that had a rule function return true, we keep the thread and its
> associated data. This can be very large when there are many ports open
> and scripts (like the skypev2-version) have "lax" portrules.
>
> We do run all the scripts at the start but the majority of them wait
> while trying to connect. This adds some to the memory cost but not
> much I would think (compared to setup, above).
>
> > We could put a limit on the number of scripts that run at once,
> > either doing them in batches like Nmap's host groups, or running up to a
> > certain number and then only starting a new thread after an existing one
> > has finished.
>
> The "optimal" fix would be to create a generator that creates threads
> on demand (most of the main function in nse_main.lua). Dependencies
> (runlevels internally) would be a little tricky.

This is the same architecture (generator) I was thinking of.

> Personally, I think this can be solved by user chosen host group
> sizes. Also, I dislike that skypev2-version has such an easily matched
> portrule...

That's part of the problem. However I think it's reasonable to have
scripts that want to run for every single port, for accounting purposes
or whatever.

David Fifield
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/