Re: Weird Crash - "WAITING_TO_RUNNING"

[Nathan <nathan.stocks () gmail com>]

On Mon, Nov 22, 2010 at 4:02 PM, David Fifield <david () bamsoftware com> wrote:
> On Mon, Nov 22, 2010 at 03:43:29PM -0700, Nathan wrote:
> > On Wed, Nov 17, 2010 at 12:34 PM, David Fifield <david () bamsoftware com> wrote:
> > > On Mon, Nov 15, 2010 at 02:53:59PM -0700, Nathan wrote:
> > > > > > > On Tue, Nov 09, 2010 at 10:32:19AM -0800, David Fifield wrote:
> > > > > > > Nathan, please try out this nse_main.lua. It's has a quick and dirty
> > > > > > > modification that prevents the creation of more than 100 script threads
> > > > > > > at a time. Run the scan so that it creates lots of spurious open ports
> > > > > > > like before. It should not use up all your memory and should eventually
> > > > > > > finish.
> > > > > > >
> > > > > > > I think we will actually set the limit higher than 100 in practice.
> > > >
> > > > Okay, it didn't change the accuracy (we didn't expect it too), so it
> > > > still thought all 65k+ ports were open.  But it certainly limited RAM
> > > > usage and actually finished!
> > > >
> > > > It was using about 55MB RAM when it ended, and it took 5m23s -- a huge
> > > > improvement over using 4GB of RAM and crashing!
> > >
> > > I just committed this as r21084. I increased the limit from 100 to
> > > 1,000. I also made some code changes so please test it again and see if
> > > it works.
> > >
> > > Patrick D.: I defined a new file-level local variable CONCURRENCY_LIMIT
> > > in nse_main.lua. Is this the best place for it?
> >
> > Okay, so I tested 2.36TEST2, which should be up to r21143, so it
> > should include your change from 100 to 1,000 concurrent service
> > detection threads.
> >
> > => The run with CONCURRENCY_LIMIT=100 BEFORE these changes took about
> > 55MB RAM and 5m23s.
> > => The run with CONCURRENCY_LIMIT=1000 AFTER these changes took about
> > 95MB RAM and 14m16s
> >
> > and then I edited nse_main.lua and lowered the limit to 100 to see if
> > that would restore the better performance, and got:
> >
> > => The run with CONCURRENCY_LIMIT=100 AFTER these changes took about
> > 69MB RAM and 12m56s
> >
> > So, assuming that ALL of the following are true:
> > - my test server is sane (it should be, I've pretty much only been
> > using it to test these nmap changes the last couple weeks)
> > - the Internet route between me and my target is pretty much the same
> > (no way to know, because I didn't take a traceroute)
> > - nothing drastic changed on the target itself (I don't think it has,
> > but who knows...)
> > - no other change to nmap affected this (beats me)
> >
> > THEN
> >
> > It appears that your latest changes affected things quite adversely,
> > and raising the concurrency limit to 1000 didn't help either.
> >
> > I think your older version worked much faster and used much less RAM
> > and that we should revert to it.  I will use it on my servers, at
> > least, for the time being.
>
> Please try to reproduce the 5-minute time with the older version. I
> really don't expect the few changes we made to affect timing like this.
> I suspect that this is sensitive to the number of open ports that are
> found, which will depend on network conditions and probably isn't easily
> reproducible.
>
> If you consistently get different results over, say, three runs, then
> I'll look at it closer.
>
> David Fifield

Sure thing.  It looks pretty consistent to me.  Here's three runs on
the same server with 5.35DC1 with your custom nse_main.lua:

Scan ending at Mon Nov 22 16:45:14 MST 2010:
5m12s - 53392KB RAM

Scan ending at Mon Nov 22 16:51:06 MST 2010:
5m8s - 54836KB RAM

Scan ending at Mon Nov 22 16:58:16 MST 2010
5m26s - 55980KB RAM

(Note that I didn't run these end-on-end, I typed some stuff into this
email in between each scan)

~ Nathan
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/