Nmap Development mailing list archives
Re: Weird Crash - "WAITING_TO_RUNNING"
From: David Fifield <david () bamsoftware com>
Date: Mon, 22 Nov 2010 16:42:45 -0800
On Mon, Nov 22, 2010 at 05:37:54PM -0700, Nathan wrote:
On Mon, Nov 22, 2010 at 5:11 PM, David Fifield <david () bamsoftware com> wrote:On Mon, Nov 22, 2010 at 04:59:24PM -0700, Nathan wrote:On Mon, Nov 22, 2010 at 4:02 PM, David Fifield <david () bamsoftware com> wrote:On Mon, Nov 22, 2010 at 03:43:29PM -0700, Nathan wrote:On Wed, Nov 17, 2010 at 12:34 PM, David Fifield <david () bamsoftware com> wrote:On Mon, Nov 15, 2010 at 02:53:59PM -0700, Nathan wrote:On Tue, Nov 09, 2010 at 10:32:19AM -0800, David Fifield wrote: Nathan, please try out this nse_main.lua. It's has a quick and dirty modification that prevents the creation of more than 100 script threads at a time. Run the scan so that it creates lots of spurious open ports like before. It should not use up all your memory and should eventually finish. I think we will actually set the limit higher than 100 in practice.Okay, it didn't change the accuracy (we didn't expect it too), so it still thought all 65k+ ports were open. But it certainly limited RAM usage and actually finished! It was using about 55MB RAM when it ended, and it took 5m23s -- a huge improvement over using 4GB of RAM and crashing!I just committed this as r21084. I increased the limit from 100 to 1,000. I also made some code changes so please test it again and see if it works. Patrick D.: I defined a new file-level local variable CONCURRENCY_LIMIT in nse_main.lua. Is this the best place for it?Okay, so I tested 2.36TEST2, which should be up to r21143, so it should include your change from 100 to 1,000 concurrent service detection threads. => The run with CONCURRENCY_LIMIT=100 BEFORE these changes took about 55MB RAM and 5m23s. => The run with CONCURRENCY_LIMIT=1000 AFTER these changes took about 95MB RAM and 14m16s and then I edited nse_main.lua and lowered the limit to 100 to see if that would restore the better performance, and got: => The run with CONCURRENCY_LIMIT=100 AFTER these changes took about 69MB RAM and 12m56s So, assuming that ALL of the following are true: - my test server is sane (it should be, I've pretty much only been using it to test these nmap changes the last couple weeks) - the Internet route between me and my target is pretty much the same (no way to know, because I didn't take a traceroute) - nothing drastic changed on the target itself (I don't think it has, but who knows...) - no other change to nmap affected this (beats me) THEN It appears that your latest changes affected things quite adversely, and raising the concurrency limit to 1000 didn't help either. I think your older version worked much faster and used much less RAM and that we should revert to it. I will use it on my servers, at least, for the time being.Please try to reproduce the 5-minute time with the older version. I really don't expect the few changes we made to affect timing like this. I suspect that this is sensitive to the number of open ports that are found, which will depend on network conditions and probably isn't easily reproducible. If you consistently get different results over, say, three runs, then I'll look at it closer.Sure thing. It looks pretty consistent to me. Here's three runs on the same server with 5.35DC1 with your custom nse_main.lua: Scan ending at Mon Nov 22 16:45:14 MST 2010: 5m12s - 53392KB RAM Scan ending at Mon Nov 22 16:51:06 MST 2010: 5m8s - 54836KB RAM Scan ending at Mon Nov 22 16:58:16 MST 2010 5m26s - 55980KB RAM (Note that I didn't run these end-on-end, I typed some stuff into this email in between each scan)And if you're still available, can you do one more with 5.36TEST2? David Fifield5.36TEST2 with CONCURRENCY_LIMIT lowered to 100: Scan ending at Mon Nov 22 17:29:20 MST 2010 13m26 - 69644KB RAM I've got to run -- I'm late leaving work, but I can rerun it without the altered CONCURRENCY_LIMIT tomorrow (I forgot I had that alteration in there).
That's fine; I think this is conclusive enough. David Fifield _______________________________________________ Sent through the nmap-dev mailing list http://cgi.insecure.org/mailman/listinfo/nmap-dev Archived at http://seclists.org/nmap-dev/
Current thread:
- Re: Weird Crash - "WAITING_TO_RUNNING", (continued)
- Re: Weird Crash - "WAITING_TO_RUNNING" Nathan (Nov 15)
- Message not available
- Fwd: Re: Weird Crash - "WAITING_TO_RUNNING" (Action Required) Nathan (Nov 15)
- Re: Weird Crash - "WAITING_TO_RUNNING" Nathan (Nov 15)
- Re: Weird Crash - "WAITING_TO_RUNNING" Nathan (Nov 15)
- Re: Weird Crash - "WAITING_TO_RUNNING" David Fifield (Nov 17)
- Re: Weird Crash - "WAITING_TO_RUNNING" Nathan (Nov 22)
- Re: Weird Crash - "WAITING_TO_RUNNING" David Fifield (Nov 22)
- Re: Weird Crash - "WAITING_TO_RUNNING" Nathan (Nov 22)
- Re: Weird Crash - "WAITING_TO_RUNNING" David Fifield (Nov 22)
- Re: Weird Crash - "WAITING_TO_RUNNING" Nathan (Nov 22)
- Re: Weird Crash - "WAITING_TO_RUNNING" David Fifield (Nov 22)
- Re: Weird Crash - "WAITING_TO_RUNNING" David Fifield (Nov 26)
- Re: Weird Crash - "WAITING_TO_RUNNING" Nathan (Dec 03)
- Re: Weird Crash - "WAITING_TO_RUNNING" David Fifield (Dec 03)
- Re: Weird Crash - "WAITING_TO_RUNNING" David Fifield (Dec 29)
- Re: Weird Crash - "WAITING_TO_RUNNING" Patrick Donnelly (Dec 13)
- Re: Weird Crash - "WAITING_TO_RUNNING" Nathan (Nov 08)