Re: Weird Crash - "WAITING_TO_RUNNING"

[Nathan <nathan.stocks () gmail com>]

On Thu, Mar 17, 2011 at 5:09 PM, Nathan <nathan.stocks () gmail com> wrote:

> On Sun, Jan 9, 2011 at 10:25 PM, David Fifield <david () bamsoftware com>wrote:
>
> > On Wed, Jan 05, 2011 at 04:07:41PM -0700, Nathan wrote:
> > > > I looked through the logs and identified revisions that are more
> > likely
> > > > to have changed something in NSE betweern 5.35DC1 and 5.36TEST1. I
> > know
> > > > this is asking a lot, but if the --datadir doesn't solve it, please
> > try
> > > > building and testing each of these revisions:
> > > >
> > > > 19514 19515 20266 20267 20632 20633
> > > >
> > > > Starting from a Subversion checkout, you will run
> > > >
> > > > svn update --ignore-externals -r 19514 . nbase ncat nping nsock zenmap
> > > >
> > > > Then build and test, using "--datadir .". Then move on to 19515,
> > 20266,
> > > > etc. (You don't have to check out again every time.)
> > >
> > > No, I was not using "--datadir ." before.
> > >
> > > NOW for the long-awaited testing:
> > >
> > > Here's the command I used to set up nmap each time (from an svn
> > checkout):
> > > $ svn update --ignore-externals -r $REVISION . nbase ncat nping nsock
> > > zenmap && make clean && ./configure && make
> > >
> > > Here's the command I'm using to test each of the versions you indicated:
> > >
> > > $ time /usr/bin/sudo ./nmap --datadir . -sS -sV -T4 -p 1-65535
> > > 74.62.92.70 -P0 -v
> > >
> > > ...while in a separate shell I run the following command to see the
> > > total RAM usage at the END of the run (obviously RAM usage varies
> > > quite a bit during the run):
> > >
> > > $ while /bin/true ; do sleep 1 && echo "---------" && date && cat
> > > /proc/meminfo  | grep MemFree ; done
> > >
> > > Note that nothing else is running on the box at the time except my SSH
> > > session to it.
> > >
> > > ==> r19514: <==
> > >
> > > Elapsed time: 14m12s
> > > Memory usage: 4GB (crashed)
> > >
> > > ==> r19515 <==
> > >
> > > Elapsed time: 13m31s
> > > Memory usage: 4GB (crashed)
> > >
> > > ==> r20266 <==
> > >
> > > Elapsed time: 14m36s
> > > Memory usage: 4GB (crashed)
> > >
> > > ==> r20267 <==
> > >
> > > Elapsed time: 13m58s
> > > Memory usage: 4GB (crashed)
> > >
> > > ==> r20632 <==
> > >
> > > Elapsed time: 14m21s
> > > Memory usage: 4GB (crashed)
> > >
> > > ==> r20633 <==
> > >
> > > Elapsed time: 13m5s
> > > Memory usage: 4GB (crashed)
> > >
> > > -------------
> > >
> > > So what in the world did I miss?  Is this "--data_dir ." option
> > > messing things up?  I wouldn't expect any crashes in revisions after
> > > the nse_main.lua change...and yet they're all crashing.  The 'svn
> > > updates' always had at least a decent number of files listed as having
> > > been modified in the update.
> >
> > This is very strange. Do you have any local changes in your working
> > directory? (Run "svn status" and see if there's anything other than "?"
> > and "X".)
> >
> > I assume the same crashes don't happen without --datadir?
> >
> > David Fifield
>
> Wow, this is got to be one of my slowest responses ever--2 months.  (sorry
> - I was pulled off to other projects)
>
> First, no there were never any local changes in the local working directory
> of nmap.  Running "svn stat" on any of the checked out versions produces:
>
> ---------------
> X      nping
> X      nsock
> X      nbase
> X      ncat
> X      zenmap
>
> Performing status on external item at 'nsock'
>
> Performing status on external item at 'nbase'
>
> Performing status on external item at 'zenmap'
>
> Performing status on external item at 'ncat'
>
> Performing status on external item at 'nping'
> ---------------
>
> Second, no, actually.  When I omit "--data-dir ." -- then the crashes still
> occur.  Perhaps the compiled-in-place-but-not-installed binary is pulling in
> libraries from the other installed versions or something (?)
>
> Anyway, now 5.51 has been released, I'll go test that tomorrow morning and
> see how that behaves.
>
> ~ Nathan

Ok, I've got some results with 5.51 on the same box.  Same command as
before:

time /usr/bin/sudo ./nmap --datadir . -sS -sV -T4 -p 1-65535 74.62.92.70 -P0
-v

(compiled in place - didn't run 'make install')

Elapsed time: 15m13s
Memory usage: 99MB

(same compile, but omitting '--datadir .' on the command, we get pretty much
the same thing)

Elapsed time: 15m21s
Memory usage: 99MB

Now here's the interesting thing, sticking to the original command, if I
edit the nse_main.lua in the local directory and set CONCURRENCY_LIMIT=100,
like you had it when we were testing your patches, we get this:

Elapsed time: 14m16s
Memory usage: 74MB

Fascinating, eh?  If you bring the concurrent threads down, you get a lower
peak memory usage AND faster total run time.  Well, that makes sense,
there's only 8 cores on this box, after all.  Looks like the concurrency was
too high and we were wasting some time and memory just managing all the
stuff in flight.  So what would happen if we brought it even closer?  Here's
what I got when I set CONCURRENCY_LIMIT=10.

Elapsed time: 21m31s
Memory usage: 67MB

Oops!  Even though we continued to trim peak memory usage, our run time just
got slaughtered.  Obviously the scripts aren't using that much processor
time, so we just artifically bottlenecked ourselves with a concurrency limit
set too low.

Fascinating stuff.  Do with it as you will.  I would expect the optimal
concurrency value to fluctuate per load of scripts run and per machine
setup, so I realize there's no globally "best" value.  Not to mention that
this is a corner case where nmap thinks every dang port is open and needs a
couple scripts run against each one.

The nice thing for me is that any way I take it, 5.51 isn't crashing and
doesn't necessarily need any monkey-patching from me.

~ Nathan
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/