Re: Weird Crash - "WAITING_TO_RUNNING"

[Nathan <nathan.stocks () gmail com>]

On Tue, Nov 2, 2010 at 4:45 PM, David Fifield <david () bamsoftware com> wrote:
> On Tue, Nov 02, 2010 at 04:24:41PM -0600, Nathan wrote:
> > On Tue, Nov 2, 2010 at 2:17 PM, Patrick Donnelly <batrick () batbytes com> wrote:
> > > Hi Nathan,
> > >
> > > On Mon, Nov 1, 2010 at 4:12 PM, Nathan <nathan.stocks () gmail com> wrote:
> > > > Anyone know why I would get the following crash using the options "-sS
> > > > -sV -T4 -p 1-65535" with nmap 5.35DC1?
> > > >
> > > > "nse_restore: WAITING_TO_RUNNING error!
> > > > not enough memory"
> > > >
> > > > I've got 16GB of RAM on the box (2.6.23 linux kernel), with 10GB
> > > > free...so "not enough memory" must not be talking about total system
> > > > RAM...
> > >
> > > It is referring to system memory. I don't know why you got that error
> > > though. Do you have ulimit turned on? Are you able to reproduce this
> > > error repeatedly and reliably?
> >
> > To my knowledge, ulimit is not turned on in any fashion:
> >
> > $ ulimit
> > unlimited
> >
> > I can indeed reproduce this error repeatedly and reliably.  Here's the
> > full output of it crashing three times in a row.  Note that it takes
> > about 13 minutes until it crashes.
> >
> > ====> Here's what I did to watch the memory during the third run.
> > You'll notice that while a ton of memory was used, there was still
> > over 2GB completely free.  There was also a lot of memory in the disk
> > cache that could have been used if necessary <====
> >
> > $ while /bin/true ; do sleep 1 && echo "---------" && date && cat
> > /proc/meminfo  | grep MemFree ; done
> > [snipped about 13 minutes worth of output right here]
> > ---------
> > Tue Nov  2 16:04:55 MDT 2010
> > MemFree:       2365096 kB              <==== (crash!)
> > ---------
> > Tue Nov  2 16:04:56 MDT 2010
> > MemFree:       5509820 kB
>
> 5509820 KB - 2365096 - KB is right at 3 GB, so you could be running out
> of per-process address space. Is this on a 32-bit machine?
>
> It looks like there's a loop somewhere that is allocating memory without
> bound.
>
> I'd like you to try running each potential script individually to see
> which one is responsible. One way is to run Nmap with the -d2 option,
> wait five minutes or so, and then press Enter in the terminal. With -d,
> Enter causes Nmap to print a stack trace of all running scripts. The
> stack trace will look like this:
>
> NSE: Running: 'html-title' (thread: 0x8feea48)
>        stack traceback:
>                [C]: in function 'receive'
>                <More...>
>
> (Another way is to run "nmap -sV -d2" to get a list of scripts, then try
> running each of them individually with --script. But I think the "press
> Enter" method will be faster.)
>
> David Fifield

"Is this on a 32-bit machine?" -- The hardware is 64-bit capable, but
the old version of linux on it is compiled in 32-bit, I think.  The OS
can address more than 4GB RAM, but the stuff running on it seems to
act very 32-bit.  Python's maxint is 32-bit, anyway:

$ python
Python 2.5.2 (r252:60911, Oct 30 2008, 18:03:18)
[GCC 4.1.2 (Gentoo 4.1.2 p1.1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
> > > import sys
> > > sys.maxint
2147483647

$ uname -a
Linux scan5 2.6.23-gentoo-r9 #4 SMP Tue Oct 12 16:02:12 MDT 2010 i686
Intel(R) Xeon(R) CPU E5507 @ 2.27GHz GenuineIntel GNU/Linux

I attached a file with the bottom ~9000 lines of the stack trace
before I hit Ctrl-C.  If you'd like be to do it again and try to get
the top of the stack trace, just let me know.  It seemed to be
spitting out a stack trace without end, which is why I eventually
killed it.

Here's a small snippet of the stack trace.  The entire ~9000 lines I
got seemed to just alternate unpredictably between skypev2-version and
jdwp-version.

NSE: Running: 'skypev2-version' (thread: 0x114e4950)
        stack traceback:
                [C]: in function 'socket_lock'
                [string "local connect, socket_lock = ...;..."]:3: in function 'connect'
                /usr/share/nmap/nselib/comm.lua:59: in function 'setup_connect'
                /usr/share/nmap/nselib/comm.lua:122: in function 'exchange'
                /usr/share/nmap/scripts/skypev2-version.nse:27: in function
</usr/share/nmap/scripts/skypev2-version.nse:26>
                (tail call): ?
NSE: Running: 'jdwp-version' (thread: 0x22ff37a0)
        stack traceback:
                [C]: in function 'socket_lock'
                [string "local connect, socket_lock = ...;..."]:3: in function 'connect'
                /usr/share/nmap/nselib/comm.lua:59: in function 'setup_connect'
                /usr/share/nmap/nselib/comm.lua:122: in function 'exchange'
                /usr/share/nmap/scripts/jdwp-version.nse:34: in function
</usr/share/nmap/scripts/jdwp-version.nse:30>
                (tail call): ?
NSE: Running: 'jdwp-version' (thread: 0x15204580)
        stack traceback:
                [C]: in function 'socket_lock'
                [string "local connect, socket_lock = ...;..."]:3: in function 'connect'
                /usr/share/nmap/nselib/comm.lua:59: in function 'setup_connect'
                /usr/share/nmap/nselib/comm.lua:122: in function 'exchange'
                /usr/share/nmap/scripts/jdwp-version.nse:34: in function
</usr/share/nmap/scripts/jdwp-version.nse:30>
                (tail call): ?
NSE: Running: 'skypev2-version' (thread: 0xf014ba0)
        stack traceback:
                [C]: in function 'socket_lock'
                [string "local connect, socket_lock = ...;..."]:3: in function 'connect'
                /usr/share/nmap/nselib/comm.lua:59: in function 'setup_connect'
                /usr/share/nmap/nselib/comm.lua:122: in function 'exchange'
                /usr/share/nmap/scripts/skypev2-version.nse:27: in function
</usr/share/nmap/scripts/skypev2-version.nse:26>
                (tail call): ?

Does this mean anything to anyone?

~ Nathan

Attachment: waiting_to_running_stack_trace.txt.bz2
Description:

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/