Re: Weird Crash - "WAITING_TO_RUNNING"

[Nathan <nathan.stocks () gmail com>]

On Wed, Nov 3, 2010 at 10:40 PM, David Fifield <david () bamsoftware com> wrote:
> On Wed, Nov 03, 2010 at 04:32:30PM -0600, Nathan wrote:
> > On Wed, Nov 3, 2010 at 1:33 PM, Nathan <nathan.stocks () gmail com> wrote:
> > > On Wed, Nov 3, 2010 at 11:29 AM, Patrick Donnelly <batrick () batbytes com> wrote:
> > > > On Wed, Nov 3, 2010 at 1:07 PM, Nathan <nathan.stocks () gmail com> wrote:
> > > > > "Is this on a 32-bit machine?" -- The hardware is 64-bit capable, but
> > > > > the old version of linux on it is compiled in 32-bit, I think.  The OS
> > > > > can address more than 4GB RAM, but the stuff running on it seems to
> > > > > act very 32-bit.  Python's maxint is 32-bit, anyway:
> > > > >
> > > > > $ python
> > > > > Python 2.5.2 (r252:60911, Oct 30 2008, 18:03:18)
> > > > > [GCC 4.1.2 (Gentoo 4.1.2 p1.1)] on linux2
> > > > > Type "help", "copyright", "credits" or "license" for more information.
> > > > > > > > import sys
> > > > > > > > sys.maxint
> > > > > 2147483647
> > > > >
> > > > > $ uname -a
> > > > > Linux scan5 2.6.23-gentoo-r9 #4 SMP Tue Oct 12 16:02:12 MDT 2010 i686
> > > > > Intel(R) Xeon(R) CPU E5507 @ 2.27GHz GenuineIntel GNU/Linux
> > > >
> > > > Your processor (running as 32 bit) is using PAE (Physical Address
> > > > Extension) to allow more than 4GB of RAM. Your process is still only
> > > > allowed ~4GB of addressable RAM. This is why Nmap is crashing for you.
> > >
> > > Okay, I'll buy that that is the _symptom_ we're seeing.  I can't
> > > imagine you're trying to imply that it's normal for an nmap scan of
> > > one target to take that much RAM, though.  At 65,535 ports, 4GB gives
> > > you 64KB PER PORT.  I could generate a small jpeg thumbnail of the
> > > traffic sent to each port at that rate.
> > >
> > > The question is, WHAT is using up that RAM, and how do we stop it?   I
> > > could run multiple instances of this same scan on this same box with
> > > nmap 4.68 (that I just upgraded from) against this target without any
> > > problems at all.
> > >
> > > Did anyone look at the stack trace???  It seriously can't be normal to
> > > run "skypev2-version" and "jdwp-version" tens of thousands of times
> > > until you run out of RAM.  Infinite loop, anyone?  Or, perhaps this is
> > > caused by nmap erroneously thinking every single port is open (which
> > > is one of the things I hoped upgrading would fix), where the
> > > underlying assumption is that very few ports will be open so we can
> > > use a ton of memory to investigate each open port?
> >
> > Found a workaround:  add "--scan-delay 1ms".  It finishes a lot
> > quicker too.  It seems much more accurate too.  At least it has a lot
> > less than 65,535 ports listed as open like nmap 4.68 thought.
> >
> > So now my question is: According to the nmap man page, nmap will "try"
> > to dynamically determine an appropriate delay time, but it's unclear
> > as to whether specifying this option overrides that dynamic
> > determination completely or if it just sets a starting point that will
> > still be dynamically adjusted if it thinks is necessary.  Does anyone
> > know which behavior I just described matches reality?
>
> It is a hard limit that doesn't change throughout the scan.
>
> The memory exhaustion you've found is clearly a bug in Nmap. If no one
> has responded constructively it's because we don't know what might be
> the cause yet.
>
> Your discovery of --scan-delay is a good clue. Also the fact that ports
> are being erroneously marked open. I can't think of a reason why
> scanning faster would cause ports to be seen as open; Nmap never marks a
> port open unless it gets some kind of response, and rate-limited replies
> would show up as filtered instead.
>
> Try doing your scan with fewer ports at full speed and see if they are
> wrongly marked open. Like this:
>
> /usr/bin/sudo /usr/bin/nmap -sS -sV -T4 --top-ports 100 74.62.92.70 -P0 -v
>
> You should not get more open ports than you got with --scan-delay. If
> you get more open ports, there's something weird going on that we have
> to figure out. Run with the -v option and you'll be notified of open
> ports in real time; that way you'll be able to see if you're getting a
> flood of them.


Okay, so I've been doing a bunch of testing.  I've identified 100
different targets that had this original WAITING_TO_RUNNING crash
because of the memory usage problem that we haven't pinned down yet.
But first, I'll go through all the emails in this thread and respond
to all the helpful stuff people have been suggesting.

Running the original scan with -v just floods the console with open
port messages:

$ /usr/bin/sudo /usr/bin/nmap -sS -sV -T4 -p 1-65535 74.62.92.70 -P0 -v

Starting Nmap 5.35DC1 ( http://nmap.org ) at 2010-11-08 14:25 MST
NSE: Loaded 6 scripts for scanning.
Initiating Parallel DNS resolution of 1 host. at 14:25
Completed Parallel DNS resolution of 1 host. at 14:25, 0.03s elapsed
Initiating SYN Stealth Scan at 14:25
Scanning remote.tjwels.net (74.62.92.70) [65535 ports]
Discovered open port 25/tcp on 74.62.92.70
Discovered open port 443/tcp on 74.62.92.70
Discovered open port 3389/tcp on 74.62.92.70
Discovered open port 53/tcp on 74.62.92.70
Discovered open port 80/tcp on 74.62.92.70
Discovered open port 51496/tcp on 74.62.92.70
Discovered open port 28873/tcp on 74.62.92.70
Discovered open port 63812/tcp on 74.62.92.70
Discovered open port 23940/tcp on 74.62.92.70
Discovered open port 55599/tcp on 74.62.92.70
Discovered open port 34806/tcp on 74.62.92.70
[snip -- it goes on for thousands of lines]

Using the "--top-ports 100" option that you mentioned also seems to
eliminate the false-positive-open-ports, but I can't use that, because
I need to do a full port scan:

$ /usr/bin/sudo /usr/bin/nmap -sS -sV -T4 --top-ports 100 74.62.92.70 -P0 -v

Starting Nmap 5.35DC1 ( http://nmap.org ) at 2010-11-08 14:23 MST
NSE: Loaded 6 scripts for scanning.
Initiating Parallel DNS resolution of 1 host. at 14:23
Completed Parallel DNS resolution of 1 host. at 14:23, 0.03s elapsed
Initiating SYN Stealth Scan at 14:23
Scanning remote.tjwels.net (74.62.92.70) [100 ports]
Discovered open port 3389/tcp on 74.62.92.70
Discovered open port 25/tcp on 74.62.92.70
Discovered open port 80/tcp on 74.62.92.70
Discovered open port 53/tcp on 74.62.92.70
Discovered open port 443/tcp on 74.62.92.70
Completed SYN Stealth Scan at 14:23, 1.35s elapsed (100 total ports)
Initiating Service scan at 14:23
Scanning 5 services on remote.tjwels.net (74.62.92.70)
Completed Service scan at 14:23, 20.20s elapsed (5 services on 1 host)
NSE: Script scanning 74.62.92.70.
Nmap scan report for remote.tjwels.net (74.62.92.70)
Host is up (0.089s latency).
Not shown: 92 closed ports
PORT     STATE    SERVICE       VERSION
21/tcp   filtered ftp
25/tcp   open     smtp          Microsoft ESMTP
53/tcp   open     domain        Microsoft DNS 6.0.6002
80/tcp   open     http          Microsoft IIS httpd 7.0
110/tcp  filtered pop3
143/tcp  filtered imap
443/tcp  open     ssl/http      Microsoft IIS httpd 7.0
3389/tcp open     microsoft-rdp Microsoft Terminal Service
Service Info: OS: Windows

Read data files from: /usr/share/nmap
Service detection performed. Please report any incorrect results at
http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 21.70 seconds
           Raw packets sent: 103 (4.532KB) | Rcvd: 97 (3.900KB)


The original workaround I found (--scan-delay 1ms) has worked (i.e.
not crashed the first time I tried running against a target) on
exactly 40 of 100 targets I found that exhibit the crashing behavior.
The other 60 still crashed with the same symptoms.  I haven't had time
to re-run the scans and see if the behavior is transient, or if the
same set of options always crashes (or doesn't) against a given host.

I'll go respond to the other emails now...

~ Nathan
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/