Re: Nmap's memory use

[David Fifield <david () bamsoftware com>]

On Thu, Nov 19, 2009 at 06:04:58PM -0700, David Fifield wrote:
> We've had some report recently about Nmap using a lot of memory.
>
> "Port memory bloat"
> http://seclists.org/nmap-dev/2009/q3/926
> "nmap 5 memory usage"
> http://seclists.org/nmap-dev/2009/q4/300
>
> In the first link above, Pavel Kankovsky observed that it is the Port
> class that is using up most of the memory in some scans. My preliminary
> tests agree with this. I used the Massif memory profiling too
> (http://valgrind.org/docs/manual/ms-manual.html) from the Valgrind suite
> as follows:
>
> Pavel provided a patch that reduces the memory usage of all the Ports in
> the default state at the end. I think that is the right idea. I am going
> to try to see if there's a way to reduce their memory use even further,
> perhaps by changing the programming interface so that the creation of
> discrete objects isn't necessary.

I've finished working on memory reduction for now. Apart from the
earlier nmap-os-db parsing memory improvement described in
http://seclists.org/nmap-dev/2009/q4/479, I just merged two more
important improvements from the nmap-mem branch in r16308.

For a bit of context, first look at this benchmark scan, all UDP ports
on four hosts. It uses a maximum of 33.63 MB, almost all of it coming
from addPort at the end of the scan.
http://www.bamsoftware.com/wiki/Nmap/Memory#a20091118

The first new improvement was a reduction in size of Port objects. I
used Pavel's idea of dynamically allocating parts of Port that aren't
need for every port, like service and RPC information. The size of a
bare Port object dropped from 92 to 40 bytes on my machine. Now the peak
memory usage is 17.66 MB.
http://www.bamsoftware.com/wiki/Nmap/Memory#r16276

The second improvement is to avoid allocating some Port objects at all.
In many scans, the majority of probes don't receive a response. The big
spike in memory at the end of the previous two graphs is thousands of
Ports being created for the open|filtered ports that didn't respond. Now
the code has a notion of a "default port state" for ports that don't get
a response. All the ports in the default state share a single Port
object. This further reduced memory use to 5.94 MB.
http://www.bamsoftware.com/wiki/Nmap/Memory#r16290

The second improvement won't help, for example, when TCP ports are
closed because of RSTs rather than being filtered, but you will still
get the benefit of smaller Port objects.

David Fifield
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/