Nmap's memory use

[David Fifield <david () bamsoftware com>]

Hi,

We've had some report recently about Nmap using a lot of memory.

"Port memory bloat"
http://seclists.org/nmap-dev/2009/q3/926
"nmap 5 memory usage"
http://seclists.org/nmap-dev/2009/q4/300

I started looking at ways to improve this. This note is to let you know
what I've found so far and to ask if anyone has tips on memory
profiling.

In the first link above, Pavel Kankovsky observed that it is the Port
class that is using up most of the memory in some scans. My preliminary
tests agree with this. I used the Massif memory profiling too
(http://valgrind.org/docs/manual/ms-manual.html) from the Valgrind suite
as follows:

valgrind --tool=massif ./nmap -p- -sU scanme.nmap.org/30 -oX scanme-30.xml --max-retries 1
ms_print massif.out.XXXX

This prints the following graph. The height of each bar shows the size
of the heap allocations. @ are detailed snapshots where a trace shoing
where heap items were allocated is available. # is the peak usaage, also
with a trace.

    MB
33.63^                                                                       #
     |                                                                       #
     |                                                                       #
     |                                                                       #
     |                                                                       #
     |                                                                       #
     |                                                                       #
     |                                                                       #
     |                                                                       #
     |                                                             @::::::@::#
     |                                                             @::::::@::#
     |                                                             @::::::@::#
     |                                                             @::::::@::#
     |                                                             @::::::@::#
     |                                                             @::::::@::#
     |                                                             @::::::@::#
     |                                                             @::::::@::#
     |                             @:::@::@@::::::@::@@::::::::@:::@::::::@::#
     | ::@@:::::::@::::::::::::::::@: :@::@ :::: :@::@ ::::::::@:::@::::::@::#
     |:: @@:::::: @: ::::::::::::: @: :@::@ :::: :@::@ ::::::::@:::@::::::@::#
   0 +----------------------------------------------------------------------->Gi
     0                                                                   8.117

The first thing I had to figure out was that the horizontal axis is not
time, but rather number of instructions. Therefore times when the CPU
was more busy are wider in the chart. We can see a gradual climb up to
about 6 MB, then a sudden jump up to 13 MB, then another up to 34 MB.
While it looks like it held steady at 13 MB for a while, I think that is
an artifact of being measured by instructions not time. All of the huge
growth at the end happened when the HostScanStats::dismissBench method
was called to mark all the ports that hadn't gotten a response as
open|filtered. This creates a huge number of short-lived port objects.
In the peak usage, such allocations accounted for 74.29% of all heap
allocations.

Pavel provided a patch that reduces the memory usage of all the Ports in
the default state at the end. I think that is the right idea. I am going
to try to see if there's a way to reduce their memory use even further,
perhaps by changing the programming interface so that the creation of
discrete objects isn't necessary.

David Fifield
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/