Re: printtableSize() causing huge spike in memory usage at end of hostgroup

[Brandon Enright <bmenrigh () ucsd edu>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, 16 Mar 2009 21:45:49 -0700
Fyodor <fyodor () insecure org> wrote:

> On Mon, Mar 16, 2009 at 10:36:45PM -0600, David Fifield wrote:
> > On Tue, Mar 17, 2009 at 02:52:37AM +0000, Brandon Enright wrote:
> >
> > It sounds like the right solution to me. It is reminiscent of
> > xml_convert, which used to allocate a worst-case six bytes per input
> > byte, even though the most bytes only take up one bytes of output.
> >
> > If C++ STL routines make this jobs easier, then use them by all
> > means. We don't have to be compatible with C in the Nmap codebase.
>
> Yeah, but if STL is used, it is important to test that it is suitably
> performant from a CPU and memory usage perspective.  Increasing the
> level of abstraction makes it easier to overlook inefficiency.  Of
> course, the current case demonstrates that lower-level code can be
> vulnerable to gross inefficiency too.  Particularly in a case like
> this where it is scaled far beyond what was ever intended when it was
> first written.
>
> So I'm not against using the STL for this table printing as long as it
> is done efficiently.  And it sounds like Brandon has some rigorous
> test cases available to check that :).  Doing it in plain C is fine
> too.
>
> Cheers,
> -F

Attached is a patch to use memory a little bit more efficiently.  The
buffer starts at 1k and any time it is too small it is doubled in
size.  I did some testing that suggests for most scans 1k is probably
enough.

Right now:
Standard Windows host + -sV: TABLE_MEM: size: 464, allocated: 961
Standard Unix Server + -sV: TABLE_MEM: size: 667, allocated: 1925
Windows Exchange + -sV: TABLE_MEM: size: 620, allocated: 1041
Windows Exchange + -sV -sC: TABLE_MEM: size: 1050, allocated: 1910
Extremely noisy host + -sV -sC: TABLE_MEM: size: 6372, allocated: 113435
Extremely noisy host + -sV -sC -d3: TABLE_MEM: size: 3260062, allocated: 86514530

With this patch:
Standard Windows host + -sV: TABLE_MEM: size: 489, allocated: 1024
Standard Unix Server + -sV: TABLE_MEM: size: 667, allocated: 1024
Windows Exchange + -sV: TABLE_MEM: size: 620, allocated: 1024
Windows Exchange + -sV -sC: TABLE_MEM: size: 1050, allocated: 2048
Extremely noisy host + -sV -sC: TABLE_MEM: size: 4661, allocated: 8192
Extremely noisy host + -sV -sC -d3: TABLE_MEM: size: 3260161, allocated: 4194304

Note that the patch is generally no worse in memory consumption in the
normal case and way better in the extreme case (saving 82 MB).  In the
normal case it should use less CPU because it doesn't have to
pre-iterate all of the table cells to guess the max table size.  At the
cost of 1 more resize in some scans, we could start the buffer at 512
bytes to save some memory.  I don't think saving 512 bytes at the start
is worth it though.

Also note that there is more memory that could be squeezed out if we
could work a way to free the cell after the memcpy.  As it stands now
the printed table duplicates (doubles) what is already in memory in the
form of many cell->str.  I didn't even bother looking into this.

Comments wanted.

Brandon

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)

iEYEARECAAYFAknANGsACgkQqaGPzAsl94KVKQCaA6H9xAYSbUOGS/XToTPr5UHC
NTsAoJAINdmEH9LXec0DVIGzD2eGiU1h
=Ky5R
-----END PGP SIGNATURE-----

Attachment: printmem.diff
Description:

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org