Re: printtableSize() causing huge spike in memory usage at end of hostgroup

[Brandon Enright <bmenrigh () ucsd edu>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Comments/results inline.

On Tue, 17 Mar 2009 23:14:25 -0600
David Fifield <david () bamsoftware com> wrote:

> On Tue, Mar 17, 2009 at 11:38:13PM +0000, Brandon Enright wrote:
> > Attached is a patch to use memory a little bit more efficiently.
> > The buffer starts at 1k and any time it is too small it is doubled
> > in size.  I did some testing that suggests for most scans 1k is
> > probably enough.
> >
> > Right now:
> > Standard Windows host + -sV: TABLE_MEM: size: 464, allocated: 961
> > Standard Unix Server + -sV: TABLE_MEM: size: 667, allocated: 1925
> > Windows Exchange + -sV: TABLE_MEM: size: 620, allocated: 1041
> > Windows Exchange + -sV -sC: TABLE_MEM: size: 1050, allocated: 1910
> > Extremely noisy host + -sV -sC: TABLE_MEM: size: 6372, allocated:
> > 113435 Extremely noisy host + -sV -sC -d3: TABLE_MEM: size:
> > 3260062, allocated: 86514530
> >
> > With this patch:
> > Standard Windows host + -sV: TABLE_MEM: size: 489, allocated: 1024
> > Standard Unix Server + -sV: TABLE_MEM: size: 667, allocated: 1024
> > Windows Exchange + -sV: TABLE_MEM: size: 620, allocated: 1024
> > Windows Exchange + -sV -sC: TABLE_MEM: size: 1050, allocated: 2048
> > Extremely noisy host + -sV -sC: TABLE_MEM: size: 4661, allocated:
> > 8192 Extremely noisy host + -sV -sC -d3: TABLE_MEM: size: 3260161,
> > allocated: 4194304
>
> What about a simple scan without -sV? 1024 seems a little high to me.

It probably is.  The results at the bottom of this email show that.

> You can cause fewer reallocations to happen in some cases by doing the
> reallocation like this:
>
>   if (cell->strlength + p + 2 > tableoutsz) {
>     tableoutsz = (cell->strlength + p + 2) * 2;
>     tableout = (char *) safe_realloc(tableout, tableoutsz);
>   }

Yeah, that's a better idea.  I changed the patch to this and tested
with this strategy.

> How often is a table reused?

Surprisingly never.

> Is it worth freeing the table's buffer
> memory, or trimming it to size, do you think?

I looked through all of the code that makes use of NmapOutputTable and
it all destroys the object shortly after the table is printed.  Since
freeing the table is part of the destructor, I don't think we need to
explicitly trim a large table.  It would be redundant to realloc() our
table smaller and the free it with the call to the destructor.

> David Fifield

To test the size of the table and how much memory gets allocated with
my patch, I ran 4 scans, each against the same /23.

./nmap <targetnet>
TABLE; Min: 39; Max: 770; Avg: 216.358024691358
MEMORY; Min: 1024; Max: 1024; Avg: 1024


./nmap -v --reason <targetnet>
TABLE; Min: 57; Max: 1311; Avg: 338.358024691358
MEMORY; Min: 1024; Max: 2062; Avg: 1062.1975308642


./nmap -v -sV <targetnet>
TABLE; Min: 48; Max: 1723; Avg: 396.450617283951
MEMORY; Min: 1024; Max: 2062; Avg: 1054.03703703704


./nmap -v -sV -sC --script=all <targetnet>
TABLE; Min: 49; Max: 47064451; Avg: 2034446.87037037
MEMORY; Min: 1024; Max: 94123680; Avg: 4068606.90123457


I think what this is telling us is that we should start the table off
at ~512 bytes rather than 1024.  In the average usage scenario this
will be almost no waste and shouldn't need to resize the table more
than once.  Note that the really huge table in the last scan was from
our DNS servers.  Every single name in the whole zone got printed with
the dns-zone-transfer script.

Attached is an update version of the patch with a 512 byte starting
buffer and a better allocation routine to eliminate the need for the
while loop.

Brandon

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)

iEYEARECAAYFAknEJKgACgkQqaGPzAsl94JnHQCeKGwM/IPZMo2AI0tCYBjIS+u/
LzoAoMgAH6t10+Z1xx8w7CvuhvOTWQdU
=hynh
-----END PGP SIGNATURE-----

Attachment: tableout_mem.patch
Description:

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org