Nmap Development mailing list archives
nmap massive memory usage
From: John Richard Moser <nigelenki () comcast net>
Date: Tue, 04 Dec 2007 10:30:10 -0500
In scanning two /24 ranges for full UDP sweeps on all 0-65535 ports, I found nmap (after fininshing the first group) held 600MB+ of RSS on 64-bit Linux. Command line: nmap -oA nmap.udp -vv -sU --min-hostgroup 64 --min-parallelism 128 \ -T4 -p0-65535 -P0 192.168.1.0/24 192.168.2.0/24 I HAVE NOT READ THE CODE. I am guessing blindly at how nmap works and making notes. This message contains long boring sections of "if I were to write a tool for this I'd do X Y and Z" so just look for the below line when you want to find the next dumb thought. In case you don't feel like reading my useless rant, core question is: why does nmap eat so much memory? As for the rest of you, I'll probably get an explanation of why anything I wrote below does or doesn't work or a flame about not sending useless crap ;) == BREAK SECTION == I calculate if nmap uses 8192 bytes of storage per bitmap to create 6 bitmaps per host (one for each open, closed, filtered, unfiltered, open|filtered, closed|filtered), it will eat 3MiB per 64 hosts (so in this case, up to 24MB) just for that. That should get flushed each time a host's information gets flushed to disk, so not a problem. 3MiB accounted for. Note: You can use 3 bit to track the state here, and have nmap use 24576 bytes instead of 49152 per host. To look up the bits, nmap would have to have an array of char[24576] as a bitmap and find the starting index by. This would even give a state for "Unscanned" to help track randomized ports (and hosts). === /* we write LTR so offsets... 00000000 01234567 Bits 00000000 00000000 01234567 89111111 012345 */ bit = (port * 3); byte = bit / 8; offset = bit % 8; val = bitmap[byte]; if (offset > 5) { /* shift over to make room 00000011 10000000 <- 00000110 10000000 */ val <<= (3 - (8 - offset)); /* OR with MSBs of next byte shifted to the right 00000110 10000000 ------> 00000110 00000001 (bit OR) 00000111 */ val |= (bitmap[byte+1] >> 8 - (3 - (8 - offset)); } else { val >>= (8 - offset) - 3; } /* clear to the 3 bits we need */ val &= 0x07; === nmap could also just use a linked list of ports, but that would quickly outgrow this: 32-Bit 64-bit Desc 4 8 Pointer to next 2 2 Short int: port 1 1 Byte: state The compiler would likely pad this to 8 bytes and 12 bytes, respectively. So after 6144 or 4096 ports (for 6 individual bitmaps), or 3072 or 2048 ports (for the above compacted single not-bitmap), memory usage would pass the bitmap. Using an array instead, and a separate list of ordered ports (states[5] versus {21,22,80,443,3306} as a port list), it's still just 1 byte each, which is better until you reach 24576 ports (and thus not important enough for a complex hack). Of course, service version scanning makes this moot; the problem is no longer simple, and the array carries tons of extra data. =================== == BREAK SECTION == =================== So what else could hold memory? Obviously the above will eat at most maybe 24MB in my entire scan. What gets stored? Packet captures? Interesting information about the system? Does any of that get flushed once the host is scanned? Once a host is done being scanned, it should immediately be analyzed and then have any relevant data not useful for further analysis freed. I can't imagine that nmap is holding anything else in there though, why would it hold packet captures and all beyond useful life? -- Bring back the Firefox plushy! http://digg.com/linux_unix/Is_the_Firefox_plush_gone_for_good https://bugzilla.mozilla.org/show_bug.cgi?id=322367 _______________________________________________ Sent through the nmap-dev mailing list http://cgi.insecure.org/mailman/listinfo/nmap-dev Archived at http://SecLists.Org
Current thread:
- nmap massive memory usage John Richard Moser (Dec 04)
- Re: nmap massive memory usage majek04 (Dec 04)
- Re: nmap massive memory usage John Richard Moser (Dec 04)
- Re: nmap massive memory usage majek04 (Dec 04)