Re: New option: --min-rate for minimum-rate scanning

[David Fifield <david () bamsoftware com>]

On Thu, Mar 27, 2008 at 02:14:53AM +0000, Brandon Enright wrote:
> Okay so I've wanted the --min-rate option for a long time now but
> haven't had as much need for it now that massping()->ultrascan() is
> done.  I've very glad to be testing --min-rate none-the-less.  Thanks a
> bunch, once again!
>
> For all of these tests, this is the command I was using:
> time sudo ./nmap --min-rate 100000 --min-hostgroup 256 -P0 -n -d -v -p-
> <target(s)>
>
> Since as David highlighted, the time spent at the end of the scan
> waiting for probes is significant, I've included the "Final times for
> host..." line because it will help the rates make more sense.
>
> Also, since details about the target host really seem to matter, I'm
> going to include details about each host before the scan info.
>
>
> Machine 1: This Linux machine is several layer-2 hops away (the magic
> of VLANs) but doesn't need to be routed (0 layer-3 hops), only switched.
> It has been tuned to send and receive packets as fast as possible.
> - --------------------------------------------------------------------------
> Overall sending rates: 87014.77 packets / s, 3828649.88 bytes / s.
> Final times for host: srtt: 169 rttvar: 7  to: 100000

Wow, nice. I can only get 15000 packets per second against localhost.

> Machine 2: Same as machine one but hasn't been tuned and has a slower
> CPU.
> - --------------------------------------------------------------------------
> Overall sending rates: 81033.16 packets / s, 3565458.99 bytes / s.
> Final times for host: srtt: 194 rttvar: 0  to: 100000
>
>
> Machine 3: This is a Linux machine 2 layer-3 hops away.
> - --------------------------------------------------------------------------
> Overall sending rates: 53396.18 packets / s, 2349431.90 bytes / s.
> Final times for host: srtt: 322 rttvar: 40  to: 100000
>
>
> Machine 4: This is a Linux machine many layer-2 hops away but only 2
> layer-3 hops away.  It has been tuned to send and receive packets as
> fast as possible.
> - --------------------------------------------------------------------------
> Overall sending rates: 50908.96 packets / s, 2239994.16 bytes / s.
> Final times for host: srtt: 551 rttvar: 83  to: 100000
>
>
> Here is machines 1-4 being scanned at the same time.
> - --------------------------------------------------------------------------
> Overall sending rates: 62243.49 packets / s, 2738713.41 bytes / s.
> Final times for host: srtt: 147 rttvar: 26  to: 100000
> Final times for host: srtt: 184 rttvar: 17  to: 100000
> Final times for host: srtt: 227 rttvar: 20  to: 100000
> Final times for host: srtt: 436 rttvar: 14  to: 100000
>
>
> Machine 5: This IP isn't in use.  There isn't a machine here.
> - --------------------------------------------------------------------------
> doAnyOutstandingRetransmits took 31ms
> doAnyOutstandingRetransmits took 31ms
> doAnyOutstandingRetransmits took 31ms
> doAnyOutstandingRetransmits took 32ms
> doAnyOutstandingRetransmits took 33ms
> doAnyOutstandingRetransmits took 32ms
> doAnyOutstandingRetransmits took 33ms
> doAnyOutstandingRetransmits took 33ms
> doAnyOutstandingRetransmits took 33ms
> doAnyOutstandingRetransmits took 34ms
> doAnyOutstandingRetransmits took 35ms
> Overall sending rates: 6703.13 packets / s, 294937.91 bytes / s.
> Final times for host: srtt: -1 rttvar: -1  to: 1000000

"doAnyOutstandingRetransmits took XXms" is a symptom of the "algorithmic
inefficiency" I mentioned. Once that is fixed these scans that have lots
of retransmits should go faster.

The debugging message only prints when the function takes longer than
30 ms. But consider that at 10000 packets per second, ideally we send a
packet every 0.1 ms. A delay of 30 ms puts us 300 packets behind, and
that likely happens many times a second.

> Machine 6: This is a Windows XP machine running a fancy "enterprise
> quality" firewall/host IPS.  Note that the firewall has an exception to
> allow the scanning machine through.  This machine is physically and
> logically adjacent to machine 4 (many layer-2 hops, 2 layer-3 hops).
> - --------------------------------------------------------------------------
> Overall sending rates: 13434.18 packets / s, 591103.81 bytes / s.
> Final times for host: srtt: 824 rttvar: 69  to: 100000
>
>
> Machine 7: Same as machine 6.
> - --------------------------------------------------------------------------
> Overall sending rates: 13028.04 packets / s, 573233.59 bytes / s.
> Final times for host: srtt: 15675 rttvar: 2380  to: 100000
>
>
> Machine 8: Disappointed with Windows speeds so far, I decided to scan
> this Windows 2003 machine that has no firewall, has been tuned to send
> and receive packets as fast as Windows can, and is several layer-2 hops
> but 0 layer-3 hops away.
> - --------------------------------------------------------------------------
> Overall sending rates: 13948.95 packets / s, 613753.59 bytes / s.
> Final times for host: srtt: 349 rttvar: 26  to: 100000
>
>
> Machine 9: Okay, so something's up with Windows so I decided to scan
> this box.  This is a several-hundred-thousand-dollar scanning appliance
> based on Windows 2000.  The vendor's engineers that support this device
> jokingly call it a "packet cannon".  It is physically and logically
> adjacent to machine 8.
> - --------------------------------------------------------------------------
> Overall sending rates: 13693.00 packets / s, 602492.01 bytes / s.
> Final times for host: srtt: 401 rttvar: 35  to: 100000
>
>
> Here is machine 1-9 being scanned at the same time.  The final times
> list is not in order.
> - --------------------------------------------------------------------------
> Overall sending rates: 15877.96 packets / s, 698630.18 bytes / s.
> Final times for host: srtt: 139 rttvar: 18  to: 100000
> Final times for host: srtt: 172 rttvar: 36  to: 100000
> Final times for host: srtt: 205 rttvar: 9  to: 100000
> Final times for host: srtt: -1 rttvar: -1  to: 1000000
> Final times for host: srtt: 794 rttvar: 190  to: 100000
> Final times for host: srtt: 465 rttvar: 26  to: 100000
> Final times for host: srtt: 630 rttvar: 67  to: 100000
> Final times for host: srtt: 251 rttvar: 61  to: 100000
> Final times for host: srtt: 302 rttvar: 85  to: 100000
>
>
> Here is a /25 network (128 machines, 2 layer-3 hops away) being scanned
> at the same time.  Most of the machines on this network are Windows, a
> few are Linux.  Only a small handful of IPs aren't in use.  Sorry, no
> final times list for this scan.
> - --------------------------------------------------------------------------
> Overall sending rates: 13009.93 packets / s, 572436.91 bytes / s.
>
>
> Analysis:
>
> The Linux scanning times are great.  It is clear that latency is the
> biggest factor in scanning Linux because Nmap needs to scale up srtt
> and wait longer at the end of the scan.  When I scan all 4 Linux boxes
> together I get close to the average of their rates which is something
> that sounds reasonable and expected to me.
>
> I learned something new about Windows networking today: it blows worse
> than I originally thought.
>
> The problem scanning Windows seems to be that the kernel's packet
> buffer gets flooded out and it drops packets.  Nmap detect this and
> increases "send delay" and "max_successful_tryno" accordingly.
>
> Here is some of the output from machine 9:
>
> Increased max_successful_tryno for <machine 9> to 1 (packet drop)
> Increased max_successful_tryno for <machine 9> to 2 (packet drop)
> Increased max_successful_tryno for <machine 9> to 3 (packet drop)
> Increased max_successful_tryno for <machine 9> to 4 (packet drop)
> Increasing send delay for <machine 9> from 0 to 5 due to max_successful_tryno increase to 4 

The message printed is misleading, because the send delay isn't honored
when --min-rate is used. The real slowdown is the increase of
max_successful_tryno. It's possible that using a higher packet send rate
will actually make the scans take longer, because retransmits mean there
are more packets to send overall even if they get sent faster. Also the
doAnyOutstandingRetransmits issue is likely interfering with this.

> What I don't understand is why when scanning many machines at once,
> Windows and non-existent IPs seems to drag down the overall sending
> rate.  Based on my understanding (and David's description), even if one
> machine's probes are done and Nmap is waiting, it can still be sending
> to other machines in the hostgroup.  I would think than Nmap would be
> able to blast away an the only time spent waiting would be at the very
> end when there are less than 100k probes left to be sent.

That is true, but if the Linux hosts finish faster (for whatever reason)
and then the Windows hosts have to finish scanning at a slower rate,
that will bring the overall average down. If you run with -d and use the
run-time interaction feature by hitting Enter during a scan, you can see
a live estimate of the current scanning rate. You might see it really
fast at the beginning and slow down at the end.

> I'm happy to try any patch, Nmap command, or network size (up to when
> Nmap runs out of memory at around /17) so feel free to ask or patch away.

Would you run the tests again with "--max-retries 0"? That will
eliminate the doAnyOutstandingRetransmits slowdown.

David Fifield

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org