Re: PortBunny - FX and Fabs at 24C3

[Brandon Enright <bmenrigh () ucsd edu>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

A very thoughtful response in the face of such a negative presentation;
thanks Fyodor.  I'm constantly impressed by the integrity and
comradeship of Nmap developers.  My comments inline.


On Sun, 13 Jan 2008 00:56:39 -0800 or thereabouts Fyodor
<fyodor () insecure org> wrote:

> On Sat, Jan 05, 2008 at 04:19:39PM +0100, kx wrote:
> > Effectively, they find responsive hosts to acts as triggers for
> > detecting packet loss. Dan Kaminsky was talking about this stuff at
> > least as far back as 2005 with Scanrand, but I am sure earlier. He
> > called them canaries as opposed to triggers.
> >
> > I watched the video, and I don't know what version of nmap they were
> > testing against. I am not sure if they looked at David's code:
> > http://seclists.org/nmap-dev/2007/q3/0433.html
>
> Thanks for letting us know!  Recurity has placed the code, slides, and
> presentation video up at
> http://www.recurity-labs.com/portbunny/portbunny.html .
>
> I watched the presentation and, to be honest, I'm not too impressed
> overall.  Maybe I'd be more charitable if Fabs didn't attack Nmap
> throughout the whole thing.  Oh wait, slide number 7 is titled "My hat
> is off to Fyodor!".  Then he attacks Nmap for the remaining 59 slides
> :).  Oh well, I'm sort of used to this after all the other
> presentations I've watched about amazing new scanners set to destroy
> Nmap.  But Scanrand and UnicornScan haven't exactly eaten our lunch,
> and I don't think we need to start quivering in fear over PortBunny
> just yet :).

Unicornscan and Scanrand have their place but anyone claiming they
should replace Nmap doesn't know what they're talking about.
Unicornscan and Scanrand are niche port scanners tuned to do one or two
things well.  Nmap is the Swiss Army Knife of port scanners.

Slide 10 of PortBunny says: "Bottom line: Do just one thing, but do it
right.".  That's great but Nmap does everything and it does it *very*
well.

But hey, if they think Nmap is the app to beat, we must be doing
something right.

> Here is the full description from the PortBunny web page:
>  
>   "PortBunny is a Linux-kernel-based port-scanner created by Recurity
>    Labs. Its aim is to provide a reliable and fast
>    TCP-SYN-port-scanner which performs sophisticated timing based on
>    the use of so called "trigger"-packets. The port-scan is performed
>    in 2 steps: First the scanner tries to find packets, to which the
>    target responds ("triggers"). Second, the actual port-scan is
>    performed. During the scan, the triggers, which were found in the
>    first scanning-phase, are used to determine the optimal speed at
>    which the target may be scanned."
>
> Clearly they consider "triggers" to be PortBunny's primary innovation.
> So I hate to break it to them that Nmap has done this for 3.5 years.
> Here is a snippet which was added to the Nmap Changelog
> (http://insecure.org/nmap/changelog.html) for the 3.70 release (August
> 2004):
>
> o Nmap now has a "port scan ping" system. If it has received at least
>   one response from any port on the host, but has not received
>   responses lately (usually due to filtering), Nmap will "ping" that
>   known-good port occasionally to detect latency, packet drop rate,
>   etc. 
>
> That sounds a lot like their "triggers", doesn't it?  It is true that
> they send more triggers which does greatly improve speed, but it can
> lead to accidentally syn-flooding the open port you use as a trigger.
> (this is not a problem if you find a closed port, but this is only
> useful for very heavily filtered hosts anyway).

"Triggers" are a good idea, even if Nmap beat them to it.  We may be
able to better borrow a few ideas and better utilize our port ping
system.

> In Slide 35, Fabs does a comparison scan with "66535 ports, mostly
> filtered, Internet".  I'm sure he means 65,535 ports, but that is just
> a typo.  Anyway, Nmap (no version number or command line given) takes
> 12 minutes, 18 seconds.  PortBunny took 15 seconds.  Of course, the
> audience cheered wildly even though (IMHO) this is a sign that
> PortBunny is WAY too aggressive.  That is an average of 4,369 probes
> per second (not counting all of their trigger probes).  Nmap can scan
> that fast too if you remove congestion control and toss out the idea
> of accuracy.

Unicornscan would have taken about 8 seconds.  Anyone actually looking
an the scan results of these scans though would see right away that
faster doesn't mean better.  Too bad they didn't include results.

> The presentation is filled with scan times, but none of the slides
> address accuracy.  It appears that they don't do any sort of
> retransmission, so if their probe to a port is dropped, they just
> count it as filtered and move on.  Nmap by default retransmits to each
> port at least once.  But if they are so concerned with speed, they can
> turn that off.  Also, they could try options like -T5 to make Nmap
> more aggressive.  Admittedly Nmap has far more timing-related options
> than PortBunny has total options, but learning the Nmap timing options
> is still much easier than writing your own port scanner.  Too bad they
> didn't spend the time improving what they see as subpar algorithms in
> Nmap's approach to dealing with heavily filtered hosts.

Patches would have been nice.  They wanted to reinvent the wheel though
(slide 2).

> Their slide 50 describes "serious bugs" in Nmap, but the algorithm
> they spotted is actually a critical feature for UDP scans (and can be
> useful in rare TCP cases too).  Maybe if their scanner actually
> handled UDP, they would understand this.

They fail to mention that not doing this will make the scan less
accurate.  Slide 50 is a good example of how desperate they are
criticize.

> Anyway, I try not to just get mad when I see presentations like this,
> and to actually try to think of ways to improve Nmap.  Here are some
> things we might look at:
>
> o Whether we should do the port scan pings more frequently.  Fabs
>   makes fun of us for only doing one every five seconds, but that slow
>   rate was chosen for a very good reason: we don't want to SYN flood
>   the ports.

I think we should increase the rate to at least 1/s for the faster
timing options.

>   Remember that many old BSD machines and crappy devices
>   can be SYN flooded with as few as 8 packets.

Maybe a tunable parameter?  I don' have a
problem with --yet-another-timing-option, this is one on the reason why
Nmap is so flexible.

>   Also, if the machine
>   you are running Nmap from is firewalled such that it isn't sending
>   RST in response to the SYN/ACK returned from the port scan ping
>   (trigger) packets, a SYN flood is even more likely.  Still, we might
>   be able to make it somewhat more aggressive.  We could also consider
>   having Nmap generate its own RST in response to SYN/ACK packets,
>   though it would be best if we only do that if the host isn't doing
>   this itself.  And that detection may be a bit of a pain.

There are a bunch of reasons why it is a very bad idea to run Nmap on a
firewalled machine.  We should probably remind people of this more often
rather than worrying about sending RSTs on our own.

> o Fabs claims that our global congestion control (which works on the
>   set of hosts were are scanning, as opposed to single targets) may be
>   dragged down undesirably when only 1 target is dropping many packets
>   but the rest of the hosts are very fast and response.  We should
>   investigate and think more about how we want Nmap to behave in such
>   situations.  Portbunny attacks the Nmap algorithms for scanning
>   hosts in parallel, but it sort of punts the issue by only scanning
>   one host at a time by default.  We could consider loosening the
>   global congestion controls, removing them, or offering an option to
>   remove them.  But that would require more data first.

This is the one argument that I agree with; but not for the same
reasons they cite. Some of this has improved with David's work moving
massping() to ultrascan() but the problem hasn't completely gone away.
My experience is that Nmap is very good at a few number of ports across
many hosts or many ports across a few hosts.

In fact, I too "solved" this problem by only scanning a single host at
a time with Nmap.  To still scan in parallel, I run many instances of
Nmap at once.  I call it poor man's threading.  If you have the
hardware to do it and you've appropriately tuned your kernel, it works
*very* well.

Another reason I do this is that Nmap used to (may have been fixed)
time out a whole scan group / host group when a single host timed out.
I'd have a single "bad" host kill the results for a whole group of 32
or more.

I suspect the original reason for global congestion control was that
the network was as much the bottleneck as a single host.  I think
slides 53 and 54 are complete bullshit now though.  The only case where
the network is a shared bottleneck is wireless networks and *really old*
hardware based networks.

Modern switching and routing gear can handle millions of pps, far more
than a single scanning machine can generate.  The bottleneck is how
quickly the target host can respond.

> o Fabs declares that "the kernel is a good place for a port scanner",
>   and indeed PortBunny is a Linux-only kernel module.  They suggest
>   that you use a dedicated box and not run other Internet applications
>   such as web browsers at the same time.

This is a terrible idea.  I think they did it in the kernel because
they wanted to and went looking for excuses why it was a good idea
later.  If speed is all you're looking for, the Unicornscan guys sure
seem to be doing well with their user-land distribute TCP/IP stack.
There is absolutely no good reason to stuff a portscanner in the kernel.

With this design, what they've made is a packet generator that happens
to tell you what ports are open.

>   I don't think we'll copy
>   this load-the-port-scanner-into-the-kernel approach :).  They
>   complain that Nmap is too bloated (do you guys really not like or
>   use OS detection, version detection, etc.?), but at least Nmap
>   doesn't bloat your kernel.

Nmap is great and it continues to get development and new features
because people like it and want it to do more.  I say as long as we keep
the quality up, new features are more than welcome.

> Anyway, those are my initial reactions.  While it is a shame that they
> took such an antagonistic we're-better-than-Nmap approach, it is
> always nice to see new open source port scanners and research!  I hope
> they continue working on it (but tone down their presentations).

I 100% agree.  They have some good ideas and have clearly thought a lot
about port scanning.  I'd like to see more research more numbers
(including accuracy info) out of them.

> Another thing I'll give them is that they do already have shirts for
> sale.  Someday we need to make bad-ass Nmap shirts :).
>
> Cheers,
> Fyodor

To summarize things I think we should look at:

* Look into increasing the port-ping rate and possibly even make it
tunable.  I'd like an option like --heartbeat-delay <ms>

* Look into reducing the effects of global congestion control, moving
to more of a per-host timing setup.  Maybe allow this to be turned on
or off with an option like --timing-per-host


I'll volunteer to provide any host/group timing information if someone
wants to get "real world" data from a large network.  We now have a
100% dedicated Nmap box that does nothing but Nmap scan like crazy all
day long.  It should be easier for me to get consistent results now
than it was when David was migrating massping().

Brandon

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.7 (GNU/Linux)

iD8DBQFHifZyqaGPzAsl94IRAth3AJ9bgJX5Hh+p0DeiW0D07MdxTHlfswCgv+su
fv34lGR4OuZ6CoWpCQxsO4w=
=+NCq
-----END PGP SIGNATURE-----

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org