OS detection improvements

[David Fifield <david () bamsoftware com>]

Hello,

You may remember that a few weeks ago I asked for hosts with known OSes
for research on improving OS detection.

http://seclists.org/nmap-dev/2008/q4/0218.html

I committed r10862, which brings in improvements due to this and other
research. These are the changes:

1. The widening of ranges for T test expressions in nmap-os-db. Any
   expressions that were not already ranges were expanded to cover plus
   and minus five of their original values.
2. The normalization of TG expressions in nmap-os-db. Nmap is only
   capable of outputting 0x20, 0x40, 0x80, and 0xFF for a TG value, but
   many fingerprints had values other than these. They have all been
   rounded to their nearest likely value.
3. The elimination of the U1.TOS and IE.TOSI tests (both having to do
   with type of service). This was effected by setting their MatchPoints
   to 0.

These changes are the result of lots of research, scanning, and analysis
over the last month or so. The goal was to improve detection across many
network hops, even when there are packet-mangling routers in the way.

The T (measured initial TTL) and TG (guessed initial TTL) tests commonly
failed, with T failing as much as 50% of the time in both random
Internet and known-hosts scans. TG didn't fail as often, but there was a
bug in one of the fingerprint tools that meant many reference
fingerprints in nmap-os-db has TG values that would never be produced by
Nmap, and would never match. This has been fixed.

UI.TOS and IE.TOSI were trickier. They were among the leading failing
tests, but it wasn't clear what do with them. It is common for the type
of service to be set to zero by network nodes; most of the mismatches
that weren't otherwise accounted for were because the field was zero.
After some discussion, Fyodor and I decided that because there aren't
many different likely results for these tests, and because they are
commonly mangled in transit, to remove the tests.

My somewhat haphazard notes and statistics are at
http://www.bamsoftware.com/wiki/Nmap/OSDetectionAnomalies.
I apologize that they are not set up for presentation, but anyway
there's no reason not to disclose them. I'll explain anything you want
to ask me about. A better presented summary of the effects of these
changes, separately and all together, is at
http://www.bamsoftware.com/wiki/Nmap/ReferenceOSScans2.

I want to thank Jason DePriest, Brandon Enright, Dave Moore, Vijay
Sankar, Matt Selsky, and Fyodor for volunteering hosts to scan.

David Fifield

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org