OS Detection Suggestions?

[Fyodor <fyodor () insecure org>]

Nmap Hackers,

I'm excited about all of the Google SummerOfCode projects reported
earlier, but am particularly interested in the OS Detection redesign!
I have been working on that with Zhao Lei for weeks.  We have been
reading RFCs, casting experimental packets in all directions, reading
related research, etc.  His latest draft proposal is below, and
implementation work is just beginning.  We are adding quite a few
useful tests, packed into as few probes as possible.  But maybe some
of you have other clever ideas?  Remember that changing the system
requires (for best results) creating a new DB from scratch, so this
won't be done often!  The current OS detection scheme has served us
well for 7 years, and I'd like the new one to last at least as long!
So now is your chance to influence the new system.  If you have any
ideas, please post 'em to nmap-dev () insecure org (or send them to me
directly, if you prefer that).  Join that list (or read it on
seclists.org) if you wish to participate in ensuing discussion.  Of
course we can't add every possible.  Here are the "ground rules" for
evaluating tests that we used for the new design:

o The tests must not require Nmap to mess with the host firewall.  So,
  for example, SYN/ACK retransmission timing is out because Nmap would
  need to block the host from receiving the SYN/ACKs and RST'ing them.

o The tests must be quick.  RTT timeout is also out because waiting a
  minute or two for RTT counting is unacceptable.  There is no absolute
  cut-off time though -- the amount of time a test takes is 
  considered against the value of information it provides.  Ideal tests
  simply require one packet which generates an immediate response.  Even
  better are tests that can be piggy-backed on an existing test by
  modifying the probe or analyzing the response more carefully.

o I'm trying to avoid structurally bogus/corrupt packets.  Packets
  which blatantly violate the RFCs are more likely to be
  dropped/scrubbed on the network, crash the target machine, be
  reliably detected by IDS systems, etc.

o I'm trying to avoid sending any fragmented IP packets, as they are
  problematic to send and are frequently defragmented en route.

o Tests should require as few new probe packets as possible.  A
  congestion control test that requires hundreds of sent packets would
  be unacceptable.

o I'm trying to avoid making full TCP connections, although I won't
  rule that out.

o The new tests must actually work :).  Testing an attribute that all
  known hosts handle the same way doesn't do much good.

Note that these rules are a gray area.  I'm willing to consider
anything if the information gathered is substantial enough.

With that out of the way, I'm including Zhao's latest draft design
below.  The grammar is imperfect because I asked him to write it in
English rather than his native Chinese.  But the ideas are good!  Some
tests are mentioned here but not described in detail -- you can get
more background information on them from
http://www.insecure.org/nmap/nmap-fingerprinting-article.html and
http://seclists.org/lists/nmap-dev/2005/Jul-Sep/0039.html .

PROPOSED NEW NMAP OS DETECTION MECHANISM
by Zhao Lei (zhaolei(a)gmail.com) and Fyodor (fyodor(a)insecure.org)

######################
# Modifications of tests #T2-T7
######################

## Description

  o Original test1 is combined with Tseq probes.

  o I'd like to use Tseq probes to do option-combination tests,
    because we can expect the target really handles and responses to
    each combination, not simply return a, say, RST with no options.
    
** The rest of this section is not changed **
  
  o The modified 6 (test2 - test7) probes of current nmap, mainly
    adding the SACK option and setting the Acknowledgement value (to
    test seq value in RST+ packets).

    * I currently set the ACK value in all of the probe packets. I
      checked RFC793 and didn't find there are any requirements for
      when ACK shouldn't be set. This seq value test works! I found
      one "3Com SuperStack II switch (OS v 2.0), AsanteHub 2072
      Ethernet hub" (IP 59.66.79.3-13) set SEQ = Probe's ACK in T5,T7
      while others not.

  o Add and change some of the tests on the responses.

## Probe Packets Specification

# Test2 - A NULL with the same options to an open port

  IP DF flag: set
  TCP Sequence: Random1
  TCP Acknowledgement: Random2
  TCP Flags: none
  TCP Options: \003\003\012\001\002\004\001\011\010\012\077\077\077\077\000\000\000\000\004\002
   o Window Scale (\012)
   o MSS (\001\011)
   o Timestamps (TSval: \077\077\077\077; TSecr: 0)
   o SACK permitted
   
# Test3 - A SYN|FIN|URG|PSH with the same options to an open port

  IP DF flag: not set
  TCP Sequence: Random1
  TCP Acknowledgement: Random2
  TCP Flags: SYN|FIN|URG|PSH
  TCP Options: \003\003\012\001\002\004\001\011\010\012\077\077\077\077\000\000\000\000\004\002
   o Window Scale (\012)
   o MSS (\001\011)
   o Timestamps (TSval: \077\077\077\077; TSecr: 0)
   o SACK permitted

# Test4 - An ACK to an open port

  IP DF flag: set
  TCP Sequence: Random1
  TCP Acknowledgement: Random2
  TCP Flags: ACK
  TCP Options: \003\003\012\001\002\004\001\011\010\012\077\077\077\077\000\000\000\000\004\002
   o Window Scale (\012)
   o MSS (\001\011)
   o Timestamps (TSval: \077\077\077\077; TSecr: 0)
   o SACK permitted

# Test5 - An SYN to a closed port

  IP DF flag: not set
  TCP Sequence: Random1
  TCP Acknowledgement: Random2
  TCP Flags: SYN
  TCP Options: \003\003\012\001\002\004\001\011\010\012\077\077\077\077\000\000\000\000\004\002
   o Window Scale (\012)
   o MSS (\001\011)
   o Timestamps (TSval: \077\077\077\077; TSecr: 0)
   o SACK permitted

# Test6 - An ACK to a closed port

  IP DF flag: set
  TCP Sequence: Random1
  TCP Acknowledgement: Random2
  TCP Flags: SYN
  TCP Options: \003\003\012\001\002\004\001\011\010\012\077\077\077\077\000\000\000\000\004\002
   o Window Scale (\012)
   o MSS (\001\011)
   o Timestamps (TSval: \077\077\077\077; TSecr: 0)
   o SACK permitted
   
# Test7 - An FIN|PSH|URG to a closed port

  IP DF flag: not set
  TCP Sequence: Random1
  TCP Acknowledgement: Random2
  TCP Flags: FIN|PSH|URG
  TCP Options: \003\003\012\001\002\004\001\011\010\012\077\077\077\077\000\000\000\000\004\002
   o Window Scale (\012)
   o MSS (\001\011)
   o Timestamps (TSval: \077\077\077\077; TSecr: 0)
   o SACK permitted

## Tests Specification

** Changed: add TTL check, add A++ in Seq check, use checksum for Rst+
Payload **

  Check the following stuffs:
   DF, TCP Window size, Seq value, Ack value, Flags, Options, Quirks, RSTData

  Different with the current nmap:
   o Add seq value check, quirks check and RST payload check.
   o Modify the options check.
  
# The Seq value (Seq=xx)

  We expect the sequence number in the response would be one of the
  following 3 different patterns:

   o Z. It is zero;
   o A. It is the same with the ack we sent;
   o A++. It is ACK+1;
   o O. Other situations.

# The options (Ops=xx)

  o SACK-permitted. Denoted as 'S'.
  
  o WScale value. I think we can simply append the value to 'W' when
    compare. For example, "W3".

  o Timestamp value. Check whether the TSval is zero and whether the
    TSecr is zero. I think we can denote this as "Txx" (x=0|1).

# The Quirks (Qrk=xx)

  We can denote the quirks as the following:

  o R. The "reserved" field of TCP is not zero;
  o U. URG pointer value when urg flag not set.

# The Payload (RCK=xx)

  If it is RST+ packet and has payload, we count the checksum of the
  data and compare the checksum. When there is no data, we use
  RCK=0000.

  * Don't do this to SYN+ACK packets, because the data in this kind of
    packets may be service related.

## Fingerprint example

  DF, TCP Window size, Seq value, Ack value, Flags, Options, Quirks,
RSTData CheckSum

  T2(Resp=N)
  T3(Resp=Y%DF=Y%TTL=64%W=7FFF%SEQ=O%ACK=S++%Flags=AS%Ops=MST11NW0%Qrk=&RCK=0000)
  T4(Resp=Y%DF=Y%TTL=64%W=0%SEQ=A%ACK=O%Flags=R%Ops=%Qrk=%RCK=0000)
  T5(Resp=Y%DF=Y%TTL=64%W=0%SEQ=Z%ACK=S++%Flags=AR%Ops=%Qrk=%RCK=0000)
  T6(Resp=Y%DF=Y%TTL=64%W=0%SEQ=A%ACK=O%Flags=R%Ops=%Qrk=%RCK=0000)
  T7(Resp=Y%DF=Y%TTL=64%W=0%SEQ=Z%ACK=S++%Flags=AR%Ops=%Qrk=%RCK=0000)

################
# T1 & TSeq test
################

## Description

   o Original 6 TSeq probes, but with different options. The time
     intervals are the same with the original.
   o Use the first probe as if it is T1, and if there is no reply,
     resend the probe after the other TSeq tests are done.
   o Now we have the left 5 probes to do option-combination probes. If
     some don't response, give them flags and retry after TSeq.
     
## Probe Packets Specification

# Packet 1

  TCP Sequence: Random1
  TCP Acknowledgement: Random2
  TCP Flags: SYN
  TCP Option Length: 24
  TCP Options: \003\003\012\001\002\004\001\011\010\012\077\077\077\077\000\000\000\000\001\001\004\002\001\001
   o Window Scale (\012)
   o MSS (\001\011)
   o Timestamps (TSval: \077\077\077\077; TSecr: 0)
   o SACK permitted

# Packet 2

  TCP Sequence: Random1
  TCP Acknowledgement: Random2
  TCP Flags: SYN
  TCP Option Length: 24
  TCP Options: \002\004\001\011\003\003\000\001\004\002\001\001\010\012\077\077\077\077\000\000\000\000\001\001
   o MSS (\001\011)
   o Window Scale (\000)
   o SACK permitted
   o Timestamps (TSval: \077\077\077\077; TSecr: 0)

# Packet 3

  TCP Sequence: Random1
  TCP Acknowledgement: Random2
  TCP Flags: SYN
  TCP Option Length: 24
  TCP Options: \010\012\077\077\077\077\000\000\000\000\001\001\004\002\001\001\003\003\005\001\002\004\001\011
   o Timestamps (TSval: \077\077\077\077; TSecr: 0)
   o SACK permitted
   o Window Scale (\005)
   o MSS (\001\011)

# Packet 4

  TCP Sequence: Random1
  TCP Acknowledgement: Random2
  TCP Flags: SYN
  TCP Option Length: 24
  TCP Options: \004\002\001\001\010\012\077\077\077\077\000\000\000\000\001\001\003\003\012\001\002\004\001\011
   o SACK permitted
   o Timestamps (TSval: \077\077\077\077; TSecr: 0)
   o Window Scale (\012)
   o MSS (\001\011)

# Packet 5

  TCP Sequence: Random1
  TCP Acknowledgement: Random2
  TCP Flags: SYN
  TCP Option Length: 24
  TCP Options: \002\004\001\011\004\002\001\001\010\012\077\077\077\077\000\000\000\000\001\001\003\003\012\001
   o MSS (\001\011)
   o SACK permitted
   o Timestamps (TSval: \077\077\077\077; TSecr: 0)
   o Window Scale (\012)

# Packet 6

  TCP Sequence: Random1
  TCP Acknowledgement: Random2
  TCP Flags: SYN
  TCP Option Length: 24
  TCP Options: \002\004\001\011\010\012\077\077\077\077\000\000\000\000\001\001\004\002\001\001\003\003\012\000
   o MSS (\001\011)
   o Timestamps (TSval: \077\077\077\077; TSecr: 0)
   o SACK permitted
   o Window Scale (\012)
   o End of List

## Tests Specification

# T1

  Do t1 test as T2-T7

# TSeq

  Do Seq test as the current nmap does.

# TOps

  Get each option and compare.

## Fingerprint example

   TSeq(Class=RI%gcd=2%SI=39623%IPID=Z%TS=100HZ)
   TOps(O1=W10MT10S%O2=MST11W0=MST11NW0%O4=MST11NW0%O5=MST11NW0)
   T1(Resp=Y%DF=Y%TTL=64%W=7FFF%SEQ=O%ACK=S++%Flags=AS%Ops=MST11NW0%Qrk=%RCK=0000)



######################
# Two ICMP echo probes
######################
  
## Description

   Two tasks for the probes:

    o To cause two IPids reply in ICMP protocol to test if the way is per
      <src, des, protocol> triple IPid couter.

    o To test if the target echo the ICMP code field back and if any of
      Identifier and Sequence Number is zero.

   Why two probes?

    o We can get one more IPid;
    o To confirm some fields are really *echoed* other than set to that
      value (eg. DF, TOS)
      
## Probes Specification

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |     Type      |     Code      |          Checksum             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Identifier          |        Sequence Number        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |     Data ...
   +-+-+-+-+-

# ICMP probe 1

  IP DF: 1
  IP TOS: 0
  ICMP type: 8
  ICMP code: \011
  Identifier: random
  Sequence Number: \011\021
  DataLen: 120 bytes

# ICMP probe 2

  IP DF: 0
  IP TOS: \004 (IP_TOS_RELIABILITY)
  ICMP type: 8
  ICMP code: 0
  Identifier: the identifier in probe1 + 1
  Sequence Number: \011\022
  DataLen: 150 bytes

## Tests Specification

# Overview

  We have two icmp echo test and expect two icmp echo replies.

  Though it is natural to store one fingerprint for each reply (see
  below) and compare separately,

    TI1(Resp=Y%DF=Y%TTL=64%TOS=0%CD=0%Seq=S%DL=S)
    TI2(Resp=Y%DF=N%TTL=64%TOS=0%CD=0%Seq=S%DL=S)

  I prefer to combine the two into one single line:

    TI(Resp=Y%DF=N%TTL=64%TOS=0%CD=0%Seq=S%DL=S)

  because:
    o this will save much place for the db file
    o It is possible that the host doesn't reply to icmp echo
  messages at all, but it is very rare that the host responds to only
  one of them.
    o there are not much information dropped during the
  combination. Also what dropped might be very rare. (for example, two
  different non-zero TOS? two different non-zero Code?)

  * I ever thought to check if Icmp ID is zero, but I changed my
    mind. It is much easier if we use id to identify our replies and
    indeed I didn't see an implcation doesn't reply the ID field.

# Response

  1) Two replies: Resp=Y;
  2) If we don't get a reply: Resp=N;
  3) If we get only one reply, resend the other probe (say, one more
  time). Still no reply? Look upon as no reply: Resp=N.
  
# Test1: DF (DF=xx)

  * In the followin text, I'll use R1_xxx and R2_xxx to denote the
    value of xxx field of in the two reply messages.

  Patterns of DF:

   o N. Both R1_DF and R2_DF are not set;
   o S. Both use the DF value that the sender uses;
   o Y. Both set;
   o O. Other.

# Test2: TTL (TTL=xx)

  Count initial ttl based on R1_TTL. (The possibility for these two
  icmp echo reply have different ttl is very low)

# Test3: TOS (TOS=xx)

  Patterns of TOS:

   o [Value]. Both set TOS to the same value [Value] (this includes
     that both set to zero);
   o S. Both use the TOS that the sender uses;
   o O. Other.

# Test4: Code (CD=xx)

  Patterns of Code:

   o [Value]. Both set Code to the same value [Value];
   o S. Both use the Code that the sender uses;
   o O. Other.

# Test5: Seq (Seq=xx)

  Patterns of Seq:

   o [Value]. Both set Seq to the same value [Value];
   o S. Both use the Seq value that the sender uses;
   o O. Other.

# Test6: DataLen (DL=xx)

  Pattens of DataLen:

   o [Value]. Both truncted to a specific value;
   o S. Both the same with the sender;
   o O. Other.

## Fingerprint Example

   TI(Resp=Y%DF=N%TTL=64%TOS=0%CD=0%Seq=S%DL=S)



#######################################
# One TCP ECN-setup SYN to an open port
#######################################

## Description

  Purpose:
  o ECN support test;
  o Test if the target echo Urg pointer value with urg flag not set;
  o Test if the target echo reserved field.

## Probe Packet Specification
  
    0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15
  +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+
  |               |               | C | E | U | A | P | R | S | F |
  | Header Length |    Reserved   | W | C | R | C | S | S | Y | I |
  |               |               | R | E | G | K | H | T | N | N |
  +---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+

  TCP Sequence: Random
  TCP Acknowledgement: 0
  Reserved(0): 1
  TCP Flags: CWR ECE SYN  (* we look upon CWR&ECE as flags here)
  TCP Options: none

## Tests Specification

# Response

  1) one reply. Resp=Y;
  2) If we get RST, retry for a second time. Still nothing? then
     Resp=N.
  3) no reply. Resp=N.

# ECN support test (ECN=xx)

  Check if ECE is set, CWR is set, and compare the result to the
  following 3 patterns:

  1) Y: Only ECE is set. Receiver that supports ECN and replys
     correctly;
  2) N: Neither is set. Receiver that doesn't support ECN and sets the
     "reserved" bit to zero;
  3) S: Both are set. Receiver that doesn't support ECN but echoes the
     "reserved" bit back;

# Reserved field echo test (RE=xx)

  Denote as RE = Y, N

# URG pointer value when URG flag not set test (UP=xx)

  Denote as UP = Y, N


## Fingerprint example:
  
  TE(Resp=Y%TTL=64%ECN=Y%RE=N%UP=N)


###############################
# UDP probe (probe not changed)
###############################

  The probe is not changed. But we'll get the IPid from the expected
  ICMP packet and test if the "unused" field is not zero. (UN = Y, N)

  PU(DF=N%TOS=0%IPLEN=38%UN=Y%RIPTL=148%RID=E%RIPCK=E%UCK=F%ULEN=134%DAT=E)


###########################
# Final Fingerprint Example
###########################

  TSeq(Class=RI%gcd=2%SI=39623%IPID=Z%TS=100HZ)
  TOps(O1=W10MT10S%O2=MST11W0=MST11NW0%O4=MST11NW0%O5=MST11NW0)
  T1(Resp=Y%DF=Y%TTL=64%W=7FFF%SEQ=O%ACK=S++%Flags=AS%Ops=MST11NW0%Qrk=%RCK=0000)
  T2(Resp=N)
  T3(Resp=Y%DF=Y%TTL=64%W=7FFF%SEQ=O%ACK=S++%Flags=AS%Ops=MST11NW0%Qrk=&RCK=0000)
  T4(Resp=Y%DF=Y%TTL=64%W=0%SEQ=A%ACK=O%Flags=R%Ops=%Qrk=%RCK=0000)
  T5(Resp=Y%DF=Y%TTL=64%W=0%SEQ=Z%ACK=S++%Flags=AR%Ops=%Qrk=%RCK=0000)
  T6(Resp=Y%DF=Y%TTL=64%W=0%SEQ=A%ACK=O%Flags=R%Ops=%Qrk=%RCK=0000)
  T7(Resp=Y%DF=Y%TTL=64%W=0%SEQ=Z%ACK=S++%Flags=AR%Ops=%Qrk=%RCK=0000)
  TI(Resp=Y%DF=N%TTL=64%TOS=0%CD=0%Seq=S%DL=S)
  PU(DF=N%TOS=0%IPLEN=38%UN=Y%RIPTL=148%RID=E%RIPCK=E%UCK=F%ULEN=134%DAT=E)

 * Some label in PU may be shortened.




_______________________________________________
Sent through the nmap-hackers mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-hackers