Re: PR#977: Version detection: version.bind / fallbacks

[Tom Sellers <nmap () fadedcode net>]

All,
  I'll likely commit the version detection changes tomorrow night.  Any additional testing or feedback would be 
appreciated.

Thanks much,
Tom

On 8/21/2017 9:17 AM, Tom Sellers wrote:
> All,
>  The following is text from Github PR#977 ( https://github.com/nmap/nmap/pull/977 ). Hopefully the formatting 
> survives the transition to email.
>
>
> The goal of this PR is to use data from a Project Sonar Internet wide survey of DNS responses to a version.bind query 
> on both TCP and UDP to improve Nmap's version detection coverage. As part of this
> effort a couple of quality and consistency issues were addressed. Full disclosure, I work on the Rapid7 team that 
> runs Project Sonar.
>
> Note: Core version detection fallback logic was changed.
>
> Summary
>
> The DNS query response packet over TCP and UDP only differ by one field. The TCP version contains a two byte length 
> field at the start of the response data. This means that we can use the same match
> lines for both probes if the regex is constructed with this in mind and fingerprint fallbacks work cross protocol. 
> This PR implements cross protocol fallbacks by making changes to
> *AllProbes::getProbeByName which is only used in the fallback process. This should not break any existing 
> functionality since fallbacks are only currently used to fall back to GetRequest.
>
> Prior to the above, match lines were implemented separately in the TCP and UDP DNSVersionBindReq probe sections. 
> Given the organic growth of these sections over time they were inconsistent in coverage
> and had many match quality issues. Many of the fingerprints could have never fired due to over broad fingerprints 
> that occurred earlier in the match process. Additionally there were many fingerprints
> that generically matched a DNS query response but that had no service specific data in the regex.
>
>
> To address this I have:
>
>  - Moved all version.bind match lines to the UDP DNSVersionBindReq probe section
>
>  - Renamed the TCP DNSVersionBindReq to TCP DNSVersionBindReqTCP and configured it to fall back to UDP 
> DNSVersionBindReq Matchlines for this response should now only occur in one section, simplifying
> maintenance. I have made similar but very limited changes to the UDP and TCP DNSStatusRequest probe.
>
>  - Created new matchlines based on a Project Sonar Internet wide survey of DNS responses to a version.bind query on 
> both TCP and UDP. The new match lines have been grouped by product/os where possible
> and roughly ordered by occurrence on the public Internet. We will be publishing some of the metrics in the near 
> future.
>
>  - Improved the accuracy and coverage of cpe elements in the matches.
>
>  - Removed many matchlines that were overbroad or simply didn't match on any product specific data.
>
>  - Tweaked the existing matches and wrote the new matches to just match on the query and answer section of the DNS 
> query response packet.
>
>  - Implemented softmatches for generic DNS errors commonly seen in responses such as Not Implemented, Refused, etc.
>
>
>
> Example output
>
> PORT   STATE SERVICE REASON               VERSION
> 53/tcp open  domain  syn-ack ttl 223      Microsoft DNS 6.0.6001 (17714650) (Windows Server 2008 SP1)
> 53/udp open  domain  udp-response ttl 223 Microsoft DNS 6.0.6001 (17714650) (Windows Server 2008 SP1)
> Service Info: OS: Windows; CPE: cpe:/o:microsoft:windows_server_2008:-:sp1
>
> PORT   STATE SERVICE REASON               VERSION
> 53/tcp open  domain  syn-ack ttl 128      Microsoft DNS 6.3.9600 (25804287) (Windows Server 2012 R2)
> 53/udp open  domain  udp-response ttl 128 Microsoft DNS 6.3.9600 (25804287) (Windows Server 2012 R2)
> Service Info: OS: Windows; CPE: cpe:/o:microsoft:windows_server_2012:r2
>
> PORT   STATE SERVICE REASON               VERSION
> 53/tcp open  domain  syn-ack ttl 128      Microsoft DNS 10.0.14393 (383905FC) (Windows Server 2016)
> 53/udp open  domain  udp-response ttl 128 Microsoft DNS 10.0.14393 (383905FC) (Windows Server 2016)
> Service Info: OS: Windows; CPE: cpe:/o:microsoft:windows_server_2016
>
> PORT   STATE SERVICE REASON              VERSION
> 53/tcp open  domain  syn-ack ttl 51      ISC BIND 9.8.2rc1 (RedHat Enterprise Linux 6)
> 53/udp open  domain  udp-response ttl 48 ISC BIND 9.8.2rc1 (RedHat Enterprise Linux 6)
> Service Info: OS: Linux; CPE: cpe:/o:redhat:enterprise_linux:6
>
> PORT   STATE SERVICE REASON              VERSION
> 53/tcp open  domain  syn-ack ttl 50      ISC BIND 9.9.5 (Debian Linux 7.0 (Wheezy))
> 53/udp open  domain  udp-response ttl 50 ISC BIND 9.9.5 (Debian Linux 7.0 (Wheezy))
> Service Info: OS: Linux; CPE: cpe:/o:debian:debian_linux:7.0
>
> PORT   STATE SERVICE REASON              VERSION
> 53/tcp open  domain  syn-ack ttl 46      ISC BIND 9.9.5 (Raspbian Linux 8.0 (Jessie based))
> 53/udp open  domain  udp-response ttl 46 ISC BIND 9.9.5 (Raspbian Linux 8.0 (Jessie based))
> Service Info: OS: Linux; CPE: cpe:/o:debian:debian_linux:8.0
>
> PORT   STATE SERVICE REASON              VERSION
> 53/tcp open  domain  syn-ack ttl 56      UltraDNS Resolver
> 53/udp open  domain  udp-response ttl 56 UltraDNS Resolver
>
>
>
> Thanks,
>  Tom Sellers
>  @TomSellers
>
> Reference:
>  - Project Sonar: https://sonar.labs.rapid7.com/
> _______________________________________________
> Sent through the dev mailing list
> https://nmap.org/mailman/listinfo/dev
> Archived at http://seclists.org/nmap-dev/

_______________________________________________
Sent through the dev mailing list
https://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/