Re: [Branch] --ignore-after

[Jay Bosamiya <jaybosamiya () gmail com>]

List,

Fyodor, Dan and I had a discussion on how --ignore-after should handle
percentages (whether it implies percentage of "total" or "so far").
After quite a lot of thinking, we decided on "so far". The reasons for
this are that it allows for fast bailing for hosts that we are (almost)
sure of being portspoofed. It has a vanishingly small probability of
falsely bailing on hosts that have a large number of valid ports open
(as opposed to really portspoofed hosts). We felt that this was a proper
balance of speed and guessing correctly (preventing false positive
bailing as much as possible).

However, we'd suggest you try out the feature from the branch (use `svn
co https://svn.nmap.org/nmap-exp/jay/ignore-after/`to get the code and
compile it `./configure && make && make install` and use it) and post
any suggestions. If any hosts that you know for certain are valid (not
portspoofed) are being ignored (in either -T4 or -T5), do let us know.

Just to specify again:

"--ignore-after XXX" means ignore any hosts with atleast XXX ports open
"--ignore-after NN%" means ignore any hosts with atleast NN% of total
scanned ports open
"--ignore-after NN%,XXX" means ignore any hosts with atleast XXX ports
open AND atleast NN% of ports scanned so far open (Note: This syntax
will be rarely used by end users but will be used by those who write
scripts that use Nmap; or internally in -T4 and -T5)

The -T4 and -T5 constants now are:
-T4 ----- 50%,80
-T5 ----- 20%,70

Increasing NN% would imply lesser false positives but would increase the
number of ports that need to be scanned before bailing.

Any suggestions/feedback/criticism is welcome as always :)

Cheers,
Jay


On Friday 19 September 2014 12:32 AM, Daniel Miller wrote:
> I definitely see what you mean: my suggestion has a pretty rotten
> worst-case. But let's keep in mind that any algorithm which ignores
> hosts is a time improvement over the current behavior. After all, the
> service discovery and script scanning phases can take up a lot of
> time. So I think the primary goal should be "properly detecting and
> ignoring a host which is lying to us." That is to say, we don't want
> to ignore a host that simply has a lot of services running. Keeping
> this in mind, see my inline replies:
>
> On Thu, Sep 18, 2014 at 9:30 AM, Jay Bosamiya <jaybosamiya () gmail com
> <mailto:jaybosamiya () gmail com>> wrote:
>
>     However, what if the first 100 ports are closed and all the rest
>     are open?
>     We'd end up bailing on the 900th open port.
>
>
> This still represents an improvement over the existing "never ignore"
> capability.
>  
>
>     Now, let's take the case of random port ordering (as Nmap does by
>     default). In this case, we would end up bailing soon on hosts that
>     have a large number of common ports open (since Nmap "cheats" by
>     placing common ports first) and would bail only after a long time
>     on hosts that have common ports closed but almost all uncommon
>     ports open.
>
>
> Nmap only moves 28 ports to the head of the queue. All other positions
> are random. So assuming a malicious host that wishes to give us the
> worst case performance, closing only these predictable ports in the
> 90%,90 case would require scanning 280 ports. In reality, the worst
> case would be having 89% of ports open, with the 28 being the closed
> ones, so that the host is not ignored yet maximizes the number of
> false open ports.
>
> This is really the selling point of your method: it is based on a
> fixed number of ports being open, and the number is low (and keeps
> getting lower as we discuss!). With my method, the malicious target
> can play a game of chance to see how close to the 90% mark they can
> keep their open ports before the running tally gets too high (like
> slipping face cards into a blackjack shoe without tipping off a dealer
> who is counting cards). A safe bet might be to spoof 70% of ports
> open, resulting in 700 fake ports slipping under the ignore filter and
> mucking up Nmap's results.
>  
>
>     Also, even if we scan the same host, due to randomization, the
>     bailing out would be very "random" since it'd have different
>     number of ports seen open before bailing. This basically gets rid
>     of reproducibility in a scan.
>     For example: take ports 1-91 open and scan order
>     1,2,3,4,5,...,1000 then Nmap would bail. However, if scan order
>     just got reversed, then Nmap would not bail. This just doesn't
>     seem right.
>
>
> I suppose this is just my overactive cautiousness, but I would rather
> use an algorithm that's hard to predict. A malicious target that knows
> your algorithm can simply choose 60 unused ports out of Nmap's default
> 1000 to spoof and they will be invisible. But if somewhere there is a
> machine with 60 real services running, it will be hidden as well. It's
> just as easy (or easier) to run portspoof and fake every port open,
> but then both methods will detect and ignore it.
>
> I think that reproducibility matters only in results, not in process.
> The result is binary: ignore or not. Whether we scan many ports or few
> ports should not matter greatly so long as the result is the same each
> time.
>
> We really only need to handle two cases:
>
> 1. All ports are open, the majority of those being fake: ignore. Some
> filtered/closed would be permitted to allow for dropped packets and
> down services allowed through the spoofing firewall.
> 2. Some significant portion of ports are not open: do not ignore. I
> think even if half the ports are closed/filtered, that's a data point
> that is important.
>
> Remember the principle of least astonishment. The only feature that
> has been requested was the ability to ignore hosts running portspoof
> (100% of ports open). Fyodor also brought up that it would be useful
> for wide-scale scans of a few services to weed out false positives (by
> specifying an unusual port in addition to the desired port and
> ignoring if it is open), but either method can provide that through
> the CLI option. Everything in between is a gray area, and I think we
> should tread lightly.
>
> All this to say, I still am not sure which way to go!
>
> Dan

_______________________________________________
Sent through the dev mailing list
http://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/