some nmap tools

[MadHat <madhat () unspecific com>]

I have the responsibility of monitoring a large number of IPs for 
security issues.  One of the most important things for me was to know 
what was listening where and of course nmap is the only real solution.  
 The problem was that my boss wanted me to be able to generate a report 
of how many new ports were opened in the last 24 hours, how many new 
hosts in the past 24 hours, or even how many hosts we have live that 
are Internet facing or web servers, etc...

I looked at a lot of the tools out there.  Many of them are great, but 
did not scale to the size I needed.  I ended up writing some simple 
scripts in perl to do what I needed to do.

http://www.unspecific.com/.go/nmap/

To give an example of what I am doing, I scan 80K+ IPs spread around 
the world in about 10 hours from a single, centralized host at this 
time.  Of these, about 20K+ are accessible from the general Internet.  
I do basic scans of the registered ports and do an OS fingerprint.  I 
have not added Version fingerprinting as of yet.

There are 2 main scripts I use daily, being the nmap-wrapper (which is 
really in constant flux trying to find the best configuration) and 
nmap-diff.

nmap-wrapper runs X number of nmap processes in parallel (I currently 
use 32).  Each process only scans a single host for a few reasons.  The 
data is taken back into the script and log files are created.  Each 
class C has it's own log file per day.  Not the best solution, but the 
easiest and most efficient I could find in a hurry. ( I have looked 
into the nmap-sql, but they don't seem to scale well for me.  But I am 
still looking.)  I scan a single host at a time because I want to know 
exactly when each host is being scanned (the start and stop time), but 
this has been fixed in later versions of nmap and verbose outputs.  I 
also used a single host this way because, at least in older versions of 
nmap, I had issues with timing.  It seemed that when trying to do it 
all at once, if it hit a block of hosts not responding, it would take 
forever to time out (which is where a lot of my tweaking has been) and 
not finish in a timely manner.  The way nmap wrapper is written if one 
process hangs, it keeps going, for as one process ends it launches 
another and tries to keep X processes running at all times.

nnmap-diff is run every day after the nmap-wrapper is completed (see 
website for example).  It does a host-by-host diff from the previous 
day (by default it is from 48 hours ago, but you can specify how many 
days back the base should be) and will gen a basic report of what 
changed (more specifically, by default it only shows what was added and 
not was went away).  You can also only look for changes on specific 
ports, or specific IPs or partial IP matching.  It also has the ability 
to mail the output to an email address, but I usually just pipe it to 
mail ;)

Using these I can detect within 24 hours of when a firewall hole is bad 
or a host has some unauthorized app listening on a non-standard port, 
etc...  in theory anyway and I have detected some good stuff.

There are a couple of other scripts up there as well.

nmap-search, which I have sent to the list before (years ago when I 
wrote it), which allows you to search a single nmap -oG output based on 
a few different things.

nmap-report, which is similar to the above, but works on the flat file 
system generated by nmap-wrapper.  I can tell it to give me all the 
hosts that have port 80 open, whether or not to include the DNS entry 
found by nmap at the time of the run, whether or not to report all the 
details found on hosts that match, match on OS, or partial IP (was 
going to be subnet, but I have not had a chance to fix it), or just 
search on a specific IP.  It also allows you to specify how many day 
ago to search.  The default searches yesterdays DB, but you can tell it 
to go back 3 days (-b3) and look for an answer then.

I have been using these for about a year and have had good success thus 
far.  I am about to start rewriting some of them and recently had the 
opportunity to speak to Fyodor about these and some ideas I had and he 
recommended I pass them on here as other may want to check them out.  
So here it is....

http://www.unspecific.com/.go/nmap/

Feedback is welcome as always.

Enjoy
--
MadHat (at) Unspecific.com


---------------------------------------------------------------------
For help using this (nmap-dev) mailing list, send a blank email to 
nmap-dev-help () insecure org . List archive: http://seclists.org