Nmap Development mailing list archives
How I scan large networks (was Re: Running Malware Scripts)
From: Brandon Enright <bmenrigh () ucsd edu>
Date: Wed, 31 Dec 2008 20:58:01 +0000
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 [ This initial discussion was about scanning a large network looking for compromised hosts. I have expanded it to include how I scan large networks in the first place. ] On Wed, 31 Dec 2008 11:39:43 -0800 or thereabouts "Rathbun, Dan" <Dan.Rathbun () aecom com> wrote:
Brandon, Does your script have any dependencies? I am running RHEL 5. -Dan
The only dependency problem I can think you could run into is missing perl's NetAddr::IP module. I don't know anything about how RHEL is managed but if you run 'cpan -i NetAddr::IP' as root, you should get the module. If you send me the error you are getting I'll tell you what you need. Brandon
Thanks. Yes it was 'NetAddr::IP' that was missing. I appreciate your response on that. I ended up killing the scan of our LA subnets after nearly 5 days. I think I will move this to a more beefy server, but I also don't think I can afford to do -PN right now. Can I ask what NMAP switches you usually prefer for this application? I need to develop a process that balances performance against results. Thank you, (and Happy New Year!) -Dan
This is a complicated question and the answer falls into two categories: "What I do today" and "What I plan to do in the near future when David finishes his work on nmap-perf". In short, today I scan with one Nmap process per IP. To achieve any speed at all I run many Nmap processes in parallel. I wrote a deadline scheduler in perl to actually handle the execution of Nmap processes. David has put a lot of work into optimizing Nmap and working on hostgroup performance so in the future I will scan groups of between 4 and 32 IPs with a single Nmap process. I will determine the optimal number for my purposes empirically. Now for the long answer: First thing I do is I maintain a list of netblocks that I want to scan as well as IPs I want to exclude. I then run (script attached): #!/bin/bash # This script lives in /home/bmenrigh/ip-lists/ cat excludes/*.txt | sort | uniq > excludes/all.txt nmap -n -iL campus_netblocks.txt --excludefile excludes/all.txt -sL -oG - | egrep '^Host' | awk '{print $2}' | sort | uniq > campus_scanable.txt cat campus_scanable.txt | sort -R > campus_scanable_rnd.txt This produces two files, one with all of the IPs, one per line listed. The other is the same list, just randomized. I then setup the file that actually does the scanning with the command line options that I want (script attached). Here it is: #!/bin/bash # This script lives in /home/bmenrigh/flexmap NMAP="/home/bmenrigh/flexmap/nmap/svn/nmap/nmap" DATADIR="/home/bmenrigh/flexmap/nmap/svn/nmap/" EXCLUDE="/home/bmenrigh/ip-lists/excludes/all.txt" cd /home/bmenrigh/flexmap/ ulimit -c unlimited ulimit -v 262144 # Get this damn memory leak under control sudo nice $NMAP --datadir=$DATADIR -p- -P0 -sV --version-all --allports --script=malware,intrusive --open -ttl 10 -O2 -vv -d -T5 --min-parallelism 128 --max-parallelism 512 --min-rtt-timeout=1 --max-rtt-timeout=300 --initial-rtt-timeout=150 --max-retries=1 --min-rate 250 --max-rate 4000 --host-timeout 900000 -oA ../flexmap/log/report_$1 --excludefile $EXCLUDE $1 > ../flexmap/log/report_$1.txt 2> ../flexmap/log/report_$1.err Now, the above script scans a single IP relatively quickly. To scan a couple hundred thousand hosts I've written a deadline scheduler (attached) to manage the parallelism and speed up or slow down accordingly. It reads IPs from STDIN and forks off the above script to Nmap single IPs (again, in the future I'll modify this to do scans in chunks). You run the script like so: $ cat /home/bmenrigh/campus_scanable_rnd.txt | ./fastnmap.pl And the script produces output like: - - ------ Scan started at Mon Dec 29 03:52:00 2008 UTC Goal end time at Mon Jan 5 03:52:00 2009 UTC 70800 IPs done (38.61%) in 231689 seconds with 51.46 average threads Average thread-work-factor: 5.94 mIST Currently using 51 threads, will finish at Mon Jan 5 02:33:16 2009 UTC Target off of goal by -78.72 minutes Network rate: TX 39759 pps (2.20 MBps); RX 577 pps (0.03 MBps) - - ------ ...this output updated every 100 hosts scanned... Now there are a few adjustable parameters that you'll need to edit in the script to get it to run to your liking. First, the script is deadline based which means it tries to get the scanning job done in a certain amount of time (the goal time). The default is 7 days but you can edit "my $finish_in = 7 * 24 * 60 * 60;" to be any reasonable amount of time. The other thing you may want to adjust is the minimum free memory amount (in K) "my $min_free_mem = 500000;". The memory stats don't take swap into account so if you use swap adjust "sub mem_stats" accordingly. Once you have things edited and run the script, it will print out the stats (above) that report how it is doing. The average threads count is the average over the lifetime of the script. It uses past performance to predict future performance and needed parallelism. The "thread-work-factor" is the measure of how efficient a thread is. 5.94 mIST (milli-IPs-per-second-per-thread) means that a thread averages scanning 5.94 hosts in 1000 seconds. Based on that performance metric, the scheduler will adjust the parallelism up and down to try to meet the goal finish time exactly. Over 7 days it is usually off by between 5 and 10 minutes. The current expected miss amount of -78.72 minutes means that at the current rate, the script will finish 78 minutes early. Different networks and different scanning machines behave differently (surprise!). UCSD is basically powered off and on vacation right now which means scanning isn't very efficient. During normal days, the thread-work-factor we get is about 6.50. By editing the Nmap options in the bash script, you can make it scan faster or slower which will directly effect the thread-work-factor. The machine I use to scan is a 8 proc, 8 GB mem, Intel Pro 1000 box. If you have less than about 6 GB of memory you'll need to keep the $min_free_mem variable set reasonably. I'm running Linux 2.6.26 x86_64 with Ethernet MULTIQUEUE and NAPI support turned on. I also tweak the networking options with sysctl: # Disables source route verification net.ipv4.conf.default.rp_filter = 0 # Disable reverse path net.ipv4.conf.all.rp_filter = 0 net.core.rmem_default = 65536 net.core.wmem_default = 65536 net.core.rmem_max = 536870912 net.core.wmem_max = 536870912 net.ipv4.tcp_rmem = 65536 65536 536870912 net.ipv4.tcp_wmem = 65536 65536 536870912 net.ipv4.tcp_mem = 33554432 134217728 1073741824 net.core.netdev_max_backlog = 16384 net.ipv4.tcp_max_syn_backlog = 16384 net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_tw_reuse = 1 net.unix.max_dgram_qlen = 64 fs.file-max = 65535 vm.min_free_kbytes = 131072 kernel.core_pattern = core-%e-%p-%t I also set the TXQUEUELEN on the NIC higher: # ifconfig eth0 txqueuelen 16384 I also max out the hardware ring buffer on the card with ethtool: $ sudo ethtool -g eth0 Ring parameters for eth0: Current hardware settings: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 4096 Our next step with this box is to upgrade the Gigabit card to a Intel 10 Gb card. Everything I've read suggests that the 10 Gb cards service packets from multiple hardware queues much better than the 1 Gb cards do. Of course, once scanning commences, I can start looking at intermediate results with npwn. I just point it at the log directory and give it a max scan age of 7 days: $ ./npwn.pl -a 14 -e awknowledged.npwn,notified.npwn -d ~/flexmap/log The awknowledged.npwn and notified.npwn files are excludes for certain issues that Npwn can report that I don't care about. I think that just about covers how I scan. Hope it helps. Brandon PS: The my (and a co-worker's) far off goal is to replace all these command line scripts with a database-driven webapp that manages, schedules, and analyses scans. Progress is slow going but ultimately we plan to release it as FOSS. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAklb3NkACgkQqaGPzAsl94JcaACgs+vqVWgF2U6FjuWbD2c2IimD CE0Anie4KYTFvu67c51gy5UrFkSby555 =+ZEe -----END PGP SIGNATURE-----
Attachment:
build_scanable.sh
Description:
Attachment:
fastnmap.pl
Description:
Attachment:
flexmap_one_ip.sh
Description:
_______________________________________________ Sent through the nmap-dev mailing list http://cgi.insecure.org/mailman/listinfo/nmap-dev Archived at http://SecLists.Org
Current thread:
- Running Malware Scripts Rathbun, Dan (Dec 24)
- Re: Running Malware Scripts Brandon Enright (Dec 24)
- Message not available
- Message not available
- Message not available
- How I scan large networks (was Re: Running Malware Scripts) Brandon Enright (Dec 31)
- Message not available
- Re: Running Malware Scripts Brandon Enright (Dec 24)