Nmap Development mailing list archives
SoC Feature Creeper and Performance Czar Tasks
From: Fyodor <fyodor () insecure org>
Date: Sat, 13 May 2006 22:55:13 -0700
While some of the Google-sponsored students will be working on a single large project for the whole summer (like a new GUI, or adding scripting support to Nmap), another group called the "feature creepers" will each have a number of smaller tasks which will allow them to explore the whole code base and contribute in many different areas. There will also be (probably) one person who focuses on performance/QA and code cleanup tasks. I have created the list below of some tasks I would like to see see done this summer. They vary from simple tasks that shouldn't take more than an hour (like -q), to much larger projects that could take a week or two (like fixed-rate packet sending, traceroute, or application scanning). I'll do as many as I can myself, and I'm sure the feature creepers will finish a bunch of them. Also, if any of you get the itch to do a little Nmap development work, this is a great list to pick ideas from. Code fixes/improvements are accepted from all quarters! It is best if you let me know when you are starting work so I can make sure someone else isn't doing the same thing. Also, if you have a pet peeve about Nmap or an improvement idea, post it to nmap-dev as a possible addition to the list. With all that, here is the initial task list: [ GENERAL TASKS ] o Fix Nmap so that regardless of order given, -T arguments are processed before specialized timing args. It may be easiest to store the specialized arguments in variables until option processing is done, and then process them. Or set flags such as max_rtt_set when you set that, then the -T option processing could check the flag before mucking with the rtt. Currently, you need to specify the -T option before any specific timing options like --max-rtt-timeout. o Add parallel traceroute support to Nmap. It should come after the port scanning and host enumeration sections, and utilize a port/protocol that Nmap has found to be accessible on the particular target. Multiple hosts should be done in parallel (as with most Nmap scans), and multiple probes done in parallel too. The Nmap parallel rDNS infrastructure should be used to look up the intermediate hosts (unless -n was specified), because that system does caching (be sure to verify that they are indeed being cached, as many of the intermediate hosts will be repeated for each machine). Printing to normal output should be concise -- for example you can show the whole path for the first system and then only show for the first point of path divergience for the next system. You may even be able to do the probes backwards and cease probing once you find a machine that you already know the path to. XML output format should give the entire route for each host, since space isn't as serious of a constraint there. o Fix UDP scan such that it doesn't find its own port open when scanning localhost (try nmap -sU -p- localhost). o Remove inet_aton.c from nbase and change all Nmap calls to inet_pton. The duplication is pointless, plus the present inet_aton.c is BSD licensed WITH ADVERTISING CLAUSE. o Remove massping() in favor of doing all host enumeration from ultra_scan(). o Once this is done, make default host host discovery more comprehensive, at least when !isdirectlyconnected(). o Maybe should add an option like --reason which adds a column to the port table explaining why each port is considered up/down/whatever. A typical reason might be "no response" or "port-unreach from XX.XX.XX.XX" o Write a general scanning engine for abusing applications for port scanning purposes. This would handle scanning through SOCKS and HTTP proxies, and the existing FTP bounce scan would also be ported to this engine. Proxy chaining must be supported. o Explore many different application protocols, trying to find ways for version detection to identify difficult services such as the new Skype protocols or DHCP. o Solar Designer notes: "When there're ports in more than two categories (say, open, closed, and filtered), the output is often way too long. For example, it is rather common to filter out ports below 1024 except those actually used for public services. Currently, Nmap would report the fact that "ports not listed are filtered", followed by a long list of all the "closed" ports it scanned with some "open" ones inbetween. Unless I grep out the "closed", it's very inconvenient to see which ports are open. And I often don't know a scan will produce this kind of result before I start it, so I don't redirect Nmap output anywhere or pipe it through grep, wait for a long time for it to complete the scan, and then have my terminal flooded..." o Solution: If there are more than, say 50 ports in a non-open state other than the ignored state, simply list them (comma separated, with ranges, like the XML output does with port lists). o Add -q (quiet) mode, which only prints open ports (maybe just a line specifying how many closed/filtered ports there are for a host, if there are any). This option may also omit certain low-priority messages. This would replace the undocumented "quash argv" functionality of -q. The code for that should accordingly be removed (if anyone uses that quash argv feature -- speak up now!). [ PERFORMANCE/QA TASKS ] General Performance Notes o While performance is very important, be wary of any performance improvements that: o Are likely to reduce Nmap accuracy in some realistic cases o Make the code/APIs significantly uglier, harder to use, easier to use insecurely, or otherwise disadventageous. o Is not portable o Performance improvements should generally be based on benchmarks showing that there is a problem in the first place, and demonstrating that the fix helps. o [PERF] Run valgrind and other memory debugging stuff. I will discover memory leaks and calls to malloc that can be optimized. Add a command-line option which has Nmap free as much memory as practical before quitting, to aid detection of true memory leaks. o [PREF] Run Windows performance/quality testing tools (maybe through trial versions where they are commercial). Benchmark Windows vs. Linux Performance and look at ways to improve areas where Windows is much slower. Poll nmap-dev for ideas as well. o There are some apparently good suggestions for improving Nmap compilation on windows (simple things, like changing optimization flags) in 7/16/03 mail from "bingle" <bingle2000(a)hotmail.com> o Windows TCP connect() scan may be limiting itself to too few sockets, because it looks like max_sd() always returns zero for Windows. Investigate. Maybe I should limit it to 9 due to the new SP2 limitations. o [PERF] Scan a large number of machines across the Internet, look at accuracy and timing issues. Try to improve algorithms without sacrificing accuracy. Do similar tests on a LAN. Since Nmap can detect whether a host is on a LAN or has to go through a router, maybe Nmap should use more aggressive default options in the former case. And even more aggressive if Nmap detects that it is scanning localhost. Nmap has functions like islocalhost() and isdirectlyconnected() to easily detect these conditions. Tune the various Nmap performance values, such as those in init_perf_values(). Be very, very careful not to sacrifice accuracy for performance. o [PERF] Run profilers (such as gprof) against Nmap, identify and fix the resource hogs o [PERF] Add a fixed-rate packet sending mode to ultra_scan() (like what UnicornScan and ScanRand offer). Must be done cleanly since this is critical Nmap engine code. o [PERF] dentify and eliminate dead and duplicate code.There are probably many functions in the Nmap code base which aren't even used. There are also functionality duplication (like having tcpip.h packet headers and the same in dnet) which could be removed by standardizing on one or the other. o [PERF] Rewrite Portlist class in a more efficient way, which takes advantage of the way it is used (e.g. out-of-order inserts are very common, searches are very common, sorted iteration is common). It uses too much CPU time right now. o [PERF] Investigate ICMP error message handling -- see ( http://xtrmntr.org/ORBman/tmp/nmap/ ) for an example patch, also see http://seclists.org/lists/nmap-dev/2006/Apr-Jun/0054.html o [PERF] Go through tcpip.h and such and remove extraneous stuff, standardize on dnet packet headers, etc. o [PERF] I need to fix pcap so that it has a proper timeout, which is set for each pcap_next. Currently, I just set a very low timeout in open_live and then my readip_pcap will call it over and over until the real timeout is reached. o Consider changing Nmap to achieve non-blocking pcap read by setting the pcap sd nonblocking on Linux rather than my patch. See 1/16/03 mail from Pavel Kankovsky to Nessus list. [ Adding a general timeout to pcap may be better. ] o [PERF] Determine empirically which TCP and UDP ports are most commonly open. Add an easy way to scan just the most popular ports. Maybe this could be an extra designation in nmap-services, or maybe ordered arrays in the code like top_tcp_ports[] and top_udp_ports. Maybe -F should scan (just) all of the popular ports, and perhaps there should be an option to specify how many popular ports you want for each scanned protocol (e.g. --top-ports 24). Cheers, Fyodor _______________________________________________ Sent through the nmap-dev mailing list http://cgi.insecure.org/mailman/listinfo/nmap-dev
Current thread:
- SoC Feature Creeper and Performance Czar Tasks Fyodor (May 13)
- Re: SoC Feature Creeper and Performance Czar Tasks doug (May 16)
- Re: SoC Feature Creeper and Performance Czar Tasks Fyodor (May 24)
- Re: SoC Feature Creeper and Performance Czar Tasks AgentSmith15 (May 24)