Nmap Development mailing list archives
Writeup on `brute.lua` Modification v.2. Resource Management Feature.
From: Sergey Khegay <g.sergeykhegay () gmail com>
Date: Thu, 9 Jun 2016 01:33:22 -0400
## `brute.lua` write up. v.2 06.08.16 Given recent feedback and further discussion with Fotis I have added a new feature to the `brute.lua`. I call it Resource Management feature. Just to recall the issue: Patrick Donnelly gave a valuable comment on my previous changes made to `brute.lua`. Particularly, he noticed that there is no much logic to let a brute `Engine` instance spawn `max_threads` number of threads if the default value of the NSE's `--max-parallelism` setting is less than `max_threads` and not changed by the user. (It makes even less sense if we run several instances of the `Engine`) After couple tests I confirmed that it was really was a bad approach, and `brute.lua` indeed allocated more resources than it could use efficiently. I also confirmed, that increasing the default value of `--max-parallelism` indeed increases bruteforce speed. Here is a short table. ||===================================================================|| || max_threads >| 20 | 20 | 50 || 50 | || || max-parallelism >| 20 | 50 | 20 || 50 | least || ||--- #accounts ----|--------|--------|--------||--------|- speedup -|| ||------------------|--------|--------|--------||--------|-----------|| || 400 | 86.34 | 91.08 | 84.85 || 75.48 | 112% || || 1000 | 178.05 | 179.28 | 167.54 || 139.79 | 119% || ||-------------------------------------------------------------------|| Table values are in seconds. Although least speedup column might not look really impressive, I think that with more accounts the numbers might look more optimistic. The reason is that with more accounts `brute.lua` will have more time to gain momentum, i.e. spawn more threads and increase average tries per second, as we do that work gradually. # RESOURCE MANAGEMENT FEATURE Now back to the Resource Management feature. The idea is simple: we just need to know if there are any threads waiting in queue to make a connection. If there are any, then there is a big possibility that we are hogging too much resources. That information can be taken from this piece of code: /* Some constants used for enforcing a limit on the number of open sockets * in use by threads. The maximum value between MAX_PARALLELISM and * o.max_parallelism is the max # of threads that can have connected sockets * (open). * * THREAD_SOCKETS is a weak keyed table of <Thread, Socket Table> pairs. * A socket table is a weak keyed table (socket keys with garbage values) of * sockets the Thread has allocated but not necessarily open). You may * test for an open socket by checking whether its nsiod field in the * socket userdata structure is not NULL. * * CONNECT_WAITING is a weak keyed table of <Thread, Garbage Value> pairs. * The table contains threads waiting to make a socket connection. */ taken from https://github.com/nmap/nmap/blob/eaf903879181463ac17702fa3e4e92b1737c31e9/nse_nsock.cc#L155-L163 Basically, we are interested in the number of entries in CONNECT_WAITING table. Information in the THREAD_SOCKETS might be useful as well and might be used for further improvements of `brute.lua`. I coded a simple hook in the nse_nsock.cc // nse_nsock.cc: 1055 /* This function also has a binding in stdnse.lua */ static int l_get_stats (lua_State *L) { lua_newtable(L); int idx = lua_gettop(L); /* the only field so far is connect_waiting - number of threads waiting for connection */ lua_pushinteger(L, nseU_tablen(L, CONNECT_WAITING)); lua_setfield(L, idx, "connect_waiting"); return 1; } //--- Which is also accessible from `stdnse.lua` library. And logically should be used like this `local stats = stdnse.get_stats()`. // stdnse.lua: 1409 --- Returns current connection statistics object -- @return A statistics object. Currently has only field "connection_waiting" function get_stats() return nmap.socket.get_stats() end //--- # RESULTS I do not have a good table to show so far, but let me describe a bit. Environment: - remote server on Amazon AWS - local virtual machine - vsftpd daemon for ftp protocol I scan both machines at the same time, so two instances of the brute `Engine` are running. Nmap runs with following settings (they are set so intensionally). max_threads: 20 max-parallelism: 10 Run WITHOUT feature: o. Both `Engine` instances tried to spawn maximum allowed number of threads, namely 20. So at some point there were ~40 threads in total. Only 10 of them could work effectively, other were put into CONNECT_WAITING table. o. Valid credentials were found on both servers. o. Running time: 55.23 sec. Run WITH feature: o. Both `Engine` instances spawned no more that 12 threads in total, which is consistent with our `--max-parallelism` limit. o. All running threads were equally split between instances. This happened only because both instances started almost at the same time, so they were equally increasing the number of threads, until the total number of threads hit the limit. Generally it might not always be the case. If one instance starts much earlier then another than the first one would have an advantage of using more threads. But I consider this a normal behavior. o. Valid credentials were found on both servers. o. Running time: 57.58 sec. So both runs completed in almost the same time, but obviously the run with feature implemented used less resources. I also performed a test with one of the servers playing a bad guy, limiting the number of connections per ip to 5, and with following settings: max_threads: 20 max-parallism: 20 As was expected the `Engine` instance which were bruteforcing a "bad guy" spawned only ~5 threads (due to connection and protocol exception adaptability mechanisms first and resource management feature later) and the other instance used the rest 15 available slots for threads. # CODE The code is available here: https://github.com/sergeykhegay/nmap/tree/gsoc-brute Files to review: nse_nsock.cc nselib/brute.lua nselib/stdnse.lua scripts/ftp-brute.nse # GENERAL CONSERN I need to find a good metric to alert the engine to stop, as for now the `Engine` is too stubborn and, theoretically, might never give up on retrying bruteforce attempts.
_______________________________________________ Sent through the dev mailing list https://nmap.org/mailman/listinfo/dev Archived at http://seclists.org/nmap-dev/
Current thread:
- Writeup on `brute.lua` Modification v.2. Resource Management Feature. Sergey Khegay (Jun 08)
- Re: Writeup on `brute.lua` Modification v.2. Resource Management Feature. Patrick Donnelly (Jun 09)
- Re: Writeup on `brute.lua` Modification v.2. Resource Management Feature. Patrick Donnelly (Jun 09)
- Re: Writeup on `brute.lua` Modification v.2. Resource Management Feature. Daniel Miller (Jun 13)
- Re: Writeup on `brute.lua` Modification v.2. Resource Management Feature. Patrick Donnelly (Jun 09)