Re: [nmap-svn] r28585 - nmap/todo

[Patrick Donnelly <batrick () batbytes com>]

On Tue, May 15, 2012 at 5:16 PM,  <commit-mailer () insecure org> wrote:
> Author: fyodor
> Date: Tue May 15 14:16:49 2012
> New Revision: 28585
>
> Log:
> Add a todo entry about fixing NSE pipelining
>
> Modified:
>   nmap/todo/nmap.txt
>
> Modified: nmap/todo/nmap.txt
> ==============================================================================
> --- nmap/todo/nmap.txt  (original)
> +++ nmap/todo/nmap.txt  Tue May 15 14:16:49 2012
> @@ -123,6 +123,27 @@
>   o Review NSE-based port scanning and RST idle scan.
>     http://seclists.org/nmap-dev/2011/q2/307. [Henri and Hani?]
>
> +o Investigate why http pipelining so often doesn't work in NSE
> +  scripts, and often NSE ends up reverting to one request at a time.
> +  Scripts may not be using it correctly, and also we wish it were more
> +  transparent and there wasn't this big API divide between pipeline
> +  and non-pipeline.  We just want it send requests as fast as it can,
> +  and get a callback when there's a response.  Maybe the http library
> +  buffers them, or pipelines them, or blocks the http.get call until
> +  there's more room.  It just seems to always degenerate to 1 request
> +  at a time.  For example:
> +  sudo nmap --script=http-enum bamsoftware.com -p80 -d2
> +  quickly (within a few seconds) gives:
> +    NSE: http-enum: Searching for entries under path '' (change with 'http-enum.basepath' argument)
> +    NSE: Total number of pipelined requests: 2081
> +    NSE: Number of requests allowed by pipeline: 100
> +    NSE: Received only 41 of 100 expected responses.
> +    Decreasing max pipelined requests to 41.
> +    NSE: Received only 1 of 41 expected responses.
> +    Decreasing max pipelined requests to 1.
> +  100 may a wildly high number of requests to attempt to pipeline.
> +  And then something else probably goes wrong after it decides 41 is okay.

My earlier thoughts on the subject (for those reading nmap-svn and curious):

http://seclists.org/nmap-dev/2011/q4/186

I think callbacks in general will be, from a programmatic perspective,
inconsistent with the rest of our libraries. In particular, the nsock
binding does not expose the callback mechanism to scripts. I would
like to see a design that avoids callbacks.

One possibility is to keep http.get as a "blocking" call but scripts
can achieve http pipelining by having multiple workers. I don't think
this is too unreasonable as we don't want more than ~5 pipelined
requests anyway. So, the http library will automatically pipeline
requests to the same host coming from multiple threads but one thread
can only have one request pending at a time.

-- 
- Patrick Donnelly
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/