Nmap Development mailing list archives
Re: Nsock read buffer
From: David Fifield <david () bamsoftware com>
Date: Tue, 11 Jan 2011 20:57:50 -0800
On Mon, Nov 15, 2010 at 10:23:15PM -0800, David Fifield wrote:
I started implementing this in the branch svn co --username guest --password "" svn://svn.insecure.org/nmap-exp/david/nmap-readbuf I attached a script that shows how it works. nsock_readbytes and nsock_readlines (and hence nmap.receive_bytes and nmap.receive_lines) return exactly what you ask for, not more and not less. If there are not enough bytes available, the functions return an error and the bytes stay in the buffer for the next read. If there are more than enough bytes available, only the first n bytes or lines are returned, and what's left over remains in the buffer. I think that the behavior of nsock_readbytes and nsock_readlines in my branch is just write. My question is, how should nsock_read work? nsock_read is used to mean both of these two things: 1. Keep reading until timeout or EOF, and return everything. 2. Return me some small chunk of data, up to one buffer's worth, but I don't care about the exact number. In practice, nsock_receive does (1), except that there a built-in limit of 589823 bytes, after which the read always finishes. (Look for the comment "spews and spews data" in nsock_core.c.) When people want (2), they have been using nsock_readbytes(1), which only does one recv and only returns up to a relatively small fixed-size buffer (like recv). My personal feeling is the nsock_read should work like (2). But we still want a function that works like (1) and returns a big block of data all at once. Service scan does this for example. But even service scan has no use for 589823 bytes; it seems like the caller should be able to set this limit. "As many bytes as are available, but not more than X." This is pretty close to how nsock_readbytes works now, in that it quits reading once enough bytes are available, except that you can still get more than you asked for (up to one fixed-size buffer's worth more).
I've been going back and forth over the API for these read functions. I've identified four use cases: 1. Return everything left on the socket (looped reading), with a sanity limit on bytes. 2. Return whatever is immediately available on the socket. 3. Return an exact number of bytes. 4. Return an exact number of lines, with a sanity limit on bytes. In the branch, nsock_readbytes and nsock_readlines do (3) and (4). nsock_read does (2), and there's no built-in way to do (1). I updated some scripts that were, for example, using socket:receive_bytes(1), to use socket:receive() instead. However I can't guarantee that this is correct in every case. Another question is what to do with sanity byte limits. We can't just read forever if someone keeps writing without sending a newline. Current nsock_readlines already has such a limit, only it is an undocumented, unspecified internal constant. If there's no newline you get back a partial line. An alternative is just to return an error and not violate the contract that we will return only the requested number of lines. Some of the migration is tricky. Here is an example from domcom-cmd.nse. The call to receive_lines(1) is expecting to get more than one line. It looks like it's expecting a line with "BeginData" and one with "EndData". local status, line = socket:receive_lines(1) if ( not(status) ) then return false, "Failed to read line" end lines = stdnse.strsplit( "\n", line ) for _, line in ipairs( lines ) do if ( not(line:match("BeginData")) and not(line:match("EndData")) ) then table.insert(result, line) end end Strictly speaking, this code is incorrect. There's no reason why receive_lines has to return everything on the socket. It just happens that line endings fall on packet boundaries so no half-lines are returned. Since the new receive_lines(1) will always return exactly one line, this has to be rewritten. What's wanted here is an nsock_read that does option (1) above. But then it becomes obvious that the code is relying on an integer number of lines being returned, and you have to write an explicit loop with receive_lines(1). This is what I have done but it's more code, and frustrating for a project that was supposed to simplify code. David Fifield _______________________________________________ Sent through the nmap-dev mailing list http://cgi.insecure.org/mailman/listinfo/nmap-dev Archived at http://seclists.org/nmap-dev/
Current thread:
- Re: Nsock read buffer David Fifield (Jan 11)
- Re: Nsock read buffer ambarisha b (Jan 12)
- Re: Nsock read buffer Patrick Donnelly (Feb 21)