Re: NSE Socket Operation on a non-socket

[Joao Correa <joao () livewire com br>]

Hi Jah,

Thanks for the reply,

On Sat, Jul 18, 2009 at 4:00 PM, jah<jah () zadkiel plus com> wrote:
> On 18/07/2009 10:19, Joao Correa wrote:
> > About your patch, I have a few considerations:
> > -             if not opts and opts.recv_before then
> > +             if opts and opts.recv_before then
> >
> > I agree that the line is incorrect, but I believe that the correct would be:
> >
> > if not opts or (opts and not opts.recv_before) then
> >
> > because what we are trying to check here is if there is no data
> > payload or recv_before option set to emit the debug message. Using
> > tryssl without any of these conditions makes impossible for the script
> > to check if a tcp connection is really working, because it depends on
> > exchanging data to discover if that socket requires SSL or not.
> Given what you say, I think this would be better:
>
> if ( not data or data:len() == 0 ) or ( opts and opts.recv_before ) then
>
> That way we explicitly check for zero-length or nil data.  And we check
> for opts.recv_before.

Since the code is already on an else scope for the test "if #data > 0
then" I believe that there is no need to test it again. Is there any
reason for testing it again explicitly? If opts.recv_before is true,
the debug message should not be emitted, because it is expected to
exist data exchange which allows the script to discover if it is a
working socket, so I think that a test that will lead to the mentioned
situation is exactly the oposite for the second test (that would be
the test I first mentioned).

> > About the bug, if any of the receive() functions did not work fine
> > (that is what happens if you open a tcp connection on a socket that
> > requires SSL) there's something wrong with our socket and maybe TCP is
> > not the correct protocol option. That's why the script attempts to
> > reconnect using SSL.
> >
> > The big issue here is that, if you try SSL on a socket that does not
> > use SSL, you get the error message instantly. But if you try TCP on a
> > socket that uses SSL, it will connect successfully, and you won't
> > notice it unless that you exchange some data with the server.
> If I try an SSL connection to 3389 (MS Terminal Service) I get a
> timeout, not an instant error message.

The same behaviour I've seen for port 445 here.

> Looking at a packet capture, I see that the TCP handshake takes place,
> the local host sends an SSLv2 client hello (which wireshark decodes as
> "TPKT"), the remote hosts ACKs the data and then nothing happens until
> the connection times-out .

That's not the usual behavior I've seen on non SSL sockets. If you
test banner script against scanme.nmap.org on port 80, you will see
that the tcp socket times-out and the second connection attempt (which
is ssl) will be closed by the server before the socket times out, not
representing a big overhead. Somehow this is what was expected from
situations like this, that the script attempts again with the
fall-back option but, in case of this option being a SSL, it is not a
significative overhead.

We will always find server that have different behaviors around the
internet, and since tryssl is sort of "heuristics" to allow
connections with transparent protocol configuration, sometimes we will
step into the worst case.

I'm not saying that we should not worry about the bug and that there
aren't ways to improve the heuristics. I'm just saying that we must
take care with it making its work.

> If I call socket:get_info() (after connect() returns nil, "TIMEOUT") it
> returns true.

While writing the script I had some problems using get_info(),
because, for example, if you set a tcp connection on a socket that
requires ssl, and does not exchange data, and performs get_info(), it
will return true for a not working socket. But I don't believe it is
the case, I'm just mentioning what made me use data exchange to ensure
socket correctness instead of get_info().

> Calling socket:close() also returns true.
> I've attached a sample script (s.nse), plus the outputs (3389.nmap) and
> a pcap so you can see what I mean.
> Same thing happens against port 445.
> I should point-out that the machine I'm running nmap on is windows too.

> > It seems that the ms service at port 445 does not send any data on
> > connect (data that should be the service banner), what makes the first
> > receive fail and makes the script test a second option (which is ssl).
> True, neither port 445 nor 3389 return a banner after a TCP connection
> and the first receive() should normally return: nil, "TIMEOUT"
> This is, of course, very common for any service which requires some data
> before it will talk.

Yes, it is normal. That's why we use two options for exchanging data
in tryssl, the first is receiving banners (for services that do not
require data before talking) and the second that is sending a payload
(for services that will not speak). The problem is that with banners
we will never use the second options and, sometimes, the first one
also won't help us.

The expected behaviour is that, when trying to set  the second socket
(with SSL), it closes the socket without a big delay, as happens with
'nmap scanme.nmap.org -p80 --script banner -d3'. In the set of cases
that the SSL socket is not closed, it will only take a few extra
seconds to timeout and the results won't be compromised.

I don't know if it is a small set, but your report is the first
situation where I don't meet the expected behaviour. I'll start
testing the function against more services just to check if this is
something more usual than I thought.

> > Normally this second attempt would not represent a considerable
> > overhead, since while trying to set a SSL connection on a non-ssl
> > socket would lead to a connection close instantly. For some reason (as
> > you noticed, and I could check here) specially this service does not
> > close the connection when a client tries to open a connection with SSL
> > support. I've tested some different ms services here, and all of them
> > closed my SSL connection attempts.
> Very strange that we're seeing two different behaviours.  What version
> of windows are you testing against?

Windows XP SP2. I could notice that, for example, port 135 closes the
SSL connection instantly. In port 445 the behaviours is the one you
mentioned.

> > About returning the socket instead of nil, one problem is that, if we
> > do not test the second option, on a situation like having telnet over
> > ssl (for example) the script would try a TCP connection, it would not
> > receive any data but the incorrect socket would remain open. That's
> > why it should retry with SSL.
> >
> > I don't think that this bug can lead to errors, but in some cases
> > (like the one of the port 445) it can represent some lose of
> > performance. In this specific case, using tryssl may lead to some lose
> > of performance, but not using it can lead to false negatives in a much
> > bigger number of situations.
> >
> > I'll work on a fix for this situation, but I can't think about a
> > solution now. It would be great if someone could post some ideas.
> I wonder if it might be a good idea to use two functions instead of
> opencon() - one that deals with TCP connections and knows what
> conditions need to be satisfied to try SSL and one for SSL connections
> which can fall back to plain TCP if SSL doesn't work.  This might make
> it easier to deal with different conditions.

Hm. Looks interesting. But I can't see how different this two
functions would be... I'll try to make a few tests, and see if I can
think about something related. If you can send a patch about how you
think this should be done. It would be great.

> There doesn't seem to be any point in passing sockets back to tryssl()
> because opencon never reuses a socket - it always creates a new one and
> it might be a good idea to ensure sockets are closed in opencon().  You
> can just pass some variable back to tryssl() instead of a socket.

tryssl() is being widely used in different scripts and not only in
get_banner(). The main objective of the function is returning a
working socket for a script, freeing the programmer from dealing with
SSL checks. This way, instead of using socket.connect(a,b,TCP) you
just use tryssl().

If you check some scripts that are using it I believe that you can
understand better its objective. I would suggest checking the function
request from http.lua or the script irc-info.nse. Anyway, there are
other places where it is being used.

> > Another strange behavior I could notice is that when I try SSL
> > connection first on 445, the second connection, which is TCP, receives
> > a fast connection close from the server before it reaches its timeout.
> > When it happens, nsock displays a message about invalid socket.
> > Perhaps the problem is a similar situation, where your script is
> > running and the socket is closed unexpectedly.
> If I do the same thing, SSL to 445 and then TCP to 445 then I see a
> completion of the 3-way handshake for the second connection and then the
> local hosts sends RST,ACK to the remote host - exactly the opposite of
> what you saw.  This causes a fatal nsock_loop error 10038 - even before
> connect() returns - which is the error I saw yesterday.

Hm. I didn't look this deep, I'll take a second look on it and reply again.

I've been running Nmap on a Ubuntu linux and on Mac OS X (against a
windows machine, of course). I'll make a second test using the windows
box, and I'll also take a deeper look on the files you sent, if
something new comes up, I'll reply again.
> So it looks like it might be the OS jumping in and messing things up.
> I'm going to pick through a capture file and see if this is what
> happened yesterday when running banner and smb-enum-shares.
> > I'm not sure I was clear enough, but if you need a better explanation
> > about anything, don't mind asking.
> Thanks Joao, very clear.
>
> jah
>
> U:\jah>nmap -d3 -PN -n -sS -p1 --max-retries 0 --script-trace --script-args p=3389 --script s.nse 192.168.1.17 > 
> 3389.nmap
>
> ***WinIP***  trying to initialize WinPcap
> Winpcap present, dynamic linked to: WinPcap version 4.0.2 (packet.dll version 4.0.0.1040), based on libpcap version 
> 0.9.5
>
> Starting Nmap 5.00 ( http://nmap.org ) at 2009-07-18 15:34 GMT Standard Time
> Fetchfile found C:\Program Files\Nmap\nmap-services
> The max # of sockets we are using is: 0
> --------------- Timing report ---------------
>  hostgroups: min 1, max 100000
>  rtt-timeouts: init 1000, min 100, max 10000
>  max-scan-delay: TCP 1000, UDP 1000, SCTP 1000
>  parallelism: min 0, max 0
>  max-retries: 0, host-timeout: 0
>  min-rate: 0, max-rate: 0
> ---------------------------------------------
> Fetchfile found C:\Program Files\Nmap\nse_main.lua
> Fetchfile found C:\Program Files\Nmap\nselib/
> Fetchfile found C:\Program Files\Nmap\scripts\script.db
> Fetchfile found C:\Program Files\Nmap\scripts\s.nse
> NSE: Script C:\Program Files\Nmap\scripts\s.nse was selected by name.
> NSE: Loaded 1 scripts for scanning.
> NSE: Loaded 's.nse'.
> doing 0.0.0.0 = 192.168.1.17
> Initiating ARP Ping Scan at 15:34
> ...
> Completed ARP Ping Scan at 15:34, 0.11s elapsed (1 total hosts)
> Overall sending rates: 9.09 packets / s, 381.82 bytes / s.
> pcap stats: 2 packets received by filter, 0 dropped by kernel.
> Initiating SYN Stealth Scan at 15:34
> ...
> Completed SYN Stealth Scan at 15:34, 0.00s elapsed (1 total ports)
> Overall sending rates: 0.00 packets / s, 0.00 bytes / s.
> pcap stats: 2 packets received by filter, 0 dropped by kernel.
> NSE: Script scanning 192.168.1.17.
> NSE: Starting runlevel 1 scan
> Initiating NSE at 15:34
> NSE: NSE Script Threads (1) running:
> NSE: Starting 's' (thread: 01030828) against 192.168.1.17.
> !! printing socket      userdata: 01031B68
> NSOCK (1.1410s) SSL/TCP connection requested to 192.168.1.17:3389 (IOD #1) EID 9
> NSOCK (1.1410s) nsock_loop() started (timeout=50ms). 1 events pending
> NSOCK (1.2030s) nsock_loop() started (timeout=50ms). 1 events pending
> NSOCK (1.2660s) nsock_loop() started (timeout=50ms). 1 events pending
> NSOCK (1.3280s) nsock_loop() started (timeout=50ms). 1 events pending
> NSOCK (1.3910s) nsock_loop() started (timeout=50ms). 1 events pending
> NSOCK (1.4530s) nsock_loop() started (timeout=50ms). 1 events pending
> NSOCK (1.5160s) nsock_loop() started (timeout=50ms). 1 events pending
> NSOCK (1.5780s) nsock_loop() started (timeout=50ms). 1 events pending
> ...
> NSOCK (30.3910s) nsock_loop() started (timeout=50ms). 1 events pending
> NSOCK (30.4530s) nsock_loop() started (timeout=50ms). 1 events pending
> NSOCK (30.5160s) nsock_loop() started (timeout=50ms). 1 events pending
> NSOCK (30.5780s) nsock_loop() started (timeout=50ms). 1 events pending
> NSOCK (30.6410s) nsock_loop() started (timeout=50ms). 1 events pending
> NSOCK (30.7030s) nsock_loop() started (timeout=50ms). 1 events pending
> NSOCK (30.7660s) nsock_loop() started (timeout=50ms). 1 events pending
> NSOCK (30.8280s) nsock_loop() started (timeout=50ms). 1 events pending
> NSOCK (30.8910s) nsock_loop() started (timeout=50ms). 1 events pending
> NSOCK (30.9530s) nsock_loop() started (timeout=50ms). 1 events pending
> NSOCK (31.0160s) nsock_loop() started (timeout=50ms). 1 events pending
> NSOCK (31.0780s) nsock_loop() started (timeout=50ms). 1 events pending
> NSOCK (31.1410s) Callback: SSL-CONNECT TIMEOUT for EID 9 [192.168.1.17:3389]
> NSE: TCP 192.168.1.15:52397 > 192.168.1.17:3389 | CONNECT
> !! printing connect()   nil     TIMEOUT
> !! printing get_info()  true    192.168.1.15    52397   192.168.1.17    3389
> NSE: TCP 192.168.1.15:52397 > 192.168.1.17:3389 | CLOSE
> !! printing close()     true
> NSE: Finished 's' (thread: 01030828) against 192.168.1.17.
> NSOCK (31.1410s) nsock_loop() started (timeout=50ms). 0 events pending
> Completed NSE at 15:35, 30.00s elapsed
> NSE: Script Scanning completed.
> Fetchfile found C:\Program Files\Nmap\nmap-mac-prefixes
> Host 192.168.1.17 is up, received arp-response (0.00s latency).
> Scanned at 2009-07-18 15:34:53 GMT Standard Time for 30s
> Interesting ports on 192.168.1.17:
> PORT  STATE  SERVICE REASON
> 1/tcp closed tcpmux  reset
> MAC Address: 00:0C:29:84:53:F5 (VMware)
> Final times for host: srtt: 0 rttvar: 3750  to: 100000
>
> Read from C:\Program Files\Nmap: nmap-mac-prefixes nmap-services.
> Nmap done: 1 IP address (1 host up) scanned in 31.38 seconds
>           Raw packets sent: 2 (86B) | Rcvd: 2 (82B)
>
> id=""
> author=""
> runlevel="1"
> description = ""
> categories = {}
>
> hostrule = function( host )
>  return true
> end
>
> action = function( host )
>  local port = nmap.registry.args and nmap.registry.args.p or 3389
>  local socket = nmap.new_socket()
>
>  print("!! printing socket",        socket)
>
>  print("!! printing set_timeout()", socket:set_timeout(30000))
>  print("!! printing connect()",     socket:connect(host.ip, port, "ssl"))
>  print("!! printing get_info()",    socket:get_info())
>  print("!! printing close()",       socket:close())
>
>  socket = nil
>  return nil
> end

Thanks for the information Jah,
Joao

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org