Re: Ncat + Lua - GSOC feedback request

[Henri Doreau <henri.doreau () gmail com>]

Hi,

A couple comments inline.

2013/6/16 Jacek Wielemborek <wielemborekj1 () gmail com>:
> On Fri, June 14, 2013 at 10:43:42 David Fifield wrote:
> > On Sun, May 12, 2013 at 05:51:27PM +0200, Jacek Wielemborek wrote:
> > > 2. Ideas for the use cases
> > >
> > > So, we've got this well-documented shiny tool that speaks Lua now.
> > > Apart from all the joy of starting a script with #!/usr/bin/ncat -L
> > > header line, what else is there? I already mentioned: 1) NSE protocol
> > > libraries being used for more personal needs and 2) better NSE testing
> > > facilities.
> >
> > Here are my three use cases. May I suggest that you create a SecWiki
> > page containing a list of use cases?
>
> Unfortunately I can't. I reported my problems here:
> http://seclists.org/nmap-dev/2013/q2/449
>
> > lua-exec option. This is perhaps the least ambitious, least
> > controversial, and easiest notion of Lua integration. I think you should
> > do it first before anything else. What it means is that you can do
> >       ncat --lua-exec program.lua
> > Ncat will fork and exec program.lua as if it were a shell script with
> > --sh-exec. In fact, the above is the same as
> >       ncat --sh-exec "lua program.lua"
> > except that you don't need a separate installation of Lua. (And also,
> > maybe Ncat's Lua interpreter will have libraries available by default or
> > something.)
>
> I like this idea as a starting point - I understand that all the the calls to
> io.write()/print() should direct to the socket instead of the stdout and all
> calls to io.read() should read from the string? (oh, by the way, is the
> forking really necessary?)
+1 for this as a starting point. As there's no exec(), forking is not
striclty necessary but it's a way to easily handle multiple
connections. I believe this first prototype should be kept simple.

> > Implement telnet negotiation and CRLF replacement. The -t and -C options
> > are implemented as special-purpose C code. (Grep for "dotelnet" and
> > "fix_line_endings".) It would be nice if these along with other things
> > could be implemented as little Lua snippets. What I'm picturing here is
> > not separate script files, but actual Lua source code as strings within
> > the Ncat source code. Each little program would be like a filter that
> > does CRLF substitution or whatever. This use case seems straightforward
> > but I think the implementation is not trivial (but neither is it super
> > hard). CRLF substitution needs to keep state across calls in case a CRLF
> > crosses a buffer boundary. Telnet negotiation needs to send data
> > in-band in response to information it receives. So neither is a simple
> > block-at-a-time byte filter.
>
> I'll look into that, at the moment I don't really know much about the
> negotiation protocol (but I'll catch up). I'm a bit worried about hardcoding
> Lua code in the Ncat core though - what if the user wants to drop Lua
> functionality? Fallback to C routines? Or should we make Lua a requirement for
> Ncat to compile?
Try to keep lua-related things together and between preprocessor
guards to let users choose whether to compile w/ or w/o lua. It
shouldn't be a requirement for ncat to compile. See nmap (grep for
NOLUA) to see how this can be done.

> > WebSocket mode. See RFC 6455. It would be nice to do
> >       ncat --websocket ws://example.com:8000/
> >       ncat --listen --websocket ws://localhost:8000/
> > What this involves is first sending a special HTTP request asking for a
> > protocol change to WebSocket. Then you have to receive a response and do
> > some SHA1 calculations and do error handling. After that you can send
> > and receive data on the same TCP connection, but not directly: each
> > little block of data needs to be encapsulated in WebSocket messages and
> > frames. This is quite a heavyweight use case. An implementation that can
> > do it, can probably do most of what we would want Lua integration to do.
> > You should also be able to use SSL WebSocket in the obvious way:
> >       ncat --websocket wss://example.com:8000/
>
> I love this one - it would add some really new functionality to Ncat and be a
> great demo of what the Lua engine could (and should) be be capable of. The
> only thing that I don't exactly understand is the usage - do you mean that
> every Lua script should be capable of messing with argv interpretation (sounds
> dangerous) or WebSocket should be a special script? Option three that comes to
> my mind is invocation like:
>
> ncat --lua-exec "websocket,url=ws://example.com:8000"
> ncat --lua-exec "websocket" example.com 8000
> ncat --lua-exec "websocket" --ssl example.com 8000
>
> ncat --lua-exec "websocket,url=ws://example.com:8000,mode=listen"
> ncat --lua-exec "websocket" -l -p 8000
>
> These examples raise questions about how much of the the Ncat internal
> interface should be exposed to the user - my initial plans suggested to allow
> the scripts to read (and perhaps change) the "options" structure, here I also
> introduce a special script parameter list. That's probably going to be quite
> some work for me to do.
Here again, you can have a look at NSE, but if such websocket code is
implemented within ncat's core, you shouldn't have too much problem.
The C code can access everything and pass whatever is needed to lua.

> > Chat mode for me is pretty much meh. If it's easy to do with some inline
> > Lua, fine. I don't think it's worth adding code to allow custom
> > nicknames or anything fancy.
>
> I pretty much considered it a nice server demo - from the beginning I assumed
> that Ncat-Lua should cover all the Ncat's functionality, but when I now think
> of it, perhaps it's better to focus on the client side first.
>
> > I don't find the use of NSE libraries compelling. You'll have to
> > convince me. I think I don't yet understand the better NSE testing idea.
>
> (I'll reply to that below, where you answered to the section 4 of my original
> e-mail, which is also about NSE)
>
> > > 3. Hard requirements
> > >
> > > * Quite a lot of planning
> > >
> > > I definitely consider it a requirement number one for this task. I
> > > don't really believe that it's possible to integrate Lua with Ncat
> > > properly without prior planning, especially if we're heading for NSE
> > > compatibility. The planning phase is when I expect to need my mentor
> > > the most, so that the code would be written well and once, with no
> > > further rewriting ever needed.
> >
> > If I may offer a different perspective, it may be that lots of
> > pre-planning (like http://c2.com/cgi/wiki?BigDesignUpFront) isn't right
> > for this job. Partly because we don't even have a list of requirements
> > yet, and only a few use cases. You would be trying to design something
> > that is under-specified. I am fine with building some prototypes and
> > throwing away some designs.
> >
> > Lots of planning up front also increases the risk that there will be
> > nothing useful at the end of the summer. I suggest you implement
> > --lua-exec first, and from there you will have a better idea of the
> > requirements.
>
> Hm, we already talked about this on IRC and I agree that might be the better
> option. So, instead of strictly following the timeline I wrote for my
> proposal, I'll be focusing on --lua-exec for now.
>
> > > 4. Open questions
> > >
> > > How much compatibility with NSE is possible for Ncat without getting
> > > too bloated?
> >
> > I think that you should not aim for NSE compatibility unless you already
> > have a worked-out design for it. What would it mean for Ncat to be
> > compatible with http-date.nse, for example? With http.lua? I don't know
> > what that even means. I fear that without a design or use cases in mind,
> > we'll have an implementation that won't be used.
>
> Honestly, I don't have any design for the NSE part (and the IRC log from my
> last e-mail suggests it would be a really difficult task). My idea of
> compability with http-date.nse meant that the user runs something like:
>
> ncat --script http-date localhost 80
>
> And the action method would be run against the server. The compability with
> http.lua would mean that it would be possible to use the library inside the
> Ncat scripts.
>
> I wouldn't mind dropping NSE support for now though; it was just an idea I
> meant to pass to you guys and find out if I it's possible to do in three months
> along with a Lua interface to the rest of Ncat's functionality (with some
> mentoring of course).
If I understand correctly, that would mean adding quite some code to
ncat just to have an alias to
"nmap --script http-date -p 80 localhost". NAK for me on that one...

Regards

--
Henri
_______________________________________________
Sent through the dev mailing list
http://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/