Structured script output merge candidate 1

[David Fifield <david () bamsoftware com>]

I have been integrating and editing Daniel Miller's patches for
structured script output. Structured script output allows scripts to
return a table of name-value pairs, which will be reflected in an
element hierarchy in XML output. The code is here:

svn co https://svn.nmap.org/nmap-exp/david/xml-output

This is what the XML output looks like. The normal output is the same as
in Nmap trunk.

https://secwiki.org/w/Nmap/Structured_Script_Output#Merge_candidate_1

These commands show sample diffs that make scripts provide structured
output:

svn diff -c 29491 https://svn.nmap.org/nmap-exp/david/xml-output
svn diff -c 29492 https://svn.nmap.org/nmap-exp/david/xml-output
svn diff -c 29493 https://svn.nmap.org/nmap-exp/david/xml-output

This merge candidate is missing one thing, which is an ordered output
table along the lines of what Patrik provided in
http://seclists.org/nmap-dev/2012/q3/289.

At the end of this message I am appending the new documentation for this
system, which will be part of scripting.xml. This documentation reflects
how I anticipate the structured output being used.

David Fifield


Structured and Unstructured Output

NSE scripts should usually return a table representing their output, one
that is nicely organized and has thoughtfully chosen keys. Such a table
will be automatically formatted for screen output and will be stored as
nested elements in XML output. Having XML output broken down logically
into keys and values makes it easier for other tools to make use of
script output. It is possible for a script to return only a string, but
doing so is deprecated. In the past, scripts could only return a string,
and their output was simply copied to the XML as a blob of text–this is
now known as “unstructured output ”.

Suppose a script called user-list returns a table as shown in this code
sample. The following paragraphs show how it appears in normal and XML
output.

local output = nmap.output_table()
output.hostname = "slimer"
output.users = {}
output.users[#output.users + 1] = "root"
output.users[#output.users + 1] = "foo"
output.users[#output.users + 1] = "bar"
return output

A Lua table is converted to a string for normal output. The way this
works is: each nested table gets a new level of indentation. Table
entries with string keys are preceded by the key and a colon; entries
with integer keys simply appear in order. Unlike normal Lua tables,
which are unordered, a table that comes from nmap.output_table will keep
its keys in the order they were inserted. Example 9.4, “Automatic
formatting of NSE structured output” shows how the example table appears
in normal output.

Example 9.4. Automatic formatting of NSE structured output

PORT     STATE SERVICE
1123/tcp open  unknown
| user-list:
|   hostname: slimer
|   users:
|     root
|     foo
|_    bar

The XML representation of a Lua table is constructed as follows. Nested
table become table elements. Entries of tables that are not themselves
tables become elem elements. Entries (whether table or elem) with string
keys get a key attribute (e.g. <elem key="username">foo</elem>); entries
with integer keys have no key element and their key is implicit in the
order in which they appear.

In addition to the above, whatever normal output the script produces
(even if automatically generated) is copied to the output attribute of
the script element. Newlines and other special characters will be
encoded as XML character entities, for example &#xa;. Example 9.5, “NSE
structured output in XML” shows how the example table appears in XML.

Example 9.5. NSE structured output in XML

<script id="t" output="&#xa;hostname: slimer&#xa;users: &#xa;  root&#xa;  foo&#xa;  bar">
  <elem key="hostname">slimer</elem>
  <table key="users">
    <elem>root</elem>
    <elem>foo</elem>
    <elem>bar</elem>
  </table>
</script>

Some scripts need more control their normal output. This is the case,
for example, with scripts that need to display complex tables. For
complete control over the output, these scripts may do either of these
things:

* return a string as second return value, or
* set the __tostring metamethod on the returned table.

The resulting string will be used in normal output, and the table will
be used in XML as usual. The formatted string may contain newline
characters to appear as multiple lines.

If the above code example were modified in this way to return a
formatted string,

local output = nmap.output_table()
output.hostname = "slimer"
output.users = {}
output.users[#output.users + 1] = "root"
output.users[#output.users + 1] = "foo"
output.users[#output.users + 1] = "bar"
local output_str = string.format("hostname: %s\n", output.hostname)
output_str = output_str .. "\n" .. stdnse.strjoin(", ", output.users)
return output, output_str

then the normal output would appear as follows:

PORT     STATE SERVICE
1123/tcp open  unknown
| user-list:
|   hostname: slimer
|_  users: root, foo, bar
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/