Updating scripts for structured output

[David Fifield <david () bamsoftware com>]

With the new structured script output, I'm sure you're asking yourself
how to augment your scripts so that people can take advantage of easily
parsed data elements in XML output. Here are some ideas.

First off, we're hoping that simple scripts can get away with just
returning a table and relying on the automatic stdnse.output_table
formatting. stdnse.output_table gives you a table that keeps its
elements in order of assignment. So for example, if you do this in your
script:
        local output_tab = stdnse.output_table()
        output_tab.ip = "127.0.0.1"
        output_tab.hostname = "localhost"
        return output_tab
The output of the script will be:
        | test:
        |   ip: 127.0.0.1
        |_  hostname: localhost

If you need more control over the output, return a string after the
table or set the __tostring metamethod on the return table.
        local output_str = "The IP is " .. output_tab.ip .. " and the hostname is " .. output_tab.hostname
        return output_tab, output_str
The output will be:
        |_test: The IP is 127.0.0.1 and the hostname is localhost

As for converting existing scripts, generally you can't just return the
table you've been passing to stdnse.format_output. You want XML that
looks like
        <elem key="ip">127.0.0.1</elem>
and not
        <elem>External IP: 127.0.0.1</elem>

If you have a script that is returning a big table of values through
stdnse.format_output, you can continue to do that for the formatted
string that you use as the second return value. You will also want to
build a table that represents the same information and has nice
queryable keys. In fact, it's probably best practice to build the table
as a primary output format, and then derive the table you pass to
format_output from that.

If you are using tab.lua, then you should build an abstraction for
adding a row that adds keys to an output table and adds the same data to
the tab.lua table. See r29580 for an example of how this was done in
traceroute-geolocation.

There's a new (yet unsupported) NSEDoc tag @xmloutput. You can
semi-automatically generate this by running your script through HTML
Tidy:
        nmap --script=myscript -oX - | tidy -xml -utf8 -indent -wrap 0
        
I think we'll start to settle on some standards for key names and data
formats in structured output. These should be done with an eye towards
easy automated processing. For example, ssl-cert shows certficate
digests with colons in normal output, but without colons in XML output.
Also I've been using the format "%Y-%m-%dT%H:%M:%SZ" for dates and
times; I don't know if that's the best but it is at least ISO 8601.

Here is the book section on structured output.
        http://nmap.org/book/nse-api.html#nse-structured-output

David Fifield
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/