Re: Suggestion: Using script output - Human-readable and machine-parseable

[Daniel Miller <bonsaiviking () gmail com>]

On 4/5/11, Daniel Miller <bonsaiviking () gmail com> wrote:
> David,
>
> I'm attaching a patch that allows for formatting output as YAML. It
> includes a new NSE library, yaml, that is a slightly modified copy of
> the libyaml binding from yaml.luaforge.net (properly attributed). The
> second diff (luayaml.diff) shows the modifications I made to it.
>
> I added a new function to stdnse, format_yaml, that takes arguments
> like format_output, but the tables should be in a different
> configuration. I modified smb-os-discovery to show the new way, and
> smb-enum-shares to show a half-assed workaround (since a multiline
> string can be represented as YAML, it just shoves the existing output
> into a YAML block scalar literal). The new yaml-test.nse script is
> just to demonstrate and test the output, since it's a prerule script
> and can be run without running a scan.
>
> As it currently stands, I'm dynamically linking against libyaml, and
> using systemwide headers for it, which I know is not the way we'll
> want to go, but it was easiest to start. I haven't worked with
> autoconfig before, so I didn't know how to package the library with
> Nmap. It's pretty stable (0.1.3 was released August 2009), and not too
> big, so it wouldn't be tough to include it. If you want to test, the
> libyaml-0-2 and libyaml-dev packages for Ubuntu were sufficient to
> build and run my patched version.
>
> For tagging the output as YAML, a YAML document always starts with
> "---", so that turns out to be a good marker to look for. The way Nmap
> prefixes script output with "| " makes it impossible to copy-and-paste
> the output as valid YAML, though. Not a huge deal, since XML output
> doesn't do that, though. Since YAML is structured already, it wouldn't
> be necessary to convert it to an XML element hierarchy, but if we
> wanted to (to save writers of Nmap XML parsers from needing a YAML
> library), see http://yaml.org/xml/index.html for the generally
> accepted way of doing it.
>
> Dan
>
> On Sat, Apr 2, 2011 at 11:29 AM, David Fifield <david () bamsoftware com>
> wrote:
> > On Thu, Mar 31, 2011 at 03:46:41PM -0500, Daniel Miller wrote:
> > > Hey list,
> > >
> > > From the Nmap TODO:
> > > > o Nmap should have a better way to handle XML script output.
> > > >  o We currently just stick the current script output text into an XML
> > > > tag.
> > > May I make a suggestion? If we format script output as YAML
> > > (http://www.yaml.org/), it will be machine-readable, while preserving
> > > the human-readable nature. There are a couple of YAML bindings for Lua:
> > > libyaml (http://yaml.luaforge.net/) and libsyck
> > > (https://github.com/indeyets/syck/tree/master/ext/lua)
> > >
> > > The quickest way to support this straight off would be to modify
> > > stdnse.format_output to output YAML. The NSE runtime could quote
> > > non-YAML return values as block-literal scalars, until such time as they
> > > are rewritten to return more useful formatted output. A downside would
> > > be that output would take a few more lines than before, though this
> > > could be overcome by using "flow styles," of which JSON is a proper
> > > subset.
> > >
> > > Benefits of doing this are:
> > > * extensibility without requiring modifications to Nmap XML schema
> > > * Human-readable nature of YAML means a single output format for human
> > > and machine consumers
> > > * existing structured-output using stdnse.format_output means very few
> > > changes for script authors
> > > * YAML bindings available for Lua, Python, Perl, Ruby, C, and many other
> > > languages.
> > >
> > > I was going to submit this idea with a patch, but I realized I don't
> > > have the first clue as to how to integrate a new Lua library binding
> > > into the Nmap source. Instead, here's some examples of YAML-formatted
> > > script output.
> >
> > How about this: Write a new stdnse.yaml function with an interface
> > similar to that of stdnse.format_output. Then we can switch scripts one
> > at a time instead of all at once.
> >
> > What I am thinking is, that stdnse.yaml can somehow tag the output as
> > being structured data and not just plain text. Then we can modify the
> > XML output to look for this tag and emit an element hierarchy instead of
> > (or in addition to) a chunk of text. But that's a second step; the first
> > is just getting YAML on the screen.
> >
> > Take a look at json.generate, which does something similar.
> >
> > David Fifield

Update: I realized I did not attach yaml-test.nse. Here it is

Dan

Attachment: yaml-test.nse
Description:

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/