Re: [NSE][RFC] format.lua library

[David Fifield <david () bamsoftware com>]

On Fri, Sep 14, 2012 at 01:42:04PM -0500, Daniel Miller wrote:
> I've attached my current version of the format.lua lib, as well as 2
> patches that enable structured output for nfs-ls and nbstat,
> respectively. These make use of some of the functions in format.lua.
> Note that the nfs-ls patch also modifies tab.lua to support
> embedding table output within outline-style text output, while
> keeping a structured XML serialization.
>
> I realize that this is mostly geared around my original "tostring"
> method of doing structured output, whereas most of the scripts that
> have been converted so far use parallel structure and string
> outputs. I think that this library could work as a consolidation
> point for functions enabling both styles.
>
> Once this is merged, I would move format_timestamp into format.lib
> as a separate commit.

Thanks Daniel. I think I want to merge this once we've addressed a few
things.

The naming of the keys here is what bothers me the most:

>             <table>
>               <elem key="PERMISSION">drwxrwxrwt</elem>
>               <elem key="SIZE">4096</elem>
>               <elem key="GID">0</elem>
>               <elem key="FILENAME">/tmp</elem>
>               <elem key="MODIFICATION TIME">2012-07-20 16:49</elem>
>               <elem key="UID">0</elem>
>             </table>

I just think this goes against the spirit of what structured output is
about, which is making script data easily usable by programmers. In
terms of a Model–View–Controller design, I think that structured output
should be a direct representation of the model, and normal unstructured
output should be a view of that model. Looking in the nfs-ls script, we
see that there already exists a fairly structured programmer's
representation of a file entry:

    local perm = f.type .. f.mode
    tab.addrow(outtab, perm, f.uid, f.gid,
               f.size, f.time, f.filename)

I think that structured output should reflect this preferred
representation. Since perm is actually made up of f.type and f.mode,
those should be separate data in structured output. (A numeric
permission is probably even more useful to a programmer; i.e., the input
to rpc.Util.FpermToString.) An exception is that it should probably use
f.mtime or f.atime or f.ctime instead of just f.time.

              <table>
                <elem key="mode">041777</elem>
                <elem key="size">4096</elem>
                <elem key="gid">0</elem>
                <elem key="filename">/tmp</elem>
                <elem key="mtime">2012-07-20T16:49</elem>
                <elem key="uid">0</elem>
              </table>

I see the appeal in automatically generating structured output from a
table, but I don't think it's sufficiently general. Even apart from keys
being different from column headers, we might want a different
representation in structured output than in normal output. In normal
output, "drwxrwxrwt" is the obvious right choice. But if I'm writing a
script to process NFS entries, I probably prefer 041777. I can think of
applications where I'd want to parse "drwxrwxrwt" back into a number to
do something interesting.

A good example of what I mean is http-auth.nse.
http.parse_www_authenticate returns a table that is basically exactly
what you want as a programmer, so you just return that directly for
structured output. For normal output, we do a little extra work to put
the parameters on a separate line for each challenge. I think that this
is how scripts should be written: concentrate on writing functions that
return data that is easy to use, then do some extra work to format it
for display.

I'm in favor of the format library, if that's the way you think about
producing structured output. After all, you're volunteering to do the
work of adapting scripts, so I want you to be able to do it in the way
that's most comfortable.

David Fifield
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/