Re: NSE structured output--normal output formatting

[Daniel Miller <bonsaiviking () gmail com>]

On 07/27/2012 09:46 AM, Aleksandar Nikolic wrote:
> Hi all,
>
> on our last nse meeting , David asked us to see if we could adapt some
> of our scripts to work well with Daniel's xml-output branch
> (https://svn.nmap.org/nmap-exp/dmiller/xml-output/).
> The idea was to see how much work would it need and to make some
> examples to see how things should
> work.
>
> I've picked up my dns-nsec3-enum script to make it suitable for xml
> output and here are my comments.
>
> As the script already builds a table for it's output, not much needed to
> be changed, and all the changes
> seem natural. Actual changes required were minimal. "Biggest" problem
> was element ordering which
> wasn't that much of a problem, as I can set ordering by something like :
>             
> stdnse.set_tostring(result,stdnse.format_generator({key_order =
> {"domain_info","hashes","total_hashes"}}))
>
> Sample xml output is:
> <table>
>      <table key="hashes">
>          <elem>nexthash jn29pj2cc5g8raagtt12pkr0lghiorgi
> jni9pg73ho9djg5uencea5fv9lce3vbh</elem>
>          <elem>nexthash 8ne56rkodj8ugu2dbrbi50g6j408l0fm
> 8qp1qh2hnh2upad8r3vkhetln7v4vbhb</elem>
>          ... (more hashes here, a lot more)
>          <elem>nexthash um77ja0arlmihc2n69mmhbddq3b9jc1j
> un2m24fdtsh5ma1a6k1npjj8bnbe09oq</elem>
>      </table>
>      <table key="domain_info">
>          <elem key="Iterations">5</elem>
>          <elem key="Salt">AABBCCDDEEFFAABB</elem>
>          <elem key="Domain">example.com</elem>
>      </table>
>      <elem key="total_hashes">1080</elem>
> </table>
>
> And sample normal ouput :
> 53/udp open|filtered domain  no-response
> | dns-nsec3-enum:
> |
> | domain_info:
> | Domain: example.com
> | Salt: AABBCCDDEEFFAABB
> | Iterations: 5
> | hashes:
> |   nexthash jn29pj2cc5g8raagtt12pkr0lghiorgi
> jni9pg73ho9djg5uencea5fv9lce3vbh
> |   nexthash ekev2le03m8altheblpgbeh4i183bso1
> ekt9utd533ad3kpf19qjkn6n4j13akqv
> |   nexthash duoiqfbg5uahurj49ltg5k49v6b8g5gd
> e12ig006mdgerilcp86uhfqlbc8ja24i
>      ... (more hashes here, a lot more)
> |   nexthash um77ja0arlmihc2n69mmhbddq3b9jc1j
> un2m24fdtsh5ma1a6k1npjj8bnbe09oq
> |_total_hashes: 1080
>
> Granted, there should be a couple of cosmetic changes, but I think this
> illustrates usage pretty well.
>
> I have to say that the changes I had to make to the script look natural
> , and weren't a problem at all.
>
> The patched script is in xml-output branch.
>
>
> Aleksandar
> _______________________________________________
> Sent through the nmap-dev mailing list
> http://cgi.insecure.org/mailman/listinfo/nmap-dev
> Archived at http://seclists.org/nmap-dev/
Aleksandar,

Thanks for taking the time to do this. I made a few changes to 
illustrate how the structured output can be handled (commit message 
attached)

1. The "nexthash abcd1234 abcd5678" lines are really 2 distinct values, 
and the "nexthash" part is implied. I put them as list items in a table, 
then used the stdnse.set_tostring to assign a "stringify" transform to 
make the same text output. This way the XML looks like:

<table key="hashes">
  <table>
    <elem>d1427bj0ahqnpi4t0t0aaun18oqpgcda</elem>
    <elem>vhnelm23s1m3japt7gohc82hgr9un2at</elem>
  </table>
</table>

2. Added an indent to the domain_info table tostring function to make a 
prettier output. This is an unfortunate side-effect of the current 
implementation: if you set a tostring handler, the top-level recursive 
tostring won't know to indent. If we had ordered-dict output structures, 
it would solve this problem in the most common case where an explicit 
tostring handler is set just to order the keys.

3. Changed the @output nsedoc section to match the new output, and added 
@xmloutput nsedoc section.

I'm glad to see that this new way of doing things didn't require much 
change for this script.

Dan
> --- Begin Message ---
>
>
> From: commit-mailer () insecure org
>
> Date: Fri, 27 Jul 2012 08:15:42 -0700 (PDT)
>
> Author: dmiller
> Date: Fri Jul 27 08:15:42 2012
> New Revision: 29384
>
> Log:
> Make some more structured output changes to dns-nsec3-enum
>
> Modified:
>    nmap-exp/dmiller/xml-output/scripts/dns-nsec3-enum.nse
>
> Modified: nmap-exp/dmiller/xml-output/scripts/dns-nsec3-enum.nse
> ==============================================================================
> --- nmap-exp/dmiller/xml-output/scripts/dns-nsec3-enum.nse      (original)
> +++ nmap-exp/dmiller/xml-output/scripts/dns-nsec3-enum.nse      Fri Jul 27 08:15:42 2012
> @@ -59,14 +59,47 @@
>  -- PORT   STATE SERVICE
>  -- 53/udp open  domain
>  -- | dns-nsec3-enum:
> --- |   domain example.com
> --- |   salt 123456
> --- |   iterations 10
> +-- |
> +-- | domain_info:
> +-- |   domain: example.com
> +-- |   salt: 123456
> +-- |   iterations: 10
> +-- | hashes:
>  -- |   nexthash d1427bj0ahqnpi4t0t0aaun18oqpgcda vhnelm23s1m3japt7gohc82hgr9un2at
>  -- |   nexthash k7i4ekvi22ebrim5b6celtaniknd6ilj prv54a3cr1tbcvqslrb7bftf5ji5l0p8
>  -- |   nexthash 9ool6bk7r2diaiu81ctiemmb6n961mph nm7v0ig7h9c0agaedc901kojfj9bgabj
>  -- |   nexthash 430456af8svfvl98l66shhrgucoip7mi mges520acstgaviekurg3oksh9u31bmb
> --- |_  Total hashes found: 8
> +-- |_total_hashes: 8
> +--
> +--@xmloutput
> +-- <script id="dns-nsec3-enum" output="...">
> +--   <table>
> +--     <elem key="total_hashes">8</elem>
> +--     <table key="domain_info">
> +--       <elem key="domain">example.com</elem>
> +--       <elem key="salt">123456</elem>
> +--       <elem key="iterations">10</elem>
> +--     </table>
> +--     <table key="hashes">
> +--       <table>
> +--         <elem>d1427bj0ahqnpi4t0t0aaun18oqpgcda</elem>
> +--         <elem>vhnelm23s1m3japt7gohc82hgr9un2at</elem>
> +--       </table>
> +--       <table>
> +--         <elem>k7i4ekvi22ebrim5b6celtaniknd6ilj</elem>
> +--         <elem>prv54a3cr1tbcvqslrb7bftf5ji5l0p8</elem>
> +--       </table>
> +--       <table>
> +--         <elem>9ool6bk7r2diaiu81ctiemmb6n961mph</elem>
> +--         <elem>nm7v0ig7h9c0agaedc901kojfj9bgabj</elem>
> +--       </table>
> +--       <table>
> +--         <elem>430456af8svfvl98l66shhrgucoip7mi</elem>
> +--         <elem>mges520acstgaviekurg3oksh9u31bmb</elem>
> +--       </table>
> +--     </table>
> +--   </table>
> +-- </script>
>  
>  author = "Aleksandar Nikolic, John Bond"
>  license = "Simplified (2-clause) BSD license--See http://nmap.org/svn/docs/licenses/BSD-simplified";
> @@ -205,8 +238,14 @@
>  
>  -- convenience function , check if table cointains an element
>  local function table_contains(tbl,element)
> -  for _, value in pairs(tbl) do
> -    if value == element then
> +  for _, value in ipairs(tbl) do
> +    local found = 0
> +    for i, v2 in ipairs(value) do
> +      if v2 == element[i] then
> +        found = found + 1
> +      end
> +    end
> +    if found == #value then
>        return true
>      end
>    end
> @@ -224,7 +263,10 @@
>                 for _, nsec3 in ipairs(auth_filter(result, "NSEC3")) do
>                         local h1 = string.lower(remove_suffix(nsec3.dname,domain))
>                         local h2 = string.lower(nsec3.hash.base32)
> -                       local nexthash = "nexthash " .. h1 .. " " .. h2
> +                       local nexthash = {h1, h2}
> +      stdnse.set_tostring(nexthash, function(h)
> +          return string.format("nexthash %s %s", h[1], h[2])
> +        end)
>                         if not table_contains(all_results.hashes,nexthash) then
>                                 table.insert(all_results.hashes, nexthash)
>                                 stdnse.print_debug("nexthash " .. h1 .. " " .. h2)
> @@ -265,10 +307,10 @@
>                         local h2 = string.lower(nsec3.hash.base32)
>                         if table_size(todo) == 0 then
>                                 local domain_info = {}
> -                               domain_info["Domain"] = domain
> -                               domain_info["Salt"] = salt
> -                               domain_info["Iterations"] = iter
> -                               stdnse.set_tostring(domain_info,stdnse.format_generator({key_order = 
> {"Domain","Salt","Iterations"}}))
> +                               domain_info["domain"] = domain
> +                               domain_info["salt"] = salt
> +                               domain_info["iterations"] = iter
> +                               stdnse.set_tostring(domain_info,stdnse.format_generator({indent="  ",key_order = 
> {"domain","salt","iterations"}}))
>                                 all_results.domain_info = domain_info
>                                 stdnse.print_debug("domain " .. domain)
>                                 stdnse.print_debug("salt " .. salt)
> @@ -308,7 +350,10 @@
>                                         end
>                                 end -- for
>                         end -- else
> -                       local nexthash = "nexthash " .. h1 .. " " .. h2
> +                       local nexthash = {h1, h2}
> +      stdnse.set_tostring(nexthash, function(h)
> +          return string.format("nexthash %s %s", h[1], h[2])
> +        end)
>                         table.insert(all_results.hashes, nexthash)
>                         stdnse.print_debug("nexthash " .. h1 .. " " .. h2)      
>                 end
>
> _______________________________________________
> Sent through the nmap-svn mailing list
> http://cgi.insecure.org/mailman/listinfo/nmap-svn
>
>
> --- End Message ---
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/