RE: XML Questions

["Jay Freeman \(saurik\)" <saurik () saurik com>]

Fyoder:

<note>Typed most of this 6 hours ago, got heavily delayed due to ISP
downtime :(.</note>

You mention:

<smurf responses="417"></smurf>  # Smurf (subnet-directed broadcast)
address.  Not <smurf/> because I might want to list the addresses someday

XML parsers consider these exactly the same thing (and often if you just
have an open and then a close during an XSLT step the parser will even
condense it automatically for you).

   <service name="http" proto="http" conf="5" method="detection"
version="Apache/1.3.12 (Unix) mod_perl/1.24"/> # Detected via some future
protocol/version detection system like the Nmap+V patch.

Instead it might be better to separate out "version" into "application" and
"version".  Also, when you start to get this many attributes attached to
something, you often want to break them into separate tags (although it
isn't necessary in this case... might still be useful to have the
<version/>/<application/> tags separated though... just because they might
contain more text, and more special characters which look better escaped in
a tag instead of an attribute).

Something to think about: one of the greatest things about using XML for the
output format is you _don't_ need to think that much about ahead
functionality.  If you leave off the entire idea of a "service" tag now, and
later want to add something for it, you just do.  The clients don't ask for
the tag, so it isn't used, and the parser ignores it (or in the case of SAX
the client chooses to ignore it).  Obviously if someone was running these
things through a funky grep command they are going to be in trouble, but
they really shouldn't be doing XML using such primitive tools.  Also, there
is an application called xml2 which will take an XML stream and convert it
into something easily grep'd (separates each tag out onto separate lines, so
you can grep for the stuff you want based on full tag name).

If push really comes to shove, you can just include XSLT files to backtrack
through versions.  As long as the new file has at least as much information
as the last one, you can just translate backwards.  Every time the XML file
changes enough that existing parsers wouldn't be able to correctly read it
(changed existing tag/attribute name, no longer output a tag/attribute
clients require, or output strange values for attributes clients won't be
expecting, hopefully infrequent events) an XSLT file could be written to
convert back to the old version.  At this point, nmap really could then ask
for what version of the file is wanted, and backtrack version by version
(which shouldn't be bad for a good XSLT engine that is internally using SAX
to stream the documents through all layers at once)... _without_ needing
large amounts of functionality inside of nmap for outputting in each
different version.

There would be a small amount of overhead, but with a good XSLT engine it is
unlikely to be noticeable, especially as this would only be necessary for
the big changes (and I even think that Xalan at least (C++ again...)
supports compiling XSLT files for faster execution... might be wrong about
that, however).

Sincerely,
Jay Freeman (saurik)
saurik () saurik com

-----Original Message-----
From: Fyodor [mailto:fyodor () insecure org]
Sent: Sunday, August 27, 2000 4:22 AM
To: Jay Freeman (saurik)
Cc: Stou Sandalski; Nmap-Dev
Subject: RE: XML Questions

On Sun, 27 Aug 2000, Jay Freeman (saurik) wrote:

> post).  One thing I would consider is moving some of the parts together
more
> into a few larger tags.  This shouldn't make much of a difference in most
> cases, but there are definitely some parsers that would be greatly helped
by
> it:

Good idea!  I have made that change (and several others people privately
suggested).  Here is my latest version:

Nmap XML Output Format Proposal

CHANGELOG

8/27 -- Added scanner="nmap" attribute to nmaprun to help support
        other scanners that want to support the same format.  (
        Suggested by Lamont Granquist (lamont () icopyright com)

8/27 -- Renamed id attribute in port tag to "port" (Apparently id
        might become some sort of reserved keyword in XSL).

8/27 -- Clumped addesses/hostnames/ports into their own XML entities
        (suggested by "Jay Freeman (saurik)" <saurik () saurik com>

I have appended the sample output.  Sorry about the long line
length.  Please send mail to nmap-dev (or just me) if you have any
suggestions for improvement.  Remember that it can be very hard to change
machine-readable output formats once they are defined.  So we must take
the opportunity to scrutinize it now.  Also note that there has to be a
balance between readability, output size, and extensibility.


<?xml version="1.0" ?>
# These pound comments will not appear in actual -oX output
# Note this may not be internally consistant (eg options used vs report
given) since I'm trying to show a diverse range of features
# Also note that this may show some features that Nmap does not currently
support (but I want to leave room for nmap enhancements).
<!-- nmap (V. 2.54BETA3) scan initiated Tue Aug 15 21:49:21 2000 as:
nmap -sT -I -sR -O -p- -oM /tmp/smploutput.mlog localhost db -->
# Info on How Nmap was run
<nmaprun scanner="nmap" args="-sT -I -sR -O -p- -oM /tmp/smploutput.mlog
localhost db" start="966401074" version="2.54BETA3" xmloutputversion="1.0" >
<scaninfo type="connect" protocol="tcp" services="1-1024,1100,1400" />
<scaninfo type="udp" protocol="udp" services="1-1024,1100,1400" />
<scaninfo type="ipproto" protocol="ip" services="1-255" />
<verbose level="1" />
<debugging level="0" />

# Info on hosts (each containing port info
# note that the addrtype attributes you see will be optional and default to
ipv4
<host status="up">
<addresses>
  <address addr="192.168.0.24" addrtype="ipv4" />
  <address addr="00:C0:F0:48:3A:54" addrtype="mac" />
</addresses>
<hostnames>
  <hostname name="amy.insecure.org" type="A" />
  <hostname name="mail.insecure.org" type="CNAME" />
  <hostname name="hopelessly.insecure.org" type="CNAME" />
</hostnames>
<os><osmatch name="Linux 2.1.122 - 2.2.16" accuracy="100%" />
    <osmatch name="FreeBSD 4.0-Release" accuracy="100%" />
    <osmatch name="Linux 2.1.84 - 2.1.121" accuracy="96%" />
    <portused state="open" proto="tcp" port="22" />  # OS detection is based
on ports it can find to test against
    <portused state="closed" proto="tcp" port="1" />
</os>
<tcpsequence index="2796433" class="random positive increments"
difficulty="Worthy Challenge" />

<ports>
<port protocol="TCP" port="22" owner="root">
   <state state="open" conf="5" /> # Conf describes the confidence that the
state is correct: 5 might mean "sure" while 2 could be a guess
   <service name="ssh" conf="3" method="table" /> # Obtained via lookup from
nmap-services (see "method"); confidence is 1-5
   <service name="xlm4" conf="3" method="table" /> # In case there are more
than one service listed as using that port
   <banner>SSH-1.99-2.0.13 (non-commercial)</banner>
</port>
<port protocol="TCP" port="80" owner="nobody">
   <state state="open" conf="5" />
   <service name="http" proto="http" conf="5" method="detection"
version="Apache/1.3.12 (Unix) mod_perl/1.24" /> # Detected via some future
protocol/version detection system like the Nmap+V patch.
</port>
<port protocol="TCP" port="32773" owner="root">
   <state state="open" conf="5" />
   <service name="ttdbserverd" proto="rpc" version="1-2" conf="5"
method="detection" /> # RPCgrind also counts as a "detection" method
</port>
<port protocol="UDP" port="31337">
   <state state="filtered" conf="5" /><filteredby><packet proto="ICMP"
type="3" code="3" name="ICMP port unreachable" srcipaddr="10.3.7.4" ip_v="4"
/></filteredby>
   <service name="backorifice" conf="3" method="table" />
</port>
</ports>
<traceroute type="udp" port="41702">
  <hop distance="0"> # Info like below but regarding src machine </hop>
  <hop distance="1">
     <address addr="10.4.1.7" addrtype="ipv4" />
     <hostname name="router.insecure.org" type="A" />
  </hop>
  <hop distance="4"> ... </hop>
</traceroute>
<timestamp end="966401076" /> # Time we finished with this host
</host>
<host status="down">
<addresses><address addr="192.168.255.255" addrtype="ipv4" /></addresses>
<smurf responses="417"></smurf>  # Smurf (subnet-directed broadcast)
address.  Not </smurf> because I might want to list the addresses someday
<extraports state="filtered" proto="TCP" ids="1-20,400-700,854" />
</host>

# Statistics on Nmap run
<runstats>
<finished time="966401078" timeout=0 />
<hosts up="4" down="16" total="20" />
<!-- Nmap run completed at Tue Aug 15 22:59:22 2000; 20 IP addresses (4
hosts up) scanned in 106 seconds -->
</runstats>
</nmaprun>



---------------------------------------------------------------------
For help using this (nmap-dev) mailing list, send a blank email to 
nmap-dev-help () insecure org . List run by ezmlm-idx (www.ezmlm.org).