CPE output merged

[David Fifield <david () bamsoftware com>]

All,

There has been some discussion about adding OS and service detection
output in Common Platform Enumeration (CPE) format.

Nmap with Common Platform Enumeration
http://seclists.org/nmap-dev/2008/q4/626
Analysis of using CPE for Nmap OS signatures
http://seclists.org/nmap-dev/2010/q3/278
Analysis of using CPE for version detection
http://seclists.org/nmap-dev/2010/q3/303

I just merged a branch that adds such output to Nmap.

== Example output

# nmap -sV -O -F scanme.nmap.org -oX cpe.xml
Nmap scan report for scanme.nmap.org (74.207.244.221)
PORT     STATE    SERVICE     VERSION
22/tcp   open     ssh         OpenSSH 5.3p1 Debian 3ubuntu7 (protocol 2.0)
80/tcp   open     http        Apache httpd 2.2.14 ((Ubuntu))
Device type: general purpose
Running: Linux 2.6.X
OS CPE: cpe:/o:linux:kernel:2.6.39
OS details: Linux 2.6.39
Network Distance: 10 hops
Service Info: OS: Linux; CPE: cpe:/o:linux:kernel

The cpe:/o:linux:kernel:2.6.39 comes from OS detection ("OS details:
Linux 2.6.39"), and the cpe:/o:linux:kernel comes from version detection
("Service Info: OS: Linux").

There is also CPE output for individual services, but only in XML. This
is what the XML CPE output looks like:

<port protocol="tcp" portid="22">
  <state state="open" reason="syn-ack" reason_ttl="53"/>
  <service name="ssh" product="OpenSSH" version="5.3p1 Debian 3ubuntu7" extrainfo="protocol 2.0" ostype="Linux" 
method="probed" conf="10">
    <cpe>cpe:/a:openbsd:openssh:5.3p1</cpe>
    <cpe>cpe:/o:linux:kernel</cpe>
  </service>
</port>
...
<os>
  <osclass type="general purpose" vendor="Linux" osfamily="Linux" osgen="2.6.X" accuracy="100">
    <cpe>cpe:/o:linux:kernel:2.6.39</cpe>
  </osclass>
  <osmatch name="Linux 2.6.39" accuracy="100" line="39262"/>
</os>

== Database format

The CPE information is stored in new fields in nmap-os-db and
nmap-service-probes that, unfortunately, are not backward-compatible
with earlier versions of Nmap. In nmap-os-db, a Class line may be
followed by a CPE line:

Fingerprint Microsoft Windows 2000 Server SP4 or Windows XP Professional SP3
Class Microsoft | Windows | 2000 | general purpose
CPE cpe:/o:microsoft:windows_2000::sp4:server
Class Microsoft | Windows | XP | general purpose
CPE cpe:/o:microsoft:windows_xp auto

The "auto" keyword at the end of a CPE line means that the CPE was
automatically derived from existing data; "auto" CPEs will be replaced
when we make enhancements to the automatic classifier. CPEs that have
been written manually, like those Henri has been adding, lack "auto" and
so will not be replaced.

In nmap-service-probes, CPE is treated like existing templates like p//
(product) and d// (device type). The slash at the end of each CPE URL is
not part of CPE, but only makes the syntax uniform with other fields.
These entries also have a way to say "auto": a trailing "a" at the end.

match ftp ... p/Microsoft IIS ftpd/ address rejected/ o/Windows/ cpe:/o:microsoft:windows/a cpe:/a:microsoft:iis/

== Current limitations

Not all database entries have a CPE classification. In nmap-os-db, we
have 2474 CPE lines for 3936 class lines (63%), and in
nmap-service-probes, we have 703 application CPEs and 1543 OS CPEs out
of 7352 match lines (10% and 21% respectively). Henri and I have been
working on the most common OSes and applications, so I think you will
get better percentages in practice. The biggest holes are hardware
versions (CPEs that start with "cpe:/h") and applications (starting with
"cpe:/a").

What you can do to help, is choose some small subset of one of the
database files (for example, all 3Com devices), and manually or
automatically generate CPE for it. Then send in a patch. You should
compare your addition to the official dictionary at
http://nvd.nist.gov/cpe.cfm (currently official-cpe-dictionary_v2.2.xml)
but the dictionary is still missing many of the entries that we have in
Nmap databases, so you should generate new CPEs following the guidelines
in http://cpe.mitre.org/specification/index.html.

David Fifield
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/