Re: NIST CPE

[David Fifield <david () bamsoftware com>]

On Thu, Mar 24, 2011 at 04:42:26AM +0530, ambarisha b wrote:
> Hi,
>                                                                    I
> have been studying the NIST CPE specification and David's reports from
> http://seclists.org/nmap-dev/2010/q3/278 - OS fingerprints
> http://seclists.org/nmap-dev/2010/q3/303 - version service probes
>
> When I first read the specification,it seemed like the standard isn't
> yet ready for adoption by the nmap database.But rethinking it , I
> guess these are the pains you take to adopt a common standard.
>
> I have studied the mockup script that David's report included.A few
> things came to my mind:
>
> 1. The script doesn't use the cpe dictionary completely ( I guess the
> vendor and vendor-family maps must have been obtained by referring the
> dictionary and manually putting it in).Shouldn't we be cross-checking
> a component name with the dictionary,because I think that the
> specification relies heavily on the dictionary and in many situations
> doesn't define clear-cut rules to express a cpe name.

It's true, the cpeify-os.py script doesn't use the dictionary, instead
only attempts to generate names that are compatible with dictionary
names. What would it mean to cross-check the names against the
dictionary? Just check if they exist, and show an error if not? That
makes sense, though I would handle that as a separate step and not as
part of the cpeify algorithm.

> 2. The script doesn't try to use the Fingerprint line from each
> fingerprint.I can see that we don't strictly follow a format,
> nevertheless , there is a specific format we "try" to stick to while
> writing the Fingerprint line.May be we can try to match the
> Fingerprint line with the human-readable tag in the dictionary(I don't
> mean a "cold" complete line match here).This ,ofcourse, would
> introduce some amount of doubt about the accuracy.

This is a good idea and it would be great to see an implementation of
it. The matching doesn't have to be perfect, only good enough to save a
human lots of work. It's fine if a few names still need to be handled
manually. Instead of matching dictionary descriptions, I would just
build another map or common patterns that we use (like "SP2") to CPE
components. This makes it a little more complicated because one
Fingerprint line can correspond to multiple CPE names. For example,
"Microsoft Windows XP SP2 - SP3" would become
        cpe:/o:microsoft:windows_xp::sp2
        cpe:/o:microsoft:windows_xp::sp3
This is even worse with names like "Linux 2.6.9 - 2.6.14".

> 3. The major concern is with the embedded device type and there is
> quite a big number of them.Mostly we're storing the details regarding
> the device in the Fingerprint line.So any progress on processing   the
> Fingerprint line will yield good results here also.

I see this as the biggest problem. Doing OS integration has given me an
appreciation for the wide variety of goofy and arbitrary naming
conventions for hardware. We could grep for anything that looks like a
model number, but even that seems dubious to me. I think that some kind
of automated first-step process would help a lot, but all this names
will still have to be reviewed by somebody.

> 4. We also need to have need to maintain consistency with the CPE
> dictionary while adding new fingerprints.This,I think, can easily be
> automated.Still,what if the cpe doesn't have a name registered yet?We
> add the fingerprint to database without the cpe name.And we will also
> have to revise the database periodically to see if any of the names
> for the fingerprints without cpe name have been  registered.

A tool that compares an Nmap database with the official dictionary,
notes any mismatches, and suggests close matches, would be very helpful.

David Fifield
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/