Re: nmap-os-db reaches 1500 signatures

[doug () hcsw org]

Hi Michael,

On Fri, Jul 25, 2008 at 05:50:21PM -0400 or thereabouts, Michael Pattrick wrote:
> If you write out your workflow, I could probably put something together.

The OS integration program looks great. I already have some
scripts for processing the service fingerprints. I'm pretty
comfortable with them and I don't want to distract you
from your other SoC stuff. If you're interested, I'm
appending a description I wrote a year or two ago.

What I think could be helpful is a script to do some
offline processing of the FPs. Often we get submissions
of obscure binary services that I don't feel comfortable
integrating because we've got too few examples. If there
was a script that searched for similar fingerprints in all
the thousands of previous submissions we might find enough
to create good match lines. I can kinda hack this sometimes
manually but an automated script would be great.

Also a script to scan the nmap-service-probes file and
look for duplicate or very similar match lines might be
nice. I bet a few near-duplicates have slipped in over
the years.

Doug




My core system is very simple. On a linux machine it should only take
a few minutes to set up. For navigating the collection of fingerprints
I use the "mutt" mail client. I have gotten very used to using mutt's
regular expressions, limits, flagging, and all its other advanced
features. In fact, it is such a useful, well designed program that I
couldn't imagine using anything else for this task OR for reading my
regular email!

The most useful mutt short cut is a simple limit which you can use to
grep through a mailbox:

Hit the lowercase l key then type (including the quotes)

~b "service: http"

This matches the regexp against the msg *b*ody fields. Note that you
can combine regexps here too (and that's why you need quotes). Oh and
it's case insensitive when you enter everything lowercase - this is great.

I link mutt to lisp through a mutt "macro" which, of course, is nothing
like a lisp macro. It's just a keyboard shortcut:

$ cat ~/.muttrc
macro pager "o"    "<pipe-message>lisp -quiet -load /home/doug/nmap/vd/utils/serviceunwrap<enter>"
macro index "o"    "<pipe-message>lisp -quiet -load /home/doug/nmap/vd/utils/serviceunwrap<enter>"

This pipes the message's entire body to my lisp program serviceunwrap
when I press o in mutt.

The program "lisp" is cmucl 19d - an amazing high-performance lisp
environment. On linux it is easy to set up with one of the numerous
binary packages. DO NOT try to compile it from source. I mean it. :)
For example, here is the official requirement for building cmucl:

"In order to build CMU CL, you will need:

a) A working CMU CL binary.  There is no way around this requirement!"

If you want something more portable, you can also use clisp which
runs just about everywhere and can be compiled from C source. In
fact I have read-time conditionals in the serviceunwrap program
that should allow it to run in both environments.

(defun shellexec (cmd)
  #+cmu (ext:process-input (ext:run-program "/bin/sh" (list "-c" cmd) :output t :wait t))
  #+clisp (ext:shell cmd))

#+ is a CL "reader macro" that lets us write portable CL code much
more elegantly than C's not-deserving-of-the-name-macro "macro pre-processor".

serviceunwrap depends on a program Fyodor wrote now in the nmap-dev
directory of SVN: servicematch.cc which lets us test the probe results
exactly as Nmap sees them.

And coupled with nvi and sometimes vim in lisp mode, that about sums up
my interactive environment.

I also use lisp to do some offline processing of the mbox files. In
addition to the serviceunwrap.lisp file I'm attaching a file mbox.lisp
which is a program I run to take a look at the different categories
of fingerprints that haven't been processed in an mbox file. I have
some other shortcuts and macros that I paste into the lisp REPL
with the mbox code loaded to get more info when needed. This is especially
useful when I'm looking for examples of fingerprints in PREVIOUS
SUBMISSIONS MBOXs. Sometimes you really need a few examples before you
can write a reliable regexp and it's always useful to test new regexps
against old fingerprints to make sure they still work.

I could organise some of these macros into programs but right now they
are just unstructured s-expressions. This is what people mean when
they say lisp is "interactive". As suprising as this concept is to,
ahem, a more structured opinion of program design, you don't need to have
running programs to experiment or even to get productive work done!

Note that for serviceunwrap.lisp I highly recommend compiling the
program to machine code because you want running that script to be
highly responsive: you'll run it thousands of times in an integration
session (debugging regexps, testing for regressions, etc). clisp can
compile to byte code but is still fairly sluggish for interactive use
of this nature (maybe 5 times slower than perl - both are due to
interpreter/compiler overheads and could be overcome with things
like persistent perl).

I usually compile a lisp image with machine code functions using the
Python compiler shipped with cmucl:

Loaded subsystems:
    Python 1.1, target Intel x86

(nothing to do with Guido's python, of course, just an unfortunate
name conflict)

These programs load as fast as a C program. In fact if you know
how to use Python by declaring a few types in your program, it
will use TYPE INFERENCE to generate machine code that often has
a signifigant performance edge over gcc for complex programs.

Because of this style of lisp development - write first and then
make it fast if needed - lisp is not only an excellent rapid
prototype language but an excellent production software language.
Lisp lets you make this performance tradeoff so much more easily
than any other language. You can compile functions in your image
once it looks unlikely they will change but leave the functions
you're working on interpreted for rapid development speed. It
is just simply far beyond any other environment in programmer
convenience and power.

Anyways I'm not sure how much of that will apply to integrating
OS detection fingerprints or frankly even somebody else integrating
version fingerprints, but it seems to work OK for me!

Best of luck,

Doug

Attachment: signature.asc
Description: Digital signature

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org