Re: Any guide for Adding match rules into nmap-service-probes?

[ryan chou <jkryanchou () gmail com>]

Hi Dainnel Miller,

   I'm so appreciated for your reply. It seems a complicated process.
Although I have read nmap-service-probes in details and known the
rule-format and what every directive does, I am still curious about how
this works. Is there any 101 tutorial for writing those match rules? How to
reduce false positive and false negtive? Especically for those private
scripts you mentioned, could you give me a brief introduction on its
principle and what they mainly do?

   Could you offer me some suggestions if I want to customize the
nmap-service-probes file ?

   Could any other developers participate in the process? If possible, how
could we contribute to it by not only just submitting fingperints
unrecognized by nmap on the pages you mentiond, but also writing match
rules and testing its converage for imporving those match rules' accuracy.
what should we prepare before start contributing? If not, could you give me
the reasons why impossible? maintainces cost? or any other reasons?

2015-06-07 22:35 GMT+08:00 Daniel Miller <bonsaiviking () gmail com>:

> Ryan,
>
>
> There is a set of scripts that streamlines the process of sorting and
> analyzing the hundreds of new service fingerprints that get submitted to
> the submitter page [1] every quarter. While the specific process and the
> scripts are private, the general process goes something like this:
>
> The fingerprint format is documented in the Nmap book and available online
> [2]. First, the fingerprint must be unwrapped, leaving a Probe name and the
> regex-escaped response which that probe elicited. In the most simple cases
> (including most "banner"-type services like FTP, SSH, and Telnet), the
> response is simply used directly as the match line, following the format
> described at [3]. Other times, some data may change in an insignificant way
> (like a timestamp or nonce) and it needs to be matched more generally,
> using regex atoms like "." or character classes.
>
> Choosing which probe response to use is a matter of optimizing some
> variables:
>
> 1. Choose a probe that is sent early, so that -sV doesn't take too long
> 2. Choose a probe that gets a unique response over one that is generic.
> This is why we have probes like FourOhFourRequest.
> 3. Choose a probe that may get fallback responses (Null or GetRequest) if
> there is some variability in the responses sent.
>
> The next step is to capture (as PCRE understands "capture") the portions
> of the response that represent interesting data that may change, like a
> hostname, version number, OS type, or other interesting info. These
> captures are then used in the version fields so that they can be presented
> to the user.
>
> The best way to understand this process (beyond reading the relevant
> chapter in the Nmap book [4]) is to read portions of the
> nmap-service-probes file.
>
> Dan
>
> [1] https://nmap.org/cgi-bin/submit.cgi?new-service
> [2] https://nmap.org/book/vscan-community.html#vscan-submit-prints
> [3] https://nmap.org/book/vscan-fileformat.html#vscan-db-match
>
> On Sat, Jun 6, 2015 at 7:24 PM, ryan chou <jkryanchou () gmail com> wrote:
>
> > and i know the fingerprint-submit page, while i want to how the
> > fingerprint-submitted be converted to match rule. any Guide or standard for
> > this. as far as i know not all fingerprint which no match rules matched
> > could be transform to match rule.: )
> >
> >
> >
> > 2015-06-07 8:16 GMT+08:00 ryan chou <jkryanchou () gmail com>:
> >
> > > Thanks Jacek, i Got it. i will resend this mail to other developers. :)
> > >
> > > 2015-06-06 23:56 GMT+08:00 Jacek Wielemborek <d33tah () gmail com>:
> > >
> > > > W dniu 06.06.2015 o 17:14, ryan chou pisze:
> > > > > Hi Daniel Miller,
> > > >
> > > > Hello,
> > > >
> > > > You sent the message to Nmap developers list, so I'd assume that you
> > > > would accept an answer from any other developer as well. Please find the
> > > > answer below.
> > > >
> > > > >      May i ask you some questions on the process of adding match rule
> > > > into
> > > > > nmap-service-probes? Are there any guide or standard for it? I only
> > > > found
> > > > > the vs-fileformat instructions in nmap offical Guide. No charpters
> > > > were
> > > > > about the process of writing regexes of rules and adding them into
> > > > > nmap-service-probes.
> > > > >
> > > > >    Could you introduce it for us with some samples included the part
> > > > of how
> > > > > to test it for improving the rules' coverage and reducing the rate of
> > > > false
> > > > > positive.
> > > > >
> > > > >    And Could it be a standard or Guide for more develpers to
> > > > participate in
> > > > > it to help improve the nmap？
> > > >
> > > > Generally, the best way to contribute is to report a fingerprint Nmap
> > > > generates to https://nmap.org/submit/ . A sample fingerprint might look
> > > > like this:
> > > >
> > > >  nmap localhost -p 31337 -sV
> > > >
> > > > Starting Nmap 6.47SVN ( http://nmap.org ) at 2015-06-06 17:54 CEST
> > > > Nmap scan report for localhost (127.0.0.1)
> > > > Host is up (0.000045s latency).
> > > > Other addresses for localhost (not scanned): ::1
> > > > PORT      STATE SERVICE VERSION
> > > > 31337/tcp open  Elite?
> > > > 1 service unrecognized despite returning data. If you know the
> > > > service/version, please submit the following fingerprint at
> > > > https://nmap.org/cgi-bin/submit.cgi?new-service :
> > > >
> > > > SF-Port31337-TCP:V=6.47SVN%I=7%D=6/6%Time=557317B1%P=x86_64-unknown-linux-
> > > > SF:gnu%r(NULL,2,"b\n");
> > > >
> > > > Service detection performed. Please report any incorrect results at
> > > > https://nmap.org/submit/ .
> > > > Nmap done: 1 IP address (1 host up) scanned in 1.20 seconds
> > > >
> > > > The part you should submit is the following:
> > > >
> > > >
> > > > SF-Port31337-TCP:V=6.47SVN%I=7%D=6/6%Time=557317B1%P=x86_64-unknown-linux-
> > > > SF:gnu%r(NULL,2,"b\n");
> > > >
> > > > (it might differ depending on the buffer you received)
> > > >
> > > > As for the nmap-service-probes modification, you will find documentation
> > > > here:
> > > >
> > > > https://nmap.org/book/vscan-fileformat.html
> > > >
> > > > Please let us know if there is anything else you would like to know
> > > > about.
> > > >
> > > > Cheers,
> > > > d33tah
> >
> > _______________________________________________
> > Sent through the dev mailing list
> > https://nmap.org/mailman/listinfo/dev
> > Archived at http://seclists.org/nmap-dev/
_______________________________________________
Sent through the dev mailing list
https://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/