[NSE] Vulnerability library

[Djalal Harouni <tixxdz () opendz org>]

Hi list,

Please find attached the vulnerability library with some scripts to
show how to use the library. There is also a tar file to show the
output samples.

The library is based on the Vulnerability Library Proposal [1], you can
also find the same proposal at:
svn://svn.insecure.org/nmap-exp/djalal/nse-vuln/vuln_library_proposal.txt


The library scales well, it can handle a large number of vulnerability
entries, thanks to Lua and NSE.
(This is true unless you are reporting Flash vulnerabilities :p )

A scan like this using the provided sample scripts:
./nmap -PN -sn -n -d3 -v --script vulns-* 10.0.2.1/24 \
--script-args="vulns.showall,vulns-test-1.number=1000,vulns-test-2.number=1200"

Will use more than 400MB to store all the vulnerabilities info into
the registry, each host will have 1200 vulnerability entries.


Tests:
------
The provided scripts are just some simple hostrule scripts that will
generate vulnerability entries, and there is a postrule script for
post-processing.


To test the library you can use these simple commands:
./nmap -PN -sn -n -d1 -v --script vulns-* 192.168.1.1/30 \
--script-args="vulns.showall,vulns-test-1.number=2,vulns-test-2.number=3"

(without the 'vulns.showall' script argument and with -d5)
./nmap -PN -sn -n -d5 -v --script vulns-* 192.168.1.1/30
--script-args="vulns-test-1.number=2,vulns-test-2.number=3"


Note: you should run './nmap --script-updatedb' to be able to use
wildcards (e.g: '--script vulns-*').


Features:
---------
Some features of the library:

o Scripts can register popular Vulnerability IDs to automatically
  construct link references.

o A simple Report class for hostrule/portrule scripts to report
  vulnerabilities.

o Vulnerability tables can be saved silently into the registry for
  post-processing.
  Currently only the following fields of a vulnerability table are saved:
  vuln_table.title, vuln_table.state, vuln_table.risk_factor,
  vuln_table.scores, vuln_table.description, vuln_table.references and
  vuln_table.dates.

  Ignored fields:
  vuln_table.check_results   Results of the vulnerability check.
  vuln_table.exploit_results  Results of the exploit attempt.
  vuln_table.extra_info  

  These fields were already reported during the hostrule/portrule scan.


  If the vulnerability state is NOT VULNERABLE then the following fields
  are also ignored:
  vuln_table.risk_factor, vuln_table.scores, vuln_table.description and
  vuln_table.dates.


o The library will always re-construct the vulnerability tables to avoid
  using existing vulnerability tables referenced by external objects.

o The library automatically handles duplicate entries, and it can also
  update the state of the vulnerabilities that were reported by several
  scripts and saved into the registry (e.g: from LIKELY_VULN to VULN).

o Virtual hosts are also handled. Two vhosts affected by the same X
  vulnerability: each one will have its vulnerability table entry.

o Vulnerability IDs can be updated automatically:
  Example: Vulnerability reported by two scripts that affects the same host
  Vulnerability X reported by script A with IDs: CVE-XXXX-XXXX
  Vulnerability X reported by script B WITH IDs: CVE-XXXX-XXXX, OSVDB:XXXX

  The OSVDB ID of the second entry will be automatically added to the IDs
  of the first entry (only if it's the same host, we avoid doing extra
  cost operations: analyze all the IDs).

o The library can detect if some of the vulnerability IDs reported by
  different scripts are referencing different vulnerabilty entries. In
  this case a debug message like this will be printed (debug level 3):

  "vulns.lua: Warning at vuln 'X': please check the vulnerability IDs
  field."

  Please note that only the first returned vulnerability entry will be used,
  we'll ignore all the other entries (avoid complexity).

  It's the job of the script writers to provide the correct vulnerability
  IDs.

o The library can also detect if all the IDs are used and are referencing
  the correct vulnerability entry. In this case a debug message like this
  will be printed (debug level 3):

  "vulns.lua: Vulnerability 'X' referenced by %d IDs from %d (Good)"

  or Bad if some of them do not reference the vulnerability entry:
  "vulns.lua: Vulnerability 'X' referenced by %d IDs from %d (Bad)"

o The library offers some functions to format and return the vulnerability
  information table in a standard way:
  vulns.format_vuln() and vulns.format_vuln_table()

o Post-processing operations: there are a lot of operations that postrule
  scripts can do:

  Note: post-processing scripts must first call the vulns.save_reports()
  to initialize the vulnerability database and to get their Filter ID.
  vulns.save_reports() should be called by a prerule function to not miss
  any vulnerability even entries reported by prerule scripts.


  - Print all the vulnerability entries that were reported during a
    scan: vulns.make_output()   Results are sorted (which eats CPU).

    This function will print the list of scripts that have tested the
    same vulnerability, and it can show if the target or host were found
    vulnerable or not according to all the vulnerability entries that
    were reported by the different scripts during the scan.


  - Get the list of all vulnerability entries according to a selection
    filter:  vulns.find()
    Example: Find all Windows SMB vulnerabilities that were reported during
    the scan.


  - Get the list of vulnerability entries by ID (affected hosts):
    Example: Get all the hosts that are affected by the vulnerability X.


For other features you should read the library documentation.


Future improvements:
--------------------
Improve the output reporting of the Script Post-scan phase.

Currently results of the scripts are returned as strings to NSE, where
they will be pushed into a special Script Result object to be shown later.
This operation will consume memory.


> From the following logic of the Script Post-scan phase we can see:
(File: nmap.cc, line: 2017)
  if (o.script) {
    script_scan(Targets, SCRIPT_POST_SCAN);
    printscriptresults(script_scan_results, SCRIPT_POST_SCAN);
    script_scan_results->clear();
    delete new_targets;
    new_targets = NULL;
  }

There are no operations between the script_scan() and the
printscriptresults() call.

We can introduce extra flags in the script_scan() call to control the
behaviour of NSE, like: print results as early as possible.

We can make scripts return results in tables, then we'll just iterate,
clean and print. Of course we must use the same output functions to
support XML etc.

I'll try to make a patch when time permits.


Comments, questions and bug reports are welcome.

Thanks.


[1] http://seclists.org/nmap-dev/2011/q3/553

-- 
tixxdz
http://opendz.org

Attachment: vulns.lua
Description:

Attachment: smtp-vuln-cve2011-1720.nse.diff
Description:

Attachment: vulns-test-2.nse
Description:

Attachment: vulns-test-1.nse
Description:

Attachment: vulns-post-process.nse
Description:

Attachment: output-samples.tar.gz
Description:

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/