Nmap Development mailing list archives
RE: [RFC] Vulnerability library proposal
From: "Rob Nicholls" <robert () robnicholls co uk>
Date: Sun, 7 Aug 2011 11:10:45 +0100
Hi Djalal, Overall it all looks very promising :) Is it possible we can parse the IDS values to automatically create the references for popular IDs (e.g. CVE, OSVDB)? Otherwise you're duplicating holding 'CVE-2010-4344' and 'http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-4344' (this also ensures that people adding an OSVDB such as 69860 won't need to remember to also add the corresponding URL as a reference, which appears to have been missed in one of your examples). Then references can be used to hold links to other third party references, such as posts on Bugtraq, Full Disclosure or relevant blog entries (e.g. blogs.msdn.com), where an ID might not (easily) exist. It also makes it easier to update the references (i.e. we won't have to trawl through hundreds of scripts) if one of the big names changes the URL used to view IDs (although I suspect they'd automatically redirects users to the new location anyway). I must admit I'm not entirely sure about making IDS a mandatory value. What if someone puts together a check for something when a CVE doesn't exist? Sure, Apache seems to always have a CVE against every security issue it fixes; but less popular software might not have such references. I know we can create our own references, which could be a way around it though. This probably goes outside the scope, but what would the XML output look like? It'd be great if we could somehow use the internal tags to create XML tags to easily identify the state/risk factor/references etc. (to save us from having to parse all of the script output first). I know Daniel Miller has suggested (and even supplied) a YAML based solution, but I find it easier to read a more traditional XML output, and I generally use XPath to extract data from XML files generated by other tools (at least Ruby has native YAML support, if I ever need to go that way). My concern is that the vulnerability data is crying out to be marked up to allow for easy data extraction, and without it we're not really improving that much over the existing output, but if we hack in support for additional XML tags now for just the vulnerability data then would we make life more difficult if we later decided to introduce YAML (or other) output for all NSE scripts. Personally, I like the idea of XML rather than YAML, especially as it allows us to easily validate the Nmap XML files. It also avoids mixing XML and YAML in the same file (I'd prefer it if the XML output just contained XML; if people want YAML due to its good data representation then maybe we should create a YAML output file?). Rob -----Original Message----- From: nmap-dev-bounces () insecure org [mailto:nmap-dev-bounces () insecure org] On Behalf Of Djalal Harouni Sent: 07 August 2011 00:41 To: nmap-dev Cc: Marc Ruef Subject: [RFC] Vulnerability library proposal Hi list, This is a proposal for a new NSE vulnerability library. The library is designed to help managing discovered vulnerabilities and to make the output more consistent. It would be really great if we can have suggestions from pen-testers and from people that integrate and use Nmap in their security tools. Thanks in advance. vulns library ------------- Table: ------ 1) Functionality. 2) portrule and hostrule scripts. 3) prerule and postrule scripts (vulnerability management scripts). Note: when we say post-processing scripts we are referring to postrule scripts. 1) Functionality: ----------------- * This library is only for managing reported vulnerabilities. * The library must produce consistent output for all vulnerabilities. * The library should be able to silently store the vulnerabilities information in the registry for post-processing. * The library should also store and report vulnerabilities that have been checked but aren't present (not vulnerable, patched, etc). * The library must handle different states: UNKNOWN: We don't know the state of the vulnerability. LIKELY_VULN: The program seems vulnerable (check and compare versions) NOT_VULN: The program was confirmed not to be vulnerable. VULN: The program was confirmed to be vulnerable. * The library will accept different IDs ('CVE', 'OSVDB', 'YOUR_DB', etc), which are used to reference and to map vulnerability entries. Duplicate entries that are reported by several scripts must also be handled, and in this case the vulnerabilities state can also be updated. (i.e. from 'LIKELY_VULN' to 'NOT_VULN' or from 'LIKELY_VULN' to 'VULN') Vulnerabilities will be saved in the registry (global database) only when a post-processing script is selected by the user. The script will set a special variable or call a function that will activate the global vulnerability database, this way we do not wast memory and extra processing, and the user will have the feature without specifying any script argument. 2) portrule and hostrule scripts: --------------------------------- Vulnerability table info: - Vulnerability title (mandatory field). - State: UNKNOWN, NOT_VULN, LIKELY_VULN, VULN (mandatory field). - IDS: CVE, OSVDB, BID ... (mandatory field). The IDs entries will help the library to reference vulnerabilities and to track duplicate ones. - "Risk factor": if present then show it (optional). - "References": reference links (optional). - "Description": vulnerability description (optional). - "Check results" (optional). This field can be the result of the vulnerable check, did the server return anything ? This can help specialists to investigate the results and decide if the program is vulnerable or not. - "Exploitation results" (optional), if present then show it. - More information will be shown according to the debug level. Vulnerability table example: vuln_info = { -- mandatory fields title = 'Exim string_format Function Remote Overflow', -- string state = vulns.State.VULN, -- number IDS = {CVE = 'CVE-2010-4344', OSVDB = '69685'}, -- IDs map -- the following fields are all optional risk_factor = 'High', -- string description = 'vulnerability description ...', -- string references = { -- list of references 'http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-4344', 'http://osvdb.org/show/osvdb/69685', }, -- these fields are all optional and they will not be saved in the -- global database (registry). check_results = '...', -- string exploit_results = '...', -- string -- will probably be the result of table.concat() extra_info = '...', -- string (result of table.concat() or whatever) ... } Sample output: Portrule/Hostrule output: --@output -- PORT STATE SERVICE -- 25/tcp open smtp -- | smtp-vuln-cve2010-4344: -- | Vulnerabilities: -- | Exim string_format Function Remote Overflow: -- | State: VULNERABLE -- | IDs: CVE-2010-4344; OSVDB 69685 -- | Risk factor: High -- | References: -- | http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-4344 -- | http://osvdb.org/show/osvdb/69685 -- | -- | Exim exim User Account Configuration File Directive Local privileges escalation: -- | State: VULNERABLE -- | IDs: CVE-2010-4345; OSVDB 69860 -- | Risk factor: High -- | References: -- |_ http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-4345 API for portrule and hostrule scripts: -------------------------------------- * Create and return a new local report object. local report = vulns.Report:new(SCRIPT_NAME, host, port) Note: SCRIPT_NAME, host and port information can be used by post-processing scripts. * Report vulnerabilities. Arguments are: valid vulnerabilities tables. return report:make_output(vuln_table, ...) * Add vulnerabilities. Arguments are: valid vulnerabilities tables. report:add(vuln_table, ...) return report:make_output() General usage: -------------- local report = vulns.Report:new(SCRIPT_NAME, host, port) return report:make_output(vuln_table) 3) prerule and postrule scripts (vulnerability management scripts): ------------------------------------------------------------------- * Scripts can make the vulns.lua store silently the vulnerability reports. The library can store silently all reported vulnerabilities, this is very useful for post-processing scripts (postrule) that produce final reports. To enable this feature these post-processing scripts must first call vulns.save_reports(), then any call to report:add(...) will automatically save vulnerabilities in the global database (registry). * Vulnerability information for post-processing scripts: vuln_info = { script_name = "the name of the script that pushed me", -- If the 'host' table was specified when calling -- vulns.Report:new() -- e.g. local report = vulns.Report:new(SCRIPT_NAME, host) host = { ip = host.ip targetname = host.targetname, bin_ip = host.bin_ip, }, -- If the 'host' and 'port' tables were specified when calling -- vulns.Report:new() -- e.g. local report = vulns.Report:new(SCRIPT_NAME, host, port) port = { number = port.number, protocol = port.protocol, service = port.service, version = port.version, } -- mandatory fields title = 'Exim string_format Function Remote Overflow', -- string state = vulns.State.VULN, -- number IDS = {CVE = 'CVE-2010-4344', OSVDB = '69685'}, -- IDs map -- the following fields are all optional risk_factor = 'High', -- string description = 'vulnerability description ...', -- string references = { -- list of references 'http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-4344', 'http://osvdb.org/show/osvdb/69685', }, } Functions list: --------------- * Instruct the vulns library to store vulnerabilities reported by all the scripts in the database (registry). This function should be called by the prerule of one of the post-processing scripts. function vulns.save_reports() The following functions have effects only if the vulns.save_reports() was called. * Add vulnerability IDs to the database. Note: the library will handle and add the appropriate ids that are found in the vuln_table, so you don't need to call this function. function vulns.add_ids() * Get the IDS table. function vulns.get_ids() * Add vulnerability entries to the database, however it's better for scripts to use the Report class (i.e. call report:add(vuln_table, ...) ). function vulns.add(script_name, vuln_table, ...) * Search for the specified vulnerability ID and return true if there are hosts that are affected or were tested against this vulnerability. (All vulnerability states are counted). function vulns.lookup_id(id_type, id) * Search and return a list of vulnerability entries according to the specified selection filter. function vulns.find(selection_filter) selection_filter is a table with the current fields: state: The state mask (optional field). hosts_filter: A function that will inspect the host table and returns true or false. (Optional field). ports_filter: A function that will inspect the host and port tables and returns true or false. (Optional field). risk_factor: The risk factor (optional). id_type: The ID type ('CVE', 'OSVDB' or whatever) (optional). id: The vulnerability ID (optional). -- Return all vulnerabilities local list = vulns.find() -- Select the returned vulnerabilities local list = vulns.find({state = bit.bor(vulns.State.VULN, vulns.State.LIKELY_VULN), -- The argument is the vuln host table if it -- was specified. hosts_filter = function(host) -- parse and match data -- return true or false end, -- The argument is the vuln host and port tables -- if they were specified. ports_filter = function(host, port) -- parse and match data -- return true or false end, risk_factor = "High", id_type = 'CVE', id ='CVE-XXXX-XXXX'}) * Return a list of vulnerability entries (hosts that are affected or were tested against the specified vulnerability ID) according to the state mask. function vulns.find_id_entries(id_type, id, state_mask) e.g. -- all states local list = vulns.find_id_entries('CVE', 'CVE-XXXX-XXXX') -- Hosts and entries that are "NOT VULNERABLE" local list = vulns.find_id_entries('CVE', 'CVE-XXXX-XXXX', vulns.State.NOT_VULN) -- Hosts and entries that are: "LIKELY VULNERABLE" and "VULNERABLE" local list = vulns.find_id_entries('CVE', 'CVE-XXXX-XXXX', bit.bor(vulns.State.LIKELY_VULN, vulns.State.VULN)) With this function you can construct a list of major vulnerability issues and see if your network is vulnerable or not. * Return a vulnerability iterator. This function will take a selection filter like the vulns.find() function. function vulns.get_vulnerabilities(selection_filter) * Return formatted output to be displayed to the user. This function will take a selection filter like the vulns.find() function. function vulns.make_output(selection_filter) -- Post-processing scripts can do: return vulns.make_output({state = vulns.State.VULN, risk_factor = "High"}) * Return formatted output to be displayed to the user. This function will take a vulnerability entry that was returned by one of the previous functions: vulns.find(), vulns.find_id_entries() and vulns.get_vulnerabilities() function vulns.make_output_vuln(vuln_table) * Register callback functions that will filter vulnerabilities before they are saved in the registry. This is a powerfull function, use with care. The callback function will take a vuln_table as an argument, and must return true or false. function vulns.register_filter(callback_filter) Usage: ------ -- Post-processing script: prerule = function() return vulns.save_reports() end postrule = function() local ids = vulns.registry_get_ids() if ids then return true end return false end prerule_action = function() return nil end postrule_action = function() -- parse, manipulate and report saved vulnerabilities -- ... -- Return all the stored vulnerabilities that have a state: -- 'VULNERABLE' or 'LIKELY_VULNERABLE' return vulns.make_output({state = bit.bor(vulns.State.VULN, vulns.State.LIKELY_VULN)}) end tactions = {prerule = prerule_action, postrule = postrule_action} action = function(...) return tactions[SCRIPT_TYPE](...) end Vulnerability database Internal data representation: ---------------------------------------------------- VULNS = { -- Vulnerability entries ENTRIES = { HOSTS = { [host_a] = { vuln_1 = { title = 'Program X vulnerability', state = vulns.State.VULN, IDS = {CVE = 'CVE-XXXX-XXXX', OSVDB = 'XXXXX'}, -- the following fields are all optional risk_factor = 'High', description = 'vulnerability description ...', references = VULNS.SHARED.REFERENCES[x], }, vuln_2 = { ... }, ... }, [host_b] = { ... }, }, NETWORK = { }, }, -- Store shared data between vulnerabilities here (type of data: tables) SHARED = { -- List of references, members will be referenced by the previous -- vulnerability entries. REFERENCES = { { ["http://..."] = true, ["http://..."] = true, }, { }, }, }, -- This is a cache to reference all the vulnerabilities information: -- tables, maps etc. Only memory addresses are stored here. -- -- This will also help us to lookup vulnerabilities by IDs, and to -- update entries. IDS = { 'CVE' = { 'CVE-XXXX-XXXX' = { references = VULNS.SHARED.REFERENCES[x], entries = { HOSTS = { -- References to the hosts affected by this vulnerability. [host_x] = VULNS.ENTRIES.HOSTS[host_x][vuln_x], [host_y] = VULNS.ENTRIES.HOSTS[host_y][vuln_a], ... }, }, 'CVE-YYYY-YYYY' = { }, }, 'OSVDB' = { 'XXXXX' = { references = VULNS.SHARED.REFERENCES[x], entries = { ... }, }, 'YYYYY' = { references = VULNS.SHARED.REFERENCES[y], entries = { ... }, }, }, 'YOUR_FAVORITE_ID' = { 'XXXXX' = { ... }, }, }, } Thanks. -- tixxdz http://opendz.org _______________________________________________ Sent through the nmap-dev mailing list http://cgi.insecure.org/mailman/listinfo/nmap-dev Archived at http://seclists.org/nmap-dev/ _______________________________________________ Sent through the nmap-dev mailing list http://cgi.insecure.org/mailman/listinfo/nmap-dev Archived at http://seclists.org/nmap-dev/
Current thread:
- [RFC] Vulnerability library proposal Djalal Harouni (Aug 06)
- RE: [RFC] Vulnerability library proposal Rob Nicholls (Aug 07)
- Re: [RFC] Vulnerability library proposal Henri Doreau (Aug 09)
- Re: [RFC] Vulnerability library proposal Djalal Harouni (Aug 09)
- Re: [RFC] Vulnerability library proposal Djalal Harouni (Aug 09)
- Re: [RFC] Vulnerability library proposal Daniel Miller (Aug 09)
- Re: [RFC] Vulnerability library proposal Djalal Harouni (Aug 09)
- Re: [RFC] Vulnerability library proposal Henri Doreau (Aug 09)
- RE: [RFC] Vulnerability library proposal Rob Nicholls (Aug 07)
- Re: [RFC] Vulnerability library proposal Christian Heinrich (Aug 09)
- Re: [RFC] Vulnerability library proposal Djalal Harouni (Aug 09)
- Re: [RFC] Vulnerability library proposal Christian Heinrich (Aug 09)
- Re: [RFC] Vulnerability library proposal Djalal Harouni (Aug 09)
- RE: [RFC] Vulnerability library proposal Rob Nicholls (Aug 09)
- Re: [RFC] Vulnerability library proposal Djalal Harouni (Aug 09)