Nmap Development mailing list archives
Re: [NSE][PATCH] datafiles.lua
From: jah <jah () zadkiel plus com>
Date: Wed, 10 Sep 2008 03:30:35 +0100
On 09/09/2008 19:51, Sven Klemm wrote:
The attached patch contains a few modifications to datafiles.lua: Make get_array() and get_assoc_array() normal functions. They were local functions in parse_lines() but there was no reason for them to be local functions and moving them out makes parse_lines() easier to read.
I didn't consider these functions useful enough to be exposed through the library, but there's no other reason why they couldn't be. Doing so does indeed make the code a little easier to read.
Move nmap.fetchfile() call to read_from_file() as it fits better there.
You're right.
Remove check for "(" in supplied pattern strings. I think the check is superfluous as a pattern does not need to have a "(" to produce a match as the complete match will be returned if there is no match group specified and strings containing "(" do not necessarily produce a match as a string might contain a literal "(" .
The check for a "(" is necessary to determine whether a string is a literal or a pattern for matching. The parse_lines() function needs to be able to reject certain combinations of key, value so that it doesn't waste time returning a table of junk - which would happen, for example, if the table passed was something like { ["hello"] = "nmap" }. This means that there is a requirement to include captures in the pattern string passed to parse_file() or parse_lines() and is the only solution I could come-up with to distinguish between a string literal and a pattern. With your patch, making the call datafiles.parse_file( "nmap-services" ) fails with "Error in datafiles.parse_lines: Invalid value for index udp." because this distinction isn't being made. The distinction between a string literal and a pattern allows us to pass a table such as { ["tcp"] = "pattern", ["udp"] = "pattern" } which returns tables such as the one returned by the original parse_services(). I think it's unlikely that a string literal used for these kinds of purposes would contain parentheses and with this limitation, the functions should work nicely. I've attached a patch which includes the other good changes you propose to make, and which includes your patch from http://seclists.org/nmap-dev/2008/q3/0707.html Regards, jah
--- The datafiles module provides functions for reading and parsing Nmap's -- data files. For example nmap-protocol, nmap-rpc, etc. These functions' -- return values are setup for use with exception handling via nmap.new_try(). -- @author Kris Katterjohn 03/2008 -- @author jah 08/2008 module(... or "datafiles", package.seeall) local stdnse = require "stdnse" --- -- Holds tables containing captures for common data files, indexed by filename. -- @type table -- @name common_files local common_files = { ["nmap-rpc"] = { [function(ln) return tonumber( ln:match( "^%s*[^%s#]+%s+(%d+)" ) ) end] = "^%s*([^%s#]+)%s+%d+" }, ["nmap-protocols"] = { [function(ln) return tonumber( ln:match( "^%s*[^%s#]+%s+(%d+)" ) ) end] = "^%s*([^%s#]+)%s+%d+" }, ["nmap-services"] = { ["tcp"] = { [function(ln) return tonumber( ln:match( "^%s*[^%s#]+%s+(%d+)/tcp" ) ) end] = "^%s*([^%s#]+)%s+%d+/tcp" }, ["udp"] = { [function(ln) return tonumber( ln:match( "^%s*[^%s#]+%s+(%d+)/udp" ) ) end] = "^%s*([^%s#]+)%s+%d+/udp" } } } --- -- This function reads and parses Nmap's nmap-protocols file. -- bool is a Boolean value indicating success. If bool is true, then the -- second returned value is a table with protocol numbers indexing the -- protocol names. If bool is false, an error message is returned as the -- second value instead of the table. -- @return bool, table|err -- @see parse_file parse_protocols = function() local status, protocols_table = parse_file("nmap-protocols") if not status then return false, "Error parsing nmap-protocols" end return true, protocols_table end --- -- This function reads and parses Nmap's nmap-rpc file. bool is a -- Boolean value indicating success. If bool is true, then the second -- returned value is a table with RPC numbers indexing the RPC names. -- If bool is false, an error message is returned as the second value -- instead of the table. -- @return bool, table|err -- @see parse_file parse_rpc = function() local status, rpc_table = parse_file("nmap-rpc") if not status then return false, "Error parsing nmap-rpc" end return true, rpc_table end --- -- This function reads and parses Nmap's nmap-services file. -- bool is a Boolean value indicating success. If bool is true, -- then the second returned value is a table containing two other -- tables: tcp{} and udp{}. tcp{} contains services indexed by TCP port -- numbers. udp{} is the same, but for UDP. You can pass "tcp" or "udp" -- as an argument to parse_services() to only get the corresponding table. -- If bool is false, an error message is returned as the second value instead -- of the table. -- @param protocol The protocol table to return. -- @return bool, table|err -- @see parse_file parse_services = function(protocol) if protocol and protocol ~= "tcp" and protocol ~= "udp" then return false, "Bad protocol for nmap-services: use tcp or udp" end local status, services_table = parse_file("nmap-services", protocol) if not status then return false, "Error parsing nmap-services" end return true, services_table end --- -- Generic parsing of datafiles. By supplying this function with a table containing captures to be applied to each line -- of a datafile a table will be returned which mirrors the structure of the supplied table and which contains any captured -- values. A capture will be applied to each line using string.match() and may also be enclosed within a table or a function. -- A function must accept a line as its paramater and should return one value derived from that line. function parse_file( filename, ... ) local data_struct -- must have a filename if type( filename ) ~= "string" or filename == "" then return false, "Error in datafiles.parse_file: No file to parse." end -- is filename a member of common_files? is second parameter a key in common_files or is it a table? if common_files[filename] then if type( arg[1] ) == "string" and common_files[filename][arg[1]] then data_struct = {{ [arg[1]] = common_files[filename][arg[1]] }} elseif type( arg[1] ) == "table" then data_struct = { arg[1] } else data_struct = { common_files[filename] } end end if type( data_struct ) ~= "table" then local t = {} for _, a in ipairs( arg ) do if type( a ) == "table" then if not next( a ) then a = { "^(.+)$" } end -- empty table? no problem, you'll get the whole line t[#t+1] = a end end if #t == 0 then return false, "Error in datafiles.parse_file: I've no idea how you want your data." end data_struct = t end -- get a table of lines local status, lines = read_from_file( filename ) if not status then return false, ( "Error in datafiles.parse_file: %s could not be read: %s." ):format( filename, lines ) end -- do the actual parsing local ret = {} for _, ds in ipairs( data_struct ) do status, ret[#ret+1] = parse_lines( lines, ds ) -- hmmm should we fail all if there are any failures? yes? ok if not status then return false, ret[#ret] end end return true, unpack( ret ) end --- -- Generic parsing of an array of strings. By supplying this function with a table containing captures to be applied to each value -- of a array-like table of strings a table will be returned which mirrors the structure of the supplied table and which contains any captured -- values. A capture will be applied to each array member using string.match() and may also be enclosed within a table or a function. -- A function must accept an array member as its paramater and should return one value derived from that member. function parse_lines( lines, data_struct ) if type( lines ) ~= "table" or #lines < 1 then return false, "Error in datafiles.parse_lines: No lines to parse." end if type( data_struct ) ~= "table" or not next( data_struct ) then return false, "Error in datafiles.parse_lines: No patterns for data capture." end local ret = {} -- traverse data_struct and enforce sensible index-value pairs. Call functions to process the members of lines. for index, value in pairs( data_struct ) do if type(index) == nil then return false, "Error in datafiles.parse_lines: Invalid index." end if type(index) == "number" or ( type(index) == "string" and not index:match("%(") ) then if type(value) == "number" or ( type(value) == "string" and not value:match("%(") ) then return false, "Error in datafiles.parse_lines: No patterns for data capture." elseif type(value) == "string" or type(value) == "function" then ret = get_array( lines, value ) elseif type(value) == "table" then _, ret[index] = parse_lines( lines, value ) else -- TEMP stdnse.print_debug("Unexpected value %s", type(value)) end elseif type(index) == "string" or type(index) == "function" then if type( value ) == "string" or type( value ) == "function" then ret = get_assoc_array( lines, index, value ) else return false, ( "Error in datafiles.parse_lines: Invalid value for index %s." ):format( index ) end else -- TEMP stdnse.print_debug("unexpexted index %s %s", type(index), type(value)) end end return true, ret end --- -- Reads a file, line by line, into a table. -- @param file String with the name of the file to read. -- @return Boolean True on success, False on error -- @return Table (array-style) of lines read from the file or error message in case of an error. function read_from_file( file ) -- get path to file local filepath = nmap.fetchfile( file ) if not filepath then return false, ( "Error in nmap.fetchfile: Could not find file %s." ):format( filename ) end local f, err, _ = io.open( filepath, "r" ) if not f then return false, ( "Error in datafiles.read_from_file: Cannot open %s for reading: %s" ):format( file, err ) end local line, ret = nil, {} while true do line = f:read() if not line then break end ret[#ret+1] = line end f:close() return true, ret end --- return an array-like table of values captured from each line --@param lines table of strings containing the lines to process --@param v_pattern pattern to use on the lines to produce the value for the array get_array = function( lines, v_pattern ) local ret = {} for _, line in ipairs( lines ) do assert( type( line ) == "string" ) local captured if type( v_pattern ) == "function" then captured = v_pattern( line ) else captured = line:match( v_pattern ) end table.insert( ret, captured ) end return ret end --- return an associative array table of index-value pairs captured from each line --@param lines table of strings containing the lines to process --@param i_pattern pattern to use on the lines to produce the key for the associative array --@param v_pattern pattern to use on the lines to produce the value for the associative array get_assoc_array = function( lines, i_pattern, v_pattern ) local ret = {} for _, line in ipairs(lines) do assert( type( line ) == "string" ) if type(i_pattern) == "function" then index = i_pattern(line) else index = line:match(i_pattern) end if index and type(v_pattern) == "function" then ret[index] = v_pattern(line) elseif index then ret[index] = line:match(v_pattern) end end return ret end
_______________________________________________ Sent through the nmap-dev mailing list http://cgi.insecure.org/mailman/listinfo/nmap-dev Archived at http://SecLists.Org
Current thread:
- [NSE][PATCH] datafiles.lua Sven Klemm (Sep 09)
- Re: [NSE][PATCH] datafiles.lua jah (Sep 09)
- Re: [NSE][PATCH] datafiles.lua Sven Klemm (Sep 10)
- Re: [NSE][PATCH] datafiles.lua jah (Sep 10)
- Re: [NSE][PATCH] datafiles.lua Sven Klemm (Sep 13)
- Re: [NSE][PATCH] datafiles.lua jah (Sep 13)
- Re: [NSE][PATCH] datafiles.lua Sven Klemm (Sep 15)
- Re: [NSE][PATCH] datafiles.lua David Fifield (Sep 15)
- Re: [NSE][PATCH] datafiles.lua Sven Klemm (Sep 10)
- Re: [NSE][PATCH] datafiles.lua jah (Sep 09)