RE: nmap plugin architecture

["Jay Freeman \(saurik\)" <saurik () saurik com>]

<summary>I like the idea of output modules for the existing nmap base, and
feel you should continue working on that as I am not even working with the
existing nmap base; but do not feel that "processing" modules would add that
much worth.</summary>

> From your description, it seems as if what you are doing is more of a
modification of the existing architecture.  What I am doing was rather
different (I went to the nth degree with the modularization, and started
totally from scratch).

My goal is a much more drastic modification.  Instead of just keeping a
port-scanning core and then passing the found ports to "processing" modules,
I wanted all of the modules to work together towards the common goal of
generating information about the end host.  My top-level modules are all at
the same level of power in the environment, and are built the same way, and
my main goal was modularizing the scans themselves, even if nmap's main
design or core functionality of port, OS, and version (I throw this in
because of my bias with nmap+V :) ) scanning isn't changed.

Exploit detection or "auto-r00ters" (as mentioned in your posting to the
list) could definitely be built into my plug-in system; but I agree with a
comment of Fyoder from a while back that if you really want exploit
detection you might as well be using Nessus, which is likely to do a better
job of it anyway.  I would much rather move towards making OS, port,
service, version, application, remote module/plug-in detection, etc. scans
work together more, making them easier to develop, and allow them to use
information that each other has come up with in a better fashion.

I like the idea of modularizing the output of the existing nmap, but I
actually disagree with simply adding "processing" modules that have ports
passed to them for the reasons above.  I consider it to be non-core nmap
functionality.  I _do_ perceive service, protocol, and even version and
other invasive tactics to see what is out there.  All that information helps
provide a better network map, and let you understand what the computers out
there are doing.  When I think of a web server out there, I don't just look
at the OS, and know it has port 80 on it running HTTP, I mentally consider
the fact that it is running Apache, and has Apache's JServ installed, as
well as PHP3.  This server could be drastically different than a Windows
2000 server running IIS, but not necessarily a Windows 2000 computer running
Apache.

Personally, I feel as if adding XML output and having external programs pick
up the cause and do other things would be an easier fit for nmap, not more
difficult.  If it's so difficult to write, I'll do it.  I don't consider the
existing logging THAT difficult, and as I have mentioned an XML format can
flux and change all the time without breaking any existing parsers, even if
you don't use my scheme of backtracking through old versions using XSLT.

That should just become easier if you continue your work on a completely
modular logging system, which I really do like and feel would be a great
improvement over the existing system of passing different flags to the
output system so we end up with the different output methods sharing half
their code.  XML is very easy to create, and very easy to parse, assuming
you use the correct tools.  If it is becoming too much of a headache for
people then nmap should be patched with nmap+V, compiled in g++, and start
using C++ to work on it.


On to what I have already (which is all about 2 months old, haven't worked
on it since): a source structure that separates modules into directories,
configure scripts and makefiles that automatically detect modules and
compile them, a module subsystem which lets you load have modules "provide",
"require", and "request" other modules (although the system which sorts this
isn't in place).  I also have a command line option subsystem that lets
modules register and wait for command line options.

Currently I have no modules that do anything more powerful than print "All
right! Module LOADED!" or "Command line option -sS processed.".  I still
want to think some more about how I want to send data around the system, XML
is a definite possibility for that, but might not be necessary... and how
output would progress was also important.  Right now I am just allocating
memory and having all the modules decide on struct and class formats.

I am also using what I am calling "dynamic memory reallocation", which is
actually more likely to cause problems than anything else in the future.  I
need to see how this would work in a completely separated set of modules (in
different files even), although I don't consider that that important.  If it
doesn't cause problems in that scenario (or I relegate that to being even
less important), then I am apt to keep it.  I am using it to allow the
registered memory regions to be classes with constructors, or just really
complex data structures that need to be setup ahead of time.

How it works:  Modules register memory regions with the memory system, along
with functions to create and destroy the memory area.  Modules can also
"donate" memory, in which case they would give the destructor for the memory
only (which could be kept separate from the original registering module, so
if that memory is removed the original module can still step in and provide
it).  When modules need the data in these memory regions, they request it.
If it isn't already available, it will be allocated on the fly by the module
that offered it in the first place.  I was using this as the basis for the
port lists, actually, and overriding default structures.  A module might, as
it loads, donate memory that nmap had itself registered to override the
default structure.  This is solving memory leaks for all these options and
abilities (as nmap will then go and ask the modules to free their memory
when the system is shutting itself down, and know _who_ to ask, as someone
else may have allocated the memory in the first place), as well as allow for
the default options flexibility.

The output is what I really need to consider... my system is modular to the
point where I am not sure what will come out of it, things like nmap's new
IP protocol scan (which was added after I had started thinking how this
would all work) kind of shot out my willingness to think of everything in
terms of hosts and "ports" and then allow modules to add more information on
what they know about either the machine as a whole or the individual ports.
I'm still not sure, however, as after seeing how nmap was handling that
internally (not so much the implementation but the names of the variables)
started to make me lean towards the "protocols are the 'ports' of IP"
description of events.  If I did go that way, at least there would be a
basic structure that other applications could count on, and attempt to get
information from.  Using XML's flexibleness, modules could add the
information they have to these setups and not cause global naming issues.

Maybe standard output module would be constructed which takes the standard
internal scan structures to create an output XML document which incorporates
the default options, and then asks other modules to serialize their
information.  Of course, there would be a console output module as well :).
The console output might be able to be done as a totally general XSLT step,
but at bare minimum I can have the modules provide their own XSLT and
include it to the end of a global one for serializing their tags to straight
text.

Sincerely,
Jay Freeman (saurik)
saurik () saurik com

-----Original Message-----
From: null () mail saurik com [mailto:null () mail saurik com]On Behalf Of H D
Moore
Sent: Sunday, August 27, 2000 2:53 PM
To: Jay Freeman (saurik)
Subject: nmap plugin architecture

Hi,

I was hoping to elicit a response from you about where you are going
(and what you already have) with modularizing nmap.  My previous email
mentioned that I worked on my own mods last night, as soon as I read
your email I stopped development until some kind of consensus was
arranged. Would you like some help with your system?  Should I continue
my own development and compare notes later? My feelings won't get hurt
either way, I just don't feel like working on something that someone
else is already doing better.

-HD


---------------------------------------------------------------------
For help using this (nmap-dev) mailing list, send a blank email to 
nmap-dev-help () insecure org . List run by ezmlm-idx (www.ezmlm.org).