Re: Nmap+V Great

["Jay Freeman \(saurik\)" <saurik () saurik com>]

OK, yes, it _is_ "great functionality" :-).  However, as attached to it as I
am, I have to admit that it isn't really itself "great".

...

"Why?", you ask? (hehe)

Please bear with me while I blur the line between my answer and some
not-so-subtle bashing of the development mentality I've been bringing to
this project :-).

It's slow.  It's _damned_ slow.  Well, there are lots of things out there
that are slower, but the big point is that it is much, much slower than it
needs to be:  it should be running it's tests on multiple ports in parallel,
while it is waiting that half a second for the first port to respond (or as
much as a few seconds if the wrong prompt was sent and we don't get anything
back) it should already be messing with the second.  I really need to sit
down and rearchitect it a little, it should be at least 10 times faster than
it is; in large-scale scanning situations, even faster... possibly even
geometrically faster considering you could really interweave multiple ports
from multiple hosts at the same time, although nmap isn't really designed
for that as an internal scan (which might be another argument for pulling it
as a seperate program that just post-processes nmap's output).  The problem
is that every time in the past that I have started working on it I've run
into the problem of over-reaching:  rather than just trying to make a better
nmap+V I try to make a better nmap, or a better vulnerability scanner, or...
oh it just gets pathetic :-P.

Another aspect of the problem is that what I have spawned from a challenge
(as most of what I do does).  It then quickly progressed as a continuation
of the challenge: people kept claiming (both directly and indirectly) that
it wouldn't be capable of binary protocols, or that it would never support
any protocol on any port (initially it only figured out what web server, not
whether it was a web server or a CVS server).  My arcane file format (which
Fyodor has likened to "line noise" in the past in my favorite e-mail on the
subject) and I have managed to overcome all but one of the actual
functionality complaints that have come up (at least that I actually
noticed, if anyone still has any issues bring them up).

I also agree with the comment by H D Moore that a modularized plug-in system
would be better; which is why each of my post-nmap+V attempts involved
turning nmap into a large plug-in architecture.  This included nmap++, the
one I got the farthest on, which really ended up dying because I didn't
understand shared objects on Unix well enough; I never figured out how to
get them to do fix-ups from the loaded library back into the library that
loaded it... I'm currently thinking that the support is just too primitive
to handle situations like that.  It is also important to note that I tend to
have a hard time finding the middle ground between "there, its arcane, but
it works better than you would likely believe it should" and "here, what I'm
working on will do everything that could possibly be needed by a security
scanner... oh, and the whole hunger thing... I'm also doing concurrent
development on version 2 which will also handle world peace".  I have a new
motto I try to keep in mind now, however, which is also now that of
CyberUniverse, Inc. (a stupid little consulting company I am part owner of
that is almost down to 0 clients):  "Start simple; then get harder."

The main reason I haven't done much work on this in a while is really
because I've been working on other projects, mainly my .NET decompiler
(which I gave my talk on today).  The .NET part of the conference is now
pretty much over, so I'm going to be switching to the Internet & Network
Security part of the conference.  I was talking to the keynote speaker today
(from Foundstone, an internet vulnerability scanner service company), and I
got excited about working on nmap again; might attempt to get something
better working this weekend :-).


The "line noise" e-mail goes into a list of things that Fyodor wanted before
merging this kind of scan.  To summarize his points:

- Syntax not strong enough for many protocols. --  This has since been dealt
with by adding binary escaping support, I still don't have powerful logic
but I have yet to come across a protocol that needed it and I would prefer
not to complicate things by adding it, either to this file format or even a
new one (unless it was a very special extraneous thing, as in all of the
other protocols used a simple format but this one used a script written in
____ scripting language).  If you want to help find one of these, a great
place to look is a protocol that requires two different connections to be
dealt with at the same time (I support dropping one and making a second one,
so it needs to be a case of juggling), or one that _requires_ a call-back.

- Syntax should support TCP & UDP. --  The syntax almost does (would need
only one extension, or just a second file, which might be better), the
implementation does not.  This is my main functionality weakness.  It is
important to note, however, that the main reason for this is that I don't
know of any UDP protocols other than DNS and NTP, which didn't seem like
that large enough of a target segment to develop the functionality against.

- Version checking may not be worth doing. --  I didn't quite get this one,
and might have misinterpreted its intent.  He brings up the hoops I had to
go through to get IRC to work, but to me that is a proof of how powerful my
line noise is and how _easy_ it was to do it (which it really shouldn't have
been).

- Implementation needs to be fast and parrallel. --  Not a functionality
complaint, but a very good point: one that I entirely agree with and the
reason I haven't been pushing merging this since this comment started coming
up in discussions.

- File format needs to be simple to understand. --  Frankly, I wish the OS
fingerprint file format was easier to understand, mine seems easy by
comparison... then again, I wrote mine :-).  Making what I have a little
easier is really just a matter of changing my single character commands into
words, adding actual delineated strings rather than my freaky brace
closures... the architecture would stay the same, the file format would
match line-by-line, it would just have a more complex parser (which, to me,
wasn't the important engineering aspect so I didn't spend much time on it).

- Tests should use port numbers as "hints" to minimize extraneous
connections, but should support protocols running on the "wrong" port. --  I
handle this as of releasing the very, very powerful (and I must say "cool")
'f' command.

- Tests should be as isolated as possible, the current implementation is a
"spaghetti" of goto statements that will be hard to administrate some
day. --  I agree with this, actually, although the way I actually _use_ the
'g', 'f', and 'p' commands (goto, fork, and branch-on-port) doesn't feel
that bad, although the glue is quite obvious to people looking at the file
format.  As an example, the hint table I use to handle one of the previous
requisits is just another section with numerous 'p' commands; it in no way
segregates this aspect of meta information from the rest of the "spaghetti"
and could contain the same functionality as any other section.  What I could
do is have a "-2" mean "goto the next brick that was marked with a 2", which
would let me keep the tests more logically seperated rather than having to
explicitely chain them as I currently do.  It also shouldn't be that much
harder to actually split it into a directory of seperate files and bounce to
the next file.

- Possible collaboration with Nessus, we should look at what they have. --
I don't remember what it was, but I remember looking into it and it not
being that complete or that elegant, rather using a full programming
language to define each check; my goal was/is to be as data-oriented as
possible.

Sincerely,
Jay Freeman (saurik)
saurik () saurik com

----- Original Message -----
From: "Secure42" <secure42 () flashmail com>
To: <nmap-dev () insecure org>
Sent: Monday, July 30, 2001 1:26 PM
Subject: Nmap+V Great


I found nmap+V really great, i have been looking for a tool like this for a
long time.
I think it would be great to add it to new nmap version and not to keep
outside like a patch. What do you think about it?
Also i miss one nmap+V page, i think reading the mailing-list archives
looking for information it is not really comfortable.

Thank you for your joob Fyordo, Jay and many others!

Secure42




---------------------------------------------------------------------
For help using this (nmap-dev) mailing list, send a blank email to 
nmap-dev-help () insecure org . List run by ezmlm-idx (www.ezmlm.org).