Re: SOC idea

[Jacek Wielemborek <d33tah () gmail com>]

Hello,

See my answer inline below.

09/01/2014 07:40:35 Jason Gerfen <jason.gerfen () gmail com>:
> I just posted on the announce list, which led me to
> http://nmap.org/soc/#creative and now this message.
>
> They say you miss 100% of the shots you never take so here it goes...

Congratulations, great attitude! I understand that you're interested in Google 
Summer of Code. If so, getting one's idea accepted is definitely not that 
difficult if you're motivated (I, for one, am a good example :)). You already 
did a few things right - got involved early, came up with an idea and 
requested comments on the mailing list. So far, way to go, IMHO :)

> I have already started a project which utilizes the current nmap
> binaries and improves the speed dramatically. The project has only
> recently started and with some preliminary benchmarks shaves 20seconds
> off simple host discovery scans.
>
> Details of the project can be found at
> https://github.com/jas-/node-libnmap &
> https://npmjs.org/package/node-libnmap.
>
> Thanks!

I took a quick look at your code and if I understand it correctly, it's 
currently limited to host discovery. There's not much written yet, but I 
already have a few potentially interesting comments:

1. Your Github readme suggests that you might be using Nmap 5.51. If this is 
the case, do upgrade to the latest version before benchmarking your code, 
otherwise you might be wasting time.

2. Consider using -oX instead of -oG. This way you will get an ouput format 
less likely to be changed in the future. As far as I know, -oG is meant for 
being read by a human, not parsed by a machine.

3. In addition to using -sn, you might also want to call -n, which disables 
rDNS queries. This might speed up the queries to a greater extent.

4. See if you're running Nmap in priviledged mode (one, though insecure, way 
to enable it is to run Nmap as root. For better solutions, see 
https://secwiki.org/w/Running_nmap_as_an_unprivileged_user). This enables many 
speedups, such as - in your discovery case - ARP ping, which is both more 
accurate and faster in case of local networks.

It might also be a good idea to do some research on cutting down the Nmap 
initialization time - for example by compiling it with a C library lighter 
than glibc.

Also, consider the use case you're planning to eventually accomodate. If your 
goal is to speed things up, there are things that you can do faster by 
spawning multiple Nmap processes than by running a single process. This 
usually not due to SMP though - currently, AFAIK, the biggest problem is that 
Nmap scanning is split into phases. There's the discovery phase, rDNS phase, 
port scanning phase, NSE, traceroute, OS discovery, service discovery etc... 
If in any of the hosts lags behind in any of the phases, all remaining ones 
have to wait. Currently everybody walks around the problem by scripting stuff 
like you do, but we all know that it's more or less a temporary solution.

There are other use cases for your library if you keep developing it - some 
things cannot currently be done with pure command line and it's possible that 
not all of them will ever be added to Nmap. One of the features missing in the 
Nmap core that we might expect to be added to Nmap soon is port specification 
that would allow to exclude certain ports. You could probably implement it in 
your library more easily than you would do that in the C code (though it's not 
the correct solution actually).

Another example is a feature that I once needed, but I would be surprised to 
ever find in Nmap core. I needed to perform a lot of OS discoveries on hosts 
whose closed/open TCP/UDP ports differed. I wrote a script myself that 
performed the probing and I believe that there are many use cases like this 
that could be made easier with a cool node.js interface. Keep in mind though 
that if you plan to go around problems that way, you have to solve many 
problems that Nmap takes care of for you.

One of the problems is timing. When I probed my first few thousands of hosts, I 
already got two automated abuse complaints from some trigger-happy IDSes. I 
quickly understood that even though I actually sent SYNs to just three ports, 
it summed up to numbers big enough for these systems to complain. I then wrote 
a script that ordered the targets so that it's less likely for two hosts from 
the same network to be probed in a short period of time. That's one of the 
things you might have to do for your tool to be useful in larger scans.

My final advice for you is that you really should read Fyodor's book. It's an 
amazing guide to Nmap's functionality, with a lot of explanations that will 
help you understand what actually happens and when will you need the features. 
It's a few hours of reading, but I guarantee you that your knowledge of Nmap 
will greatly increase, which you're bound make use of in the future :) 
Especially that the default "nmap scanme.nmap.org" turns on quite a lot of 
options and many of them are very risky if you want to stay stealthy.

Whoa, quite a lengthy post. I hope I helped and didn't discourage you. Keep up 
your work, it might actually prove useful if you think it over!

Yours,
Jacek Wielemborek

Attachment: signature.asc
Description: This is a digitally signed message part.

_______________________________________________
Sent through the dev mailing list
http://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/