Fwd: SOC idea

[Jason Gerfen <jason.gerfen () gmail com>]

---------- Forwarded message ----------
From: Jason Gerfen <jason.gerfen () gmail com>
Date: Thu, Jan 9, 2014 at 5:31 PM
Subject: Re: SOC idea
To: Jacek Wielemborek <d33tah () gmail com>


Thanks for the response!

On Thu, Jan 9, 2014 at 10:21 AM, Jacek Wielemborek <d33tah () gmail com> wrote:
> Hello,
>
> See my answer inline below.
>
> 09/01/2014 07:40:35 Jason Gerfen <jason.gerfen () gmail com>:
> > I just posted on the announce list, which led me to
> > http://nmap.org/soc/#creative and now this message.
> >
> > They say you miss 100% of the shots you never take so here it goes...
>
> Congratulations, great attitude! I understand that you're interested in Google
> Summer of Code. If so, getting one's idea accepted is definitely not that
> difficult if you're motivated (I, for one, am a good example :)). You already
> did a few things right - got involved early, came up with an idea and
> requested comments on the mailing list. So far, way to go, IMHO :)

Thanks.

> > I have already started a project which utilizes the current nmap
> > binaries and improves the speed dramatically. The project has only
> > recently started and with some preliminary benchmarks shaves 20seconds
> > off simple host discovery scans.
> >
> > Details of the project can be found at
> > https://github.com/jas-/node-libnmap &
> > https://npmjs.org/package/node-libnmap.
> >
> > Thanks!
>
> I took a quick look at your code and if I understand it correctly, it's
> currently limited to host discovery. There's not much written yet, but I
> already have a few potentially interesting comments:
>
> 1. Your Github readme suggests that you might be using Nmap 5.51. If this is
> the case, do upgrade to the latest version before benchmarking your code,
> otherwise you might be wasting time.

I am currently using and testing against 5.51, so green there.

> 2. Consider using -oX instead of -oG. This way you will get an ouput format
> less likely to be changed in the future. As far as I know, -oG is meant for
> being read by a human, not parsed by a machine.

Am I wrong to assume -oG will allow the following?

$ nmap -sn -oG - 10.0.2.0/24 | awk 'up { print }'

Really due to the nature of the language, in this case node.js which
relies heavily upon Objects, and more specifically JSON formatted
objects a simple regex combined with a capture group of the results is
going to be faster than traversing XML elements.

> 3. In addition to using -sn, you might also want to call -n, which disables
> rDNS queries. This might speed up the queries to a greater extent.

This change has already been applied, thanks for that I did not see
that in the docs, but may have over looked it.

> 4. See if you're running Nmap in priviledged mode (one, though insecure, way
> to enable it is to run Nmap as root. For better solutions, see
> https://secwiki.org/w/Running_nmap_as_an_unprivileged_user). This enables many
> speedups, such as - in your discovery case - ARP ping, which is both more
> accurate and faster in case of local networks.

While I do appreciate the information about this type of setup,
because the project really should work as a 'plug and play' turn key
solution for simple scanning purposes this would be left to the sys
admin and can be applied with the optional flags a developer can use
with the project.

> It might also be a good idea to do some research on cutting down the Nmap
> initialization time - for example by compiling it with a C library lighter
> than glibc.

This again is something that will be left to the sys admin in this
projects case.

> Also, consider the use case you're planning to eventually accomodate. If your
> goal is to speed things up, there are things that you can do faster by
> spawning multiple Nmap processes than by running a single process. This
> usually not due to SMP though - currently, AFAIK, the biggest problem is that
> Nmap scanning is split into phases. There's the discovery phase, rDNS phase,
> port scanning phase, NSE, traceroute, OS discovery, service discovery etc...
> If in any of the hosts lags behind in any of the phases, all remaining ones
> have to wait. Currently everybody walks around the problem by scripting stuff
> like you do, but we all know that it's more or less a temporary solution.

Yeah, I can see why it is easier to split the subnet, range etc into
chunks and create individual process for each block of address to help
optimize each processor core.

> There are other use cases for your library if you keep developing it - some
> things cannot currently be done with pure command line and it's possible that
> not all of them will ever be added to Nmap. One of the features missing in the
> Nmap core that we might expect to be added to Nmap soon is port specification
> that would allow to exclude certain ports. You could probably implement it in
> your library more easily than you would do that in the C code (though it's not
> the correct solution actually).
Good to know.

> Another example is a feature that I once needed, but I would be surprised to
> ever find in Nmap core. I needed to perform a lot of OS discoveries on hosts
> whose closed/open TCP/UDP ports differed. I wrote a script myself that
> performed the probing and I believe that there are many use cases like this
> that could be made easier with a cool node.js interface. Keep in mind though
> that if you plan to go around problems that way, you have to solve many
> problems that Nmap takes care of for you.

Haven't run into any yet, but I am sure as the project progresses I
might. I have been looking at the py-libnode implementation as a sort
of guide except I am focusing this project on optimizing the scans
using node.js non-blocking IO functionality combined with separate
processes for each subnet chunk.

> One of the problems is timing. When I probed my first few thousands of hosts, I
> already got two automated abuse complaints from some trigger-happy IDSes. I
> quickly understood that even though I actually sent SYNs to just three ports,
> it summed up to numbers big enough for these systems to complain. I then wrote
> a script that ordered the targets so that it's less likely for two hosts from
> the same network to be probed in a short period of time. That's one of the
> things you might have to do for your tool to be useful in larger scans.

Never considered this, as most IDS are at perimeters in our env.

> My final advice for you is that you really should read Fyodor's book. It's an
> amazing guide to Nmap's functionality, with a lot of explanations that will
> help you understand what actually happens and when will you need the features.
> It's a few hours of reading, but I guarantee you that your knowledge of Nmap
> will greatly increase, which you're bound make use of in the future :)
> Especially that the default "nmap scanme.nmap.org" turns on quite a lot of
> options and many of them are very risky if you want to stay stealthy.

This is definitely good information, didn't know there was a book.

> Whoa, quite a lengthy post. I hope I helped and didn't discourage you. Keep up
> your work, it might actually prove useful if you think it over!

Thank you again sir, much appreciated.

> Yours,
> Jacek Wielemborek
> _______________________________________________
> Sent through the dev mailing list
> http://nmap.org/mailman/listinfo/dev
> Archived at http://seclists.org/nmap-dev/
_______________________________________________
Sent through the dev mailing list
http://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/