Re: Output Format Changes (RE: Nmap 4.10 Released for Testing)

[Fyodor <fyodor () insecure org>]

On Tue, Jun 13, 2006 at 03:12:57PM -0400, Yudson, Marc CTR MDA/DOCN wrote:
>       About four months ago, I pushed a series of custom scripts into
> production here manipulating and cataloging data output from Nmap 4.03
> with the following flags:
>
>       nmap -sS -A -n x.x.0.0/12 -T3 -oG $WORK_DIR$RESULTS_FILE

Wow, that is one huge scan :).

> discovery as well as systems history. I'm really a huge fan of Nmap for
> the ease of use, the efficiency and speed of the scans, and I especially
> love the self throttling feature. Scanning a /12 really wouldn't be
> feasible without it. Puts little to no load on the network, and our
> infrastructure guys really don't notice it. Even with the  -T5 option
> there were no complaints. In contrast to other historical tools, we have
> trouble maintaining two weeks of data in a terabyte of space. Here, I
> have 100 days worth of data in under 1.5 Gigs. Hotness.

Thanks for the interesting report, and I'm glad you find Nmap useful.

>       The point of this e-mail was really just to support leaving the
> -OG output the same, so I don't have to go back and edit the various
> scripts scrubbing the output data. It wouldn't be a huge deal, but the
> potential for changing out the output format with every distribution
> could prove to be tiresome.

Don't worry, there aren't any imminent plans to change -oG.  Though
evey you would probably acknowledge that using XML results and
exporting them to a DB might scale better and be easier to query.
Though apparently you have scaled quite well already to ecompass a
scan of half a million machines nightly and store the results in 1.5GB
of flat files.

>       I'm also more than happy to share what I've written with anyone
> who is interested, if there is a medium to do so. I also threw together
> a ghetto web front end, little more than a basic query through the
> flatfiles. There are searchable system profiles, designed to implement
> device or system type-casting when I finally have an opportunity to
> script it out. Kinda ugly, but everything works very dependably. Also,
> everything is in Perl. 

Neat.  The system actually sounds a lot like Madhat's Perl Nmap tools
( http://www.unspecific.com/nmap/ ), which he used to use for scanning
a large enterprise nightly and reporting changes.  If you ever get
around to posting some of your tools and notes from your experiences,
please do send them here.

Cheers,
Fyodor


_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev