How I scan large networks (was Re: Running Malware Scripts)

[Brandon Enright <bmenrigh () ucsd edu>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

[ This initial discussion was about scanning a large network looking
for compromised hosts.  I have expanded it to include how I scan
large networks in the first place. ]


On Wed, 31 Dec 2008 11:39:43 -0800 or thereabouts "Rathbun, Dan"
<Dan.Rathbun () aecom com> wrote:

> > > Brandon,
> > >
> > > Does your script have any dependencies?  I am running RHEL 5.
> > >
> > > -Dan
> > >   

> > The only dependency problem I can think you could run into is
> > missing
> > perl's NetAddr::IP module.  I don't know anything about how RHEL
> > is
> > managed but if you run 'cpan -i NetAddr::IP' as root, you should
> > get
> > the module.  If you send me the error you are getting I'll tell
> > you
> > what you need.
> >
> > Brandon
> >   

> Thanks.  Yes it was 'NetAddr::IP' that was missing.  I appreciate your
> response on that.
>
> I ended up killing the scan of our LA subnets after nearly 5 days.  I
> think I will move this to a more beefy server, but I also don't think
> I can afford to do -PN right now.  Can I ask what NMAP switches you
> usually prefer for this application?  I need to develop a process that
> balances performance against results.
>
> Thank you, (and Happy New Year!)
>
> -Dan
>   


This is a complicated question and the answer falls into two
categories: "What I do today" and "What I plan to do in the near future
when David finishes his work on nmap-perf".

In short, today I scan with one Nmap process per IP.  To achieve any
speed at all I run many Nmap processes in parallel.  I wrote a deadline
scheduler in perl to actually handle the execution of Nmap processes.
David has put a lot of work into optimizing Nmap and working on
hostgroup performance so in the future I will scan groups of between 4
and 32 IPs with a single Nmap process.  I will determine the optimal
number for my purposes empirically.

Now for the long answer:

First thing I do is I maintain a list of netblocks that I want to scan
as well as IPs I want to exclude.  I then run (script attached):

#!/bin/bash

# This script lives in /home/bmenrigh/ip-lists/
cat excludes/*.txt | sort | uniq > excludes/all.txt
nmap -n -iL campus_netblocks.txt --excludefile excludes/all.txt -sL -oG - | egrep '^Host' | awk '{print $2}' | sort | 
uniq > campus_scanable.txt 
cat campus_scanable.txt | sort -R > campus_scanable_rnd.txt


This produces two files, one with all of the IPs, one per line listed.
The other is the same list, just randomized.

I then setup the file that actually does the scanning with the command
line options that I want (script attached).  Here it is:

#!/bin/bash

# This script lives in /home/bmenrigh/flexmap

NMAP="/home/bmenrigh/flexmap/nmap/svn/nmap/nmap"
DATADIR="/home/bmenrigh/flexmap/nmap/svn/nmap/"
EXCLUDE="/home/bmenrigh/ip-lists/excludes/all.txt"

cd /home/bmenrigh/flexmap/

ulimit -c unlimited
ulimit -v 262144    # Get this damn memory leak under control

sudo nice $NMAP --datadir=$DATADIR -p- -P0 -sV --version-all --allports --script=malware,intrusive --open -ttl 10 -O2 
-vv -d -T5 --min-parallelism 128 --max-parallelism 512 --min-rtt-timeout=1 --max-rtt-timeout=300 
--initial-rtt-timeout=150 --max-retries=1 --min-rate 250 --max-rate 4000 --host-timeout 900000 -oA 
../flexmap/log/report_$1 --excludefile $EXCLUDE $1 > ../flexmap/log/report_$1.txt 2> ../flexmap/log/report_$1.err


Now, the above script scans a single IP relatively quickly.  To scan a
couple hundred thousand hosts I've written a deadline scheduler
(attached) to manage the parallelism and speed up or slow down
accordingly.  It reads IPs from STDIN and forks off the above script to
Nmap single IPs (again, in the future I'll modify this to do scans in
chunks).

You run the script like so:

$ cat /home/bmenrigh/campus_scanable_rnd.txt | ./fastnmap.pl

And the script produces output like:

- - ------
Scan started at Mon Dec 29 03:52:00 2008 UTC
Goal end time at Mon Jan  5 03:52:00 2009 UTC
70800  IPs done (38.61%) in 231689   seconds with 51.46 average threads
Average thread-work-factor: 5.94 mIST
Currently using 51 threads, will finish at Mon Jan  5 02:33:16 2009 UTC
Target off of goal by -78.72 minutes
Network rate: TX  39759 pps (2.20 MBps); RX    577 pps (0.03 MBps)
- - ------
...this output updated every 100 hosts scanned...


Now there are a few adjustable parameters that you'll need to edit in
the script to get it to run to your liking.  First, the script is
deadline based which means it tries to get the scanning job done in a
certain amount of time (the goal time).  The default is 7 days but you
can edit "my $finish_in = 7 * 24 * 60 * 60;" to be any reasonable
amount of time.  The other thing you may want to adjust is the minimum
free memory amount (in K) "my $min_free_mem = 500000;".  The memory
stats don't take swap into account so if you use swap adjust "sub
mem_stats" accordingly.

Once you have things edited and run the script, it will print out the
stats (above) that report how it is doing.  The average threads count
is the average over the lifetime of the script.  It uses past
performance to predict future performance and needed parallelism.  The
"thread-work-factor" is the measure of how efficient a thread is.  5.94
mIST (milli-IPs-per-second-per-thread) means that a thread averages
scanning 5.94 hosts in 1000 seconds.  Based on that performance
metric, the scheduler will adjust the parallelism up and down to try to
meet the goal finish time exactly.  Over 7 days it is usually off by
between 5 and 10 minutes.  The current expected miss amount of -78.72
minutes means that at the current rate, the script will finish 78
minutes early.

Different networks and different scanning machines behave differently
(surprise!).  UCSD is basically powered off and on vacation right now
which means scanning isn't very efficient.  During normal days, the
thread-work-factor we get is about 6.50.  By editing the Nmap options
in the bash script, you can make it scan faster or slower which will
directly effect the thread-work-factor.

The machine I use to scan is a 8 proc, 8 GB mem, Intel Pro 1000 box.
If you have less than about 6 GB of memory you'll need to keep the
$min_free_mem variable set reasonably.

I'm running Linux 2.6.26 x86_64 with Ethernet MULTIQUEUE and NAPI
support turned on.  I also tweak the networking options with sysctl:

# Disables source route verification
net.ipv4.conf.default.rp_filter = 0
# Disable reverse path
net.ipv4.conf.all.rp_filter = 0

net.core.rmem_default = 65536
net.core.wmem_default = 65536
net.core.rmem_max = 536870912
net.core.wmem_max = 536870912
net.ipv4.tcp_rmem = 65536 65536 536870912
net.ipv4.tcp_wmem = 65536 65536 536870912
net.ipv4.tcp_mem = 33554432 134217728 1073741824
net.core.netdev_max_backlog = 16384
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1

net.unix.max_dgram_qlen = 64
fs.file-max = 65535
vm.min_free_kbytes = 131072
kernel.core_pattern = core-%e-%p-%t

I also set the TXQUEUELEN on the NIC higher:

# ifconfig eth0 txqueuelen 16384

I also max out the hardware ring buffer on the card with ethtool:

$ sudo ethtool -g eth0
Ring parameters for eth0:
Current hardware settings:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             4096


Our next step with this box is to upgrade the Gigabit card to a Intel 10
Gb card.  Everything I've read suggests that the 10 Gb cards service
packets from multiple hardware queues much better than the 1 Gb cards
do.

Of course, once scanning commences, I can start looking at intermediate
results with npwn.  I just point it at the log directory and give it a
max scan age of 7 days:

$ ./npwn.pl -a 14 -e awknowledged.npwn,notified.npwn -d ~/flexmap/log

The awknowledged.npwn and notified.npwn files are excludes for certain
issues that Npwn can report that I don't care about.

I think that just about covers how I scan.  Hope it helps.

Brandon


PS:  The my (and a co-worker's) far off goal is to replace all these
command line scripts with a database-driven webapp that manages,
schedules, and analyses scans.  Progress is slow going but ultimately we
plan to release it as FOSS.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)

iEYEARECAAYFAklb3NkACgkQqaGPzAsl94JcaACgs+vqVWgF2U6FjuWbD2c2IimD
CE0Anie4KYTFvu67c51gy5UrFkSby555
=+ZEe
-----END PGP SIGNATURE-----

Attachment: build_scanable.sh
Description:

Attachment: fastnmap.pl
Description:

Attachment: flexmap_one_ip.sh
Description:

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org