Re: Tudor's Status Report - #15 of #17

[Daniel Miller <bonsaiviking () gmail com>]

Tudor,

I have been putting some thought into how you could best accomplish the
scanning. Since a big part of the hassle of Internet-wide scanning is
managing exclusion lists, I suggest you take one address at random from
each /24 in your "alive hosts" list and do a scan against each. This will
get complaints from lots of folks without being as resource-intensive as a
full scan.

At this point, you could reduce your target list by removing any addresses
from /24 blocks that resulted in complaints. Also, I would suggest removing
any addresses from /24 blocks that resulted in all ports open (or 900/1000
open or similar). These are likely not real targets.

You can also split up the ports to get a statistically-significant sample
instead of scanning all 65535 ports on every target. For each port,
sampling 5000 targets will give an accurate frequency +/-2 at 99%
confidence, so with 100M potential targets, you could break them into 20K
groups and scan a different 4 ports on each one. That seems a bit strange,
so maybe do 64 groups of 1024 ports against 5000-ish hosts each.
Significantly less workload, but decent results.

Dan

On Tue, Aug 9, 2016 at 12:38 PM, Tudor-Emil COMAN <
tudor_emil.coman () cti pub ro> wrote:

> @Daniel <bonsaiviking () gmail com>
>
> That was a mistake indeed, I was scanning the first 1000 ports.
>
> I recalculated how long it should take to scan for 65536 ports and it
> looks like it would take a lot of time, I'm trying to see if I can do some
> significant improvements before starting again.
>
> I'm probably going to end up scanning a random subset of those 115 million
> IP's.
>
>
> @d33tah
>
> I used research.nmap.org.
>
>
> It has:
>
>
>    - Intel Xeon E3-1230 v1/2 (4-core)
>    - 32GB RAM
>    - CentOS 7 OS
>    - 2TB SATA drive
>    - 1Gbps ethernet
>
> Bandwidth utilization would fluctuate a lot but I've seen it go as high as
> 79 Mbps.
>
> CPU would be at about 100%
>
> Memory at 0.2% so about 640 Mbytes.
>
>
> The scan was:
>
> ./nmap 0.0.0.0/0 --min-rate 140000 --min-hostgroup 8192 -T5 -n -Pn -p 80
> --max-retries 0 &> /dev/null -oG mass.log -sS --excludefile
> /etc/zmap/blacklist.conf
>
>
>
> ------------------------------
> *From:* Jacek Wielemborek <d33tah () gmail com>
> *Sent:* Tuesday, August 9, 2016 7:08:26 PM
> *To:* Tudor-Emil COMAN; dev () nmap org
> *Subject:* Re: Tudor's Status Report - #15 of #17
>
> W dniu 09.08.2016 o 08:46, Tudor-Emil COMAN pisze:
> > Hello folks,
> >
> >
> >
> > Scanning the entire internet on port 80 finished and took 353716.09
> seconds.
>
> Hi Tudor,
>
> Four days, nice! What was the command you used? What were the specs of
> the server you used and how much resources (bandwidth, transfer, memory)
> did it use?
>
> I'd love to hear more about this.
>
> Cheers,
> d33tah
_______________________________________________
Sent through the dev mailing list
https://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/