Tudor's Status Report - #14 of 17

[Tudor-Emil COMAN <tudor_emil.coman () cti pub ro>]

Hi,



This week I've had some unsuccessful attempts to improve some areas of ultra_scan:


1. I tried caching the probes that are allocated in doAnyNewProbes() so that only the first hostgroup allocates memory 
for probes and the rest would use those objects from the previous.

For scanning big networks like (65536 hosts) on all ports there doesn't seem to be any time gained, there might be some 
milliseconds but I don't think it's worth making the code that much more complicated as there is not a good enough way 
that I've found of passing those probes, except with a global variable or something.


2. I tried improving doAnyOutstandingRetransmits(). There I found a local probe cache that is searched in at every 
retransmission. The cache is a map where the key is a host and the value is a probe iterator so searching in it is fast 
but I tried making the search in constant time by adding to each host a variable that contains the iterator so you 
don't need to search for it anywhere. Doing this saves us from constructing a potentially big map and searching in it 
but I just couldn't see any performance benefits in testing. Retransmissions seem pretty rare anyway.



Also about regression testing last week's improvements:


1. Reducing calls to target_needs_new_hostgroup, malloc and alloc_vsprintf shouldn't add overhead in any situation.


2. Transforming incompleteHosts and completedHosts from a list to a set could possibly make the creation process more 
cumbersome because inserting in the front of a list was O(1) and inserting in a set is O(logN) but testing for smaller 
networks (256 hosts) show that using a set is still faster because the added benefit of searching faster, in the case 
that the search wouldn't come into play (all hosts are down, no packets are received), about 10-50 extra milliseconds 
are added to the scan.



Accomplishments:

- Raised o.ping_group_sz to match the value of --min-hostgroup if that is higher.

- I started a scan that would be equivalent to zmap scanning the entire internet on port 80.

./nmap 0.0.0.0/0 --min-rate 140000 --min-hostgroup 8192 -T5 -n -Pn -p 80 --max-retries 0 &> /dev/null -oG mass.log -sS 
--excludefile /etc/zmap/blacklist.conf

I calculated that it should take no longer than 5 days.

After this scan is over I'll use the IP's where port 80 was either open or closed (like -PS80) to do a bigger scan on 
all ports to update our --top-ports.


Priorities:

- Merge the two target_needs_new_hostgroup() functions.


Thanks,

Tudor






_______________________________________________
Sent through the dev mailing list
https://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/