Re: Tudor's Status Report - #17 of 17

[Tudor-Emil COMAN <tudor_emil.coman () cti pub ro>]

And the guide I forgot to attach.

________________________________
From: dev <dev-bounces () nmap org> on behalf of Tudor-Emil COMAN <tudor_emil.coman () cti pub ro>
Sent: Tuesday, August 23, 2016 12:44:00 AM
To: dev () nmap org
Subject: Tudor's Status Report - #17 of 17


Hi,



This is my last status report, it's been a wonderful summer and I really enjoyed working at Nmap.

I'm going to use this final status report to summarize my work and use it's link on seclists.org for the GSoC final 
submission.

I have been quite stretched out, working on Nsock, Nmap and a bit NSE, but I did manage to leave a mark on all these 
projects.



Nsock:

For Nsock on Windows I have created the IOCP engine which greatly reduces CPU usage.

The engine supports UDP, PCAP, SSL and it can be used to do service scans on localhost with the -sT parameter for port 
discovery.


In order to test out it's performance I created a script that reads a file with IP addresses, establishes a TCP 
connection with each one, sends a HTTP Get request and waits for the answer.

While network throughput is the same for this simple example, looking at the CPU utilization IOCP uses 1% and poll uses 
about 20% and sometimes even more.

This greatly reduced CPU overhead explains why service scans took 20% less because PCRE could use more CPU to do probe 
matching.


This being said, probably every project using Nsock on Windows has an extra 20% CPU to spare making it a better 
alternative than select or poll.

Taking this into account I've made it the default engine for Windows. Being this efficient might encourage folks to use 
Nsock for other projects.


In order to properly integrate IOCP in Nsock some small changes had to be made to the API, these changes include: 
adding an extra variable to struct nevent and defining struct io_operations which will specify the network I/O 
operations to be used for connecting, sending and receiving data on a socket. The engine itself has less than 800 lines 
of code and it is contained inside a file named engine_iocp.c.


The code has been added in revision 36171.



Other optimizations:

A great tool for detecting some bottlenecks in Nmap is Callgrind from the Valgrind toolset.

Even though it adds a lot of overhead it is far better than gprof which can only catch functions that take a lot of 
time to finish as opposed to short functions that get called a lot.


This being said I committed some improvements to the main trunk:

[r36083] - Made  get_ping_pcap_result() return the proper value. It was always returning 0 thus making ultrascan 
process only one packet per loop which was pretty terrible for performance.

[r36085] - alloc_vsprintf was called unnecessarily when trying to print XML tags (but -oX wasn't set) so those tags 
would never be printed.

[r36022] - CONCURRENCY_LIMIT in NSE can be increased above 1000 with a higher --min-parallelism value.

[r36089] - The hostgroup size during the host discovery phase can be increased from the default 4096 to the 
--min-hostgroup value if that is bigger.


In [r36089] I also unified target_needs_new_hostgroup() from nmap.cc with the function with the same name in targets.cc 
because they were essentially the same function. target_needs_new_hostgroup is now declared in targets.h and defined in 
targets.cc.

[r36089] also had a small hack to limit the calls to this function, bringing some performance gains for large hostgroup 
sizes but after causing crashes when doing arp scans on Windows I reverted that part in [r3611].


And probably the most important performance increase I did represents speeding up the findHost() function.

[r36088] - incompleHosts and completedHosts are converted from a list to a set ordered by the host's IP address. In 
findHost() instead of iterating through the entire list and comparing each IP, std::lower_bound() is used to call the 
IP compare function in logarithmic time while using the iterator in linear time.

[r36095] - Switched from std:lower_bound() to the find() method from std::set that is much faster when moving through 
the set's elements.


This is a complexity change, it speeds up scans by as much as 200% for scanning /16 IP blocks and has many subtle 
influences on how well Nmap can handle high min-rates and high hostgroup sizes making it better behaved.



Mass scans:

With these optimizations it took almost 4 days to scan every single IPv4 address on port 80 finding about 115 million 
online hosts.


The scan showed some limitations in Nmap that I fixed in:

[r36084] - o.numhosts_scanned and o.numhosts_up are made unsigned ints.

[r36086] - fixed a compiler warning.

[r36094] - o.numhosts_scanned and o.numhosts_up are now printed as unsigned ints with %u



Following Brandon's suggestion I created a small performance guide people could use to select the right performance 
parameters for their mass scans.

I attached the .pdf version to this report.



Git Links:

[r36171] - https://github.com/nmap/nmap/commit/1e1f744186113454d21a8b2226eb54abf3b685cc

[r36083] - https://github.com/nmap/nmap/commit/89e4901d496dae158826b40d35368a56ac807068

[r36085] - https://github.com/nmap/nmap/commit/c8607320977240821aeb8629f4a7e60dae35bdf8

[r36022] - https://github.com/nmap/nmap/commit/6c8a753013685bbecd787a3c120853709344130d

[r36089] - https://github.com/nmap/nmap/commit/3ba4a87c75b3322b7251d395ff7ca70d95534147

[r36088] - https://github.com/nmap/nmap/commit/7c0280382e8682c151e73cd2afa560d304603462

[r36095] - https://github.com/nmap/nmap/commit/074f99b291c37402e8d66218d6e3ee2e79ff5bcd

[r36084] - https://github.com/nmap/nmap/commit/78435476df9473488c7eae952b2a9e6d39a1f562

[r36086] - https://github.com/nmap/nmap/commit/5d942ab1bd02707edbda10d7e295ec085e383cb9

[r36094] - https://github.com/nmap/nmap/commit/7f1ec2b806fdf48293ac1c2851bcaf58b9ef72b1



Last week's accomplishments:

- Found out what was wrong with IOCP event termination, changed the logic a bit and fixed it.

- Created the guide.

- I did process the XML logs I talked about last week but I found the results to be polluted by hosts that appeared to 
have all their ports open.


Priorities:

- Receive feedback for IOCP and fix problems as they come up.

- Complete the final evaluation.


Cheers,

Tudor

Attachment: guide.pdf
Description: guide.pdf

_______________________________________________
Sent through the dev mailing list
https://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/