Nmap Announce mailing list archives
Nmap 3.70 Released: Core Scan Engine Rewrite!
From: Fyodor <fyodor () insecure org>
Date: Tue, 31 Aug 2004 13:17:27 -0700
-----BEGIN PGP SIGNED MESSAGE----- Nmap hackers, Nmap's 7th birthday is tomorrow (Sept. 1). Since the proper way to commemorate it is with a major new release, I am pleased to announce the immediate availability of Nmap 3.70. This release brings dramatic changes. The main port scanning engine has been rewritten from scratch to be faster, scan many hosts in parallel, and be gentler against target hosts and networks. Service/version detection also now functions against many hosts in parallel. The UDP system has been overhauled to work in conjunction with version detection and a new "open|filtered" state to avoid false open reports against filtered systems. Nmap now estimates completion time for port/service scans in verbose mode (-v) when they are expected to take at least a couple minutes. A "port scan ping" system can dramatically improve scan times against heavily filtered hosts. There is also a new --exclude option that allow you to skip given hosts or networks in a broad scan. For example, a host may be too sensitive or critical to scan at a given time, or a security admin may not be responsible for certain subnetworks. Or maybe you want to cease scanning CW.Com IPs because they keep sending abuse reports to your ISP :). There is a workaround for the Windows SP2 problems (improved from the workaround in 3.55-SP2), and MAC address reporting now works on Windows. There are dozens of other changes, which you can read about in the Changelog entries below. Making Nmap faster was one of your top 5 priorities in the last Nmap user survey, and I hope 3.70 will not disappoint. Timing varies dramatically based on network/firewall characteristics, but almost all of the pre-release feedback has been positive. For example, Bill Peterson, an Information Security Analyst at Alcatel, regularly scans a million IP addresses over the Internet to keep the company secure. He reported that with 3.55, "my scans were running for more than two weeks". He switched to a 3.70 pre-release (incidentally on a somewhat beefier machine), optimized the option flags a bit, and was soon finishing the scans in under a day. I've been doing my own testing against thousands of machines as well. The time taken for the command "nmap localhost" on my primary development machine improved from 3 seconds to less than three TENTHS of a second. The command "nmap -T4 scanme.insecure.org" (a filtered-by-default machine) over my home aDSL line improved from 31.8 seconds to 19.7. Results involving multiple machines or UDP scanning are often far more dramatic than these. The official motto for this release is: So fast it deserves a CERT Advisory :). If you do find a case where the UNIX version 3.70 is slower than 3.55, let me know. I'm afraid that speeds of the Windows version of Nmap may not have improved as dramatically as on Linux/BSD/Mac OS X. I had to spend time working around MS SP2 nonsense rather than focusing on optimizing for that platform. You might expect that such dramatic changes to the core of Nmap would load the new release with bugs. There are always some, but I hope you will be pleasantly surprised by 3.70's stability. The nmap-dev list has admirably tested many pre-releases over the last few weeks. I would particularly like to thank Gisle Vanem, Eric of Catastrophe.net, Andy Lutomirski, Dana Epp, Mark-David McLaughlin, William McVey, Arturo "Buanzo" Busleiman, Bill Petersen, and Tom Duffy for major contributions to 3.70. Here is the full list of significant changes: o Rewrote core port scanning engine, which is now named ultra_scan(). Improved algorithms make this faster (often dramatically so) in almost all cases. Not only is it superior against single hosts, but ultra_scan() can scan many hosts (sometimes hundreds) in parallel. This offers many efficiency/speed advantages. For example, hosts often limit the ICMP port unreachable packets used by UDP scans to 1/second. That made those scans extraordinarily slow in previous versions of Nmap. But if you are scanning 100 hosts at once, suddenly you can receive 100 responses per second. Spreading the scan amongst hosts is also gentler toward the target hosts. Nmap can still scan many ports at the same time, as well. If you find cases where ultra_scan is slower or less accurate, please send a report (including exact command-lines, versions used, and output, if possible) to Fyodor. o Added --max_hostgroup option which specifies the maximum number of hosts that Nmap is allowed to scan in parallel. o Added --min_hostgroup option which specifies the minimum number of hosts that Nmap should scan in parallel (there are some exceptions where Nmap will still scan smaller groups -- see man page). Of course, Nmap will try to choose efficient values even if you don't specify hostgroup restrictions explicitly. o Rewrote TCP SYN, ACK, Window, and Connect() scans to use ultra_scan() framework, rather than the old pos_scan(). o Rewrote FIN, Xmas, NULL, Maimon, UDP, and IP Protocol scans to use ultra_scan(), rather than the old super_scan(). o Overhauled UDP scan. Ports that don't respond are now classified as "open|filtered" (open or filtered) rather than "open". The (somewhat rare) ports that actually respond with a UDP packet to the empty probe are considered open. If version detection is requested, it will be performed on open|filtered ports. Any that respond to any of the UDP probes will have their status changed to open. This avoids a the false-positive problem where filtered UDP ports appear to be open, leading to terrified newbies thinking their machine is infected by back orifice. o Nmap now estimates completion times for almost all port scan types (any that use ultra_scan()) as well as service scan (version detection). These are only shown in verbose mode (-v). On scans that take more than a minute or two, you will see occasional updates like: SYN Stealth Scan Timing: About 30.01% done; ETC: 16:04 (0:01:09 remaining) New updates are given if the estimates change significantly. o Added --exclude option, which lets you specify a comma-separated list of targets (hosts, ranges, netblocks) that should be excluded from the scan. This is useful to keep from scannig yourself, your ISP, particularly sensitive hosts, etc. The new --excludefile reads the list (newline-delimited) from a given file. All the work was done by Mark-David McLaughlin (mdmcl(a)cisco.com> and William McVey ( wam(a)cisco.com ), who sent me a well-designed and well-tested patch. o Nmap now has a "port scan ping" system. If it has received at least one response from any port on the host, but has not received responses lately (usually due to filtering), Nmap will "ping" that known-good port occasionally to detect latency, packet drop rate, etc. o Service/version detection now handles multiple hosts at once for more efficient and less-intrusive operation. o Nmap now wishes itself a happy birthday when run on September 1 in verbose mode! The first public release was on that date in 1997. o The port randomizer now has a bias toward putting commonly-accessible ports (80, 22, etc.) near the beginning of the list. Getting a response early helps Nmap calculate response times and detect packet loss, so the scan goes faster. o Host timeout system (--host_timeout) overhauled to support host parallelization. Hosts times are tracked separately, so a host that finishes a SYN scan quickly is not penalized for an exceptionally slow host being scanned at the same time. o When Nmap has not received any responses from a host, it can now use certain timing values from other hosts from the same scan group. This way Nmap doesn't have to use absolute-worst-case (300bps SLIP link to Uzbekistan) round trip timeouts and such. o Enabled MAC address reporting when using the Windows version of Nmap. Thanks to Andy Lutomirski (luto(a)stanford.edu) for writing and sending the patch. o Workaround crippled raw sockets on Microsoft Windows XP SP2 scans. I applied a patch by Andy Lutomirski (luto(a)stanford.edu) which causes Nmap to default to winpcap sends instead. The winpcap send functionality was already there for versions of Windows such as NT and Win98 that never supported Raw Sockets in the first place. o Changed how Nmap sends Arp requests on Windows to use the iphlpapi SendARP() function rather than creating it raw and reading the response from the Windows ARP cache. This works around a (reasonable) feature of Windows Firewall which ignored such unsolicited responses. The firewall is turned on by default as of Windows XP SP2. This change was implemented by Dana Epp (dana(a)vulscan.com). o Fixed some Windows portability issues discovered by Gisle Vanem (giva(a)bgnett.no). o Upgraded libpcap from version 0.7.2 to 0.8.3. This was an attempt to fix an annoying bug, which I then found was actually in my code rather than libpcap :). o Removed Ident scan (-I). It was rarely useful, and the implementation would have to be rewritten for the new ultra_scan() system. If there is significant demand, perhaps I'll put it back in sometime. o Documented the --osscan_limit option, which saves time by skipping OS detection if at least one open and one closed port are not found on the remote hosts. OS detection is much less reliable against such hosts anyway, and skipping it can save some time. o Updated nmapfe.desktop file to provide better NmapFE desktop support under Fedora Core and other systems. Thanks to Mephisto (mephisto(a)mephisto.ma.cx) for sending the patch. o Further nmapfe.desktop changes to better fit the freedesktop standard. The patch came from Murphy (m3rf(a)swimmingnoodle.com). o Fixed capitalization (with a perl script) of many over-capitalized vendor names in nmap-mac-prefixes. o Ensured that MAC address vendor names are always escaped in XML output if they contain illegal characters (particularly '&'). Thanks to Matthieu Verbert (mve(a)zurich.ibm.com) for the report and a patch. o Changed xmloutputversion in XML output from 1.0 to 1.01 to note that there was a slight change (which was actually the MAC stuff in 3.55). Thanks to Lionel CONS (lionel.cons(a)cern.ch) for the suggestion. o Many Windows portability fix and bug fixes, thanks to patch from Gisle Vanem (giva(a)bgnett.no). With these changes, he was able to compile Nmap on Windows using MingW + gcc 3.4 C++ rather than MS Visual Studio. o Removed (addport) tags from XML output. They used to provide open ports as they were discovered, but don't work now that the port scanners scan many hosts at once. They did not specify an IP address. Of course the appropriate (port) tags are still printed once scanning of a target is complete. o Configure script now detects GNU/k*BSD systems (whatever those are), thanks to patch from Robert Millan (rmh () debian org) o Fixed various crashes and assertion failures related to the new ultra_scan() system, that were found by Arturo "Buanzo" Busleiman (buanzo(a)buanzo.com.ar), Eric (catastrophe.net), and Bill Petersen (bill.petersen(a)alcatel.com). o Fixed some minor memory leaks relating to ping and list scanning as well as the Nmap output table. These were found with valgrind ( http://valgrind.kde.org/ ). o Provide limited --packet_trace support for TCP connect() (-sT) scans. o Fixed compilation on certain Solaris machines thanks to a patch by Tom Duffy (tduffy(a)sun.com) o Fixed some warnings that crop up when compiling nbase C files with a C++ compiler. Thanks to Gisle Vanem (giva(a)bgnett.no) for sending the patch. o Tweaked the License blurb on source files and in the man page. It clarifies some issues and includes a new GPL exception that explicitly allows linking with the OpenSSL library. Some people believe that the GPL and OpenSSL licenses are incompatable without this special exception. o Fixed some serious runtime portability issues on *BSD systems. Thanks to Eric (catastrophe.net) for reporting the problem. o Changed the argument parser to better detect bogus arguments to the -iR option. o Removed a spurious warning message relating to the Windows ARP cache being empty. Patch by Gisle Vanem (giva(a)bgnett.no). o Removed some C++-style line comments (//) from nbase, because some C compilers (particularly on Solaris) barf on those. Problem reported by Raju Alluri <Raju.Alluri(a)Sun.COM> As usual, 3.70 is available from http://www.insecure.org/nmap/nmap_download.html, including Windows (.zip format) binaries. For the more paranoid (smart) members of the list, here are the md5 hashes: d22f9944db8e3e33c223badf08b62f48 nmap-3.70-1.i386.rpm 5543e2857c02f5b6d4ed8f423c3a9bef nmap-3.70-1.src.rpm b9a92e48324cd393edaef773b294f36b nmap-3.70.tar.bz2 c886ee052940b6ea90bb6431cb3285ce nmap-3.70.tgz b2ffe4d856e728c1e65c5eb36c131dd2 nmap-3.70-win32.zip 7a7f17812e19746336ff8980effc4f1e nmap-frontend-3.70-1.i386.rpm These release notes should be signed with my PGP key, which is available at http://www.insecure.org/fyodor_gpgkey.txt . The key fingerprint is: 97 2F 93 AB 9C B0 09 80 D9 51 40 6B B9 BC E1 7E Enjoy! And please let me know if you find any problems. Cheers, Fyodor -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.4 (GNU/Linux) Comment: For info see http://www.gnupg.org iQCVAwUBQTTcc84dPqJTWH2VAQFYdQP/UOYTQHsfEuzQtYt2ejNAPMp4BhXSB5Gh RAG6HnoVCeHNj6JdyleS2eugisRQAJtxxnY32JLk5gdoFbljg7CmURa20TzHzlVZ b9uQN6ZBozEbACOhPXhIBa6drnaWD0V8mEJoHaO5mUn8cTesB/kWrebYFtq3PcD3 EF++ahQ7MHY= =2M0r -----END PGP SIGNATURE----- -------------------------------------------------- For help using this (nmap-hackers) mailing list, send a blank email to nmap-hackers-help () insecure org . List archive: http://seclists.org
Current thread:
- Nmap 3.70 Released: Core Scan Engine Rewrite! Fyodor (Aug 31)