Re: Idea for discussion about NSE scripts

[Daniel Miller <bonsaiviking () gmail com>]

Kostas,

Thanks so much for your contributions and for your ideas surrounding NSE
development. I'm going to try to explain a bit about how Nmap development
is going and what some of the challenges are, how your proposal may affect
them, and some things you may not have considered. I want to be responsive
to the Nmap user and development community, but also realistic about some
of the challenges I feel stand in the way of faster development cycles.

First, I want to apologize for not getting to your and many others' pull
requests sooner. For the past 2 years or so, I have been increasingly
focused on Npcap, the Nmap Project's replacement for the unsupported
WinPcap library. Npcap is required for Nmap to work on Windows, where a
considerable portion of our user base is. It also is an increasingly
important source of licensing revenue that pays for Nmap Project
development efforts and infrastructure. We are fairly sure that we are
nearing maintenance stage on the Npcap codebase, with just a few remaining
stability problems reported by a handful of users. Once we make the Npcap
1.0 release, I expect I will be able to focus more of my time on Nmap,
including review of outstanding pull requests.

Of course, I am not the only Nmap developer, nor am I the only one
approving pull requests. There are many folks who have commit access to
Nmap's SVN repository [1], and I want to thank nnposter and Paulino
Calderon in particular for helping us keep up with NSE-related pull
requests and bug reports. This is important work, and I encourage any Nmap
committers (including former GSoC students) who have some time available to
review and merge code to take a look at the backlog and see if there's
anything that you can do to help out: https://github.com/nmap/nmap/pulls

NSE is indeed in many ways a separate entity from Nmap core, and that was a
big motivator for the design of the NSE system. A memory-safe, interpreted
language with access to lots of Nmap's great networking features allows for
a faster development cycle, and we've had great success in the past with
scanners for Heartbleed, Shellshock, MS12-020, and other vulnerabilities
being released shortly after the vulnerability announcement. What has
changed lately in this regard is mostly that Nmap did not participate in
Google Summer of Code 2018, and has no plans to participate in 2019 either.
This decision was made based on personal availability of Fyodor and myself
and on the level of effort going in to stabilizing and improving Npcap.
When Nmap participates in GSoC, we get a handful of new committers
(students) and dedicated reviewers (mentors) who work on adding new code
and reviewing relevant outside submissions, largely focused on NSE
scripting. These efforts are subsidized by Google. Without it, we have to
rely more on time volunteered by our network of developers, all of whom
have outside commitments and are not being compensated.

A complicating factor in the separation of NSE from Nmap core is that the
engine itself as well as several important NSE libraries are compiled in to
Nmap itself. It's not possible to completely divorce the NSE version from
the Nmap release version. This is a big part of why the previous
nmap-update project was stalled: maintaining a NSE updates feed apart from
core Nmap would require more effort than developing new code along with
Nmap core, because new NSE features mean a break in compatibility with
older Nmap releases. This also comes down to availability of effort, but
even if there were enough labor to commit to this effort, there would have
to be a change in architecture or processes to support tracking releases
separately from core.

One more concern worth noting regarding NSE code contributions is that even
though Lua is a memory-safe language, the only recorded security
vulnerabilities in Nmap have been in NSE scripts. Separating NSE into its
own repository would not solve the fact that we need code reviewers who can
spot issues like these that affect Nmap's users.

I hope this has helped you understand where the project is at regarding
NSE, overall development of Nmap, and some of the challenges in separating
NSE development from core Nmap. I know I tend to come across as pessimistic
when it comes to new ideas, so please understand that I am not trying to
shut down discussion at all, but rather I am putting my concerns up front
so we can continue to discuss. I also want to point out that Nmap has a
very long development history (21 years last September!) and that the core
development team has no intention of letting it waste away. We are aware of
many of the causes of the current reduction in NSE code output, and are
actively taking steps to address them. We look forward to working with you
and the rest of Nmap's global user/developer community to produce the best
scanner possible.

Dan

[1] Github is currently a read-only mirror of SVN. All commits are still
made the way they always have been, via SVN, so any former Nmap devs who
still hold SVN commit access can feel free to jump in with code
contributions directly. More complete discussion here:
https://github.com/nmap/nmap/blob/master/CONTRIBUTING.md

On Mon, Jan 7, 2019 at 6:44 AM Kostas Milonas <milonas.ko () gmail com> wrote:

> Hello everyone and happy new year!
>
> During the last 2-3 months I contributed with a few NSE scripts which
> unfortunately didn't get merged
> or reviewed. I know everyone is putting their efforts whenever possible
> and I totally appreciate their time.
> But I'm not here to complain! I just wanted to propose an idea for
> discussion!
>
> I think that NSE scripts, because of their nature, are more approachable
> to contribute than the core.
> Also, contributing with NSE scripts sounds more frequent, as a new script
> could be added to the project
> every time a new vulnerability is revealed.
>
> So, could the NSE scripts (along with or without the lib) benefit from
> being in a separate repo?
> - pull requests could get merged more frequently
> - could have separate administration from the core nmap repo (more people
> reviewing and merging,
>   without the fear of others messing with the core nmap repository)
> - in the future, could more easily lead to allow users to update their
> system with new scripts and update
>   the existing, using a simple nmap command or just do so using their
> package manager.
>
> I know that this would trigger changes on the distribution (in the sense
> of distro packages) and the build
> process, in order to meet the current state but thought that it could
> worth the discussion.
>
> I'd love to hear your feedback.
> Thanks in advance,
> Kostas
>
> _______________________________________________
> Sent through the dev mailing list
> https://nmap.org/mailman/listinfo/dev
> Archived at http://seclists.org/nmap-dev/
_______________________________________________
Sent through the dev mailing list
https://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/