Nmap and SVN Externals

[Fyodor <fyodor () insecure org>]

Hi Folks.  The developers here know that the Nmap repository is set up
as follows:

1) There is the /nmap trunk with the main Nmap code

2) There are five self-contained and ostensibly sharable code
   libraries, programs, and modules which are pulled in under /nmap
   using the SVN "externals" feature.  These all come from the same
   svn.insecure.org repository, but they have their own separate
   directories off the repository root.  The five are:

   o Nsock parallel sockets library is pulled into /nmap/nsock from
     /nsock

   o Nbase portability library is pulled into /nmap/nbase from /nbase

   o Zenmap Nmap GUI is pulled into /nmap/zenmap from /zenmap

   o Ncat utility is pulled into /nmap/ncat from /ncat

   o Nping utility is pulled into /nmap/nping from /nping

We also have a couple directories which I suppose could be externals,
but aren't (they fully live in /nmap).  These are 'ndiff' and
'libnetutil'.

I set things up this way initially because it seemed like a more
logical structure.  If things like Nsock and Nbase are to be used by
other programs than just Nmap, it seemed better to give them their own
directories rather than stuff them under /nmap.  For example, they are
used by Ncrack and also used by Ncat and Nping even before they were
integrated into the Nmap package.  Since you don't want every tool
(Nmap, Ncrack, etc.) to fork their own copy of these libraries and get
them out of sync, SVN externals seemed to be the best solution.  It
tells Nmap to grab these five from their canonical location when you
do an svn checkout or update, and commits go there too (if you do the
commit in the right place).

But working with the externals has proven rather awkward.  While svn
checkouts and updates work as I would expect (grab the latest code
from all of the externals), other SVN functions don't work that way.
If I do a "commit" in the Nmap directory, it doesn't recurse into the
externals and commit there.  Similarly, an svn log to find the recent
changes doesn't show those.  And when I use svn cp to make a branch in
/nmap-exp or wherever, you only get Nmap proper and your branch still
loads things like /nsock from their externals location.  This means
you may have to make several branches, and keep them up-to-date with
each (because svn merge only deals with one external at a time).

Now there are some cases where the "one at a time" behavior of svn
commit, log, merge, diff, and cp is desirable.  That would be
especially true if we were pulling in code maintained by 3rd parties
and we rarely or never made changes to it.  But in the case of Nmap's
externals, we maintain them all and I usually wish these commands
dealt with everything at once.  For example, if I'm trying to create a
"stable" branch off the current trunk, I want to make a branch of the
Nmap trunk code AND every external.  It is possible that SVN has some
sort of a feature to handle this better, but I don't know about it.

The externals also can cause problems for some external code.  For
example, David sometimes codes in a local git repo before updating
Nmap's svn.  But the git svn clone feature doesn't handle svn
externals, so he has to use a 3rd party git-svn-clone-externals.rb
tool.

So those are the problems and benefits of externals as they are used
now.  Maybe someone can pipe up with other pros and cons they have
seen with the way Nmap svn is organized.  But there are other options
we could do instead.  The ones which come to mind immediately are:

1) We could svn mv /nsock, /ncat, /nping, and the like all under
   /nmap.  This would "solve" the problem of dealing with externals
   for Nmap, but I think it is a bit worse of an organizational
   structure.  And problems like Ncat wouldn't be helped.  They'd just
   have to change their svn:externals /nmap/nsock and /nmap/nbase
   rather than /nsock and /nbase.  Still the vast majority of our work
   is Nmap-related, so this technique is worth considering.

2) I thought that if we got ride of the svn:externals and had users
   manually checkout nmap and then cd there and checkout /nsock,
   /nbase, and the like underneath, that would work better.  But upon
   further testing, that works worse than the externals approach.  It
   means you have to specify the "external" directories even for svn
   update, and svn status doesn't tell you about changes in the
   subdirs.  So scratch that.

3) Switch to another revision control system, such as git.  I don't
   think git can handle per-user read access control for the private
   parts of the repository (I have system configuration files and all
   sorts of other crap there), but we could potentially split off the
   public Nmap stuff into its own repository.  Of course switching to
   a new RCS would be a major pain.

Or maybe there are other ways to handle this situation in SVN that I'm
not aware of?  Or we could just continue dealing with it.  We do have
workarounds for the issues, they are just a bit annoying.

If anyone has thoughts on this issue, speak up!  I particularly want
to know whether the current (svn externals) bothers anyone else very
much and what they see as potential solutions.

Cheers,
Fyodor
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/