Re: [RFC] NSE pack/unpack library

["Philip Pickering" <pgpickering () gmail com>]

Hi!

2008/6/20 Fyodor <fyodor () insecure org>:
>  I assume that the lpack library
> you are starting with is as described at:
>
> http://lua-users.org/wiki/LuaPack
> http://www.tecgraf.puc-rio.br/~lhf/ftp/lua/

yes, the second one (the first is an outdated derivate, IIRC)


> Are there reasons that we can't use the current lpack library as-is?
> When we incorporate third party libraries, it is best to reduce
> changes to the minimum necessary.  Among other advantages, that makes
> it easy to upgrade them.  When we fix or improve things, it is best to
> submit those to the upstream maintainers in the hope that they will
> incorporate our changes so we don't have to keep maintaining them.
>
> Of course, sometimes we need functional changes which aren't desirable
> upstream.  And sometimes upstream is not maintained and so we end up
> maintaining all our changes.  But we try to limit such cases.
>
> Also, I'm not sure there is any need for Perl format compatability,
> unless the Perl version is much more powerful in a way which is
> important to us.  It may seem that we prefer Perl, since we have a
> Perl Compatable Regular Expression library in NSE.  But that was done
> because PCRE was already part of Nmap when NSE was added, and Lua's
> "patterns" just didn't seem powerful enough for our needs.
>
> Another advantages of limiting the ways that our version diverges is
> that we may be able to simply reuse or link to the source library's
> documentation.
>
> Now I don't know anything about lpack and I rarely use Perl's
> pack/unpack.  So maybe there are important reasons why we need to use
> Perl-style.  But if there aren't, I think we should stick with the
> format of the source library we use.

Well, I understand your concerns. But I try to explain, why modifying lpack
seems necessary to me.

lpack lacks some functionality we need. These features are
at least H for hex strings, B for bit strings and x for null/ignoring bytes.
By adding these features the original operator characters get misleading.

For example, the original short-operator was 'h'/'H' instead of 's'/'S'.
If I add a hex operator and call it 'H', we have a problem. If I call it
'X' and the null-byte 'n' to avoid confusion, then we also have a problem,
since 'n' is also already used by lpack as a lua native number.
Of course it's possible to call heX 'X', nUll byte 'U' and biT strings 'T', it
just doesn't seem intuitive anymore (but that's just personal taste).

Still, the main problem is the missing hex, bits and the null byte, rather
simple features I believe, but still needed.

Therefore adding lpack without modifications doesn't seem to be enough
(the same is true with an alternative pack/unpack lib called struct).
Especially since lpack registers itself at string.pack and string.unpack, which
folks at #nmap-nse opposed, at least minor changes are necessary.

Another interesting detail is the possible addition of pack/unpack
to Lua 5.2, using either lpack or struct as a base (again putting it
under string.*, as the discussion on the Lua mailing list suggests)
which will most likely miss the features, too.

I guess, we have several alternatives:
- modify lpack, send it to its maintainer and hope the features get added
- use lpack unmodified (except registering in under bin.*) without new features
- use lpack unmodified, write a wrapper around it and add needed features there

Right now I'm not really enthusiastic about the last alternative. Although
it solves both requirements (adding features and leaving as much
as possible unmodified) it does this by adding quite a bit of overhead.

The first advantages/disadvantages which come to mind are

(+) upgrading to a new lpack or pack/unpack in 5.2 would be rather easy

(-) it results in some kind of complicated design, calling a function to call
another one, both working on different aspects of the same job. But then
again it could be argued, that this is just the way functions work most of
the time :)

I hope the situation got clearer now. I'd like to hear your opinion on it.

Thanks,
Philip

_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://SecLists.Org