Re: Nmap host based registry

[Patrick Donnelly <batrick () batbytes com>]

Hi Patrik,

On Thu, Feb 16, 2012 at 6:09 PM, Patrik Karlsson <patrik () cqure net> wrote:
> Hi all,
>
> I've been working on a patch to add a host based registry to Nmap.
> One of the drivers behind it is to reduce memory usage and to allow Nmap to
> clear out registry entries that are no longer needed once a scan of a host
> group is completed.
> The patch is quite small and works by adding a registry entry to the host
> table passed to the host- and portrule functions.

I like the idea of this patch and the implementation is right.

As a historical aside on this subject:

We've tried to address memory management before for script data. The
nmap.registry table is in many ways a bad thing since any data inside
it will only be freed when either nmap ends or a script explicitly
removes the data. At one point I suggested having a _SCRIPT table that
holds data multiple instances of a script share. That didn't solve the
data lifetime issue because the _SCRIPT table has the same lifetime as
nmap.registry. It does help narrow the scope of data access.

I also suggested [1] having host and port userdata. This meant each
script received the same host userdata (no more tcopy in nse_main.lua)
and that you could associate some other data in a weak table with the
host userdata. When the host userdata is no longer used (strongly
referenced) by any script, the other data would be collected. Using a
registry for host tables is sort of the same thing (and is easier to
implement and understand).

Another common memory management issue is local versus global
variables. Global variables are a constant source of headache because
scripts/libraries set them on accident (or just don't know better) and
this causes all sorts of concurrency issues with scripts overwriting
other scripts variables. It also used to be that a script (e.g.
html-title) would have a unique global namespace shared by all of its
instance threads. So setting/getting a global variable in such a
script would break other instances of the script (a very common bug at
the time). In the end, we lessened the scope of the problem by all
script threads having unique environments; however, libraries still
have this problem (as a library is an entry point for multiple
scripts). This is why the nse_check_globals script [2] exists.

Anyway, I'd like someone to take away after reading that is (a)
scoping data access is good because (b) collecting unreferenced data
helps keep memory usage down. Ideally, scripts would share only the
bare-minimum with other scripts and when all instances of a script
expire, so does all the data it references (except the bare-minimum it
wants to export). Ditto for libraries. Globals are bad; Locals are
good.

[1] http://seclists.org/nmap-dev/2009/q1/47
[2] http://seclists.org/nmap-dev/2009/q3/70

-- 
- Patrick Donnelly
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/