FYI: Discussion regarding potential redesign of httpspider.lua

[Daniel Miller <bonsaiviking () gmail com>]

List,

I've been adding some tasks to our bug tracker [1] that anyone can jump in
and start working on, but I wanted to draw attention to one in particular
that Patrick and I have been discussing for a day or two.

The issue is #82, "httpspider redesign to only spider each site once" [2].
I have brought this sort of idea up before (linked in the issue
description), but the intention is to have all web spider-type scripts rely
on a single spidering of the target. Our current system of using cached
responses works pretty well on a small scale, but I've seen larger scans
outstrip the size of the cache, resulting in duplication of effort.

I bring this up here for a few reasons. First, I'd like the input of other
devs, especially those who have written some of the spider scripts. Second,
it's an educational look at how NSE script phases work, and the conceptual
code uses a few different NSE parallelism techniques [3] that folks may
want to learn more about.

Dan

[1] http://issues.nmap.org/
[2] http://issues.nmap.org/82
[3] http://nmap.org/book/nse-parallelism.html

_______________________________________________
Sent through the dev mailing list
https://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/