Re: [NSE] http-archive.nse

[Patrick Donnelly <batrick () batbytes com>]

On Thu, Nov 21, 2013 at 4:09 PM, Henri Doreau <henri.doreau () gmail com> wrote:
> 2013/11/21 Henri Doreau <henri.doreau () gmail com>:
> > 2013/11/21 Patrick Donnelly <batrick () batbytes com>:
> > > Hi George,
> > >
> > > On Fri, Nov 15, 2013 at 12:30 PM, George Chatzisofroniou
> > > <sophron () latthi com> wrote:
> > > > Hi guys,
> > > >
> > > > I've finally finished two http-archive* scripts.
> > > >
> > > > http-archive.nse: This scripts acts as a time machine and brings old archives
> > > > for the target website (one per archived year).
> > > >
> > > > http-achive-liveness: This one takes it a bit further and crawls through these
> > > > previous versions. When it encounters a URL, it checks if it still exists on the
> > > > target website and adds it to the list. It will return the archived version
> > > > (along with its links) only if it contains newly discovered URLs. By using this,
> > > > you may find hidden links.
> > > >
> > > > You will find both scripts attached, so you can check the NSEDoc area for more
> > > > details.
> > >
> > > This is very cool. Thanks for coming back and finishing up the
> > > scripts. My main comment so far is that I'd like to see
> > > http-archive-liveness depend on http-archive (see [1]). I'd like to
> > > reduce code repetition. [Unless you have a good reason to keep them
> > > like this?]
> > >
> > > Also, I'm having issues running against scanme.nmap.org, you may have
> > > uncovered an NSOCK (?) bug:
> > >
> > > Running this:
> > >
> > > $ ./nmap -vv -d --script
> > > "$HOME/nmap/http-archive.nse,$HOME/nmap/http-archive-liveness.nse"
> > > scanme.nmap.org -ddd
> > >
> > > Nmap hangs with this final output:
> > >
> > > NSOCK DEBUG [2.6260s] msevent_delete(): msevent_delete (IOD #2) (EID #218)
> > > NSE: TCP 72.14.184.61:47605 > 207.241.224.26:80 | CLOSE
> > > NSOCK INFO [2.6300s] nsi_delete(): nsi_delete (IOD #2)
> > > NSE: Final http cache size (195423 bytes) of max size of 1000000
> > >
> > > None of the usual terminal keys work so it's trapped in some function.
> > > I'd run gdb on it but my installation is having issues. Is anyone else
> > > having problems running this? Any ideas Henri?
> > >
> > > [1] http://nmap.org/book/nse-script-format.html#nse-format-dependencies
> > >
> > > --
> > > Patrick Donnelly
> > Hi,
> >
> > looks like it. I can reproduce and will have a closer look ASAP.
> >
> > Regards
> we were wrong, nsock seems innocent and I'd rather suspect a weak
> regexp. Please find a backtrace attached. I'm sorry I can't dig
> further right now. I lack time for proper troubleshooting.

Looks right. Thanks for the backtrace!

George:

o Lines 141 and 142 (http-archive.nse) should use string.match and not
string.gmatch (unrelated to bug).

o Lines 138 and 139 (http-archive-liveness.nse) are causing the hang
bug. The initial ".*" in the pattern is unnecessary.

-- 
Patrick Donnelly
_______________________________________________
Sent through the dev mailing list
http://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/