Re: Nmap Website Directory Tree Help

[Dave Horsfall <dave () horsfall org>]

On Sun, 26 Jul 2015, Hur wrote:

> Hi all, I  would really appreciate if you can help me with something.How 
> do I use nmap to view website directory tree? So that I can see all 
> possible links on that server.

Any web spider/crawler (that is what the tool is called) needs to honour 
any "NOFOLLOW" tags that may be present.  At least one web script, 
Wpoison, generates random pages of random gibberish with random email 
addresses to defeat just such things (I use it myself).

At least one crawler, MJ12bot, happily ignores these tags, and can get 
trapped for hours...  Hmmm; Bingbot seems to be another (I can see it as I 
type).

In short, if you ignore the warning signs about entering a minefield, 
well...

-- 
Dave Horsfall DTM (VK2KFU)  "Those who don't understand security will suffer"
Watson never said "I think there is a world market for maybe five computers."
_______________________________________________
Sent through the dev mailing list
https://nmap.org/mailman/listinfo/dev
Archived at http://seclists.org/nmap-dev/