Re: httpspider lib and hostnames with special characters

[Gutek <ange.gutek () gmail com>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Le 05/03/2012 16:39, Djalal Harouni a écrit :
> On Mon, Mar 05, 2012 at 04:32:50PM +0100, Djalal Harouni wrote:
> > Hi,
> >
> > On Mon, Mar 05, 2012 at 03:30:43PM +0100, Gutek wrote:
> > > It works (no error triggered in debug), but I have noticed something :
> > > when the hostname is just an alphanum format (eg: azerty.com), no problem
> > > when the hostname has hyphen (eg: az-erty.com), httpspider treats all
> > > links as "Link is not within host: http://az-erty.com/foo.bar";, and I
> > > have to add the withindomain argument to make it work correctly.
> > >
> > > Am I doing something wrong here, or missing a key point  ?
> > You're right: hostnames may contain hyphens but at the end.
> Ah, I mean not at the end.
>
> > -- 
> > tixxdz
> > http://opendz.org
> > _______________________________________________
> > Sent through the nmap-dev mailing list
> > http://cgi.insecure.org/mailman/listinfo/nmap-dev
> > Archived at http://seclists.org/nmap-dev/

Thanks Djalal.
It sounds to me like a weakness in httpspider's efficiency. Let's
consider a practical example with h-online.com and a
httpspider-dependant script, let's say http-backup-finder.
With a simple command like nmap -v -Pn -p80 -n --script
http-backup-finder www.h-online.com  it (silently) won't work because a
debug reveals that every link will be discarded, maybe fooling the user
into thinking that no backup was found:
- ----------
NSE: httpspider: Spidering limited to: maxdepth=3; maxpagecount=20;
withinhost=www.h-online.com
...
NSE: httpspider: Link is not within host:
http://www.h-online.com/nettools/tools/spam-list-query
NSE: httpspider: Link is not within host:
http://www.h-online.com/security/services/Reserved-IPv4-addresses-732899.html
NSE: httpspider: Link is not within host:
http://www.h-online.com/Contact-273335.html
NSE: httpspider: Link is not within host:
http://www.h-online.com/Privacy-Policy-of-h-online-com-273337.html
- -----------

Now, with a script arg to override this withinhost issue, it will work
as intended:
nmap -v -Pn -p80 -n --script http-backup-finder --script-args
http-backup-finder.withindomain=www.h-online.com -d2 www.h-online.com

- -----------
NSE: httpspider: Spidering limited to: maxdepth=3; maxpagecount=20;
withindomain=h-online.com
- -----------

As I understand it, a withindomain argument is mandatory when users want
to deal with hyphened hostnames ? if it's an intented behavior and not a
bug, maybe this should be explicitely stated in the documentation ?

Regards,

A.G.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.16 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk9U6QMACgkQ3aDTTO0ha7jFBQCeJh5Yqjv/aBf/5IQnLV5UrgvP
Pu8An2cVWtMLVTdrwxFvCuBSk0cpm4qN
=YKFo
-----END PGP SIGNATURE-----
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/