Re: [NSE] Two new scripts: http-google-email and http-reverse-ip

[Martin Holst Swende <martin () swende se>]

On 07/28/2011 01:08 PM, Patrik Karlsson wrote:
> On Jul 27, 2011, at 11:01 PM, Shinnok wrote:
>
> > Hi nmap-dev,
> >
> > As stated in some of my previous e-mails, I started tinkering with NSE
> > and LUA a bit, just so I don't miss this part of Nmap fun and I want to
> > share with you two scripts I've written:
> >
> > 1. Google Web Search and Google Groups Search e-mail discovery
> >
> > This one attempts to search for e-mails pertaining to a specific domain
> > in Google's Web search engine(google.com) and Google Groups search
> > engine(groups.google.com).
> >
> > Sample output:
> > ./nmap -p80 --script http-google-email insecure.org
> >
> > Starting Nmap 5.59BETA1 ( http://nmap.org ) at 2011-07-27 23:53 EEST
> > Nmap scan report for insecure.org (74.207.254.18)
> > Host is up (0.19s latency).
> > rDNS record for 74.207.254.18: web.insecure.org
> > PORT   STATE SERVICE
> > 80/tcp open  http
> > | http-google-email:
> > | nmap-dev () insecure org
> > | nmap-svn () insecure org
> > |_fyodor () insecure org
> >
> > Nmap done: 1 IP address (1 host up) scanned in 4.01 seconds
> >
> >
> >
> > 2. Reverse domain ip lookup using Bing
> >
> > This one attempts to find domains that are hosted on a specific ip
> > address using Bing's ip: operator.
> >
> > Sample output:
> > ./nmap -p80 --script http-reverse-ip  insecure.org
> >
> > Starting Nmap 5.59BETA1 ( http://nmap.org ) at 2011-07-27 23:55 EEST
> > Nmap scan report for insecure.org (74.207.254.18)
> > Host is up (0.20s latency).
> > rDNS record for 74.207.254.18: web.insecure.org
> > PORT   STATE SERVICE
> > 80/tcp open  http
> > | http-reverse-ip:
> > | nmap.org
> > | insecure.org
> > | secwiki.org
> > |_images.insecure.org
> >
> > Nmap done: 1 IP address (1 host up) scanned in 1.75 seconds
> >
> > The both take a "pages" argument that specifies the number of results
> > pages to request from search engines(the default is 5) and a domain and
> > host argument respectively, that can be used to query a different target
> > then the one nmap scans.
> >
> > I used http-google-malware.nse as a reference and the results are far
> > from perfect and since they are my first attempts at NSE I am welcome to
> > suggestions, improvements and conformance hints.
> >
> > Thanks,
> > Shinnok
> > <http-google-email.nse><http-reverse-ip.nse>_______________________________________________
> > Sent through the nmap-dev mailing list
> > http://cgi.insecure.org/mailman/listinfo/nmap-dev
> > Archived at http://seclists.org/nmap-dev/
>
> Hi Shinook,
>
> First off, welcome to the great world of NSE :)
> I've had some experience in the past with "wrapping" google search with tools such as wget.
> What usually happens after a bunch of queries is that Google block the search possibility from that IP or redirect 
> you to a captcha site or a site making heavy use of javascript to prevent further "automated searches".
> In the past you were able to get a google API key to get around this problem and use a dedicated 3rd party API that 
> wouldn't impose these restrictions.
> I'm not sure how this works now. I also think Bing had some similar restrictions requiring an API key.
>
> I know that Martin has done some work in this area and may have some newer information.
>
> //Patrik

Hi!

I've done some work using the API:s available for Bing, Yahoo and
Google. I saw that the scripts by Shinnok does not use the API:s, but
the regular search functionality. That may certainly work, but there may
also be issues. I know that google does its best to stop automated
searches using the normal querying facilitites, it wouldn't surprise me
if Yahoo/Bing did the same.

There are also other issues, from what I understand owasp efforts in
creating tools for automated google searches were aborted due to the
fact that performing automated searches may be some kind of violations
against the TOS. But I don't know any deatails about it, so don't take
my word for it.

If one wants to do it the 'proper' way, all three have great json api's
that they want applications to use. All three used to require API keys,
but google later abandoned this approach and opened it up. But not
completely - I don't have any links to it, but there is an open issue
with it that it does not deliver the same search results as the real
thing, and that it the search results are capped (you just won't get
everything!).

I implemented three apps in python
GHDB checker: http://www.swende.se/index.php/2010/04/googlyhacks-ghdb/
Subdomain-finder: http://www.swende.se/index.php/2010/06/finding-subdomains/
App finder (well, really the same as reverse-ip) :
http://www.swende.se/index.php/2010/07/finding-applications/

These three would be very simple to move over to Lua, the problematic
thing is that the latter two requires api keys (which are simple for an
individual to obtain though). However, if the feature that reads options
from a file could be used for this, so a user does not have to specify
his key every time he does this, it would be pretty sweet to have.

Regards,
Martin
ps. Shinnok, thanks for the post about Qt! Definitely going to check it out!
_______________________________________________
Sent through the nmap-dev mailing list
http://cgi.insecure.org/mailman/listinfo/nmap-dev
Archived at http://seclists.org/nmap-dev/