Re: google search threshold

["Paul S." <contact () winterei se>]

DO's SG range is allocated out of a single /64 (I think?) and Google 
basically asks for captcha on every single request over IPv6. :(

We're using it as a corporate vpn.

On 3/1/2016 01:49 AM, Keenan Tims wrote:
> FWIW I have seen the captchas more often on IPv6 both from home and the office than when both networks were using a 
> single shared IPv4; not sure if this is just related to chronology or a real effect. Once a month or so I seem to get 
> them for a couple of days, then they go away.
>
> No idea what's triggering it. It would be *really* helpful if Google could provide some useful technical details beyond a generic FAQ page. As 
> it is I just get annoyed by it and have no way to troubleshoot or correct the constant false positives. How is Google detecting "robots"? 
> My sense is that I tend to trigger the captcha thing when iterating similar search terms (particularly due to removal of the + operator and extremely 
> poor "change my search terms because you think you know better than I do what I want to search for" behaviour. My search patterns 
> haven't really changed since turning up IPv6 everywhere, so I have to think either the captcha trigger has gotten more aggressive, or somehow 
> prefers to blacklist IPv6 users.
>
> In any case, just going to IPv6 is definitely not a complete fix for this. It seems to be related to search behaviour 
> and $blackbox_magic.
>
> Keenan Tims
> Stargate Connections
> ________________________________________
> From: NANOG <nanog-bounces () nanog org> on behalf of Philip Lavine via NANOG <nanog () nanog org>
> Sent: February 29, 2016 7:53 AM
> To: Damian Menscher
> Cc: nanog () nanog org
> Subject: Re: google search threshold
>
> I have about 2000 users behind a single NAT. I have been looking at netflow, URL filter logs, IDS logs, etc. The 
> traffic seems to be legit.
>
> I am going to move more users to IPv6 and divide some of the subnets into different NATS and see if that alleviates the 
> traffic load.
> Thanks for the advice.
> -Philip
>
>
>        From: Damian Menscher <damian () google com>
>   To: Philip Lavine <source_route () yahoo com>
> Cc: "nanog () nanog org" <nanog () nanog org>
>   Sent: Friday, February 26, 2016 6:05 PM
>   Subject: Re: google search threshold
>
> On Fri, Feb 26, 2016 at 3:01 PM, Philip Lavine via NANOG <nanog () nanog org> wrote:
>
> Does anybody know what the threshold for google searches is before you get the captcha?I  am trying to decide if I need 
> to break up the overload NAT to a pool.
>
>
> There isn't a threshold -- if you send automated searches from an IP, then it gets blocked (for a while).
>
> So... this comes down to how much you trust your machines/users.  If you're a company with managed systems, then you can have 
> thousands of users share the same IP without problems.  But if you're an ISP, you'll likely run into problems much earlier 
> (since users like their malware).
> Some tips:   - if you do NAT: try to partition users into pools so one abusive user can't get all your external IPs blocked  
> - if you have a proxy: make sure it inserts the X-Forwarded-For header, and is restricted to your own users  - if you're an 
> ISP: IPv6 will allow each user to have their own /64, which avoids shared-fate from abusive ones
> Damian (responsible for DDoS defense)-- Damian Menscher :: Security Reliability Engineer :: Google :: AS15169