RE: What are these Google IPs hammering on my DNS server?

[Michael Hare via NANOG <nanog () nanog org>]

Damian-

Not Google or ISCs fault, our customers have made some decisions that have exasperated the issues.  By and away the 
biggest problem facing my customers is that they have chosen a stateful border firewall that collapses due to session 
exhaustion and they put everything, including aDNS, behind said firewall.  “If it hurts, don’t do it” comes to mind, 
but out of my hands.

At quick glance following the ISC link I didn’t see the compute infrastructure [core count] needed to get 1Mpps.  There 
is an obvious difference between 99% load of ~500rps and 1M, so we can maybe advise to not undersize ADNS if that's an 
issue.

I'm an ISP engineer and am generally not the directly affected party, so I don't get to pick these implementation 
details for my customers.  I appreciate the background and suggestions from you and others on this thread like Mark.  
That's an interesting comment about DNSSEC that I hadn't considered.

-Michael

From: Damian Menscher <damian () google com>
Sent: Monday, December 4, 2023 12:21 PM
To: Michael Hare <michael.hare () wisc edu>
Cc: John R. Levine <johnl () iecc com>; nanog () nanog org
Subject: Re: What are these Google IPs hammering on my DNS server?

Google Public DNS (8.8.8.8) attempts to identify and filter abuse, and while we think we're fairly effective for large 
attacks (eg, those above 1Mpps), it gets more challenging (due to risk of false positives) to adequately filter small 
attacks.  I should note that we generally see the attack traffic coming from botnets, or forwarding resolvers that 
blend the attack traffic with legitimate traffic.

Based on ISC BIND load-tests [0], a single DNS server can handle O(1Mpps).  Also, no domain should be served by a 
single DNS server, so O(1Mpps) seems like a safe lower-bound for small administrative domains (larger ones will have 
more redundancy/capacity).  Based on these estimates, we haven't treated mitigation of small attacks as a high 
priority.  If O(25Kpps) attacks are causing real problems for the community, I'd appreciate that feedback and some 
hints as to why your experience differs from the ISC BIND load-tests.  With a better understanding of the pain-points, 
we may be able to improve our filtering a bit, though I suspect we're nearing the limits of what is attainable.

Since it was mentioned up-thread, I'd caution against dropping queries from likely-legitimate recursives, as that will 
lead to a retry storm that you won't like (based on a few reports of authoritatives who suffered outages, the retry 
storm increased demand by 30x and they initially misdiagnosed the root cause as a DDoS).  The technically correct (if 
not entirely practical) mitigation for a DNS cache-busting attack laundered through open recursives is to deploy DNSSEC 
and issue NSEC/NSEC3 responses to allow the recursives to cache the non-existence of the randomized labels.

[0] https://www.isc.org/blogs/bind-performance-september-2023/

Damian
--
Damian Menscher :: Security Reliability Engineer :: Google :: AS15169

On Sun, Dec 3, 2023 at 1:22 PM Michael Hare via NANOG <nanog () nanog org<mailto:nanog () nanog org>> wrote:
John-

This is little consolation, but at AS3128, I see the same thing to our downstream at times, claiming to come from both 
13335 and 15169 often simultaneously at the tune of 25Kpps , "assuming it's not spoofed", which is pragmatically 
impossible to prove for me given our indirect relationships with these companies.  When I see these events, I typically 
also see a wide variety of country codes participating simultaneously.  Again, assuming it's not spoofed.  To me it 
just looks like effective harassment with 13335/15169 helping out.  I pine for the internet of the 1990s.

Recent events in GMT for us were the following, curious if you see the same
~ Nov 26 05:40
~ Nov 30 00:40
~ Nov 30 05:55

Application agnostic, on the low $ end for "fixes", if it's either do something or face an outage, I've found some 
utility in short term automated DSCP coloring on ingress paired with light touch policing as close to the end host as 
possible, which at least keeps things mostly working during times of conformance.  Cheap/fast and working ... most of 
the time.  Definitely not great or complete at all, and a role I'd rather not play as an educational ISP/enterprise.

So what are most folks doing to survive crap like this?  Nothing/waiting it out?  Oursourcing DNS?  Scrubbing 
appliance?  Poormans stuff like I mention above?

-Michael

> -----Original Message-----
> From: NANOG <nanog-bounces+michael.hare=wisc.edu () nanog org<mailto:wisc.edu () nanog org>> On
> Behalf Of John R. Levine
> Sent: Sunday, December 3, 2023 1:18 PM
> To: Peter Potvin <peter.potvin () accuristechnologies ca<mailto:peter.potvin () accuristechnologies ca>>
> Cc: nanog () nanog org<mailto:nanog () nanog org>
> Subject: Re: What are these Google IPs hammering on my DNS server?
>
> > Did a bit of digging on Google's developer site and came across this:
> > https://developers.google.com/speed/public-
> dns/faq#locations_of_ip_address_ranges_google_public_dns_uses_to_send_
> queries
> > Looks like the IPs you mentioned belong to Google's public DNS resolver
> > based on that list on their site. They could also be spoofed though from a
> > DNS AMP attack, so keep that in mind.
>
> Per my recent message, the replies are tiny so if it's an amplification
> attack, it's a very incompetent one.  The queries are case randomized so I
> guess it's really Google.  Sigh.
>
> If anyone is wondering, I have a passive aggressive countermeasure against
> some overqueriers that returns ten NS referral names, and then 25 random
> IP addresses for each of those names, but I don't do that to Google.
>
> R's,
> John
>
> > ------------------------------------------------------------------------------
> > *Accuris Technologies Ltd.*
> >
> >
> > On Sun, Dec 3, 2023 at 1:51 PM John Levine <johnl () iecc com<mailto:johnl () iecc com>> wrote:
> >
> > > At contacts.abuse.net<http://contacts.abuse.net>, I have a little stunt DNS server that provides
> > > domain contact info, e.g.:
> > >
> > > $ host -t txt comcast.net.contacts.abuse.net<http://comcast.net.contacts.abuse.net>
> > > comcast.net.contacts.abuse.net<http://comcast.net.contacts.abuse.net> descriptive text "abuse () comcast 
> > > net<mailto:abuse () comcast net>"
> > >
> > > $ host -t hinfo comcast.net.contacts.abuse.net<http://comcast.net.contacts.abuse.net>
> > > comcast.net.contacts.abuse.net<http://comcast.net.contacts.abuse.net> host information "lookup" 
> > > "comcast.net<http://comcast.net>"
> > >
> > > Every once in a while someone decides to look up every domain in the
> > > world and DoS'es it until I update my packet filters. This week it's
> > > been this set of IPs that belong to Google. I don't think they're
> > > 8.8.8.8. Any idea what they are? Random Google Cloud customers? A
> > > secret DNS mapping project?
> > >
> > >  172.253.1.133
> > >  172.253.206.36
> > >  172.253.1.130
> > >  172.253.206.37
> > >  172.253.13.196
> > >  172.253.255.36
> > >  172.253.13.197
> > >  172.253.1.131
> > >  172.253.255.35
> > >  172.253.255.37
> > >  172.253.1.132
> > >  172.253.13.193
> > >  172.253.1.129
> > >  172.253.255.33
> > >  172.253.206.35
> > >  172.253.255.34
> > >  172.253.206.33
> > >  172.253.206.34
> > >  172.253.13.194
> > >  172.253.13.195
> > >  172.71.125.63
> > >  172.71.117.60
> > >  172.71.133.51
> > >
> > > R's,
> > > John
>
> Regards,
> John Levine, johnl () taugh com<mailto:johnl () taugh com>, Primary Perpetrator of "The Internet for
> Dummies",
> Please consider the environment before reading this e-mail. https://jl.ly