Re: Amazon network engineering contact? re: DDoS traffic

[John <jw () nuclearfallout net>]

Zach,

As mentioned before, I am open to peering (where possible) but have not 
received a response.

My goal is to connect with someone at Amazon and work with them on a 
technical solution, which is why I posted asking for that here. Various 
factors mean that we can't just upgrade our way out of this one, and 
manual whac-a-mole on the sources has shown to have limited use.

These attacks also impact Amazon and the networks in between the sources 
and targets, and they take time to handle by the abuse teams, so there 
are good reasons to investigate them further and find better ways to 
mitigate or prevent them.

-John

On 11/8/2018 1:12 PM, Zach Puls wrote:
> Makes sense, that's understandable. Do you peer with AWS? If not, maybe opening up a peering agreement will give you a 
> better contact, and a bit more pull when attacks occur? I know someone with a peering agreement with AWS, and they have been 
> able to get resolutions fairly quickly when issues arise.
>
> Other than that, I'm not sure of a solution other than more IP transit.
>
> Thanks,
>
> Zach Puls
> Network Engineer | MEF-CECP
> KsFiberNet
>
> -----Original Message-----
> From: John Weekes <jw () nuclearfallout net>
> Sent: Thursday, November 08, 2018 15:03
> To: Zach Puls <zpuls () ksfiber net>
> Cc: nanog () nanog org
> Subject: Re: Amazon network engineering contact? re: DDoS traffic
>
> Zach,
>
> Yes, RTBH is used to distribute the null-routes that I mentioned.
>
> Unfortunately, even brief saturation events lasting just 5-10 seconds (a typical amount of time to detect the loss, issue 
> the null-route, and see the traffic start to fall off as it is distributed upstream) can cause real damage to those 
> customers who are sensitive to latency and packet loss. So while null-routes limit the duration of the impact, they 
> can't eliminate it entirely. And, of course, the actual target of the attack
> -- the now-null-routed IP address -- becomes unreachable, which was presumably the goal of the attacker.
>
> -John
>
> On 11/8/2018 12:54 PM, Zach Puls wrote:
> > No idea about an Amazon abuse contact, but do you have RTBH communities enabled with your upstream provider(s)? As a 
> > general practice, when you detect a (D)DoS attack in progress, it would help to automatically advertise that prefix to 
> > your upstream(s) with the black-hole community. This would at least help mitigate the effects of the attacks when they 
> > do occur, even if they come from a different source than AWS.
> >
> > Thanks,
> >
> > Zach Puls
> > Network Engineer | MEF-CECP
> > KsFiberNet
> >
> > -----Original Message-----
> > From: NANOG <nanog-bounces () nanog org> On Behalf Of John Weekes
> > Sent: Thursday, November 08, 2018 14:44
> > To: nanog () nanog org
> > Subject: Amazon network engineering contact? re: DDoS traffic
> >
> > We've been seeing significant attack activity from Amazon over the last two months, involving apparently compromised 
> > instances that commonly send 1-10G of traffic per source and together generate Nx10G of total traffic. Even when our overall 
> > upstream capacity exceeds an attack's overall size, the nature of load-balancing over multiple 10G upstream links means that 
> > an individual link can be saturated by multiple large flows, forcing our systems to null-route the target to limit impact.
> >
> > We've sent an abuse notification about every traffic source to Amazon, and specific sources seem to stop their 
> > involvement over time (suggesting that abuse teams are following up on them), but there is an endless parade of new 
> > attackers, and each source participates in many damaging attacks before it is shut down.
> >
> > Is there anyone at Amazon who can help with an engineering solution in terms of programmatically detecting and 
> > rate-limiting attack traffic sources, to our networks or overall? Or applying the kludge of a rate-limit for all Amazon 
> > traffic to our networks? Or working with us on some other option?
> >
> > At least one other large cloud provider has an automatic rate-limiting system in place that is effective in reducing 
> > the damage from repeat high-volume attacks.
> >
> > Emails to the Amazon NOC, peering contacts (since that would be another possible solution), and abuse department have 
> > not connected me with anyone.
> >
> > Thanks,
> > John