Re: CDN Overload?

[Mike Hammett <nanog () ics-il net>]

https://docs.google.com/spreadsheets/d/1Jdm0dOBf81kSnXEvVfI6ZJbWFNt5AbYUV8CDxGwLSm8/edit?usp=sharing 

I have made the anonymized answers public. This will obviously have some bias to it given that I mostly know fixed 
wireless operators, but I'm hoping this gets some good distribution to catch more platforms. 




----- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

----- Original Message -----

From: "Mike Hammett" <nanog () ics-il net> 
To: "NANOG" <nanog () nanog org> 
Sent: Wednesday, September 21, 2016 9:08:55 AM 
Subject: Re: CDN Overload? 

https://goo.gl/forms/LvgFRsMdNdI8E9HF3 

I have made this into a Google Form to make it easier to track compared to randomly formatted responses on multiple 
mailing lists, Facebook Groups, etc. 




----- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

----- Original Message ----- 

From: "Mike Hammett" <nanog () ics-il net> 
To: "NANOG" <nanog () nanog org> 
Sent: Monday, September 19, 2016 12:34:48 PM 
Subject: CDN Overload? 


I participate on a few other mailing lists focused on eyeball networks. For a couple years I've been hearing complaints 
from this CDN or that CDN was behaving badly. It's been severely ramping up the past few months. There have been some 
wild allegations, but I would like to develop a bit more standardized evidence collection. Initially LimeLight was the 
only culprit, but recently it has been Microsoft as well. I'm not sure if there have been any others. 

The principal complaint is that upstream of whatever is doing the rate limiting for a given customer there is 
significantly more capacity being utilized than the customer has purchased. This could happen briefly as TCP adjusts to 
the capacity limitation, but in some situations this has persisted for days at a time. I'll list out a few situations 
as best as I can recall them. Some of these may even be merges of a couple situations. The point is to show the general 
issue and develop a better process for collecting what exactly is happening at the time and how to address it. 

One situation had approximately 45 megabit/s of capacity being used up by a customer that had a 1.5 megabit/s plan. All 
other traffic normally held itself within the 1.5 megabit/s, but this particular CDN sent excessively more for extended 
periods of time. 

An often occurrence has someone with a single digit megabit/s limitation consuming 2x - 3x more than their plan on the 
other side of the rate limiter. 

Last month on my own network I saw someone with 2x - 3x being consumed upstream and they had *190* connections 
downloading said data from Microsoft. 

The past week or two I've been hearing of people only having a single connection downloading at more than their plan 
rate. 


These situations effectively shut out all other Internet traffic to that customer or even portion of the network for 
low capacity NLOS areas. It's a DoS caused by downloads. What happened to the days of MS BITS and you didn't even 
notice the download happening? A lot of these guys think that the CDNs are just a pile of dicks looking to ruin 
everyone's day and I'm certain that there are at least a couple people at each CDN that aren't that way. ;-) 




Lots of rambling, sure. What do I need to have these guys collect as evidence of a problem and who should they send it 
to? 




----- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP