Re: Google's peering, GGC, and congestion management

["Patrick W. Gilmore" <patrick () ianai net>]

On Oct 14, 2015, at 1:07 PM, Baptiste Jonglez <baptiste () bitsofnetworks org> wrote:

> In its peering documentation [https://peering.google.com/about/traffic_management.html],
> Google claims that it can drive peering links at 100% utilisation:
>
> > Congestion management
> >
> > Peering ports with Google can be run at 100% capacity in the short term,
> > with low (<1-2%) packet loss. Please note that an ICMP ping may display
> > packet loss due to ICMP rate limiting on our platforms. Please contact
> > us to arrange a peering upgrade.
>
> How do they achieve this?

The 100% number is silly. My guess? They’re at 98%.

That is easily do-able because all the traffic is coming from them. Coordinate the HTTPd on each of the servers to 
serve traffic at X bytes per second, ensure you have enough buffer in the switches for micro-bursts, check the NICs for 
silliness such as jitter, and so on. It is non-trivial, but definitely solvable.

Google is not the only company who can do this. Akamai has done it far longer. And Akamai has a much more difficult 
traffic mix, with -paying customers- to deal with.


> More generally, is there any published work on how Google serves content
> from its CDN, the Google Global Cache?  I'm especially interested in two
> aspects:
>
> - for a given eyeball network, on which basis are the CDN nodes selected?

As for picking which GGC for each eyeball, that is called “mapping”. It varies among the different CDNs. Netflix drives 
it mostly from the client. That has some -major- advantages over other CDNs. Google has in the past (haven’t checked in 
over a year) done it by giving each user a different URL, although I think they use DNS now. Akamai uses mostly DNS, 
although they have at least experimented with other ways. Etc., etc.


> - is Google able to spread traffic over distinct peering links for the
>  same eyeball network, in case some of the peering links become
>  congested?  If so, how do they measure congestion?

Yes. Easily.

User 1 asks for Stream 1, Google sends them them to Node 1. Google notices Link 1 is near full. User 2 asks for Stream 
2, Google sends them to Node 2, which uses Link 2.

This is possible for any set of Users, Streams, Nodes, and Links.

It is even possible to send User 2 to Node 2 when User 2 wants Stream 1. Or sending User 1 to Node 2 for their second 
request despite the fact they just got a stream from Node 1. There are few, if any, restrictions on the combinations.

Remember, they control the servers. All CDNs (that matter) can do this. They can re-direct users with different URLs, 
different DNS responses, 302s, etc., etc. It is not BGP.

Everything is much easier when you are one of the end points. (Or both, like with Netflix.) When you are just an ISP 
shuffling packets you neither send nor receive, things are both simpler and harder.

--
TTFN,
patrick

Attachment: signature.asc
Description: Message signed with OpenPGP using GPGMail