nanog mailing list archives
Re: ECN
From: Owen DeLong <owen () delong com>
Date: Wed, 13 Nov 2019 12:04:24 -0800
Like it or not (and I really don’t), the majority of modern CDNs are using TCP over Anycast. It’s ugly and it’s prone to problems like this. It’s nice to see a customer with know-how actually publicizing and digging into the problem. Until now, I believe an unknown number of customers have been suffering in silence or relegated to the ISPs “We can’t reproduce you problem” bin without resolution. I’ve had lots of discussions on the subject and the usual end result is “It’s too hard to measure or quantify and there’s no visible contingent of impacted users”. Now we at least have one visible impacted user. Owen
On Nov 13, 2019, at 09:19 , Anoop Ghanwani <anoop () alumni duke edu> wrote: Not to condone what cloudflare is doing, but... An ECN connection will have different bits on various packets for the duration of the connection -- pure ACKs (ACKs not piggybacking on data) will have the ECN bits as 00b, while all other packets will have either 01b, 10b (when no congestion was experienced) or 11b (when congestion was experienced). So using the ECN bits as part of the hash would affect performance throughout the life of the connection. On Wed, Nov 13, 2019 at 9:00 AM Matt Corallo <nanog () as397444 net <mailto:nanog () as397444 net>> wrote: Not ideal, sure, but if it’s only for the SYN (as you seem to indicate), splitting the flow shouldn’t have material performance degradation?On Nov 13, 2019, at 11:51, Toke Høiland-Jørgensen <toke () toke dk <mailto:toke () toke dk>> wrote: On 13 November 2019 17:20:18 CET, Matt Corallo <netadmin () as397444 net <mailto:netadmin () as397444 net>> wrote: This sounds like a bug on Cloudflare’s end (cause trying to do anycast TCP is... out of spec to say the least), not a bug in ECN/ECMP.Even without anycast, an ECMP shouldn't hash on the ECN bits. Doing so will split the flow over multiple paths; avoiding that is the whole point of doing the flow-based hashing in the first place. Anycast "only" turns a potential degradation of TCP performance into a hard failure... :) -Toke
Current thread:
- Re: ECN, (continued)
- Re: ECN Warren Kumari (Nov 13)
- TCP and anycast (was Re: ECN) Anoop Ghanwani (Nov 13)
- Re: TCP and anycast (was Re: ECN) Bill Woodcock (Nov 14)
- Re: TCP and anycast (was Re: ECN) William Herrin (Nov 14)
- Re: TCP and anycast (was Re: ECN) Randy Bush (Nov 14)
- Re: TCP and anycast (was Re: ECN) Christopher Morrow (Nov 14)
- Re: TCP and anycast (was Re: ECN) Randy Bush (Nov 14)
- Message not available
- Re: ECN Toke Høiland-Jørgensen via NANOG (Nov 13)
- Re: ECN Matt Corallo (Nov 13)
- Re: ECN Anoop Ghanwani (Nov 13)
- Re: ECN Owen DeLong (Nov 13)
- Re: ECN Toke Høiland-Jørgensen via NANOG (Nov 14)
- Re: ECN Jon Lewis (Nov 13)