RE: Level3 Internet service, out of order packets causing issues

[Mark Wicker <MWicker () esri com>]

Hi Jason,

I think that this was it. The first octet of our router’s Level3 facing interface MAC is a 6. Changed it to 0 and it 
does look much better now. Thanks!


--
Mark Wicker | Senior Network Engineer
Esri | 380 New York St | Redlands, CA  92354 | USA
T 909 793 2853 x2741 | mwicker () esri com<mailto:mwicker () esri com> | esri.com<http://www.esri.com/>

From: Jason Rokeach [mailto:jason () rokeach net]
Sent: Tuesday, January 17, 2017 6:57 PM
To: Mark Wicker <MWicker () esri com>
Cc: nanog () nanog org
Subject: Re: Level3 Internet service, out of order packets causing issues

Hi Mark,
I'm going to throw out a guess here.  By any chance, is the first octet of your router's MAC address a 4 or a 6?
In general, modern routers do not load balance per-packet, which is what caused out-of-order issues in days gone.  Load 
balancing is usually done based on a hash of the source and/or destination IP of the packet, MPLS label, or Ethernet 
(on a switched interface).  The most common cause for actual unordering of packets/frames in a modern service provider 
network, in my experience, is actually this hashing mechanism.  Many vendor's hashing implementations assume, based on 
position in the frame, that a frame with a MAC address beginning with 4 or 6 is an IPv4 or IPv6 frame, not an MPLS 
frame.  This can result in out of order packets.  The most common fix is control word being applied on a pseudowire 
(assuming you are being carried across a pseudowire in the SP network), but if this is what is occurring, you could 
also resolve the issue by changing your MAC address.


On Tue, Jan 17, 2017 at 2:12 PM, Mark Wicker <MWicker () esri com<mailto:MWicker () esri com>> wrote:
Hi,

I have 1G Level3 ethernet dedicated internet service as one of my ISP's at my company based in the Los Angeles (Inland 
Empire) area. After seeing strange application behavior while using this circuit, I failed it out of service and have 
been troubleshooting it with a directly connected machine (publically addressed, no firewall, nothing between this 
machine and our Level3 router). I have taken several packet captures while accessing various sites and have noticed 
large numbers of out of order packets which are wreaking havoc with TCP connections and other traffic. In my 
experience, per-packet load balancing across various different links can cause this issue. I do not see this behavior 
with my other ISP's. I have had several tickets opened with Level3 but have had no success. Any help here? Anyone out 
there seen this and have any contacts that may be able to help?


FYI - we own our own public IP space and advertise via BGP to Level3. Currently I am using a dedicated /24 of our space 
advertised to Level3 only to ensure that the return path is through Level3 and not another ISP. Also, everything is 
single linked from a layer 2 and 3 perspective from the router to the test machine to ensure that the cause of any out 
of order packets is not on our end.


Thanks,


--
Mark Wicker | Senior Network Engineer
Esri | 380 New York St | Redlands, CA  92354 | USA
T 909 793 2853 x2741 | mwicker () esri com<mailto:mwicker () esri com><mailto:mwicker () esri com<mailto:mwicker () 
esri com>> | esri.com<http://esri.com><http://www.esri.com/>