Re: Level(3) ex-twtelecom midwest packet loss (4323)

[Mike Hammett <nanog () ics-il net>]

08/28/2015 3:08 AM GMT 
Event Conclusion Summary 

Start: August 27, 2015 13:20 GMT 
Stop: August 28, 2015 00:00 GMT 

Root Cause: A protocol issue impacted IP services in multiple markets. 
Fix Action: Adjustments were made to clear the errors. 

Summary: 
The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was 
causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a 
site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations 
Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the 
errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any 
additional impact be experienced, please contact the Level 3 Technical Service Center. 

What the hell is a "protocol issue"? 

I'm not an idiot, you can tell me specifically what happened... 




----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

----- Original Message -----

From: "Ryan Gelobter" <ryan.g () atwgpc net> 
To: "Mel Beckman" <mel () beckman org> 
Cc: "<nanog () nanog org>" <nanog () nanog org> 
Sent: Thursday, August 27, 2015 3:14:59 PM 
Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323) 

If you have access to the Level3 portal you should see ticket #9639047 
under Network Events now. 

Event Summary:IP Network Event ~ Multiple Markets 

08/27/2015 8:05 PM GMT 

Level 3 Tier III and Operations Engineering teams have identified Internet 
Protocols dropping, affecting customer services. Restoration efforts are 
in progress, but an estimated time of restoral is not available at this 
time. 

08/27/2015 6:36 PM GMT 

IP and Transport Tier III, Operations Engineering and Field Services 
continue collaboratively working the issue. 


08/27/2015 4:59 PM GMT 

Operations Engineering is engaged and Field Services is on site in Chicago, 
IL investigating the issue. 


08/27/2015 4:38 PM GMT 

The engineers are currently migrating traffic in efforts of restoring 
services while troubleshooting continues. Field Services is being 
dispatched to a Chicago, IL site to assist. 

08/27/2015 4:21 PM GMT 

IP services are affected across multiple markets and the root cause is 
currently under investigation. The IP NOC and IP and Transport Tier III are 
actively troubleshooting and working to isolate the cause. The engineers 
have detected peering issues which are resulting in packet loss for 
customers. Please be advised that updates will be provided at minimum of 
hourly unless otherwise noted.