Re: Level(3) ex-twtelecom midwest packet loss (4323)

[Jason Canady <jason () unlimitednet us>]

Mike, I would take it to mean someone screwed something up and they 
don't want to admit to it.  :-)  That's just a guess.

--

Jason Canady
Unlimited Net, LLC
Responsive, Reliable, Secure

www.unlimitednet.us
jason () unlimitednet us
twitter: @unlimitednet

On 8/28/15 12:00 AM, Mike Hammett wrote:
> 08/28/2015 3:08 AM GMT
> Event Conclusion Summary
>
> Start: August 27, 2015 13:20 GMT
> Stop: August 28, 2015 00:00 GMT
>
> Root Cause: A protocol issue impacted IP services in multiple markets.
> Fix Action: Adjustments were made to clear the errors.
>
> Summary:
> The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was 
> causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a 
> site in Chicago, IL to assist with investigations. Troubleshooting identified a protocol issue, and Operations 
> Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the 
> errors cleared. The traffic load was also lowered on cards in Chicago to alleviate any further issues. Should any 
> additional impact be experienced, please contact the Level 3 Technical Service Center.
>
> What the hell is a "protocol issue"?
>
> I'm not an idiot, you can tell me specifically what happened...
>
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> ----- Original Message -----
>
> From: "Ryan Gelobter" <ryan.g () atwgpc net>
> To: "Mel Beckman" <mel () beckman org>
> Cc: "<nanog () nanog org>" <nanog () nanog org>
> Sent: Thursday, August 27, 2015 3:14:59 PM
> Subject: Re: Level(3) ex-twtelecom midwest packet loss (4323)
>
> If you have access to the Level3 portal you should see ticket #9639047
> under Network Events now.
>
> Event Summary:IP Network Event ~ Multiple Markets
>
> 08/27/2015 8:05 PM GMT
>
> Level 3 Tier III and Operations Engineering teams have identified Internet
> Protocols dropping, affecting customer services. Restoration efforts are
> in progress, but an estimated time of restoral is not available at this
> time.
>
> 08/27/2015 6:36 PM GMT
>
> IP and Transport Tier III, Operations Engineering and Field Services
> continue collaboratively working the issue.
>
>
> 08/27/2015 4:59 PM GMT
>
> Operations Engineering is engaged and Field Services is on site in Chicago,
> IL investigating the issue.
>
>
> 08/27/2015 4:38 PM GMT
>
> The engineers are currently migrating traffic in efforts of restoring
> services while troubleshooting continues. Field Services is being
> dispatched to a Chicago, IL site to assist.
>
> 08/27/2015 4:21 PM GMT
>
> IP services are affected across multiple markets and the root cause is
> currently under investigation. The IP NOC and IP and Transport Tier III are
> actively troubleshooting and working to isolate the cause. The engineers
> have detected peering issues which are resulting in packet loss for
> customers. Please be advised that updates will be provided at minimum of
> hourly unless otherwise noted.