Services Affected
Internet
Wide Orbit
SBS
BOC Services
Internal Phone Services
Sites Affected
Colorado Springs
Time Affected
10:45AM - 12:31PM ( 106 Minutes )
Resolution
Cause of the outage was traced back to a routing failure in our Comcast ISP. This routing failure caused traffic to be directed over to our redundant ISP, Peak. A routing misconfiguration in their network caused the traffic to be routed back to Comcast. This in turn, created a loop whereby traffic was passed between Comcast and Peak without ever being sent to KRDO. Peak identified their misconfiguration and corrected it. This allowed internet services to be restored, while Comcast continued to be degraded. Peak and Comcast have verified their configurations and this type of routing failure is not possible any more. Below are two diagrams, showing normal routing conditions and the failure mode experienced during the outage.
Normal Routing Conditions
Failed Routing Conditions
Comments
Article is closed for comments.