At approximately 16:24 today we experienced a failure on 2 of our 3 border routers which after investigation was attributed to a bogus route advertisement from a network in Romania
Engineers worked quickly to identify and filter this bogus network, as well as to identify how it managed to get past our existing filters and sanity checks.
The outage caused a number of Tier 1 network providers to experience problems, and caused a high volume of route flaps as networks fell off the internet and then attempted to reconnect, re-learn the bogus route and fail again.
Engineers around the globe co-ordinated efforts and a fix was applied roughly 30 minutes after the initial failure was first detected.
It took a further 10-15 minutes for all network flaps to settle and full routing to be restored.
We are working to find more automated / intelligent ways to spot and mitigate these potential issues.
We would like to apologise to customers who experienced an outage as a result of this issue.
Jon
1 comment for “Network Outage 16.02.09”