Routing issues between 19:00 and 20:05 BST – 27 Jul 2012
We are currently investigating the causes of a failure in THN which affected a number of customers who are routed through this node.
Initial investigations point to a Cisco switch which experienced a failure and we believe caused a cascade of CEF table pollution (Cisco Express Forwarding) which impacted routing between core routers and the bridge between London 1 and London 3 nodes.
Engineers were on site quickly (within 10 minutes) to investigate, however owing to the complex nature of the problem it took some time to investigate and resolve. Following a reboot of the switches which form a segment of the core network, normal service was resumed by 20:05 (and much sooner for some customers).
The outage was limited to the THN (London 3) center and nodes / customers routed through this node.
Please accept our apologies for any inconvenience this outage may have caused customers. We are actively working to minimise the impact of any such failure in the future.