At approximately 15:30 today a 3rd party engineer installing additional fibre to one of our core racks in Telehouse North accidentally caused the power to one of our core routers to be lost.
This problem was identified quickly and power was restored, however the power spike caused by the failure seems to have resulted in a hardware failure of the router.
Specialist engineers were dispatched to site and arrived at approx 17:00 hours, and a temporary fix was initiated until replacement parts can be installed.
We hope to be able to install these additional components at 19:30 this evening bringing full resilience back to the network.
In the mean time we are at risk should another router fail.
During the failure and router restarts customers may have noticed blips on the network as routes re-converge.
Normally when we plan maintenance to take a router out of service all sessions are cleanly shut down and disabled. Owing to the nature of the failure today this was not possible which will cause OSPF and BGP route convergence (or brief stutters)