At approximately 18:15 BST whilst engineers were working in one of our racks in IP House a power cable feeding one of our core routers became dislodged from the distribution board.
We have previously experienced problems with this particular distribution board and power cables working their way lose almost unaided, and have been planning an orderly swap to a new distribution panel which is considerably better designed.
As a result of this unplanned outage we relocated the power connection to the new distribution panel which is already live. This work was originally planned for a maintenance window in October.
Whilst we have redundant routers and so this failure should in its own right not have been a major problem; when routes failed over to the backup router in IP House it appears that this triggered a bug in the BGP routing daemon on the backup router which then also failed causing some routes to black hole. This was quickly detected and this backup router was also power cycled (and the power cable also move to the new distribution board) causing the backup router in Harbour Exchange to take over whilst we waited for the IP House routers to recover.
Total outage time was approximately 15 minutes, however as a result of the secondary router failing in IP House this had a knock on effect to our Broadband network which would have taken slightly longer to recover owing to the nature of our connections into the BT Wholesale network.
We would like to apologise for any inconvenience this outage has caused, and can assure you by moving the power connections for both routers we have now hopefully mitigated problems with the old faulty power connectors, and are also working to ascertain why the OSPF fail over caused our BGP daemon to fail on the backup router in IP House.