At approximately 02:00 Thursday morning we experienced a failurein one of our racks located in TeleHouse North.
An APC power bar experienced a terminal failure which took out 4 machines that were connected to it. It appears that the power bar then recycled and reset, causing a surge/spike which caused one of the 8 amp feeds into the rack to trip.
Engineers were dispatched to site, and arrived on site at 03:10 where they identified the problem and worked to restore power to key services within the rack (infrastructure first).
Upon further investigation they discovered the failed RPB (remote power bar) and worked to isolate it, whilst at the same time sourcing spares from our central stores. These spares were due on site at 6am however owing to traffic problems and road closures did not make it to site until 07:30 am. In the mean time, temporary power was provided to the equipment in order to minimise any physical outage / disruption.
We performed an emergency power down of the affected equipment at 08:00 and swapped in the new RPB, and checked load / distribution. Full service was restored by 08:10. Total outages were 1h30m (02:00 – 03:30) and 10m (08:00 – 08:10).
FidoNet would like to apologise to customers who were affected by this outage and the subsequent emergency maintenance required to revert the network to its usual state. Should you have any questions regarding this maintenance, please contact support@fido.net for further information.