We are experiencing what appears to be a repeat failure on the replacement line card which was installed earlier this week in Telehouse
Engineers are en route to investigate and attempt a work around. Current estimates for an ETA are 2am
Attempts have already been made to route around the affected area of the network however not all segments have re-routed resulting in customers hanging off thn-edge-1 and thn-edge-3 to also be affected to some degree
We will update this post with more detail once we have investigated further
UPDATE
Engineers arrived on site at approx 23:57
After initial investigations they identified what appeared to be an issue with the fibre and spent some time diagnosing this issue. After proving both the fibre and the fibre modules, they started to dig deeper and applied firmware updates. None of these efforts lead to an actual resolution, and after further investigation it was decided the issue was either the power supply or the chassis itself.
We were able to source a replacement and this was installed as quickly as possible and the configs carried across to the replacement hardware.
Following on from this, we monitored the connection and after 2 hours with no further reboots or port resets we deemed the fault to be cleared.
We now have a TAC open with the router manufacturer and they are working on identifying the cause of the failure on a relatively new (4 months old) hardware unit.