At approximately 11:37 this morning we received reports that new connections to our DSL network were failing. Upon investigation it transpired that the Network Server in London 1 (Hex) was experiencing problems with its line card.
Under normal circumstances we would disable the faulty server and allow the backup to take over, however the backup was offline and being rebuilt/upgraded. Attempts to fix the faulty LNS server failed and further attempts to restore the server were also unsuccessful.
At this stage we took the view that we would be better focusing on completing the upgrade of the backup server. This work was originally due to be completed Wednesday evening when the backup LNS server was going to be promoted to live status so we could then upgrade the old “live” server.
Efforts were redirected and the replacement server was completed and installed shortly after 13:45 today, at which stage connections were re-established and service was restored to normal.
We would like to apologise for any inconvenience caused by today’s failures. We are replacing the faulty server and hope to have full redundancy restored before the end of the week.