At approximately 01:45 this morning we experienced a major failure in our Telehouse West location.
Engineers were dispatched to site and worked to bring power back to the impacted servers, replacing failed components as they went.
All physical hardware was restored by approximately 09:30 am however we were then left with huge database and disk level corruption which engineers proceeded to work to repair over the following 8+ hours.
During this time, a number of sites and services which are fed from Telehouse West were impacted, including some Broadband, Web and Email hosting services including Fido Glide and Redmail.Com services, all fidonet.com and fido.net websites, a number of virtualisation platforms, dedicated customers and more.
Engineers are still on site and are planning additional emergency maintenance to bypass the equipment which we believe caused the issue. This will result in a further planned outage this evening between 22:00 and 06:00 as equipment is powered down, relocated if necessary and then brought back into service in a managed fashion.
We can only apologise for this extended disruption which was caused by a number of issues outside of our direct control