We are investigating the cause of a service failure with the new PCS6 Cloud 6 Front end.
A number of VMs have shut down, and we are investigating the cause of the failure currently
Update – the problem has been identified and resolved. This issue should not happen again.
Background: 09:45 an auto policy check ran and identified a phantom issue with one of the front end nodes which correctly shut down as a result to protect the network. Owing to a mis-configuration the failover node did not take over all of the responsibilities of the primary which had shut down. This mis-configuration has now been addressed, along with the phantom issue which caused the shut down in the first place.
Service was restored to the majority of nodes by 09:55 and all nodes by 10:02am
Please accept our apologies for this brief outage. We are working on updating procedures to ensure that this misconfiguration will not happen again.