What follows is a letter sent today from Jon Morby, Managing Director of FidoNet to our customers.
This week has been a difficult week thanks to DDoS attacks and hardware failures. We have weathered well and, except for a few minor instances, customers thankfully haven’t really noticed the carnage.
We did experience a major internal outage at one point this week which meant our phone system stopped working for a while, and even our ability to deal with support tickets (and your ability to raise support tickets) was hampered for a few hours.
Throughout all of this though our service status page at https://status.fido.net/ was alive and we were communicating through this “blog” with details of what was going on and how to remedy issues that might crop up (such as SSL certificate changes as a result of our having to install a new SSL certificate on web-11).
What has come to light is that not all customers seem to know about this particular website, despite it being linked in portals and in emails and itself having existed for more than 15 years! – but then I guess you never really need to know about these things until everything is on fire and you can’t access your email 🙂
So please do bookmark the web page https://status.fido.net/ just in case you need it in the future.
With this in mind, I’m writing to all our customers first and foremost to apologise if you were impacted by any of the issues we have experienced this week, and secondly to ask that you bookmark https://status.fido.net/ so that you can check here in case of any issues in the future.
When things break (thankfully this doesn’t happen very often) it isn’t possible for us to always answer the phones and talk to you directly, we’re usually snowed under trying to make sure that we have things fixed as quickly as possible. At these times calls tend to overflow to an external messaging service who are supposed to point you to the support site and tell you to raise a ticket there so that someone can be in touch in due course … This doesn’t always happen and often they seem to promise a callback. When we finally get to the messages at 4 am having resolved all the issues, it isn’t usually practical for us to call you back although if we had a support ticket we can then send you an explanation and keep in touch that way. This is why we ask you to try and raise a ticket. If your email isn’t working you can still open and read tickets online in our support portal at https://help.fido.net/
We have recently upgraded our support centre and it is (on the whole) much more responsive and easier to use than the old Kayako based system. If you haven’t logged into it yet then we encourage you to do so and to have a look around. We are busy populating the knowledge base with useful facts and you will often find an answer to a generic problem in there, avoiding the need to raise a support ticket at all!
Moving forwards, we are busy working on addressing the underlying issues which caused the disruption earlier this week. The main issues relate to a single point of failure behind our load balancers where they were all linked to our centralised (S3 compatible) storage cluster. When this became degraded it caused our clustered load-balancers to fail which in turn caused our internal NS cache servers to fail and without internal DNS resolution, many many things ground to a halt.
Plans are underway to rebuild the load balancers and remove their reliance on the centralised storage cluster (mainly for logging), and also to add more resilience to our ns-cache servers.
I would like to take this opportunity to personally apologise for any disruption you may have experienced this week, and would like to assure you that we are continuing to invest heavily in both training and our network infrastructure in order to ensure that these sorts of issues just don’t happen.
Thank you for continuing to choose Fido
Jon Morby
Managing Director