We are investigating an issue in the Italian datanceter.
This announcement will be updated as we have more information.
Update 10:52:
It was a switch issue, the failover didn't work for some reason either, we are investigating in order to avoid the repeat of such an outage.
The good news is that there was no power failure so the servers were up all the time, just the network was down.
Everything should be up at this time.
Update: March 27
Since the issue has been resolved, there have been scattered reports about problems with the network. We have been trying to find the cause but it is very complicated because some services and nodes on a port are working perfectly and others have problems, such as packet loss, short outages, very low throughput. They were not immediately apparent so we thought they were small, separate, issues, but the coincidence is too big at this time so we are investigating all ideas we can think of, alas, an intermittent problem is very hard to diagnose.
After more than a week of largely unsuccessful attempts to track down the issue, we are thinking of a general reboot of the network stack which is a very complicated operation and might involve some hours of downtime as everything must be shutdown and started in a particular order with the aim to avoid data loss while minimizing the downtime.
We are still investigating and keeping this solution as a last resort, if a scheduled maintenance would be deemed necessary, then you will be notified by email about the schedule and estimated duration.
Update: April 17
Since we were unable to solve this elusive issue for so long, we have decided to apply a radical solution, reinitializing all switches, drop all MAC tables and rebuild. As of now, it looks like the network stabilized. Please report any ongoing problems. We were reluctant to apply this solution because the number of customers affected by this switching issue was small and intermittent, but it persisted for too long and all the other attempts at fixing it failed.
Tuesday, February 18, 2025
Powered by WHMCompleteSolution
