2022/09/30 17:23 network has been restored. If you are still experiencing some issue please use the web console to power off the server and then power on it again

2022/09/29 11:27 using the few available ports of the OOB router we were able to connect some interfaces. Some servers are now accessible, as well as two of the three storages. 

 

 

We are experiencing some problems with a NL server which became unresponsive albeit some services are still running.

At this time looks like a storage issue but we are investigating which is difficult with an unresponsive server.

Details about the progress of the situation and ETAs for a resolution will be posted here.

At this time we do not have an ETA.

 

2022/09/28 07:35 the issue is affecting most of the equipments in the POP. We requested the datacenter on site guys to investigate.

2022/09/28 11:10 the pop is isolated as core switches stack stopped forwarding packets and it's not booting after reset.

2022/09/28 17:14 we were unable to resume the both core switch. no more options for now.

2022/09/28 18:16 ETA for deliver of current switch replacements at present is 7-10 days. We are still evaluating alternatives

2022/10/01 00:56 We have continuously worked with the materials accessible to improve the situation. The services in NL should be available to our customers. This is a temporary solution until the new equipment arrives. There will be at least another interruption when we will install them if everything works fine and the current router, not designed for this task, will hold.

 

Post-mortem

The storage was the first to fail internally, probably because of the higher traffic, then the rest of the network went down.
Both routers failed the same time, which is highly unusual, maybe a firmware issue (we will know more when we receive the routers/switches in Italy to check thoroughly) and the remote hands were unable to diagnose/solve the issue.
We will have to replace both routers but quotes we received involve delivery in more than a week. We are trying to find alternative solutions with the DC in the meantime.
We were unprepared for the redundant router to fail as well, I am sorry for this situation.

 

 



Wednesday, September 28, 2022

« Back

Powered by WHMCompleteSolution