Hábrók network outage

Resolved — Cluster is operational again.

Posted July 17, 2023 16:55 by Fokke

Monitoring — Except for a couple of nodes that are still having issues, the cluster nodes are back online and running jobs again. If you do still encounter any issues, please let us know.

Posted July 13, 2023 09:30 by Bob

Identified — We are currently reinstalling the compute nodes of the cluster to get them into the correct state again. This will at least take the rest of the day.

Posted July 12, 2023 13:08 by Fokke

Identified — Unfortunately we haven't been able to restore the network yet. Again our sincere apologies for the inconvenience this is causing.

Posted July 12, 2023 07:48 by Bob

Identified — Due to a misconfiguration around 16:00 today, the network settings of Habrok were removed. This resulted in the cluster's baremetal nodes losing their outside and inside network. Unfortunately, all running jobs will be lost. Currently we are working on restoring the network, after which we expect Habrok to be working again. We apologize for the inconvenience this will have caused and are taking steps to make sure this does not happen again.

Posted July 11, 2023 19:05 by Bob

Investigating — All Hábrók compute nodes (except for the V100 GPU nodes) are currently unreachable due to a network issue. We're investigating the issue.

Posted July 11, 2023 16:15 by Bob