Past Incidents

We have lost connectivity to our Sydney location and are currently awaiting a response from our provider.

13:21 GMT: Expected resolution time is within 30 minutes.

13:25 GMT: Services are back online.

We've noted slowness when logging into CreeperPanel, this is likely due to high load from repeated login attempts.

We are installing additional cooling to many of our new Buffalo nodes, this will result in rolling reboots.

We identified an issue regarding one of our nodes where vms did not have network connectivity. Upon rebooting the node to attempt to resolve this, it did not come up correctly.

We are liaising with the remote data centre and will update when we have further information.

Update: 2020-02-06 17:54 UTC

This has now been resolved.

We have taken our Los Angeles API endpoint down as there is currently a network routing loop causing communications on the return path from some of our locations (namely, Miami) to fail, meaning the API will wrongfully report services as offline and be unable to control them.

Traffic will be routed automatically to other API endpoints, but this may result in slower control panel performance than usual.

We've worked with our transit provider to exclude our routes from the offending upstream.

Buffalo node #152 was detected as frozen.

6:30 am GMT; Upon reboot it is displaying memory errors. 6:52 am GMT; Remote hands have been requested to reseat and if still failing, replace the faulty memory module(s). 7:29 am GMT; Reseating the memory has allowed the machine to boot. 8:10 am GMT; We've taken the machine down to fully test the memory to ensure this does not recur. 8:47 am GMT; Passed first test, bringing machine back online and will monitor.