Some systems are experiencing issues

Past Incidents

1st August 2021

No incidents reported

31st July 2021

No incidents reported

30th July 2021

No incidents reported

29th July 2021

No incidents reported

28th July 2021

No incidents reported

27th July 2021

No incidents reported

26th July 2021

No incidents reported

25th July 2021

No incidents reported

24th July 2021

No incidents reported

23rd July 2021

Compute Nodes Intermittent Power Cuts

Our Grantham location is experiencing intermittent brownouts. This has caused one of our AC units to fail - we have techs on-site investigating immediately.

19/07/2021

23:28 BST - Our AC engineers are en route to get the unit back online. We have powered down some non-essential machines to lower the heat output.

23:37 BST - Our AC engineers are on-site and are working to get the unit back online.

20/07/2021

00:19 BST - AC engineers have been unable to bring the unit back online. We are now bringing in additional backup units.

00:35 BST - We have had to start shutting down some host machines to avoid overheating the room and causing any data loss due to unexpected shutdowns.

00:41 BST - Backup AC units have arrived and are being set up to lower the temperature of the room.

00:52 BST - The units are now cooling the room and we are hoping to keep it stable enough to keep most machines online.

09:00 BST - We have shut down a small number of our Grantham compute nodes in order to relocate them to keep temperatures manageable. We are awaiting further information from Stulz regarding our air conditioning unit.

16:08 BST - We have started to bring nodes back online in alternate server rooms. We are expecting this to take a few more hours. We also have more temporary AC units arriving in the next few hours to help alleviate the temperature issues.

21/07/2021

10:38 BST - We managed to bring all but two of our compute nodes last night after the temporary AC units cooled the room down enough to be operational. Due to the high temperatures in the room as well as the extreme heatwave we are continuing to monitor around the clock with staff on-site at all times. Our main AC unit manufacturer is scheduled to come on-site on Friday to hopefully resolve the main issue. GHM45 and GHM46 are currently down whilst we attempt to bring them back online, we are working to avoid any data loss so will not be rushing the process.

22/07/2021

09:37 BST - The two nodes we failed to bring back online early yesterday were eventually fixed and all customer nodes should now be back online as of yesterday. We are still working to bring dedicated machines back online as and when possible. Our local AC company are commissioning three backup AC units that were fitted yesterday to hopefully bring temperatures back down to normal. The main AC unit manufacturer has confirmed they will be coming tomorrow and we hope they will be able to resolve the issue without needing to schedule another call out.

I'd like to thank everyone for their patience and understanding during this time, our staff have been up around the clock doing extended shifts to monitor and secure the premises. It is unfortunate that the power cuts caused our main unit to fail despite being operational without a single hiccup for 8 years now; we'll be glad to get the issue sorted and see everything return to normal. - Jake Evans, COO.

23/07/2021

10:29 BST - The three backup AC units were fully commissioned yesterday and brought temperatures back down to near normal. That allowed us to get some very well deserved rest last night as the immediate panic was under control. Our main AC unit is currently having its main board replaced as it appears that the power cuts caused it to blow. We hope to have the unit back in operation shortly which will bring this incident to an end.

If you are still experiencing issues with your service in our Grantham location, please reach out so that we can take a closer look.

10:50 BST - The nodes that were previously relocated as of our update on 20/07/2021 16:08 BST are being moved back to the main server room due to the success of the backup AC units and the imminent repair of our main AC unit. We do not expect this to take longer than a few hours

13:34 BST - A majority of the nodes are now back up. There are 4 nodes currently down, and we are working on getting them back up.

14:00 BST - Only two nodes are currently down - we are working on resolving this.

22nd July 2021

No incidents reported

21st July 2021

No incidents reported

20th July 2021

No incidents reported

19th July 2021

No incidents reported

18th July 2021

No incidents reported

17th July 2021

No incidents reported

16th July 2021

No incidents reported

15th July 2021

No incidents reported

14th July 2021

No incidents reported

13th July 2021

No incidents reported

12th July 2021

No incidents reported

11th July 2021

No incidents reported

10th July 2021

No incidents reported

9th July 2021

No incidents reported

8th July 2021

No incidents reported

7th July 2021

No incidents reported

6th July 2021

No incidents reported

5th July 2021

No incidents reported