Our Grantham location is experiencing intermittent brownouts.
This has caused one of our AC units to fail - we have techs on-site investigating immediately.
19/07/2021
23:28 BST - Our AC engineers are en route to get the unit back online. We have powered down some non-essential machines to lower the heat output.
23:37 BST - Our AC engineers are on-site and are working to get the unit back online.
20/07/2021
00:19 BST - AC engineers have been unable to bring the unit back online. We are now bringing in additional backup units.
00:35 BST - We have had to start shutting down some host machines to avoid overheating the room and causing any data loss due to unexpected shutdowns.
00:41 BST - Backup AC units have arrived and are being set up to lower the temperature of the room.
00:52 BST - The units are now cooling the room and we are hoping to keep it stable enough to keep most machines online.
09:00 BST - We have shut down a small number of our Grantham compute nodes in order to relocate them to keep temperatures manageable. We are awaiting further information from Stulz regarding our air conditioning unit.
16:08 BST - We have started to bring nodes back online in alternate server rooms. We are expecting this to take a few more hours. We also have more temporary AC units arriving in the next few hours to help alleviate the temperature issues.
21/07/2021
10:38 BST - We managed to bring all but two of our compute nodes last night after the temporary AC units cooled the room down enough to be operational. Due to the high temperatures in the room as well as the extreme heatwave we are continuing to monitor around the clock with staff on-site at all times. Our main AC unit manufacturer is scheduled to come on-site on Friday to hopefully resolve the main issue. GHM45 and GHM46 are currently down whilst we attempt to bring them back online, we are working to avoid any data loss so will not be rushing the process.
22/07/2021
09:37 BST - The two nodes we failed to bring back online early yesterday were eventually fixed and all customer nodes should now be back online as of yesterday. We are still working to bring dedicated machines back online as and when possible. Our local AC company are commissioning three backup AC units that were fitted yesterday to hopefully bring temperatures back down to normal. The main AC unit manufacturer has confirmed they will be coming tomorrow and we hope they will be able to resolve the issue without needing to schedule another call out.
I'd like to thank everyone for their patience and understanding during this time, our staff have been up around the clock doing extended shifts to monitor and secure the premises. It is unfortunate that the power cuts caused our main unit to fail despite being operational without a single hiccup for 8 years now; we'll be glad to get the issue sorted and see everything return to normal. - Jake Evans, COO.
23/07/2021
10:29 BST - The three backup AC units were fully commissioned yesterday and brought temperatures back down to near normal. That allowed us to get some very well deserved rest last night as the immediate panic was under control. Our main AC unit is currently having its main board replaced as it appears that the power cuts caused it to blow. We hope to have the unit back in operation shortly which will bring this incident to an end.
If you are still experiencing issues with your service in our Grantham location, please reach out so that we can take a closer look.
10:50 BST - The nodes that were previously relocated as of our update on 20/07/2021 16:08 BST are being moved back to the main server room due to the success of the backup AC units and the imminent repair of our main AC unit. We do not expect this to take longer than a few hours
13:34 BST - A majority of the nodes are now back up. There are 4 nodes currently down, and we are working on getting them back up.
14:00 BST - Only two nodes are currently down - we are working on resolving this.
