15th January 2020

buf159 drive replacement, scheduled 1 month ago

This node is to be taken down for a drive replacement. We do not expect the downtime to last longer than half an hour.

The node drive was replaced, however the RAID array is rebuilding so there may be some slight performance drops whilst this occurs. We are keeping an eye on the situation.

16/01/2020 00:00 We are experiencing issues getting the node to boot after the replacement, please stand by

16/01/2020 00:12 The node is booting. We will continue to monitor the RAID rebuild process.

16/01/2020 15:03 Unfortunately after multiple attempts the rebuild of the RAID array failed. We are going to use another drive. We do not anticipate any downtime whilst this is carried out.

16/01/2020 15:51 The machine WILL need to go down for this drive to be replaced due to circumstances beyond our control. We are scheduling this for 12:00 EST, or 17:00 UK time.

16/01/2020 21:00 Unfortunately, the previous maintenance was unsuccessful. To minimise the amount of disruption, our partner data centre will be performing maintenance on 17/01/2020 at 6:30 EST, or 11:30 UK time.

16/01/2020 12:10 The maintenance has completed and it appears to be successful thus far. It is likely the RAID array will finish rebuilding within the next 2 hours. Until this time, you may experience performance issues. This was accomplished with about 25 minutes of downtime.

We will be keeping an eye on the rebuild process and will mark this as fixed when it is complete.

16/01/2020 12:39 The rebuild completed successfully and drive status is now optimal. This is now fixed!