Power Incident @ Sept 19, 1:06 PM MDT
Posted by Jay Sudowski on 19 September 2020 01:47 PM
Date: September 19, 2020|
Time: 1:06 PM MDT
At approximately 1:06PM MDT, our DTC data center experienced a brief power outage on one side of the critical load. We are having all available staff meet on-site to triage and remediate any remaining service impacting issues.
Update: 2:19 PM MDT - Here is the latest information from H5:
Please note that during our planned maintenance activity we experienced a brief interruption to load supported UPS2 system. No impact was seen to UPS system 1. All dual corded critical loads would have remained in service. Loads are currently supported on Generator power at this time.
We have a large staff presence on-site at the moment triaging issues with any single corded loads. If you are experiencing a service disruption, please open a ticket so we can address it.
Update: September 23rd, 2020
We now have a clear timeline of the power outage that occurred on Saturday.
- Routine maintenance activity - annual preventive maintenance on UPS2. Scheduled from noon - 4pm on September 19, 2020
- UPS1 system was not affected and was not part of the maintenance activity
- Maintenance started at 12:15pm. Moved critical loads on UPS2 to generator. Put the UPS into maintenance bypass mode.
- Subsequent to putting the systems into maintenance bypass, Eaton began performing UPS maintenance.
- At 13:04, H5 and Eaton noticed that generator switch gear opened and closed again, which created a brief power interruption to the loads being carried by those generators.
- Power was restored to loads connected to UPS2 in just a few seconds.
- UPS2 maintenance window continued on uneventfully, albeit it on a slight delay due to Eaton collecting logs from the switchgear.
- At 17:00, maintenance was completed
- At 17:30, started return to normal operations.
- At 18:00, everything back in normal operation status.
Further analysis of the issue is still taking place. During normal operations, or even during unscheduled loss of utility power, there is no risk of this issue occurring again. Additionally, H5 will not be conducting any regular UPS maintenance until Eaton can identify the root cause and implement a proper fix.