[Completed] Emergency Network Maintenance - August 13, 2019 @ 9:00PM - 2:00AM
Posted by Jay Sudowski on 13 August 2019 11:08 AM
Completion: Wed Aug 14 00:55:37 MDT 2019|
All post-maintenance checks have completed, 99% of VMs are in normal working order, and the network is stable.
Maintenance is concluded; any VMs not in normal working order have been determined to have pre-existing issues unrelated to the maintenance, and will have a relevant ticket opened with the customer, shortly.
Update: 12:18 AM
All network maintenance is complete. There was a short outage, roughly 3-5 minutes in duration due to an issue we experienced with the NSSU upgrade. Since we had engineers engaged on-site, we were able to address the problem very quickly.
All VMs that were shutdown as part of this maintenance are now restarted. A few are throwing alerts, our helpdesk/NOC team is addressing them now.
NSSU update is completed. We are bringing offline VMS on the WEHOSTVPS2 cluster back online now.
NSSU update is in progress.
We are almost complete with re-enabling redundancy on dist3.denver2. Once this is complete, we will perform an NSSU upgrade to the chassis cluster.
Update: 9:09 PM
Our work is beginning. We are powering off VMs on the WEHOSTVPS2 cluster now.
Date: August 13, 2019
Time: 9:00 PM - 2:00 AM (Mountain Time)
Purpose of Work:
After the network incident on August 12, 2019, we have some critical tasks that we need to carry out this evening to restore redundancy to our network core.
Impact of Work:
This work will impact our clients in two distinct manners:
We will update this post regularly during the course of our maintenance. Please contact us with any questions.