SEARCH
RSS Feed
News
Aug
13
[Completed] Emergency Network Maintenance - August 13, 2019 @ 9:00PM - 2:00AM
Posted by Jay Sudowski on 13 August 2019 11:08 AM
Completion: Wed Aug 14 00:55:37 MDT 2019

All post-maintenance checks have completed, 99% of VMs are in normal working order, and the network is stable. 

Maintenance is concluded; any VMs not in normal working order have been determined to have pre-existing issues unrelated to the maintenance, and will have a relevant ticket opened with the customer, shortly.




Update: 12:18 AM
All network maintenance is complete.  There was a short outage, roughly 3-5 minutes in duration due to an issue we experienced with the NSSU upgrade.  Since we had engineers engaged on-site, we were able to address the problem very quickly.

All VMs that were shutdown as part of this maintenance are now restarted.  A few are throwing alerts, our helpdesk/NOC team is addressing them now.

Update: 11:23PM
NSSU update is completed.  We are bringing offline VMS on the WEHOSTVPS2 cluster back online now.

Update: 10:30PM
NSSU update is in progress.

Update: 10:13PM
We are almost complete with re-enabling redundancy on dist3.denver2.  Once this is complete, we will perform an NSSU upgrade to the chassis cluster.

Update: 9:09 PM
Our work is beginning.  We are powering off VMs on the WEHOSTVPS2 cluster now.


Date: August 13, 2019
Time: 9:00 PM - 2:00 AM (Mountain Time)

Purpose of Work:
After the network incident on August 12, 2019, we have some critical tasks that we need to carry out this evening to restore redundancy to our network core.
  1. Restore redundancy to our dist3.denver2 switch stack.

  2. Upgrade JunOS on our dist3.denver2 switch stack to the latest JTAC recommended JunOS.  

Impact of Work:
This work will impact our clients in two distinct manners:
  1. If you are hosted on a virtualized environment on the WEHOSTVPS2 Hyper-V cluster, we will be shutting down all virtual machines within this environment at the start of the network maintenance.  This cluster is most dramatically impacted by minimal network disruptions, and when things go sideways, they require hours of manual intervention to bring hundreds of VMs online, which are generally in an unclean state, require chkdsk/fsck, or have severe file system corruption.

  2. If you are a private cloud, colocation, or dedicated server customer, you will experience network disruptions up to 15 minutes in duration during this maintenance window. Theoretically, it should be possible to do the work we need to do with very little network impact, but after the events which transpired on Monday, it's hard to say that with any level of confidence.

We will update this post regularly during the course of our maintenance.  Please contact us with any questions.
Comments (0)
Post a new comment
 
 
Full Name:
Email:
Comments:
CAPTCHA Verification 
 
Please enter the text you see in the image into the textbox below (we use this to prevent automated submissions).