Feb 9 |
Emergency Security Patching for Fully-Managed Windows 2012+ servers - February 9, 2021
Posted by David Cunningham on 09 February 2021 11:40 PM |
Purpose of Work: Read more » | |
Jan 18 |
Network Maintenance - January 22, 2021
Posted by Jay Sudowski on 18 January 2021 04:19 PM |
Date: January 22, 2021 Time: 8:00 PM - 12:00 AM (Mountain Standard Time) Purpose of Work: On Friday, January 22 we will be continuing the network maintenance started the previous night. The work conducted during this maintenance window will complete the physical removal of dist3.denver2 from our network, assuming such work was not completed the previous night. Impact of Work: When we remove dist3.denver2 from the network, there may be a few brief periods of latency and packet loss impacting the whole network while we turn down links connecting dist3.denver to our network and repatch those links to dist3.dtc1, as needed. Read more » | |
Jan 18 |
[COMPLETED] Network Maintenance - January 21, 2021 8pm - 12am
Posted by Jay Sudowski on 18 January 2021 04:17 PM |
Update 11:05 PM - We are complete with our network maintenance for the evening. We will conduct the physical topology changes required tomorrow evening. Please contact our helpdesk for any issues. Date: January 21, 2021 Time: 8:00 PM - 12:00 AM (Mountain Standard Time) Purpose of Work: We will be performing the work necessary to remove our 1801 California Street / Downtown Denver Data Center from our dark fiber ring. The work that will be conducted Thursday night is as follows: 1. Groom VLANs from dist3.denver2 to dist3.dtc1. During this work, our networking team will logically migrate the layer 3 interfaces for routed VLANs that are still homed to dist3.denver2. We will be performing this work gradually, migrating a few VLANs every few minutes. 2. If time allows, we will move on to physically removing the switch stack from our network. Impact of Work: During the VLAN grooming work, there will be a short 3-5 minute network disruption on a per-network basis as the layer 3 interfaces are shutdown on dist3.denver2 and activated on dist3.dtc1. When we remove dist3.denver2 from the network, there may be a few brief periods of latency and packet loss impacting the whole network while we turn down links connecting dist3.denver to our network and repatch those links to dist3.dtc1, as needed. Read more » | |
Jan 12 |
[Complete] Emergency Security Patching for Fully-Managed Windows 2012+ servers - January 12, 2021
Posted by David Cunningham on 12 January 2021 10:54 PM |
Completion [Thu 14 Jan 2021 12:17:47 AM MST] Our post update night audit reveals that the vast majority of servers successfully updated overnight. A few one-off reboots will need to be conducted from here, but they will be done after-hours or scheduled separately where needed. Purpose of Work: Read more » | |
Dec 8 |
[Complete] Emergency Security Patching for Fully-Managed Windows 2012+ servers - December 10, 2020
Posted by David Cunningham on 08 December 2020 10:30 PM |
Completion [Wed 09 Dec 2020 11:35:49 PM MST] Our post update night audit reveals that the vast majority of servers successfully updated overnight. A few one-off reboots may need to be conducted from here, but they will be done after-hours, and not on hypervisor hosts or tonight. Update, [Tue 08 Dec 2020 10:31:19 PM MST] Correction: "All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 9:30PM, with some exceptions." 10:30PM. Reboots will begin shortly. Purpose of Work: December's Patch Tuesday has come around, and while it's a lighter patch volume than usual, there's still vulnerabilities that all subscribers running windows should be aware of. First of all, there are several RCE vulnerabilities for Microsoft exchange which seem to require authentication to be leveraged. Here's one example: https://msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-17132 Second, there appears to be a Hyper-V vulnerability that allows Hyper-V guests to force the Hypervisor to run arbitrary code by sending it an invalid SMB packet. This affects Server 2016+, and I would consider it a higher priority patch, of those I see this cycle: https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17095 Third, there is another Kerberos security feature bypass vulnerability to be patched. We'll be applying and testing this on our internal domains that many managed servers are a part of: https://msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-16996 Fourth, there's an RCE/EOP vulnerability that requires SMB read access to a host (which means it could affect any host that allows SMB, but not NTFS access to the 'everyone' identity). Once an adversary has this level of access, they can then send specially crafted packets over the network or locally to get the affected host to run arbitrary code as the system identity. This affects Server 2012 and up: https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17096 Impact of Work: Our exchange host will be rebooted at least once tonight to propagate security fixes. This may interfere with our ability to send and receive mail intermittently, while patches are being applied. A direct ticket update via the helpdesk portal will still work, as will a phone call. All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 9:30PM, with some exceptions. Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them. Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters. Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density. Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately. Please contact us with any questions / comments / concerns. Read more » | |