Update: Wed Jan 15 00:37:28 MST 2020 

While the maintenance event was kicked off at 9PM as planned, recent changes to windows server 2019 (and certain 2016 builds) seem to have reduced our ability to force servers to check for updates when on a domain.

Most of the servers we planned to apply this update to tonight have automatically rebooted by now, and will continue to over the course of the night.

We will be checking in tomorrow to see if any are pending, and will continue to update servers overnight as needed.

Hypervisors and clients with their own WSUS environments (with servers that would be impacted by this vulnerability) will still be scheduled separately.



Purpose of Work:
A spoofing vulnerability (CVE-2020-0601) affecting the Cryptography API in servers running Windows Server 2016 / Windows 10 or newer has been discovered.  Said exploit allows attackers to spoof valid code-signing of arbitrary executables, allowing any malware to evade detection by typical means and bypass built-in protections by masquerading as legitimate programs, and allowing attackers to MITM encrypted connections far more easily by impersonating legitimate services.

Due to the ability of this vulnerability to subvert trusted services and exacerbate any future RCE vulnerabilities immensely, we will be patching and rebooting all affected, fully-managed hosts overnight.  

Standalone hypervisors would be a general exception to this, and customer-owned Windows HVs that host unmanaged VMs should have their maintenance scheduled with us, separately.

Customers with their own update infrastructure will also be scheduled separately.


You can read more about the exploit (and patches mitigating it), here: https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-0601

The NSA has made a post here regarding this exploit: https://www.nsa.gov/News-Features/News-Stories/Article-View/Article/2056772/a-very-important-patch-tuesday/


We will update you as maintenance begins.

Impact of Work:
All affected hosts will be rebooted automatically / ASAP to propagate fixes, starting at 9PM MDT on Tuesday the 14th.

Internal systems on windows 2016 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.

Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.

Any hosts not on our fully-managed domain (usually because they have their own domain) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.

Hello,

Overnight staff is working remotely tonight due to the poor road conditions as cdot has advised avoiding travel until normal commuting hours. During this time non-emergency hands-on support (such as hardware upgrades or changeouts that do not involve a failing host) will be delayed until Wednesday morning.

Normal helpdesk support is not affected.