[Complete, Wed 11 Nov 2020 11:03:13 PM MST] Patch compliance for managed hosts without special scheduling requirements is at 100%.  Further off-schedule update reboots should not be necessary.

[Update, Wed 11 Nov 2020 01:47:16 AM MST]: Our exchange server has finished applying the latest CU, and mail service is restored.  There will be a few minor updates that will be applied to it over the course of the night, but further instances of downtime should be limited to 10m or less.

Regarding https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17051: most if not all managed clients should be unaffected by it, as this is for the "Server for NFS" linux compatibility feature of the Windows File Server role, which is pretty niche and not installed by default. 

Service-impacting reboots on non-hypervisors will continue as the night progresses, and stop before 5am MST.  Tomorrow night, we will address hosts that failed to install this round of updates and shoot for 100% patch compliance in our managed environment.


[Update, Tue 10 Nov 2020 10:40:51 PM MST]: Updating efforts are well on their way, with many hosts having already rebooted.  I will now begin the work on our exchange server, which means mail to our helpdesk sent through email clients will be temporarily deferred.  You can still update helpdesk tickets through a direct login, or call us on our support line.

Purpose of Work:

Patch Tuesday is here for November, and there's a few standout vulnerabilities among the crowd of those we'll be dealing with on fully managed servers that all subscribers running windows should be aware of.  


First of all, there's an Escalation of Privilege vulnerability currently being exploited in the wild that leverages the Kernel Cryptography driver on Windows Server 2008+.  This one was publicly disclosed by google in Late October, and will be a priority this patch cyle: https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17087

Second, there's a remote code execution vulnerability Exchange server. We'll be applying this security update, so you might not be able to reach us over email for a short period during the maintenance: https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17084

Third, there's an RCE vulnerability affecting NFS on Windows Server 2008+.  There's not alot of public detail out there for this one, but it has a CVSS of 9.8, leverages NFS, has low attack complexity, and does not require privileges or user interaction in its description... so suffice to say, it's looking like a wormable threat: https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17051

Fourth, there's a 'security feature bypass' Vulnerability in Hyper-V for Windows Server 2012R2+.  While there's not alot of detail for what it is, the network attack vector, CVSS score of 6.5, low attack complexity, and lack of need for privileges or user interaction are noteworthy, and may imply the ability for VMs to modify the hypervisor directly, which is reason enough to include Hyper-V servers in this update cycle.  https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17040

Fifth, there's a Kerberos Security Feature bypass vulnerability that would affect Active Directory.  There's not alot of info out for this one yet, but it is the sort of patch that requires additional mitigation in the form of a registry edit to enforce new Kerberos Ticket security signature standards on domain controllers, so we'll be assessing this one in our internal AD domains and reaching out to managed clients who have their own with our findings.

While that's what I would consider the highlights, there are a few other fixes that would impact 2019+ servers, including a few EOP vulnerabilities.  All vulnerabilities will be patched tonight, with rollbacks occurring selectively if required.



Impact of Work:

Our exchange host will be rebooted a few times tonight to propagate security fixes.  This may interfere with our ability to send and receive mail intermittently, while patches are being applied.


All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 9:30PM, with some exceptions.

Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.


Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.  Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density.

Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.

[Completion, Sun 18 Oct 2020 11:40:29 PM MDT] A final update audit confirms that all managed hosts with applicable security updates are either updated successfully, or scheduled to be soon.  Several procedural improvements were made to streamline and speed up the handling of future Patch Tuesdays that seem to involve an unusually high number of zero-day exploits, as this one did.

[Update 2, Fri 16 Oct 2020 11:26:55 PM MDT] A subset of fully managed windows servers outside of our usual update management solution have been brought into it; some reboots will happen for those of these that did not receive the security updates on Tuesday night.  We will perform a final audit tomorrow night before considering this window for out-of-band overnight maintenance reboots closed.

[Update 1]: About half of domain-managed VMs have been updated.  We will continue to monitor hosts that have yet to update to ensure they come up quickly, and manually update where required. 

All reboots will be finished before business hours.

Updates will also continue tommorow night.  Managed servers not on the domain will be the primary focus, at that point.

Purpose of Work:

Patch Tuesday has come around again, and there's a few standout vulnerabilities among the crowd of those we'll be dealing with on fully managed servers that all subscribers running windows should be aware of.  


First off, there's a remote code execution vulnerability affecting Windows Server 2019 hosts (and more recent versions of windows 10).  This one leverages the TCP/IP stack within windows, via specially crafted ICMPv6 Router Advertisement packets.  Because it leverages the TCP/IP stack, it would execute any injected code in the SYSTEM identity context, which means it's a wormable, EOP/RCE exploit you'll want to patch well before exploits are developed using it: https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-16898 

Second, there's a remote code execution vulnerability affecting Hyper-V on Windows Server 2008+ hosts.  This one could allow an attacker on a VM to run a specially crafted application that then forces the hypervisor itself to execute arbitrary code in an elevated user context.  Hyper-V host owners will want to patch this out ASAP: fully managed customers with hyper-V servers that have special scheduling requirements will contacted specifically about this one. https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-16891

Third, there's a privilege escalation vulnerability affecting Windows Server 2016+ (and windows 10) that leverages Windows Error Reporting.  All privilege escalation vulnerabilities are a big risk on webservers, and other such hosts with a large, publicly accessible attack surface area.  https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-16909

Fourth up, there's another privilege escalation vulnerability affecting Windows Server 2008+.  This one leverages the Windows Network Connections service, and of course should be addressed ASAP, like with most EOP vulnerabilities: https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-16887



Impact of Work:

Our exchange host will be rebooted a few times tonight to propagate security fixes.  This may interfere with our ability to send and receive mail intermittently, while patches are being applied.


All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 9:00PM on 10/13/20, with some exceptions.

Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.


Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.  Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density.

Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.

Update 3 (Thu 10 Sep 2020 01:59:38 AM MDT)
All updates scheduled for tonight are complete.  A few servers remain without special scheduling consideration, and will be done tomorrow night.  Customers will be alerted directly, where required, as this is a smaller subset of hosts.


Update 2 (Wed 09 Sep 2020 08:57:56 PM MDT):
A subset of servers were missed last night, and our exchange host will require more updating.

We will begin these updates shortly; you may notice a reboot on your windows server, or some temporary delivery deferrals for our helpdesk.

Update 1 (Wed Sep  9 00:50:34 MDT 2020):
Automatic reboots of hosts in the discussed scope have begun, and will be occurring over the next few hours.  We will monitor servers to ensure they come back up without incident.

Purpose of Work:
This patch Tuesday has a few highlights, in addition to various, less noteworthy security updates.

The first is a system-level RCE vulnerability affecting all recent versions of exchange server, which we will be mitigating on our internal server ASAP, overnight.  https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-16875

The second is an RCE vulnerability affecting all supported versions of windows server.  This one leverages a COM interaction with Javascript, and thus could affect any RDS server or webserver where a user or application pool might end up opening a maliciously crafted file: https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-0922

This would result in code being run as the identity of whatever user opened said file.

The third is an RCE vulnerability affecting Server 2016 and up that leverages how  Microsoft Windows Codecs Library handles objects in its memory.  Again, webservers and RDS servers would be particularly vulnerable to this: https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-1129



Standalone hypervisors would be a general exception to this, and customer-owned Windows HVs that host unmanaged VMs, but also run Windows 2012+ should have their maintenance scheduled with us, separately.

Customers with their own update infrastructure will also be scheduled separately.


We will update you as maintenance begins.


Impact of Work:

Our exchange host will be rebooted a few times tonight to propagate security fixes.  This may interfere with our ability to send and receive mail intermittently, while patches are being applied.


All affected hosts will be rebooted automatically / ASAP to propagate fixes, starting at 11:30PM on 9/8/20.

Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.

Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.

Any hosts not on our fully-managed domain (usually because they have their own domain) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.