RSS Feed
Latest Updates
Nov
10

[Complete, Wed 11 Nov 2020 11:03:13 PM MST] Patch compliance for managed hosts without special scheduling requirements is at 100%.  Further off-schedule update reboots should not be necessary.

[Update, Wed 11 Nov 2020 01:47:16 AM MST]:
Our exchange server has finished applying the latest CU, and mail service is restored.  There will be a few minor updates that will be applied to it over the course of the night, but further instances of downtime should be limited to 10m or less.

Regarding https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17051: most if not all managed clients should be unaffected by it, as this is for the "Server for NFS" linux compatibility feature of the Windows File Server role, which is pretty niche and not installed by default. 

Service-impacting reboots on non-hypervisors will continue as the night progresses, and stop before 5am MST.  Tomorrow night, we will address hosts that failed to install this round of updates and shoot for 100% patch compliance in our managed environment.


[Update, Tue 10 Nov 2020 10:40:51 PM MST]:
Updating efforts are well on their way, with many hosts having already rebooted.  I will now begin the work on our exchange server, which means mail to our helpdesk sent through email clients will be temporarily deferred.  You can still update helpdesk tickets through a direct login, or call us on our support line.

Purpose of Work:

Patch Tuesday is here for November, and there's a few standout vulnerabilities among the crowd of those we'll be dealing with on fully managed servers that all subscribers running windows should be aware of.  


First of all, there's an Escalation of Privilege vulnerability currently being exploited in the wild that leverages the Kernel Cryptography driver on Windows Server 2008+.  This one was publicly disclosed by google in Late October, and will be a priority this patch cyle: https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17087

Second, there's a remote code execution vulnerability Exchange server. We'll be applying this security update, so you might not be able to reach us over email for a short period during the maintenance: https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17084

Third, there's an RCE vulnerability affecting NFS on Windows Server 2008+.  There's not alot of public detail out there for this one, but it has a CVSS of 9.8, leverages NFS, has low attack complexity, and does not require privileges or user interaction in its description... so suffice to say, it's looking like a wormable threat: https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17051

Fourth, there's a 'security feature bypass' Vulnerability in Hyper-V for Windows Server 2012R2+.  While there's not alot of detail for what it is, the network attack vector, CVSS score of 6.5, low attack complexity, and lack of need for privileges or user interaction are noteworthy, and may imply the ability for VMs to modify the hypervisor directly, which is reason enough to include Hyper-V servers in this update cycle.  https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17040

Fifth, there's a Kerberos Security Feature bypass vulnerability that would affect Active Directory.  There's not alot of info out for this one yet, but it is the sort of patch that requires additional mitigation in the form of a registry edit to enforce new Kerberos Ticket security signature standards on domain controllers, so we'll be assessing this one in our internal AD domains and reaching out to managed clients who have their own with our findings.

While that's what I would consider the highlights, there are a few other fixes that would impact 2019+ servers, including a few EOP vulnerabilities.  All vulnerabilities will be patched tonight, with rollbacks occurring selectively if required.



Impact of Work:

Our exchange host will be rebooted a few times tonight to propagate security fixes.  This may interfere with our ability to send and receive mail intermittently, while patches are being applied.


All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 9:30PM, with some exceptions.

Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.


Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.  Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density.

Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.


Read more »



Oct
13

[Completion, Sun 18 Oct 2020 11:40:29 PM MDT] A final update audit confirms that all managed hosts with applicable security updates are either updated successfully, or scheduled to be soon.  Several procedural improvements were made to streamline and speed up the handling of future Patch Tuesdays that seem to involve an unusually high number of zero-day exploits, as this one did.

[Update 2, Fri 16 Oct 2020 11:26:55 PM MDT] A subset of fully managed windows servers outside of our usual update management solution have been brought into it; some reboots will happen for those of these that did not receive the security updates on Tuesday night.  We will perform a final audit tomorrow night before considering this window for out-of-band overnight maintenance reboots closed.

[Update 1]: About half of domain-managed VMs have been updated.  We will continue to monitor hosts that have yet to update to ensure they come up quickly, and manually update where required. 

All reboots will be finished before business hours.

Updates will also continue tommorow night.  Managed servers not on the domain will be the primary focus, at that point.


Purpose of Work:

Patch Tuesday has come around again, and there's a few standout vulnerabilities among the crowd of those we'll be dealing with on fully managed servers that all subscribers running windows should be aware of.  


First off, there's a remote code execution vulnerability affecting Windows Server 2019 hosts (and more recent versions of windows 10).  This one leverages the TCP/IP stack within windows, via specially crafted ICMPv6 Router Advertisement packets.  Because it leverages the TCP/IP stack, it would execute any injected code in the SYSTEM identity context, which means it's a wormable, EOP/RCE exploit you'll want to patch well before exploits are developed using it: https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-16898 

Second, there's a remote code execution vulnerability affecting Hyper-V on Windows Server 2008+ hosts.  This one could allow an attacker on a VM to run a specially crafted application that then forces the hypervisor itself to execute arbitrary code in an elevated user context.  Hyper-V host owners will want to patch this out ASAP: fully managed customers with hyper-V servers that have special scheduling requirements will contacted specifically about this one. https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-16891

Third, there's a privilege escalation vulnerability affecting Windows Server 2016+ (and windows 10) that leverages Windows Error Reporting.  All privilege escalation vulnerabilities are a big risk on webservers, and other such hosts with a large, publicly accessible attack surface area.  https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-16909

Fourth up, there's another privilege escalation vulnerability affecting Windows Server 2008+.  This one leverages the Windows Network Connections service, and of course should be addressed ASAP, like with most EOP vulnerabilities: https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-16887



Impact of Work:

Our exchange host will be rebooted a few times tonight to propagate security fixes.  This may interfere with our ability to send and receive mail intermittently, while patches are being applied.


All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 9:00PM on 10/13/20, with some exceptions.

Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.


Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.  Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density.

Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.


Read more »



Sep
19
Power Incident @ Sept 19, 1:06 PM MDT
Posted by Jay Sudowski on 19 September 2020 01:47 PM
Date: September 19, 2020

Time: 1:06 PM MDT

At approximately 1:06PM MDT, our DTC data center experienced a brief power outage on one side of the critical load. We are having all available staff meet on-site to triage and remediate any remaining service impacting issues.

Update: 2:19 PM MDT - Here is the latest information from H5:

Please note that during our planned maintenance activity we experienced a brief interruption to load supported UPS2 system. No impact was seen to UPS system 1. All dual corded critical loads would have remained in service. Loads are currently supported on Generator power at this time.

We have a large staff presence on-site at the moment triaging issues with any single corded loads.  If you are experiencing a service disruption, please open a ticket so we can address it.

Update: September 23rd, 2020

We now have a clear timeline of the power outage that occurred on Saturday.

- Routine maintenance activity - annual preventive maintenance on UPS2. Scheduled from noon - 4pm on September 19, 2020

- UPS1 system was not affected and was not part of the maintenance activity

- Maintenance started at 12:15pm. Moved critical loads on UPS2 to generator. Put the UPS into maintenance bypass mode.

- Subsequent to putting the systems into maintenance bypass, Eaton began performing UPS maintenance.

- At 13:04, H5 and Eaton noticed that generator switch gear opened and closed again, which created a brief power interruption to the loads being carried by those generators.

- Power was restored to loads connected to UPS2 in just a few seconds.

- UPS2 maintenance window continued on uneventfully, albeit it on a slight delay due to Eaton collecting logs from the switchgear.

- At 17:00, maintenance was completed

- At 17:30, started return to normal operations.

- At 18:00, everything back in normal operation status.

Further analysis of the issue is still taking place. During normal operations, or even during unscheduled loss of utility power, there is no risk of this issue occurring again.  Additionally, H5 will not be conducting any regular UPS maintenance until Eaton can identify the root cause and implement a proper fix. 



Read more »



Sep
8

Update 3 (Thu 10 Sep 2020 01:59:38 AM MDT)
All updates scheduled for tonight are complete.  A few servers remain without special scheduling consideration, and will be done tomorrow night.  Customers will be alerted directly, where required, as this is a smaller subset of hosts.


Update 2 (Wed 09 Sep 2020 08:57:56 PM MDT):
A subset of servers were missed last night, and our exchange host will require more updating.

We will begin these updates shortly; you may notice a reboot on your windows server, or some temporary delivery deferrals for our helpdesk.

Update 1 (Wed Sep  9 00:50:34 MDT 2020):
Automatic reboots of hosts in the discussed scope have begun, and will be occurring over the next few hours.  We will monitor servers to ensure they come back up without incident.

Purpose of Work:

This patch Tuesday has a few highlights, in addition to various, less noteworthy security updates.

The first is a system-level RCE vulnerability affecting all recent versions of exchange server, which we will be mitigating on our internal server ASAP, overnight.  https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-16875

The second is an RCE vulnerability affecting all supported versions of windows server.  This one leverages a COM interaction with Javascript, and thus could affect any RDS server or webserver where a user or application pool might end up opening a maliciously crafted file: https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-0922

This would result in code being run as the identity of whatever user opened said file.

The third is an RCE vulnerability affecting Server 2016 and up that leverages how  Microsoft Windows Codecs Library handles objects in its memory.  Again, webservers and RDS servers would be particularly vulnerable to this: https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-1129



Standalone hypervisors would be a general exception to this, and customer-owned Windows HVs that host unmanaged VMs, but also run Windows 2012+ should have their maintenance scheduled with us, separately.

Customers with their own update infrastructure will also be scheduled separately.


We will update you as maintenance begins.


Impact of Work:

Our exchange host will be rebooted a few times tonight to propagate security fixes.  This may interfere with our ability to send and receive mail intermittently, while patches are being applied.


All affected hosts will be rebooted automatically / ASAP to propagate fixes, starting at 11:30PM on 9/8/20.

Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.

Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.

Any hosts not on our fully-managed domain (usually because they have their own domain) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.


Read more »



Aug
20
Completion ( Sun 23 Aug 2020 10:08:10 PM MDT ) All updates to production non-cluster servers that did not require a separate maintenance window are complete.

Update 3 (Sat 22 Aug 2020 21:30:24 PM MDT)
After 10 PM MDT tonight we will be continuing with updates, including updates to some standalone hypervisors. We will be monitoring and ensuring that servers come back up after the reboots.

Update 2 (Fri 21 Aug 2020 00:35:56 PM MDT)
Automatic reboots of hosts in the discussed scope have completed for tonight and all servers have come back up. We will proceed with additional updates tomorrow evening after 10 PM MDT.

Update 1 (Thu 20 Aug 2020 10:24:16 PM MDT):

Automatic reboots of hosts in the discussed scope have begun, and will be occurring over the next hour.  We will monitor servers to ensure they come back up without incident.

Purpose of Work:
A privilege elevation vulnerability (CVE-2020-1530 and CVE-2020-1537) affects all supported versions of windows server so far.  This vulnerability exists when Windows Remote Access improperly handles memory or file operations. The exploit requires an attacker to have execution capabilities on the victim system. Systems hosting websites or with web-accessible services are particularly vulnerable.

Due to the ability of this vulnerability to allow privilege escalation and the wide attack surface, we will be patching and rebooting all affected, fully-managed hosts overnight.  

Standalone hypervisors would be a general exception to this, and customer-owned Windows HVs that host unmanaged VMs, but also run Windows 2012+ should have their maintenance scheduled with us, separately.

Customers with their own update infrastructure will also be scheduled separately.


You can read more about the exploits (and patches mitigating it), here: https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-1530 https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-1537

We will update you as maintenance begins.


Impact of Work:
All affected hosts will be rebooted automatically / ASAP to propagate fixes, starting at 10:10PM MDT on Thursday the 20th.

Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.

Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.

Any hosts not on our fully-managed domain (usually because they have their own domain) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.




Read more »