RSS Feed
Latest Updates
Jan
14

Update: Wed Jan 15 00:37:28 MST 2020 

While the maintenance event was kicked off at 9PM as planned, recent changes to windows server 2019 (and certain 2016 builds) seem to have reduced our ability to force servers to check for updates when on a domain.

Most of the servers we planned to apply this update to tonight have automatically rebooted by now, and will continue to over the course of the night.

We will be checking in tomorrow to see if any are pending, and will continue to update servers overnight as needed.

Hypervisors and clients with their own WSUS environments (with servers that would be impacted by this vulnerability) will still be scheduled separately.



Purpose of Work:
A spoofing vulnerability (CVE-2020-0601) affecting the Cryptography API in servers running Windows Server 2016 / Windows 10 or newer has been discovered.  Said exploit allows attackers to spoof valid code-signing of arbitrary executables, allowing any malware to evade detection by typical means and bypass built-in protections by masquerading as legitimate programs, and allowing attackers to MITM encrypted connections far more easily by impersonating legitimate services.

Due to the ability of this vulnerability to subvert trusted services and exacerbate any future RCE vulnerabilities immensely, we will be patching and rebooting all affected, fully-managed hosts overnight.  

Standalone hypervisors would be a general exception to this, and customer-owned Windows HVs that host unmanaged VMs should have their maintenance scheduled with us, separately.

Customers with their own update infrastructure will also be scheduled separately.


You can read more about the exploit (and patches mitigating it), here: https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-0601

The NSA has made a post here regarding this exploit: https://www.nsa.gov/News-Features/News-Stories/Article-View/Article/2056772/a-very-important-patch-tuesday/


We will update you as maintenance begins.

Impact of Work:
All affected hosts will be rebooted automatically / ASAP to propagate fixes, starting at 9PM MDT on Tuesday the 14th.

Internal systems on windows 2016 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.

Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.

Any hosts not on our fully-managed domain (usually because they have their own domain) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.


Read more »



Dec
29
Limited Support availability, Dec 29 2019, 16:30-20:30 MDT
Posted by David Cunningham on 29 December 2019 01:54 PM
Our staff is attending a company event and support team responses may be delayed during this time, with troubleshooting limited except in response to emergency matters.  Remote monitoring for such issues will still be done as usual.


This event is scheduled to take place from 4:30-8:30PM tonight.  Operations should resume as normal around 8:30PM Denver time, more or less.

Thank you; we appreciate you and your understanding!
Read more »



Nov
25
Limited hands on support at H5 overnight
Posted by Gina Coleman on 25 November 2019 08:53 PM
Hello,

Overnight staff is working remotely tonight due to the poor road conditions as cdot has advised avoiding travel
During this time non-emergency hands-on support (such as hardware upgrades or changeouts that do not involve a failing host) will be delayed until Tuesday afternoon.

Normal helpdesk support is not affected.

Read more »



Oct
29

Hello,

Overnight staff is working remotely tonight due to the poor road conditions as cdot has advised avoiding travel until normal commuting hours. During this time non-emergency hands-on support (such as hardware upgrades or changeouts that do not involve a failing host) will be delayed until Wednesday morning.

Normal helpdesk support is not affected.


Read more »



Oct
23
Limited Support Availability 2019-10-23 18:15-22:00 MDT
Posted by David Cunningham on 23 October 2019 05:57 PM
Limited Support Availability 2019-10-23 18:15-22:00 MDT

Our staff is attending a company event and support team responses may be delayed during this time, with troubleshooting limited except in response to emergency matters.

Operations should resume as normal around 10PM Denver time.

Thank you; we appreciate you and your understanding!
Read more »