RSS Feed
News
Jan
19
Purpose of Work:

January's Patch Tuesday was last week, and the released security updates were roughly double the amount in previous January patch Tuesdays. 

Unfortunately, this seems to have come at the expense of quality control: the initial releases for these patches caused various issues, including boot looping domain controllers, Hyper-V servers that couldn't run their workload, ReFS volumes that aren't detected, and Microsoft L2VPN clients that would not work with existing tunnels.

It does look like Microsoft has followed up with some out-of-band patches for each of these issues on every OS, so we'll be applying the patches, and the fixes for those patches side-by-side on our managed environment today.  

Here's a quick summary of updates:


First up, we have a wormable RCE flaw leveraging the HTTP stack on Server 2019 and up ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-21907 ).  Wormable flaws in general have the most urgent fix requirements. Luckily, this only affects Server 2022 by default, and we'd gone ahead and disabled the functionality allowing for this vulnerability on all 2022 servers we manage, as of the 11th. Today, we'll patch this up with a more permanent fix.

Secondly, a few exchange RCE vulnerabilities for all supported versions ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-21846 ). This seems to require adjacency within the same network, and as such, the risk isn't as high as it could be.  We'll still be patching this out tonight on our managed exchange servers.  During that, you'll be unable to reach us via email, but the helpdesk app and phones will still work.

Third of all, there's an EOP vulnerability affecting all versions of active directory ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-21857 ).  This is a high priority patch, that unfortunately was breaking domain controllers until about 36 hours ago.  Microsoft's site doesn't list it as publicly known, so tonight's patching should still be timely.

There are plenty more flaws this January, but rather than sum up a big sample of them, we'll just get started on deploying them shortly.

Impact of Work:


All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 9:00PM, with some exceptions.

Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.  Mail delivery to our helpdesk may be temporarily halted while our mail servers are updated as part of this patch cycle.  If you receive a delivery failure, you can still reach us by logging directly into the helpdesk and submitting a ticket directly via the portal, or calling us at 303-414-6910 x2, for emergencies.  


Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.  Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density.

Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.
Read more »



Jan
11
Due to a myriad of reported issues with specific Windows updates deployed by Microsoft today, we'll be holding off on our usual manual patching run until further information is out.

Updates to follow.


For the handful of 2022 hosts in our managed environment, we'll be manually mitigating https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-21907 tonight, and recommend anyone out of our management scope do the same.
Read more »



Dec
14
Purpose of Work:

December's Patch Tuesday has arrived.  While Microsoft (luckily) hasn't made as many waves in the news cycle as the Log4j vulnerability, there's still a few standouts that we consider alarming enough to justify same-night patching.

First up, we have an remote code execution vulnerability leveraging Microsoft Office ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2021-43905 ), with no OS version specified; this is a vulnerability that seems to require user interaction, and is not yet seen as exploited in the wild.  We'll be updating our RAS workloads accordingly.

Secondly, there's a number of elevation of privilege vulnerabilities this month: one leveraging NTFS on Server 2022 and Win 10 build 1909+ ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2021-43240 ), and two on Server 2008+ that leverage the windows installer and print spooler, respectively.  Of course, all elevation of privilege vulnerabilities inherently make other vulnerabilities far more dangerous (even something as simple as a compromised website, if your attacker is clever), so we'll be dealing with these tonight.

Third of all, there's a remote code execution vulnerability affecting sharepoint, version 2013+ (https://msrc.microsoft.com/update-guide/vulnerability/CVE-2021-42309 ).  This vuln seems to require that a user exploiting it has the Manage Lists privilege, but if they do, they can create their own site with full permissions, and the capability of getting the server to do whatever they want.  We have few supported sharepoint workloads, but we'll be reaching out to the relevant clients about this, shortly.


Impact of Work:


All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 9:30PM, with some exceptions.

Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.  Mail delivery to our helpdesk may be temporarily halted while our mail servers are updated as part of this patch cycle.  If you receive a delivery failure, you can still reach us by logging directly into the helpdesk and submitting a ticket directly via the portal, or calling us at 303-414-6910 x2, for emergencies.  


Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.  Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density.

Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.
Read more »



Dec
9
Nimble SAN Firmware Upgrade - December 10, 2021
Posted by Pete Carstensen on 09 December 2021 01:29 PM

[Completed, 17:30 12-10-21]:  This maintenance is complete. No impact to workloads was observed. 


Date:  Friday, December 10, 2021
Time:  5:00PM MT

Purpose of Work:
Nimble Storage has informed us that our All-Flash storage arrays are running on a version of NimbleOS that has a software defect which could lead to false predictive failure alerts for member drives in the storage network.  We will be upgrading our storage arrays to newer firmware that resolves this issue.

Impact of Work:
Historically, we have performed many Nimble Storage upgrades which have been transparent to any workloads consuming storage from our arrays.  Accordingly, we do not anticipate any impact, but there is always a slim possibility that connectivity to the storage arrays will be disrupted, resulting in workloads relying on the storage to hard reboot.

Please contact us with any questions / comments / concerns.
Read more »



Nov
9
Monthly Security Patching for Fully-Managed Windows 2012+ servers - November 9, 2021
Posted by David Cunningham on 09 November 2021 07:58 PM
Purpose of Work:

November's Patch Tuesday is here, and as usual there's a few standout vulnerabilities we'll be patching tonight.

First off, we have another Exchange RCE vulnerability: https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17084, which sounds like it's some kind of regression bug, exposing an old vulnerability again.  This will require we reboot our exchange server to apply, so mail will be temporarily halted for a brief window during maintenance tonight.  You can still submit tickets to the helpdesk via the web interface, or call in if there's an urgent issue.   Our helpdesk mail will be queued up and delivered when the server is back up, if not.

Secondly, there's an RCE vulnerability for windows NFS server (all versions): https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17051. This one isn't going to apply to any host we manage, as far as I'm aware.  However, the network vector, no privileges required, 'likely' exploitation assessment, and low complexity all point to a vulnerability that is likely wormable... so those of you running NFS shares on windows for use with Linux hosts, beware.  No exploitation detected in the wild, but when it hits, it'll hit fast.

Third off, there's a local elevation of privilege vulnerability affecting the Windows Kernel (2008+): https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17087.  This one is actually out in the wild as an exploit, so we'll want to patch that ASAP; 0-day vulnerabilities like this are how compromised websites become compromised servers, in a hurry.

Fourth, we have yet another print spooler RCE vulnerability ( 2008+): https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17042. Details are a bit thin as to how this one works right now, but, probably in a similar way to PrintNightmare.  No exploitation detected in the wild, yet.

Fifth, but not least, we have an RCE vulnerability for remote desktop clients (Windows 7+): https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2021-38666.  This would not affect the majority of our managed servers, since we don't make a habit of using them to RDP elsewhere... but anyone reading this is going to want to update their workstation operating system, and soon.  This sort of thing is how compromised servers become compromised workstations.


Impact of Work:


All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 9:30PM, with some exceptions.

Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.  Mail delivery to our helpdesk may be temporarily halted while our mail servers are updated as part of this patch cycle.  If you receive a delivery failure, you can still reach us by logging directly into the helpdesk and submitting a ticket directly via the portal, or calling us at 303-414-6910 x2, for emergencies.  


Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.  Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density.

Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.
Read more »



Oct
22
Completion, Fri 22 Oct 2021 09:25:05 PM MDT: This maintenance appears to have been a success: we can no longer reproduce the issues we were seeing during mass live migration traffic, during our testing.  All changes were made without production impact.

We'll be performing some updates on the cluster tonight for the purpose of further testing, and observing carefully to ensure this is the case: this would not be the first time initial testing gave us the impression the issue was resolved.



Update, Fri 22 Oct 2021 08:30:21 PM MDT: We are now proceeding with this maintenance, and will provide further updates as appropriate.

Update, Fri 22 Oct 2021 07:00:08 PM MDT:
This maintenance will be postponed for the moment, pending the completion of a pre-maintenance backup.  We'll update this thread when maintenance begins.


Purpose of Work:

We will be making some after-hours changes with our primary shared Hyper-V failover cluster, which hosts a smaller proportion of highly-available VPS instances that customers without a dedicated private cloud may rely on.

First, a single node will be paused and drained of its workload, after which time some network interface options will be changed on said node.

Secondly, the storage network will be set to handle cluster communication, while the cluster communication network is adjusted.  Cluster communication will then be handled by the CC network, again.


Once this is complete, a stress test of the primary cluster will ensue, with sensitive workloads moved away from problem hosts.


Impact of Work:

Work will begin at 7PM (MDT) tonight.


No impact should occur, in theory, but it is possible that a subset of VMs will experience brief outages if the instability we're attempt to resolve is not fixed as a result of this maintenance.

If that is the case, we will implement mitigations immediately, possibly blending both maintenance events to try and prevent any more incidents while we're at it.

Any customer VMs that experience issues as a result of this maintenance will be recovered ASAP, with customers informed individually if their VM is going to experience a longer-than-reboot outage as a result of any events.


We will inform you when maintenance is complete.

Please contact us with any questions / comments / concerns.
Read more »