|
Jan 19 |
Monthly Security Patching for Fully-Managed Windows 2012+ servers - January 19, 2022
Posted by David Cunningham on 19 January 2022 08:32 PM |
|
Purpose of Work: January's Patch Tuesday was last week, and the released security updates were roughly double the amount in previous January patch Tuesdays. Unfortunately, this seems to have come at the expense of quality control: the initial releases for these patches caused various issues, including boot looping domain controllers, Hyper-V servers that couldn't run their workload, ReFS volumes that aren't detected, and Microsoft L2VPN clients that would not work with existing tunnels. It does look like Microsoft has followed up with some out-of-band patches for each of these issues on every OS, so we'll be applying the patches, and the fixes for those patches side-by-side on our managed environment today. Here's a quick summary of updates: First up, we have a wormable RCE flaw leveraging the HTTP stack on Server 2019 and up ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-21907 ). Wormable flaws in general have the most urgent fix requirements. Luckily, this only affects Server 2022 by default, and we'd gone ahead and disabled the functionality allowing for this vulnerability on all 2022 servers we manage, as of the 11th. Today, we'll patch this up with a more permanent fix. Secondly, a few exchange RCE vulnerabilities for all supported versions ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-21846 ). This seems to require adjacency within the same network, and as such, the risk isn't as high as it could be. We'll still be patching this out tonight on our managed exchange servers. During that, you'll be unable to reach us via email, but the helpdesk app and phones will still work. Third of all, there's an EOP vulnerability affecting all versions of active directory ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-21857 ). This is a high priority patch, that unfortunately was breaking domain controllers until about 36 hours ago. Microsoft's site doesn't list it as publicly known, so tonight's patching should still be timely. There are plenty more flaws this January, but rather than sum up a big sample of them, we'll just get started on deploying them shortly. Impact of Work: All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 9:00PM, with some exceptions. Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them. Mail delivery to our helpdesk may be temporarily halted while our mail servers are updated as part of this patch cycle. If you receive a delivery failure, you can still reach us by logging directly into the helpdesk and submitting a ticket directly via the portal, or calling us at 303-414-6910 x2, for emergencies. Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters. Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density. Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately. Please contact us with any questions / comments / concerns. Read more » | |
|
Jan 11 |
Postponed: Monthly Security Patching for Fully-Managed Windows 2012+ servers - January 11, 2022
Posted by David Cunningham on 11 January 2022 10:16 PM |
|
Due to a myriad of reported issues with specific Windows updates deployed by Microsoft today, we'll be holding off on our usual manual patching run until further information is out. Updates to follow. For the handful of 2022 hosts in our managed environment, we'll be manually mitigating https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-21907 tonight, and recommend anyone out of our management scope do the same. Read more » | |
|
Dec 14 |
Monthly Security Patching for Fully-Managed Windows 2012+ servers - December 14, 2021
Posted by David Cunningham on 14 December 2021 07:16 PM |
|
Purpose of Work: December's Patch Tuesday has arrived. While Microsoft (luckily) hasn't made as many waves in the news cycle as the Log4j vulnerability, there's still a few standouts that we consider alarming enough to justify same-night patching. First up, we have an remote code execution vulnerability leveraging Microsoft Office ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2021-43905 ), with no OS version specified; this is a vulnerability that seems to require user interaction, and is not yet seen as exploited in the wild. We'll be updating our RAS workloads accordingly. Secondly, there's a number of elevation of privilege vulnerabilities this month: one leveraging NTFS on Server 2022 and Win 10 build 1909+ ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2021-43240 ), and two on Server 2008+ that leverage the windows installer and print spooler, respectively. Of course, all elevation of privilege vulnerabilities inherently make other vulnerabilities far more dangerous (even something as simple as a compromised website, if your attacker is clever), so we'll be dealing with these tonight. Third of all, there's a remote code execution vulnerability affecting sharepoint, version 2013+ (https://msrc.microsoft.com/update-guide/vulnerability/CVE-2021-42309 ). This vuln seems to require that a user exploiting it has the Manage Lists privilege, but if they do, they can create their own site with full permissions, and the capability of getting the server to do whatever they want. We have few supported sharepoint workloads, but we'll be reaching out to the relevant clients about this, shortly. Impact of Work: All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 9:30PM, with some exceptions. Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them. Mail delivery to our helpdesk may be temporarily halted while our mail servers are updated as part of this patch cycle. If you receive a delivery failure, you can still reach us by logging directly into the helpdesk and submitting a ticket directly via the portal, or calling us at 303-414-6910 x2, for emergencies. Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters. Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density. Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately. Please contact us with any questions / comments / concerns. Read more » | |
|
Dec 9 |
Nimble SAN Firmware Upgrade - December 10, 2021
Posted by Pete Carstensen on 09 December 2021 01:29 PM |
|
[Completed, 17:30 12-10-21]: This maintenance is complete. No impact to workloads was observed. Date: Friday, December 10, 2021 Time: 5:00PM MT Purpose of Work: Nimble Storage has informed us that our All-Flash storage arrays are running on a version of NimbleOS that has a software defect which could lead to false predictive failure alerts for member drives in the storage network. We will be upgrading our storage arrays to newer firmware that resolves this issue. Impact of Work: Historically, we have performed many Nimble Storage upgrades which have been transparent to any workloads consuming storage from our arrays. Accordingly, we do not anticipate any impact, but there is always a slim possibility that connectivity to the storage arrays will be disrupted, resulting in workloads relying on the storage to hard reboot. Please contact us with any questions / comments / concerns. Read more » | |
|
Nov 9 |
Monthly Security Patching for Fully-Managed Windows 2012+ servers - November 9, 2021
Posted by David Cunningham on 09 November 2021 07:58 PM |
|
Purpose of Work: November's Patch Tuesday is here, and as usual there's a few standout vulnerabilities we'll be patching tonight. First off, we have another Exchange RCE vulnerability: https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17084, which sounds like it's some kind of regression bug, exposing an old vulnerability again. This will require we reboot our exchange server to apply, so mail will be temporarily halted for a brief window during maintenance tonight. You can still submit tickets to the helpdesk via the web interface, or call in if there's an urgent issue. Our helpdesk mail will be queued up and delivered when the server is back up, if not. Secondly, there's an RCE vulnerability for windows NFS server (all versions): https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17051. This one isn't going to apply to any host we manage, as far as I'm aware. However, the network vector, no privileges required, 'likely' exploitation assessment, and low complexity all point to a vulnerability that is likely wormable... so those of you running NFS shares on windows for use with Linux hosts, beware. No exploitation detected in the wild, but when it hits, it'll hit fast. Third off, there's a local elevation of privilege vulnerability affecting the Windows Kernel (2008+): https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17087. This one is actually out in the wild as an exploit, so we'll want to patch that ASAP; 0-day vulnerabilities like this are how compromised websites become compromised servers, in a hurry. Fourth, we have yet another print spooler RCE vulnerability ( 2008+): https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17042. Details are a bit thin as to how this one works right now, but, probably in a similar way to PrintNightmare. No exploitation detected in the wild, yet. Fifth, but not least, we have an RCE vulnerability for remote desktop clients (Windows 7+): https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2021-38666. This would not affect the majority of our managed servers, since we don't make a habit of using them to RDP elsewhere... but anyone reading this is going to want to update their workstation operating system, and soon. This sort of thing is how compromised servers become compromised workstations. Impact of Work: All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 9:30PM, with some exceptions. Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them. Mail delivery to our helpdesk may be temporarily halted while our mail servers are updated as part of this patch cycle. If you receive a delivery failure, you can still reach us by logging directly into the helpdesk and submitting a ticket directly via the portal, or calling us at 303-414-6910 x2, for emergencies. Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters. Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density. Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately. Please contact us with any questions / comments / concerns. Read more » | |