|
Apr 13 |
[Completed] Monthly Security Patching for Fully-Managed Windows 2012+ servers - April 13, 2021
Posted by David Cunningham on 13 April 2021 05:58 PM |
|
[Update, Tue 13 Apr 2021 10:08:07 PM MDT]: Exchange servers are all fully updated, and further mail disruption is not expected tonight. The main server reboots have been initiated as of about 40 minutes ago, and we'll be monitoring updated hosts overnight to ensure they come up normally, then auditing for any hosts that did not update successfully in the morning, once the bulk of the updating is completed. Purpose of Work: April's Patch Tuesday has arrived, and it's not fooling around. As is typical, there's a number of vulnerabilities patched this cycle that we deem worth patching ASAP, and a few stand out among the crowd, even compared to months previous. https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2021-28310 First up, there's an escalation of privilege vulnerability affecting the kernel on Windows 10 (build 1803+) and Server 2019 that has been detected as in use by attackers' exploits 'in the wild' by Kapersky. This exploit, like most EOP exploits, could be used to turn a relatively low-stakes compromise (think, a single website getting hacked) into something more serious and server-wide. https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2021-28480 https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2021-28481 Secondly, there's another round of RCE vulnerabilities affecting up-to-date installations of Exchange Server 2013 and up which could be used to compromise those hosts and use them to laterally attack anything in the same domain. We'll be updating our internal exchange and any customer exchange servers early, as such. https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2021-28329 Third of all, there's a number of remote code execution vulnerabilities affecting the RPC service on pretty much all versions of windows (including 2008). There are in fact 27 individual CVEs being addressed by this month's security patch regarding the same or similar vulnerabilities for a variety of operating systems. The RPC service runs as system, so this would be an instant system compromise if leveraged. It does at least appear to require some level of system privileges in the CVE listing, so it may not be wormable, at least. Impact of Work: Managed exchange hosts (including our internal exchange) will be rebooted and updated at 6:30pm. Mail delivery to our helpdesk will be temporarily halted while this maintenance is occurring. You can still reach us by logging directly into the helpdesk and submitting a ticket directly via the portal, or calling us at 303-414-6910 x2, for emergencies. All other affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 9:00PM, with some exceptions. Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them. Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters. Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density. Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately. Please contact us with any questions / comments / concerns. Read more » | |
|
Apr 5 |
[Completed] Security Patching for Managed SolusVM Hypervisor, April 6, 2021
Posted by David Cunningham on 05 April 2021 09:19 PM |
|
Completion [ Tue 06 Apr 2021 10:09:40 PM MDT ] : Maintenance is completed, with all virtual servers confirmed as restored to service and apparently functioning normally. Update [ Tue 06 Apr 2021 09:28:39 PM MDT]: Maintenance pre-tasks are complete, and SolusVM guests are being shut down. Virtual servers meeting the previously outlined criteria will be impacted until the host finishes rebooting. Purpose of Work: We've done some vulnerability scanning on our internal network, and found that our SolusVM hypervisor is in need of some patching. During this maintenance, we'll be updating the management scripts of SolusVM, as well as the kernel and all packages on both the management and hypervisor nodes. This will take place at 9:15PM on April 6, 2021. Impact of Work: All non-HA VPS (identifiable by having an IP on subnet 23.239.222.0/25) will be temporarily shut down when the time to reboot the HV arrives. Impact should last until the reboot is complete, at which time VPS will be brought back up and monitored closely. Questions from affected customers can be fielded during the maintenance, with updates being provided during maintenance milestones on request. Read more » | |
|
Mar 9 |
[Completed] Monthly Security Patching for Fully-Managed Windows 2012+ servers - March 9, 2021
Posted by David Cunningham on 09 March 2021 10:48 PM |
|
Completion, Thu 11 Mar 2021 12:31:26 AM MST We've finished updating all hosts except a few that are highly available (and thus should not result in any actual customer downtime), so no further impacts due to windows updates are expected. Update, Wed 10 Mar 2021 12:44:03 AM MST: Many hosts have been rebooted, and we are continuing to monitor to ensure they come up without incident. We'll continue to manage reboots throughout the night, checking for any hosts that were unable to update and finishing their updates tomorrow night. Purpose of Work: March's patch Tuesday is underway, and while Microsoft did get alot of urgent Exchange-related patches released early this month ( See: https://helpdesk.handynetworks.com/supportsuite/index.php?/News/NewsItem/View/299/completed-emergency-out-of-band-security-patching-for-fully-managed-exchange-mail-servers-march-2nd-2021 ) , there's still a few vulnerabilities of note that we'll be taking care of on fully managed hosts, tonight. Firstly, a few of these exchange patches have new revisions out ( https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2021-26855 ). We'll be reapplying these to the handful of exchange hosts we manage. Second, there's another DNS RCE exploit that affects 2008+ ( https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2021-26897 ). This is, as always, a priority patch. The DNS service runs as local system, so effectively, RCE exploits targeting the DNS server can be used to compromise a host outright. The executive summary implies this might be related to (or at least, exacerbated by) the configure of insecure dynamic updates on Windows DNS servers. While the exploit does affect 2008, which will not be patched, I will note that this configuration is not the default on any hosting panel or platform we manage, at least. 2012+ hosts will be patched. Third, there's an IE memory corruption vulnerability that affects IE 9+ (and Edge): ( https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2021-26411 ). The executive summary clarifies that this requires user interaction, so ASP sites that use underlying explorer libraries are probably unaffected by this. This would be more of an issue on RDS servers that have users who run explorer or edge. Fourth, there's a generic escalation of privilege affecting windows 2008+ (https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2021-27077). The executive summary has little detail, but of course, these kinds of vulnerabilities could potentially exacerbate small-scale website or application compromises and turn them into host compromises, if left unpatched. Impact of Work: All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 11:00PM, with some exceptions. Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them. Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters. Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density. Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately. Please contact us with any questions / comments / concerns. Read more » | |
|
Mar 2 |
[Completed] Emergency Out-of-band Security Patching for fully managed Exchange mail servers, March 2nd, 2021
Posted by David Cunningham on 02 March 2021 09:48 PM |
|
Completed, Tue 02 Mar 2021 08:50:03 PM MST: All exchange hosts we manage are now updated, with ours having been updated within 20m of the maintenance window. No issues with the patches have been detected. We strongly recommend all self-managed clients running their own exchange servers update before EOD. I will add that in-bind updates (on patch tuesday) will still be carried out on hosts that require them, next week. Purpose of Work: Microsoft has announced that it has detected multiple exchange servers in the wild that were compromised by one specific APT group, using several 0-day exploits that were patched out of band today: https://msrc-blog.microsoft.com/2021/03/02/multiple-security-updates-released-for-exchange-server/ They have advised applying the patches ASAP, and we'll be handling this for fully managed exchange servers. Impact of Work: All fully managed exchange hosts will be rebooted automatically / ASAP to propagate fixes, starting from 6pm MST, onwards. There are not many of these hosts, but they include our own mail server. Post-patch, all hosts will be audited for signs of the compromise signatures Microsoft has disclosed, and fully managed clients will be notified with the results. Internal mail of affected systems may be temporarily impacted in the time it takes to reboot them, including our own. Submitting a ticket directly via the portal will still work, as will phone calls, during the work window. Please contact us with any questions / comments / concerns. Read more » | |
|
Feb 9 |
[Completed] Emergency Security Patching for Fully-Managed Windows 2012+ servers - February 9, 2021
Posted by David Cunningham on 09 February 2021 11:40 PM |
|
Purpose of Work: Read more » | |