RSS Feed
Latest Updates
Sep
22
Firmware updates for all-flash SAN storage - September 22, 2022
Posted by David Cunningham on 22 September 2022 07:03 PM
Purpose of Work:

Tonight, at 9 PM MDT, we will be performing a routine firmware upgrade on one of our all-flash SAN units, to get it on the latest firmware and security update level.



Impact of Work:

No impact is expected.  The unit will be automatically updating a single controller at a time, meaning it should be failing over gracefully to the other controller at any given phase of the updates.

We have performed this upgrade on two non-production SAN units earlier today, and no impact occurred.


If any impact occurs, you may see degraded performance or an outage on certain standalone hypervisors, or lower-volume clusters.

We will be monitoring the process closely for any issues to mitigate and will update this news post, if we notice anything of the sort.


Please contact us with any questions / comments / concerns.
Read more »



Sep
13

Purpose of Work:

September's Patch Tuesday has arrived, and while there's a few standout vulnerabilities, volume seems normal. There are 64 windows vulnerabilities being patched this month, compared to 66 in September 2021.

Preliminary reports from early adopters indicate no apparent widespread issues, which is about what we'd expect.  Still, the test environment will be updated first and monitored for any showstoppers.

Of those 64 vulnerabilities, only one is under active attack, and another one is a proper zero-day threat, in that it's wormable and affects a wide scope of hosts.


To get into the highlights, we'll start with the vulnerability under active exploitation: a local code execution vulnerability affecting Microsoft Edge and other chromium-based browsers ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-3075 ).  You'd have to interact with a malicious webpage for this to affect you, from what I'm seeing, so this is more of a concern on workstations, VDIs and RDS servers.

Second on the list, we have the wormable vulnerability: a Remote Code Execution vulnerability leveraging the TCP/IP service, on all supported versions of Windows Server and Windows ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-34718 ).  Since TCP/IP support is in the windows kernel, all code executed through this vulnerability would be automatically elevated.  This vulnerability requires attackers are able to send a given host a specially crafted IPv6 packet, so hosts with no IPv6 address or protocol support enabled will be safe.  That said, since IPv6 (and an APIPA address) are defaults in most cases, it's important to roll this one out quickly, since it still has a potentially broad scope of applicability.

Third up, there are two more wormable vulnerabilities, with what may be a more limited scope: Remote Code Execution vulnerabilities leveraging the IKE Protocol Extensions stack, on all supported versions of Windows Server and Windows ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-34721 https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-34722 ).  The executive summary for these ones is a little vague about the details, beyond that an attacker can send a specially crafted IKEv1 packet to any windows host with IPsec enabled to more or less open an RCE backdoor to the host.  As with the previous vulnerability, these ones will automatically run privilege elevated code.  The scope is a little more ambiguous, however: it's unclear if you simply need to have IKE supported in the networking protocol stack, or if you have have an actual IKE listener set up (as part of a tunnel or IPSec VPN dial in client).  We'll be erring on the safe side.

Fourth up, there's a Denial of Service vulnerability leveraging the Windows DNS Server service, on all supported versions of Windows Server ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-34724 ).  While not as high-stakes as the parade of RCE vulnerabilities above, giving attackers an easy way to bring any server with the DNS roll installed and running down is of course, to be avoided.  In particular, environments with a domain controller will want to see this deployed quickly.

Number 5 is a slightly more obscure one with less information: an Arbitrary Code Execution vulnerability affecting all supported versions of .net framework ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-26929 ).  It sounds like on the client level, a user would have to download and run a specially crafted file for this to be exploited, but a website running .net may be targeted in the same way, if you can get it to access said file.  Something for the webservers to watch out for.

The sixth vulnerability is an Elevation of Privilege vulnerability leveraging the Windows Common Log File System Driver, and affecting all supported versions of Windows Server and Windows ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-37969 ).  This one has been reported by multiple agencies, and may be seeing some use in the wild, but it hasn't yet been reported as such.  As is the case with all EOP vulnerabilities, an attacker must first have a foothold to take advantage of it, but with the low attack complexity, it may not need to be much of one.

The final vulnerability I'll expand upon is a Remote Code Execution vulnerability leveraging the Remote Procedure Call Runtime, and affecting all supported versions of Windows Server (  https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-35830 ).   This one looks like it could be wormable and wide-scoped, based on the fact that it's targeting a core windows service,  the Network attack vector, and no user interaction or privileges required.  The only thing holding it back from being more of a showstopper is the apparently 'high' attack complexity, explained thusly: "Successful exploitation of this vulnerability requires an attacker to invest time in repeated exploitation attempts through sending constant or intermittent data.  An unauthenticated attacker on local networks could spoof their IP address as localhost and access functionality in portmap.sys intended to only be reachable from localhost."

I'll add that none of the vulnerabilities this month (including yet more Print Spooler and Kerberos Elevation of Privilege vulnerabilities) seem to require additional action, beyond installing a given patch.

That's all for the highlights, but as usual, there's plenty more where that came from, all reviewable at https://msrc.microsoft.com/update-guide with the proper filtering.


Impact of Work:


All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 10:00PM, with some exceptions.

Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.  Mail delivery to our helpdesk may be temporarily halted while our mail servers are updated as part of this patch cycle.  If you receive a delivery failure, you can still reach us by logging directly into the helpdesk and submitting a ticket directly via the portal, or calling us at 303-414-6910 x2, for emergencies.  


Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.  Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density.

Hypervisors in DR scenarios may be updated up to one hour early, as they are not running active workloads.

Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.


Read more »



Aug
9

Purpose of Work:

August's Patch Tuesday has arrived, and it's certainly not a slow one.  There are 121 vulnerabilities being patched this month, compared to 44 in August 2021, and 120 in August 2020.  This makes this month's Patch Tuesday the 2nd most dense month of patching this year, just behind April.  Preliminary reports from early adopters indicate no apparent widespread issues despite Microsoft's heavy patching workload this month, but we'll be sure to roll things out later in the day, and keep an eye out in various external channels and our test environments, in case that changes.

Of these 121 vulnerabilities, two are listed as publicly known, one of those two is under active exploitation, and the exploited vulnerability isn't wormable... so, while the amount of overall vulnerabilities is high (and there are some standouts, as usual), the urgency is about average.

Kicking off the highlights, I'll start with the vulnerability that's already being exploited: a Local Code Execution vulnerability leveraging the Microsoft Windows Support Diagnostic Tool, and affecting all supported versions of Windows and Windows Server ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-34713 ).  While the vulnerability is listed as a 'remote code execution', the CVSS summary makes it clear the vector is local.  Similar to previous MSDT vulnerabilities, however, the code execution can be invoked by an MSDT URL called by any MSDT-aware application, such as Microsoft Word, meaning social engineering and malicious emails are possible 'remote' methods of exploiting this local vector via user interaction.  Those with a lot of end users to look after (such as in VDI or RDS environments) will want to patch this ASAP.

Second on the list is a trio of Elevation of Privilege vulnerabilities, all affecting every supported version of Exchange Server ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-24477 , https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-24516 , https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-21980 ).  Of the 3 in that group, CVE-2022-30134 is publicly disclosed.  There's not a clear picture of how these vulnerabilities work from any authoritative sources, but the fact that it requires administrators patching against it enable Extended Protection ( https://microsoft.github.io/CSS-Exchange/Security/Extended-Protection/ ) implies it's some kind of authentication bypass or MITM attack.  User interaction is listed as required on the CVSS breakdown, so my guess would be the latter.

Third up, we have yet another Remote Code Execution vulnerability leveraging SMB client / Server, and (curiously) only affecting Windows 11 x64 and Arm64 ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-35804 ).  If this vulnerability affected more versions of Windows or Windows server, it'd be of great concern, since it's definitely a wormable vulnerability.  However, it doesn't... so, I have to assume this is specific to some bleeding edge implementation of SMBv3 compression, based on the CVE summary from MSRC.  Windows 11 users should apply patches ASAP, of course.

Fourth on the list, we have an RCE vulnerability leveraging Hyper-V Server on Windows Server 2012 R2 / Windows 10 and newer ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-34696 ).  As is the case with most RCE leveraging Hyper-V, the vector can be a VM on a Hyper-V server, which then sends its payload up the stack to change scope and execute on the Hyper-V server itself.  As is also typical of these vulnerabilities, the complexity is listed as 'high', and a race condition must be met for it to work, meaning even if executed perfectly, probability still plays a role in successful exploitation.

Fifth on the list, there's a Denial of Service vulnerability leveraging Outlook Client, and affecting all supported versions of Outlook ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-35742 ).  This one isn't a showstopper like many vulnerabilities, but it's got the potential to really annoy some end users just trying to use their mail.  Any email crafted to carry a payload that can exploit this vulnerability will cause outlook to crash, then fail to launch.  No amount of cache clearing will help: you'll have to identify and delete the malicious email to get outlook to start, even if it's never opened.  Naturally, patching is preferable to having to do that for a number of users, for those with a lot of end users to look after (VDI or RDS workloads).

Finally, there's more patching against Elevation of Privilege vulnerabilities leveraging Active Directory Certificate Services in all support versions of Windows Server ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-34691 ).  This appears to mention the same hardening measures that were initially rolled out in May 2022, but it's not listed as a revision, nor are these hardening measures being made mandator yet, so we'll be conducting more reviews to see exactly what's being done here.  That said, compatibility mode (mentioned here: https://support.microsoft.com/en-us/topic/kb5014754-certificate-based-authentication-changes-on-windows-domain-controllers-ad2c23b0-15d8-4340-a468-4d4f3b188f16 ) seems to still be in effect.

That's all for the highlights, but as usual, there's plenty more where that came from, all reviewable at https://msrc.microsoft.com/update-guide with the proper filtering.


Impact of Work:


All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 10:45PM, with some exceptions.

Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.  Mail delivery to our helpdesk may be temporarily halted while our mail servers are updated as part of this patch cycle.  If you receive a delivery failure, you can still reach us by logging directly into the helpdesk and submitting a ticket directly via the portal, or calling us at 303-414-6910 x2, for emergencies.  


Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.  Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density.

Hypervisors in DR scenarios will be updated one hour early, as they are not running active workloads.

Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.


Read more »



Jul
12
Purpose of Work:

Patch Tuesday has rolled around, and  The volume of fixes is a bit lower than July of last year, and there are no early reports of issues caused by this round of patches.  Of the 87 fixes released in this batch, only two are reported as utilized in active attacks (and one of which is for Microsoft Edge). 

While there's also no patched vulnerabilities marked as publicly disclosed other than these two, it's worth noting that there's plenty of Windows bugs that have been publicly disclosed this month not included in the patching cycle, so we'll be keeping an eye out for out of band windows patches.


To start off with the one actively targeted windows vulnerability: this one is an Elevation of Privilege vulnerability leveraging the CSRSS component, and affecting all supported versions of Windows and Windows server ... and likely a few unsupported ones ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-22047 ).  Microsoft is sparse on the details for this one, but complexity and privileges required are low, so if paired with any kind of remote code execution able to interact with CSRSS (likely including compromised websites), it can be easily used to elevate that code to running as system, as is evidenced by the active attack status.  I recommend you patch this one quickly, if you're running any application that 

Next up, we have a Tampering vulnerability leveraging the 'Server' service (aka, SMB) that affects Windows server 20H2 and 2022 ( or, Windows 10 20H2 and up) https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-30216 .  Tampering vulnerabilities are pretty flexible in their potential impact ( they can be used to elevate, execute code, or disclose information).  That said, this one does seem to require some level of authentication despite having a network vector, so I'd treat it like an Elevation vulnerabilty.

Third on the list, there's a Remote Code Execution vulnerability leveraging the RPC service, affecting Windows Server 2012 and up (or, Windows 8 and up): https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-22038 .  This one is fairly alarming, with all code run using it being automatically elevated, a network vector, and no user interaction or authentication required.  The only thing potentially stopping it from being a wormable threat, is that the complexity is marked 'high', with the following note provided by MS: "Successful exploitation of this vulnerability requires an attacker to invest time in repeated exploitation attempts through sending constant or intermittent data."

Forth on the list, there's an Elevation of Privilege vulnerability leveraging IIS server, affecting all supported versions of Windows: https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-30209 .  This one's a bit odd: it's listed as requiring no privileges, and it seems to imply that while you can bypass authentication to get privileged information from the server service, you can't disrupt the service, sounding a bit more like information disclosure than elevation of privilege. There's also the note that successful exploitation of this vulnerability requires an attacker to take additional actions prior to exploitation to prepare the target environment.

That's it for the highlights this month, though as usual, there are plenty of other vulnerabilities (including some more Elevation of Privilege affecting the Printer Spooler service, and more Windows Network File System Remote Code execution vulnerabilities, for the small percentage of users running that).



Impact of Work:


All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 9:30PM, with some exceptions.

Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.  Mail delivery to our helpdesk may be temporarily halted while our mail servers are updated as part of this patch cycle.  If you receive a delivery failure, you can still reach us by logging directly into the helpdesk and submitting a ticket directly via the portal, or calling us at 303-414-6910 x2, for emergencies.  


Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.  Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density.

Hypervisors in DR scenarios will be updated one hour early, as they are not running active workloads.

Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.
Read more »



Jun
14
Purpose of Work:

It's that time again: Patch Tuesday. The volume of fixes released is similar to June of last year, and there are no early reports of issues caused by this round of patches.  Today also marks the day before internet explorer is officially out of support on Windows 10.  You can read more about that here: https://docs.microsoft.com/en-us/lifecycle/announcements/internet-explorer-11-end-of-support

Of the 55 fixes released in this batch, only one vulnerability addressed is reported as utilized in active attacks, or even publicly disclosed.  


We'll start off with said vulnerability: an arbitrary code execution vulnerability leveraging the Microsoft Diagnostic Tool ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-30190 ), affecting Server 2008 R2 and up (or, Windows 7 and up for the workstation equivalents). 

With this one being actively exploited, there's a bit of additional context to it. This vulnerability was given the designation "follina" by security researchers, and the most common vector for it is through Microsoft office products, such as Word.  What's unique about this particular vulnerability is that the nature of the MSD Tool allows it to be executed via MSDT URL protocol, even by Office documents that have macro support disabled... so even opening a malicious Office document with such an attack embedded with full precautions won't guard against it like normal.  You can read more about it here: https://msrc-blog.microsoft.com/2022/05/30/guidance-for-cve-2022-30190-microsoft-support-diagnostic-tool-vulnerability/.  


Second highlighted fix this month is for a remote code execution vulnerability leveraging Hyper-V ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-30163), affecting Server 2008 R2 and up, as well as Windows 7 and up.  This one is nasty in that the RCE is initiated from a guest on an affected Hyper-V host, then jumps the gap up to the HV's OS.  While the attack complexity is high (apparently needing events out of the control of the exploiter to happen in a certain order to work... ie, a race condition), this is still something we'll be patching on our managed hypervisors ASAP, and will encourage clients with strict maintenance window requirements to authorize patching with similar expedience.

Third patched vulnerability of note is a remote code execution vulnerability leveraging Microsoft LDAP ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-30139 ) and affecting Server 2016 / Windows 10 and up.  This would likely affect Domain Controllers in particular. This one has high listed complexity, owing to the fact that it's only exploitable if the MaxReceiveBuffer LDAP policy is set to a value higher than the default value. Systems with the default value of this policy would not be vulnerable, meaning our own domain controllers are currently not affected.  The mechanism of action is not made clear in the article, though it may rely on a buffer overflow of some sort.  

Fourth patch of note deals with an elevation of privilege vulnerability leveraging Windows' Kerberos implementation ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-30165), and affecting Server 2016 / Windows 10 and up.  This would be able to affect any domain-joined server, but appears to only affect s
ystems configured to activate both of the following features in Windows Server: CredSSP (Credential Security Service Provider) and RCG (Remote Credential Guard).  This being only EOP, an attack would still need low-privilege local access to carry this out, meaning webservers are a likely vector for it.

The last of the highlights for this month is a fix targeting a remote code execution vulnerability leveraging and affecting SQL Server 2014 and up ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-29143 ) .  The listed complexity is 'high', with low privilege required to even launch it.  Reading up on it, it seems to require specific pre-existing table structure to be possible, but if successfully launched, it would double as an elevation of privilege attack, depending on the level of privilege the SQL server service identity possesses.  We will be ensuring that the security fix at least is detected by and pushed to all our managed SQL hosts.
That's the end of the highlights, but not the fixes, of course.


Impact of Work:


All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 10:15PM, with some exceptions.

Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.  Mail delivery to our helpdesk may be temporarily halted while our mail servers are updated as part of this patch cycle.  If you receive a delivery failure, you can still reach us by logging directly into the helpdesk and submitting a ticket directly via the portal, or calling us at 303-414-6910 x2, for emergencies.  


Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.  Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density.

Hypervisors in DR scenarios will be updated one hour early, as they are not running active workloads.

Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.
Read more »