RSS Feed
News
Jun
14
Purpose of Work:

It's that time again: Patch Tuesday. The volume of fixes released is similar to June of last year, and there are no early reports of issues caused by this round of patches.  Today also marks the day before internet explorer is officially out of support on Windows 10.  You can read more about that here: https://docs.microsoft.com/en-us/lifecycle/announcements/internet-explorer-11-end-of-support

Of the 55 fixes released in this batch, only one vulnerability addressed is reported as utilized in active attacks, or even publicly disclosed.  


We'll start off with said vulnerability: an arbitrary code execution vulnerability leveraging the Microsoft Diagnostic Tool ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-30190 ), affecting Server 2008 R2 and up (or, Windows 7 and up for the workstation equivalents). 

With this one being actively exploited, there's a bit of additional context to it. This vulnerability was given the designation "follina" by security researchers, and the most common vector for it is through Microsoft office products, such as Word.  What's unique about this particular vulnerability is that the nature of the MSD Tool allows it to be executed via MSDT URL protocol, even by Office documents that have macro support disabled... so even opening a malicious Office document with such an attack embedded with full precautions won't guard against it like normal.  You can read more about it here: https://msrc-blog.microsoft.com/2022/05/30/guidance-for-cve-2022-30190-microsoft-support-diagnostic-tool-vulnerability/.  


Second highlighted fix this month is for a remote code execution vulnerability leveraging Hyper-V ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-30163), affecting Server 2008 R2 and up, as well as Windows 7 and up.  This one is nasty in that the RCE is initiated from a guest on an affected Hyper-V host, then jumps the gap up to the HV's OS.  While the attack complexity is high (apparently needing events out of the control of the exploiter to happen in a certain order to work... ie, a race condition), this is still something we'll be patching on our managed hypervisors ASAP, and will encourage clients with strict maintenance window requirements to authorize patching with similar expedience.

Third patched vulnerability of note is a remote code execution vulnerability leveraging Microsoft LDAP ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-30139 ) and affecting Server 2016 / Windows 10 and up.  This would likely affect Domain Controllers in particular. This one has high listed complexity, owing to the fact that it's only exploitable if the MaxReceiveBuffer LDAP policy is set to a value higher than the default value. Systems with the default value of this policy would not be vulnerable, meaning our own domain controllers are currently not affected.  The mechanism of action is not made clear in the article, though it may rely on a buffer overflow of some sort.  

Fourth patch of note deals with an elevation of privilege vulnerability leveraging Windows' Kerberos implementation ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-30165), and affecting Server 2016 / Windows 10 and up.  This would be able to affect any domain-joined server, but appears to only affect s
ystems configured to activate both of the following features in Windows Server: CredSSP (Credential Security Service Provider) and RCG (Remote Credential Guard).  This being only EOP, an attack would still need low-privilege local access to carry this out, meaning webservers are a likely vector for it.

The last of the highlights for this month is a fix targeting a remote code execution vulnerability leveraging and affecting SQL Server 2014 and up ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-29143 ) .  The listed complexity is 'high', with low privilege required to even launch it.  Reading up on it, it seems to require specific pre-existing table structure to be possible, but if successfully launched, it would double as an elevation of privilege attack, depending on the level of privilege the SQL server service identity possesses.  We will be ensuring that the security fix at least is detected by and pushed to all our managed SQL hosts.
That's the end of the highlights, but not the fixes, of course.


Impact of Work:


All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 10:15PM, with some exceptions.

Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.  Mail delivery to our helpdesk may be temporarily halted while our mail servers are updated as part of this patch cycle.  If you receive a delivery failure, you can still reach us by logging directly into the helpdesk and submitting a ticket directly via the portal, or calling us at 303-414-6910 x2, for emergencies.  


Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.  Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density.

Hypervisors in DR scenarios will be updated one hour early, as they are not running active workloads.

Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.
Read more »



May
10

Purpose of Work:

May's Patch Tuesday has arrived, and it's a relatively average one in terms of patch volume.  That said, there's a few publicly known vulnerabilities in the mix (one with with active exploitation detected), so we'll be proceeding on schedule.  Initial reports of post-patch experience are relatively normal.

Starting off the highlights this month, we have an LSA Spoofing vulnerability affecting all supported versions of Windows Server ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-26925).  This one is detected as being in the wild, and the vulnerability's complexity is labelled 'high', and reading through it, that tracks.  The most dangerous use case of this seems to be that an attacker gets a domain controller to authenticate against a malicious host using unauthenticated LSARPC calls.  From there, they can use captured credentials to access anything in the domain.  We'll be applying this patch, and giving the guidance article linked in that CVE a thorough review for any additional practices we or fully managed clients should implement to make NTLM relaying itself much more difficult.

Second off, we have an elevation of privilege vulnerability leveraging Active directory certificate services, and affecting Windows Server 2012 / Windows 8 and up ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-26923 ).  This one has been publicly disclosed, and allows a user with delegation rights for any computer accounts the ability to issue themselves a certificate that would grant domain administrator rights.  We'll be applying this on our own domain controllers, and will evaluate which clients on an independent update schedule are most vulnerable to this, tommorow.

Third, we have another remote code execution vulnerability leveraging Windows NFS Server, and affecting all supported versions of windows ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-26937 ).  We do not run NFS Server in our environment, but anybody who does (likely somebody with a tightly integrated windows/linux environment), will want to read up on this vulnerability, since it's wormable. 

Fourth up, there's an elevation of privilege vulnerability affecting all supported versions of Exchange Server ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-21978).  This one requires administrator roles and rights on the exchange server (and the host it's on), but can potentially elevate a user's rights to the domain administrator level.  Naturally, we'll be patching all managed exchange servers tonight.


There's more, as usual, but those are the ones of particular note.


Impact of Work:


All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 10:15PM, with some exceptions.

Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.  Mail delivery to our helpdesk may be temporarily halted while our mail servers are updated as part of this patch cycle.  If you receive a delivery failure, you can still reach us by logging directly into the helpdesk and submitting a ticket directly via the portal, or calling us at 303-414-6910 x2, for emergencies.  


Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.  Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density.

Hypervisors in DR scenarios will be updated one hour early, as they are not running active workloads.

Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.


Read more »



Apr
25
Date: April 28, 2022
Time: 9:00PM MDT - 1:00 AM MDT

Purpose of Work:
Upgrade dist1.dtc1 and dist2.dtc1 to latest recommended versions of JunOS.

Impact of Work:
These two switches function as a redundant pair.  Maintenance work will be performed on one switch at a time.  During the maintenance window, there may be a few periods of increased packet loss and latency when switch reboots are performed and routes converge.
Read more »



Apr
21

Update: 
12:36AM MDT - Final checks are all clear.  We are closing this maintenance window as complete and successful, with no customer impact.

Update:
12:30AM MDT - Our work is complete and we are making final checks.

Update:
The problematic switch is almost completely isolated.

Reminder:
This activity will begin in about 15 minutes.

Date: 
April 21, 2022
Time: 9:00 PM MDT - 1:00 AM MDT

Purpose:

Troubleshoot and resolve issues with dist3.dtc1 fpc1, which is throwing parity errors.  The cause of parity errors, according to Juniper, maybe the following reasons:

1. Emission of alpha particles from tiny amounts of radioactive materials present in the chips
2. Cosmic rays creating energetic neutrons and protons
3. A bug in the current version of JunOS running on the switch.

Description of Work:
During this maintenance window, we will physically isolate the impacted node and reboot it.  While this is a redundant node, and we don't expect any major network disruptions, this is an unusual problem we haven't faced before.  While the node is isolated, there may be brief periods of connectivity disruptions while traffic is rerouted, which may impact iSCSI traffic for certain customers.  Once the node is physically isolated, we will reboot it.  If the errors resolve themselves, we will re-establish connectivity to the device.  If the errors continue, we will physically replace the device with a spare.
Read more »



Apr
12

Purpose of Work:

April's Patch Tuesday is here, and while there's definitely some vulnerabilities of concern being addressed in this drop, only one is being actively exploited.

The patch volume is average, for this time of year (if a little high in critical and important security updates), and no widespread issues have been reported with the patches.


A few of the particularly concerning ones:

First off, we have a system-level remote code execution vulnerability that leverages the Remote Procedure Call Runtime, and affects seemingly all supported versions of windows ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-26809 ).  This one is particularly concerning, because it doesn't require authentication, and any code executed would automatically execute in a system context, making it a wormable vulnerability that could very quickly compromise an unpatched network of windows hosts if it were exploited.  This one is not yet in the wild, so we'll be following our usual patching schedule to deal with it.  If you are unable to patch this on on a self-managed environment, our recommendation is you block port 445 from untrusted hosts.

Second on the shortlist is a local elevation of privilege vulnerability that leverages the "Common Log File System Driver", also affecting seemingly all support versions of windows ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-24521 ).  This requires local login with a low-privilege user, so it's only dangerous when paired with an RCE exploit.  However, it has been detected as actively exploited, so this is one self-managed environments will want to ensure is patched ASAP, as it could be the difference between a compromised website and a ransomware situation.

Third up, we have several system-level local code execution vulnerabilities that leverage the Hyper-V service, on Windows 2012+ between the three of them ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-24537 https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-23257 https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-22008 ).  These vulnerabilities appear capable of jumping the guest/HV divide, requiring interaction by a low-privileged user on the guest, and a race condition to pass ( introducing an element of random chance) to exploit.  However, with enough tries, this could pose a hazard to Hyper-V environments very easily.  It is recommended that all Hyper-V nodes are updated ASAP; we'll be doing that for those we control scheduling for.

Fourth on the list two seemingly related RCE vulnerabilities, leveraging SMB Server and SMB Client on all supported versions of windows ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-24541 https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-24500 ).  Both vulnerabilities appear to require low-privilege user interaction with a malicious SMB server to compromise either component, so this is one that's likely to be used in phishing attacks and malicious websites.

Fifth, there's a publicly disclosed elevation of privilege vulnerability leveraging the User Profile Service, and affecting all supported versions of windows.  Despite being publicly disclosed, Microsoft reports no use of an exploit detected in the wild yet... though it's only a matter of time.  The attack complexity for this one is high, requiring the attacker to win a race condition (more or less, a dice roll influenced by the order of events in a target responding to the exploit).  However, a determined attacker with a non-privileged user foothold could certainly use this to compromise a host on a system level with time.

Impact of Work:


All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 9:30PM, with some exceptions.

Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.  Mail delivery to our helpdesk may be temporarily halted while our mail servers are updated as part of this patch cycle.  If you receive a delivery failure, you can still reach us by logging directly into the helpdesk and submitting a ticket directly via the portal, or calling us at 303-414-6910 x2, for emergencies.  


Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.  Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density.

Hypervisors in DR scenarios will be updated one hour early, as they are not running active workloads.

Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.


Read more »



Mar
8

Purpose of Work:

March's Patch Tuesday has arrived, and while there's a few vulnerabilities of note, none are being actively exploited at time of writing.  The patch volume is average, for this time of year, and no widespread issues have been reported with the patches.

A few of the highlights: 

First up, there's an RCE vulnerability affecting Remote Desktop Client on Windows 10-11 (all builds) and Server 2016, respectively ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-21990 ).  This one has been publicly disclosed, but seems to require user interaction; somebody would have to actively connect to a malicious RDP server in order for it to be successfully exploited.  That said, once it's being exploited in the wild, expect higher than average social engineering attempts or malicious rdp files targeting end users, and patch accordingly.

Secondly, there's an RCE vulnerability affecting all supported versions of Microsoft Exchange ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-23277 ).  This vulnerability requires authentication, so it's not going to be an extremely widespread risk, even when it's in the wild.  However, we'll still be patching it tonight, on supported workloads.

Third up, there's an RCE vulnerability affecting SMBv3 (both the server and client component) on Windows 10-11 (all builds after 20H2) and Windows Server 2022 ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-24508 ).  This vulnerability would require low privileges to exploit, meaning it will not be easily wormable.  Still, it could result in a compromised user account turning into a compromised server account (since SMB server runs as SYSTEM), so it is recommended this vulnerability is patched ASAP for affected OS.  Conversely, a compromised server could turn into compromised user endpoints.  There does seem to be a workaround that mitigates this vulnerability, for those not looking to patch their workloads tonight.

Fourth of all, there's a DOS vulnerability affecting Hyper-V on server 2016 and up ( https://msrc.microsoft.com/update-guide/vulnerability/CVE-2022-21975 ).  The details on this one are thin, and attack complexity is cited as 'high', with the added note that 'Successful exploitation of this vulnerability requires an attacker to win a race condition.', meaning: it will not be reproducible every time, and there's some random element to the sequencing of events that leads to a successful attack.  Still, I recommend those who won't be getting their hyper-V workloads patched tonight, plan to do so soon, given the stakes of having that workload brought offline from a single compromised VM.



Impact of Work:


All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 9:30PM, with some exceptions.

Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.  Mail delivery to our helpdesk may be temporarily halted while our mail servers are updated as part of this patch cycle.  If you receive a delivery failure, you can still reach us by logging directly into the helpdesk and submitting a ticket directly via the portal, or calling us at 303-414-6910 x2, for emergencies.  


Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.  Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density.

Hypervisors in DR scenarios will be updated one hour early, as they are not running active workloads.

Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.


Read more »