RSS Feed
Latest Updates
Feb
9

Purpose of Work:

February's Patch Tuesday is underway, and there are some notably bad vulnerabilities this time around.

First and foremost, there's CVE-2021-24078This one is a remote code execution vulnerability that affects a privileged service, the DNS server, on Server 2008+.  With a low attack complexity that can be done over the network, and requiring no user interaction, this is likely a wormable vulnerability, and any code run using it will run in the System context automatically.   Microsoft has yet to provide a mitigation or workaround in their executive summary.  To those fully managed customers yet to move away from a host running Windows Server 2008 R2 or earlier: these kinds of vulnerabilities are exactly the reason to move away from end-of-life operating systems that will not be patched ASAP.

Second, there's CVE-2021-24094 and CVE-2021-24074.  These are both remote code execution vulnerabilities affecting the TCP/IP Stack (ipv4 and ipv6, respectively) on Server 2008+.  Like the previous vulnerability, these are pre-authentication, network-accessible vulnerabilities that would allow injected code to run in a privileged service context.   What's different here is that Microsoft has provided workarounds in their executive summary for both, and by default, the IPv4 mitigation should already be in place.  We look into confirming this on all managed hosts we are unable to patch this cycle.

Third, there's CVE-2021-26701, a remote code execution vulnerability affecting .net 5.0 and certain versions of  .net core.  There is little information about this one, but the attack complexity is high, and this likely will result in websites using the listed frameworks being easily compromised and used to attempt to take over a host.  The vulnerability does not include escalation of privilege on its own, unlike the last two.

Fourth, there's CVE-2021-1732an escalation of privilege exploit leveraging the kernel on Server 2019+ and Windows 10 1803+.  This one appears to have functional exploits that have already been detected in the wild, making it truly 0-day.  We will be patching it tonight alongside the rest.



Impact of Work:

All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 11:15PM, with some exceptions.

Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.


Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.  Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density.

Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.


Read more »



Jan
18
Network Maintenance - January 22, 2021
Posted by Jay Sudowski on 18 January 2021 04:19 PM
Date: January 22, 2021 
Time: 8:00 PM - 12:00 AM (Mountain Standard Time)

Purpose of Work:
On Friday, January 22 we will be continuing the network maintenance started the previous night.  The work conducted during this maintenance window will complete the physical removal of dist3.denver2 from our network, assuming such work was not completed the previous night.

Impact of Work:
When we remove dist3.denver2 from the network, there may be a few brief periods of latency and packet loss impacting the whole network while we turn down links connecting dist3.denver to our network and repatch those links to dist3.dtc1, as needed.
Read more »



Jan
18
[COMPLETED] Network Maintenance - January 21, 2021 8pm - 12am
Posted by Jay Sudowski on 18 January 2021 04:17 PM
Update 11:05 PM - We are complete with our network maintenance for the evening.  We will conduct the physical topology changes required tomorrow evening.  Please contact our helpdesk for any issues.

Date: January 21, 2021 
Time: 8:00 PM - 12:00 AM (Mountain Standard Time)

Purpose of Work:
We will be performing the work necessary to remove our 1801 California Street / Downtown Denver Data Center from our dark fiber ring.  The work that will be conducted Thursday night is as follows:

1. Groom VLANs from dist3.denver2 to dist3.dtc1.  During this work, our networking team will logically migrate the layer 3 interfaces for routed VLANs that are still homed to dist3.denver2.  We will be performing this work gradually, migrating a few VLANs every few minutes.

2. If time allows, we will move on to physically removing the switch stack from our network.


Impact of Work:

During the VLAN grooming work, there will be a short 3-5 minute network disruption on a per-network basis as the layer 3 interfaces are shutdown on dist3.denver2 and activated on dist3.dtc1.

When we remove dist3.denver2 from the network, there may be a few brief periods of latency and packet loss impacting the whole network while we turn down links connecting dist3.denver to our network and repatch those links to dist3.dtc1, as needed.
Read more »



Jan
12

Completion [Thu 14 Jan 2021 12:17:47 AM MST]  Our post update night audit reveals that the vast majority of servers successfully updated overnight.  A few one-off reboots will need to be conducted from here, but they will be done after-hours or scheduled separately where needed.

Purpose of Work:

2021's first patch Tuesday has arrived, and there's a few notable vulnerabilities to be dealt with this time around.

First off, there's a publicly-disclosed escalation of privilege vulnerability affecting windows 2012+ that was introduced with a patch for a similar vulnerability.  This publicly disclosed vulnerability has a proof-of-concept exploit, and will likely be exploited in the wild, soon: https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2021-1648

Secondly, there's a remote desktop security feature bypass vulnerability affecting windows 2012+.  Microsoft has yet to disclose what is being bypassed, but the high CVSS score of 8.8/10 and low complexity is worrisome, implying perhaps the ability to spoof or bypass an authentication: https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2021-1674

Third, Microsoft has found a remote code execution vulnerability in Windows Defender that was being exploited in the wild upon their discovery.  This one has likely already been patched on any host with internet access and the default automatic definition updates that windows defender has, but is worth mention for those who have overridden those settings.  Fully managed clients should be covered already. https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2021-1647

Fourth, there is a Hyper-V escalation of privilege vulnerability affecting server 2012+.  Details are sparse, but it does appear to require local access to the host.  That said, any hypervisor running a web-accessible service might be at risk: https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2021-1704
.
And fifth of all, there is a remote code execution vulnerability affecting the RPC service on 2008+ that can be done over the network and has a CVSS score of 8.8, as well as low privilege requirement.  While that doesn't seem to be wormable (since it require some level of privileges), it is worth patching ASAP.  There are actually a few CVEs for similar exploits: https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2021-1658 https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2021-1660 https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2021-1702 


Impact of Work:

All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 11:15PM, with some exceptions.

Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.


Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.  Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density.

Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.


Read more »



Dec
8
Completion [Wed 09 Dec 2020 11:35:49 PM MST]  Our post update night audit reveals that the vast majority of servers successfully updated overnight.  A few one-off reboots may need to be conducted from here, but they will be done after-hours, and not on hypervisor hosts or tonight.


Update, [Tue 08 Dec 2020 10:31:19 PM MST]

Correction: "All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 9:30PM, with some exceptions."

10:30PM.  Reboots will begin shortly.


Purpose of Work:

December's Patch Tuesday has come around, and while it's a lighter patch volume than usual, there's still vulnerabilities that all subscribers running windows should be aware of.  


First of all, there are several RCE vulnerabilities for Microsoft exchange which seem to require authentication to be leveraged.  Here's one example: https://msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-17132

Second, there appears to be a Hyper-V vulnerability that allows Hyper-V guests to force the Hypervisor to run arbitrary code by sending it an invalid SMB packet.  This affects Server 2016+, and I would consider it a higher priority patch, of those I see this cycle: https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17095

Third, there is another Kerberos security feature bypass vulnerability to be patched.  We'll be applying and testing this on our internal domains that many managed servers are a part of: https://msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-16996

Fourth, there's an RCE/EOP vulnerability that requires SMB read access to a host (which means it could affect any host that allows SMB, but not NTFS access to the 'everyone' identity).  Once an adversary has this level of access, they can then send specially crafted packets over the network or locally to get the affected host to run arbitrary code as the system identity.  This affects Server 2012 and up: https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17096



Impact of Work:

Our exchange host will be rebooted at least once tonight to propagate security fixes.  This may interfere with our ability to send and receive mail intermittently, while patches are being applied.  A direct ticket update via the helpdesk portal will still work, as will a phone call.


All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 9:30PM, with some exceptions.

Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.


Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.  Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density.

Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.
Read more »