RSS Feed
News
Jan
18
Network Maintenance - January 22, 2021
Posted by Jay Sudowski on 18 January 2021 04:19 PM
Date: January 22, 2021 
Time: 8:00 PM - 12:00 AM (Mountain Standard Time)

Purpose of Work:
On Friday, January 22 we will be continuing the network maintenance started the previous night.  The work conducted during this maintenance window will complete the physical removal of dist3.denver2 from our network, assuming such work was not completed the previous night.

Impact of Work:
When we remove dist3.denver2 from the network, there may be a few brief periods of latency and packet loss impacting the whole network while we turn down links connecting dist3.denver to our network and repatch those links to dist3.dtc1, as needed.
Read more »



Jan
18
Network Maintenance - January 21, 2021 8pm - 12am
Posted by Jay Sudowski on 18 January 2021 04:17 PM
Date: January 21, 2021 
Time: 8:00 PM - 12:00 AM (Mountain Standard Time)

Purpose of Work:
We will be performing the work necessary to remove our 1801 California Street / Downtown Denver Data Center from our dark fiber ring.  The work that will be conducted Thursday night is as follows:

1. Groom VLANs from dist3.denver2 to dist3.dtc1.  During this work, our networking team will logically migrate the layer 3 interfaces for routed VLANs that are still homed to dist3.denver2.  We will be performing this work gradually, migrating a few VLANs every few minutes.

2. If time allows, we will move on to physically removing the switch stack from our network.


Impact of Work:

During the VLAN grooming work, there will be a short 3-5 minute network disruption on a per-network basis as the layer 3 interfaces are shutdown on dist3.denver2 and activated on dist3.dtc1.

When we remove dist3.denver2 from the network, there may be a few brief periods of latency and packet loss impacting the whole network while we turn down links connecting dist3.denver to our network and repatch those links to dist3.dtc1, as needed.
Read more »



Jan
12

Completion [Thu 14 Jan 2021 12:17:47 AM MST]  Our post update night audit reveals that the vast majority of servers successfully updated overnight.  A few one-off reboots will need to be conducted from here, but they will be done after-hours or scheduled separately where needed.

Purpose of Work:

2021's first patch Tuesday has arrived, and there's a few notable vulnerabilities to be dealt with this time around.

First off, there's a publicly-disclosed escalation of privilege vulnerability affecting windows 2012+ that was introduced with a patch for a similar vulnerability.  This publicly disclosed vulnerability has a proof-of-concept exploit, and will likely be exploited in the wild, soon: https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2021-1648

Secondly, there's a remote desktop security feature bypass vulnerability affecting windows 2012+.  Microsoft has yet to disclose what is being bypassed, but the high CVSS score of 8.8/10 and low complexity is worrisome, implying perhaps the ability to spoof or bypass an authentication: https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2021-1674

Third, Microsoft has found a remote code execution vulnerability in Windows Defender that was being exploited in the wild upon their discovery.  This one has likely already been patched on any host with internet access and the default automatic definition updates that windows defender has, but is worth mention for those who have overridden those settings.  Fully managed clients should be covered already. https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2021-1647

Fourth, there is a Hyper-V escalation of privilege vulnerability affecting server 2012+.  Details are sparse, but it does appear to require local access to the host.  That said, any hypervisor running a web-accessible service might be at risk: https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2021-1704
.
And fifth of all, there is a remote code execution vulnerability affecting the RPC service on 2008+ that can be done over the network and has a CVSS score of 8.8, as well as low privilege requirement.  While that doesn't seem to be wormable (since it require some level of privileges), it is worth patching ASAP.  There are actually a few CVEs for similar exploits: https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2021-1658 https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2021-1660 https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2021-1702 


Impact of Work:

All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 11:15PM, with some exceptions.

Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.


Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.  Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density.

Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.


Read more »



Dec
8
Completion [Wed 09 Dec 2020 11:35:49 PM MST]  Our post update night audit reveals that the vast majority of servers successfully updated overnight.  A few one-off reboots may need to be conducted from here, but they will be done after-hours, and not on hypervisor hosts or tonight.


Update, [Tue 08 Dec 2020 10:31:19 PM MST]

Correction: "All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 9:30PM, with some exceptions."

10:30PM.  Reboots will begin shortly.


Purpose of Work:

December's Patch Tuesday has come around, and while it's a lighter patch volume than usual, there's still vulnerabilities that all subscribers running windows should be aware of.  


First of all, there are several RCE vulnerabilities for Microsoft exchange which seem to require authentication to be leveraged.  Here's one example: https://msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-17132

Second, there appears to be a Hyper-V vulnerability that allows Hyper-V guests to force the Hypervisor to run arbitrary code by sending it an invalid SMB packet.  This affects Server 2016+, and I would consider it a higher priority patch, of those I see this cycle: https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17095

Third, there is another Kerberos security feature bypass vulnerability to be patched.  We'll be applying and testing this on our internal domains that many managed servers are a part of: https://msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-16996

Fourth, there's an RCE/EOP vulnerability that requires SMB read access to a host (which means it could affect any host that allows SMB, but not NTFS access to the 'everyone' identity).  Once an adversary has this level of access, they can then send specially crafted packets over the network or locally to get the affected host to run arbitrary code as the system identity.  This affects Server 2012 and up: https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17096



Impact of Work:

Our exchange host will be rebooted at least once tonight to propagate security fixes.  This may interfere with our ability to send and receive mail intermittently, while patches are being applied.  A direct ticket update via the helpdesk portal will still work, as will a phone call.


All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 9:30PM, with some exceptions.

Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.


Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.  Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density.

Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.
Read more »



Nov
10

[Complete, Wed 11 Nov 2020 11:03:13 PM MST] Patch compliance for managed hosts without special scheduling requirements is at 100%.  Further off-schedule update reboots should not be necessary.

[Update, Wed 11 Nov 2020 01:47:16 AM MST]:
Our exchange server has finished applying the latest CU, and mail service is restored.  There will be a few minor updates that will be applied to it over the course of the night, but further instances of downtime should be limited to 10m or less.

Regarding https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17051: most if not all managed clients should be unaffected by it, as this is for the "Server for NFS" linux compatibility feature of the Windows File Server role, which is pretty niche and not installed by default. 

Service-impacting reboots on non-hypervisors will continue as the night progresses, and stop before 5am MST.  Tomorrow night, we will address hosts that failed to install this round of updates and shoot for 100% patch compliance in our managed environment.


[Update, Tue 10 Nov 2020 10:40:51 PM MST]:
Updating efforts are well on their way, with many hosts having already rebooted.  I will now begin the work on our exchange server, which means mail to our helpdesk sent through email clients will be temporarily deferred.  You can still update helpdesk tickets through a direct login, or call us on our support line.

Purpose of Work:

Patch Tuesday is here for November, and there's a few standout vulnerabilities among the crowd of those we'll be dealing with on fully managed servers that all subscribers running windows should be aware of.  


First of all, there's an Escalation of Privilege vulnerability currently being exploited in the wild that leverages the Kernel Cryptography driver on Windows Server 2008+.  This one was publicly disclosed by google in Late October, and will be a priority this patch cyle: https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17087

Second, there's a remote code execution vulnerability Exchange server. We'll be applying this security update, so you might not be able to reach us over email for a short period during the maintenance: https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17084

Third, there's an RCE vulnerability affecting NFS on Windows Server 2008+.  There's not alot of public detail out there for this one, but it has a CVSS of 9.8, leverages NFS, has low attack complexity, and does not require privileges or user interaction in its description... so suffice to say, it's looking like a wormable threat: https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17051

Fourth, there's a 'security feature bypass' Vulnerability in Hyper-V for Windows Server 2012R2+.  While there's not alot of detail for what it is, the network attack vector, CVSS score of 6.5, low attack complexity, and lack of need for privileges or user interaction are noteworthy, and may imply the ability for VMs to modify the hypervisor directly, which is reason enough to include Hyper-V servers in this update cycle.  https://msrc.microsoft.com/update-guide/en-US/vulnerability/CVE-2020-17040

Fifth, there's a Kerberos Security Feature bypass vulnerability that would affect Active Directory.  There's not alot of info out for this one yet, but it is the sort of patch that requires additional mitigation in the form of a registry edit to enforce new Kerberos Ticket security signature standards on domain controllers, so we'll be assessing this one in our internal AD domains and reaching out to managed clients who have their own with our findings.

While that's what I would consider the highlights, there are a few other fixes that would impact 2019+ servers, including a few EOP vulnerabilities.  All vulnerabilities will be patched tonight, with rollbacks occurring selectively if required.



Impact of Work:

Our exchange host will be rebooted a few times tonight to propagate security fixes.  This may interfere with our ability to send and receive mail intermittently, while patches are being applied.


All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 9:30PM, with some exceptions.

Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.


Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.  Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density.

Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.


Read more »



Oct
13

[Completion, Sun 18 Oct 2020 11:40:29 PM MDT] A final update audit confirms that all managed hosts with applicable security updates are either updated successfully, or scheduled to be soon.  Several procedural improvements were made to streamline and speed up the handling of future Patch Tuesdays that seem to involve an unusually high number of zero-day exploits, as this one did.

[Update 2, Fri 16 Oct 2020 11:26:55 PM MDT] A subset of fully managed windows servers outside of our usual update management solution have been brought into it; some reboots will happen for those of these that did not receive the security updates on Tuesday night.  We will perform a final audit tomorrow night before considering this window for out-of-band overnight maintenance reboots closed.

[Update 1]: About half of domain-managed VMs have been updated.  We will continue to monitor hosts that have yet to update to ensure they come up quickly, and manually update where required. 

All reboots will be finished before business hours.

Updates will also continue tommorow night.  Managed servers not on the domain will be the primary focus, at that point.


Purpose of Work:

Patch Tuesday has come around again, and there's a few standout vulnerabilities among the crowd of those we'll be dealing with on fully managed servers that all subscribers running windows should be aware of.  


First off, there's a remote code execution vulnerability affecting Windows Server 2019 hosts (and more recent versions of windows 10).  This one leverages the TCP/IP stack within windows, via specially crafted ICMPv6 Router Advertisement packets.  Because it leverages the TCP/IP stack, it would execute any injected code in the SYSTEM identity context, which means it's a wormable, EOP/RCE exploit you'll want to patch well before exploits are developed using it: https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-16898 

Second, there's a remote code execution vulnerability affecting Hyper-V on Windows Server 2008+ hosts.  This one could allow an attacker on a VM to run a specially crafted application that then forces the hypervisor itself to execute arbitrary code in an elevated user context.  Hyper-V host owners will want to patch this out ASAP: fully managed customers with hyper-V servers that have special scheduling requirements will contacted specifically about this one. https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-16891

Third, there's a privilege escalation vulnerability affecting Windows Server 2016+ (and windows 10) that leverages Windows Error Reporting.  All privilege escalation vulnerabilities are a big risk on webservers, and other such hosts with a large, publicly accessible attack surface area.  https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-16909

Fourth up, there's another privilege escalation vulnerability affecting Windows Server 2008+.  This one leverages the Windows Network Connections service, and of course should be addressed ASAP, like with most EOP vulnerabilities: https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-16887



Impact of Work:

Our exchange host will be rebooted a few times tonight to propagate security fixes.  This may interfere with our ability to send and receive mail intermittently, while patches are being applied.


All affected hosts that are 2012 and up will be rebooted automatically / ASAP to propagate fixes, starting at 9:00PM on 10/13/20, with some exceptions.

Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.


Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.  Hypervisors not in a failover cluster will either be updated overnight, or have their updates scheduled, depending on customer policy / VM density.

Any hosts where updates are managed directly by the customer (or an approval process is required for zero-day updates) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.


Read more »