RSS Feed
News
Sep
19
Power Incident @ Sept 19, 1:06 PM MDT
Posted by Jay Sudowski on 19 September 2020 01:47 PM
Date: September 19, 2020

Time: 1:06 PM MDT

At approximately 1:06PM MDT, our DTC data center experienced a brief power outage on one side of the critical load. We are having all available staff meet on-site to triage and remediate any remaining service impacting issues.

Update: 2:19 PM MDT - Here is the latest information from H5:

Please note that during our planned maintenance activity we experienced a brief interruption to load supported UPS2 system. No impact was seen to UPS system 1. All dual corded critical loads would have remained in service. Loads are currently supported on Generator power at this time.

We have a large staff presence on-site at the moment triaging issues with any single corded loads.  If you are experiencing a service disruption, please open a ticket so we can address it.
Read more »



Sep
8

Update 3 (Thu 10 Sep 2020 01:59:38 AM MDT)
All updates scheduled for tonight are complete.  A few servers remain without special scheduling consideration, and will be done tomorrow night.  Customers will be alerted directly, where required, as this is a smaller subset of hosts.


Update 2 (Wed 09 Sep 2020 08:57:56 PM MDT):
A subset of servers were missed last night, and our exchange host will require more updating.

We will begin these updates shortly; you may notice a reboot on your windows server, or some temporary delivery deferrals for our helpdesk.

Update 1 (Wed Sep  9 00:50:34 MDT 2020):
Automatic reboots of hosts in the discussed scope have begun, and will be occurring over the next few hours.  We will monitor servers to ensure they come back up without incident.

Purpose of Work:

This patch Tuesday has a few highlights, in addition to various, less noteworthy security updates.

The first is a system-level RCE vulnerability affecting all recent versions of exchange server, which we will be mitigating on our internal server ASAP, overnight.  https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-16875

The second is an RCE vulnerability affecting all supported versions of windows server.  This one leverages a COM interaction with Javascript, and thus could affect any RDS server or webserver where a user or application pool might end up opening a maliciously crafted file: https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-0922

This would result in code being run as the identity of whatever user opened said file.

The third is an RCE vulnerability affecting Server 2016 and up that leverages how  Microsoft Windows Codecs Library handles objects in its memory.  Again, webservers and RDS servers would be particularly vulnerable to this: https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-1129



Standalone hypervisors would be a general exception to this, and customer-owned Windows HVs that host unmanaged VMs, but also run Windows 2012+ should have their maintenance scheduled with us, separately.

Customers with their own update infrastructure will also be scheduled separately.


We will update you as maintenance begins.


Impact of Work:

Our exchange host will be rebooted a few times tonight to propagate security fixes.  This may interfere with our ability to send and receive mail intermittently, while patches are being applied.


All affected hosts will be rebooted automatically / ASAP to propagate fixes, starting at 11:30PM on 9/8/20.

Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.

Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.

Any hosts not on our fully-managed domain (usually because they have their own domain) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.


Read more »



Aug
20
Completion ( Sun 23 Aug 2020 10:08:10 PM MDT ) All updates to production non-cluster servers that did not require a separate maintenance window are complete.

Update 3 (Sat 22 Aug 2020 21:30:24 PM MDT)
After 10 PM MDT tonight we will be continuing with updates, including updates to some standalone hypervisors. We will be monitoring and ensuring that servers come back up after the reboots.

Update 2 (Fri 21 Aug 2020 00:35:56 PM MDT)
Automatic reboots of hosts in the discussed scope have completed for tonight and all servers have come back up. We will proceed with additional updates tomorrow evening after 10 PM MDT.

Update 1 (Thu 20 Aug 2020 10:24:16 PM MDT):

Automatic reboots of hosts in the discussed scope have begun, and will be occurring over the next hour.  We will monitor servers to ensure they come back up without incident.

Purpose of Work:
A privilege elevation vulnerability (CVE-2020-1530 and CVE-2020-1537) affects all supported versions of windows server so far.  This vulnerability exists when Windows Remote Access improperly handles memory or file operations. The exploit requires an attacker to have execution capabilities on the victim system. Systems hosting websites or with web-accessible services are particularly vulnerable.

Due to the ability of this vulnerability to allow privilege escalation and the wide attack surface, we will be patching and rebooting all affected, fully-managed hosts overnight.  

Standalone hypervisors would be a general exception to this, and customer-owned Windows HVs that host unmanaged VMs, but also run Windows 2012+ should have their maintenance scheduled with us, separately.

Customers with their own update infrastructure will also be scheduled separately.


You can read more about the exploits (and patches mitigating it), here: https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-1530 https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-1537

We will update you as maintenance begins.


Impact of Work:
All affected hosts will be rebooted automatically / ASAP to propagate fixes, starting at 10:10PM MDT on Thursday the 20th.

Internal systems on Windows 2012 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.

Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.

Any hosts not on our fully-managed domain (usually because they have their own domain) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.




Read more »



Jul
14
End of Maintenance, 2:37am:
All fully-managed hosts that do not have special scheduling requirements are either updated, or well on their way to being updated.  We will reach out to clients with special scheduling requirements, and audit all hosts that were automatically updated tonight to confirm patching compliance.  The bulk of the remaining automatic reboots should be finished within 30 minutes.


Update, 12:05am:
Updates have been deadlined, and servers meeting the previously outlined criteria will begin to reboot shortly.  We will monitor all servers to ensure they come back up normally, and install the update properly.



Purpose of Work:
A remote code execution vulnerability (CVE-2020-1350) affects all versions of the  DNS Server role in all versions of windows server so far.  This vulnerability is wormable, and would run under the system context, giving any attackers full control of an affected DNS host.

Due to the ability of this vulnerability to quickly spread through and take control of any hosts running the DNS Server service on a network, we will be patching and rebooting all affected, fully-managed hosts overnight.  

Standalone hypervisors would be a general exception to this, and customer-owned Windows HVs that host unmanaged VMs, but also run a DNS server should have their maintenance scheduled with us, separately.

Customers with their own update infrastructure will also be scheduled separately.


You can read more about the exploit (and patches mitigating it), here: https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2020-1350

We will update you as maintenance begins.


Impact of Work:
All affected hosts will be rebooted automatically / ASAP to propagate fixes, starting at 10:30PM MDT on Tuesday the 14th.

Internal systems on windows 2008 and up (such as the management portal) may be temporarily impacted in the time it takes to reboot them.

Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.

Any hosts not on our fully-managed domain (usually because they have their own domain) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.

Read more »



Jun
26
cPanel users, please read - cPanel-Security Notice TSR-2020-0003
Posted by Anthony Kolka on 26 June 2020 10:16 AM

Tickets from several clients have made us aware of a phishing scam.
Several clients are receiving an email that appears to be from cpanel with the subject of "cPanel-Security Notice TSR-2020-0003" this advisory was sent out originally in May.
The email making its rounds now contains a large orange button that says "Update your cPanel & WHM Installations". This link takes you to a Chinese phishing site where they attempt to obtain your cPanel credentials.
Please do not click that link, and remember to be wary of and check the destination of any links or buttons you recieve in emails.


Read more »



Jun
5
myLittleAdmin vulnerability on Plesk with MSSQL
Posted by Anthony Kolka on 05 June 2020 09:56 AM

The DB GUI access software myLittleAdmin that is installed along with MSSQL on plesk installations has been found to have a severe remote access and rights escalation vulnerability.

Plesk recommends uninstalling myLittleAdmin completely. We are in the process of performing this action for all fully managed clients using Plesk.

https://support.plesk.com/hc/en-us/articles/360013996240
https://ssd-disclosure.com/ssd-advisory-mylittleadmin-preauth-rce/


Read more »