RSS Feed
News
Jul
12
Nimble SAN Firmware Upgrade - July 16, 2019 (COMPLETE)
Posted by Tyler Molamphy on 12 July 2019 02:28 PM
[Completed, 23:21 7-16-19]:  This maintenance is complete on all Nimble Storage Arrays.  As expected, no impact to workloads was observed. 

Date: 
 Tuesday, July 16, 2019
Time:  10:30PM MT

Purpose of Work:
Nimble Storage has let us know that a couple of our storage arrays are running on a version of NimbleOS that has a software defect which could lead to unexpected storage behavior in some circumstances.  We will be upgrading our storage arrays to newer firmware that resolves this issue.

Impact of Work:
Historically, we have performed many Nimble Storage upgrades which have been un-noticeable to any workloads consuming storage from our arrays.  Accordingly, we do not anticipate any impact, but there is always a slim possibility that connectivity to the storage arrays will be disrupted, resulting in workloads relying on the storage to hard reboot.

Please contact us with any questions / comments / concerns.
Read more »



Jun
19
[Complete] Zero-Day Emergency Security Patching for SolusVM HV - June 27, 2019
Posted by David Cunningham on 19 June 2019 10:28 PM
[Thu Jun 27 3:19:12 MDT 2019] - All VMs have been returned to service; we will be monitoring these VMS to see if anything else needs to be done, beyond syncing server times.


[Thu Jun 27 3:09:40 MDT 2019] - The SolusVM HV is back up, and VMs are being restored from saved states.  If your VM at 23.239.222.0/25 is not yet up, it should be, shortly.   I will update this ticket when all is resolved.


[Thu Jun 27 1:58:51 MDT 2019] - Our Solus HV did not recover as expect on reboot.   I have arrived at our DTC location and am troubleshooting it.  A conservative ETR based on what I know now is approximately 1 hour.   I will update you with further developments that affect this estimated time of resolution.

[Thu Jun 27 00:19:52 MDT 2019] - Maintenance is underway.  We will proceed with update and reboot on the SolusVM hypervisor shortly.

[Tue Jun 25 23:03:12 MDT 2019] - 
SolusVM hypervisor updates will be postponed until tomorrow night, as out of band access is not functioning as expected.  We will update you as we begin.

[Tue Jun 25 21:31:56 MDT 2019] - We will be initiating maintenance on our SolusVM hypervisor shortly.  Again, if you do not have a non-"Highly Available" cPanel server (often immediately identifiable by having only 1-3 IPs, in the subnet 23.239.222.0/25), this will not impact you.



Read more »



May
14

[Update, Wed May 15 03:21:50 MDT 2019] All reboots for emergency updates are complete. We will be reaching
out to those with fully managed services and their own Active Directory domains separately in the morning. It is highly
recommended that all clients using Windows server 2003-2008R2 on self-managed hosts push through the updates securing a
gainst CVE-2019-0708 ASAP.

[Update, Wed May 15 01:38:19 DST 2019]   While all updates were initiated by the deadline, several are still completing.  Reboots may extend into the next couple of hours, but will be over before business hours, MDT. 

Date: 
 May 14, 2019
Time:  9:30PM MDT - 11:30PM MDT


Purpose of Work:
A pre-authentication vulnerability targeting Remote Desktop Protocol in servers running Windows Server 2008 R2 or older has been discovered that allows for Remote Code Execution.

Because the vulnerability requires absolutely no authentication, it could be spread rapidly within a network via use of 'Worm' style malware, at which point the exploiter would effectively have full control of all infected hosts.

Due to the ease of exploitation, and the impact of exploitation, we will be patching and rebooting all affected, fully-managed hosts overnight.

You can read more about the exploit (and patches mitigating it), here: https://blogs.technet.microsoft.com/msrc/2019/05/14/prevent-a-worm-by-updating-remote-desktop-services-cve-2019-0708/

Impact of Work:
Windows 2003 hosts on our domain will be rebooted immediately to propagate a manual update, and 2008 / 2008 R2 hosts on our domain will be scheduled to reboot later tonight, after installing the updates automatically.

Any hosts not on our fully-managed domain (usually because they have their own domain) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.


Read more »



Apr
23
Nimble SAN Firmware Upgrade - April 24, 2019
Posted by Pete Carstensen on 23 April 2019 12:50 PM
[Completed, 23:22 4-24-19]:  This maintenance is complete on all Nimble Storage Arrays.  As expected, no impact to workloads was observed.  

Date: 
April 24, 2019
Time:  10:30PM MT

Purpose of Work:
Nimble Storage has let us know that a couple of our storage arrays are running on a version of NimbleOS that has a security vulnerability that could result in remote code executions and remote denial of service attack.  We will be upgrading our storage arrays to newer firmware that is not impacted by this vulnerability. (CVE-2017-9765)

Impact of Work:
Historically, we have performed many Nimble Storage upgrades which have been completely hitless and not at all noticeable to any workloads consuming storage from our arrays.  Accordingly, we do not anticipate any impact, but there is always a slim possibility that connectivity to the storage arrays will be disrupted, resulting in workloads relying on the storage to hard reboot.

Please contact us with any questions / comments / concerns.
Read more »



Apr
12
Network Disruption RFO / RCA - April 11, 2019
Posted by Arick Francis on 12 April 2019 05:40 PM
Unfortunately, Yesterday at 10:46 AM to 11:11 AM Mountian Time a network disruption occurred. We have prepared an RFO / RCA detailing the incident. We sincerely apologize for any inconvenience the network disruption has caused, and as always we are available 24/7 via the helpdesk to answer any questions and concerns you may have.

Network Disruption RFO 4.11.19.pdf

Read more »



Mar
13
Inclement Weather - Both Datacenter Locations - Until Midnight
Posted by Jeff Shotnik on 13 March 2019 11:25 AM
Denver, Colorado is currently experiencing blizzard-like weather conditions. The NWS has issued a warning that will last until midnight tonight. We have NOC engineers on site at each of our locations and will continue to monitor all systems closely. We do not expect any service impacting issues.

You can track weather for our Denver location here - https://darksky.net/forecast/39.7479,-104.9892/us12/en
You can track weather for our DTC location here - https://darksky.net/forecast/39.618,-104.8916/us12/en
Read more »