RSS Feed
News
May
14

[Update, Wed May 15 03:21:50 MDT 2019] All reboots for emergency updates are complete. We will be reaching
out to those with fully managed services and their own Active Directory domains separately in the morning. It is highly
recommended that all clients using Windows server 2003-2008R2 on self-managed hosts push through the updates securing a
gainst CVE-2019-0708 ASAP.

[Update, Wed May 15 01:38:19 DST 2019]   While all updates were initiated by the deadline, several are still completing.  Reboots may extend into the next couple of hours, but will be over before business hours, MDT. 

Date: 
 May 14, 2019
Time:  9:30PM MDT - 11:30PM MDT


Purpose of Work:
A pre-authentication vulnerability targeting Remote Desktop Protocol in servers running Windows Server 2008 R2 or older has been discovered that allows for Remote Code Execution.

Because the vulnerability requires absolutely no authentication, it could be spread rapidly within a network via use of 'Worm' style malware, at which point the exploiter would effectively have full control of all infected hosts.

Due to the ease of exploitation, and the impact of exploitation, we will be patching and rebooting all affected, fully-managed hosts overnight.

You can read more about the exploit (and patches mitigating it), here: https://blogs.technet.microsoft.com/msrc/2019/05/14/prevent-a-worm-by-updating-remote-desktop-services-cve-2019-0708/

Impact of Work:
Windows 2003 hosts on our domain will be rebooted immediately to propagate a manual update, and 2008 / 2008 R2 hosts on our domain will be scheduled to reboot later tonight, after installing the updates automatically.

Any hosts not on our fully-managed domain (usually because they have their own domain) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.


Read more »



Apr
23
Nimble SAN Firmware Upgrade - April 24, 2019
Posted by Pete Carstensen on 23 April 2019 12:50 PM
[Completed, 23:22 4-24-19]:  This maintenance is complete on all Nimble Storage Arrays.  As expected, no impact to workloads was observed.  

Date: 
April 24, 2019
Time:  10:30PM MT

Purpose of Work:
Nimble Storage has let us know that a couple of our storage arrays are running on a version of NimbleOS that has a security vulnerability that could result in remote code executions and remote denial of service attack.  We will be upgrading our storage arrays to newer firmware that is not impacted by this vulnerability. (CVE-2017-9765)

Impact of Work:
Historically, we have performed many Nimble Storage upgrades which have been completely hitless and not at all noticeable to any workloads consuming storage from our arrays.  Accordingly, we do not anticipate any impact, but there is always a slim possibility that connectivity to the storage arrays will be disrupted, resulting in workloads relying on the storage to hard reboot.

Please contact us with any questions / comments / concerns.
Read more »



Apr
12
Network Disruption RFO / RCA - April 11, 2019
Posted by Arick Francis on 12 April 2019 05:40 PM
Unfortunately, Yesterday at 10:46 AM to 11:11 AM Mountian Time a network disruption occurred. We have prepared an RFO / RCA detailing the incident. We sincerely apologize for any inconvenience the network disruption has caused, and as always we are available 24/7 via the helpdesk to answer any questions and concerns you may have.

Network Disruption RFO 4.11.19.pdf

Read more »



Mar
13
Inclement Weather - Both Datacenter Locations - Until Midnight
Posted by Jeff Shotnik on 13 March 2019 11:25 AM
Denver, Colorado is currently experiencing blizzard-like weather conditions. The NWS has issued a warning that will last until midnight tonight. We have NOC engineers on site at each of our locations and will continue to monitor all systems closely. We do not expect any service impacting issues.

You can track weather for our Denver location here - https://darksky.net/forecast/39.7479,-104.9892/us12/en
You can track weather for our DTC location here - https://darksky.net/forecast/39.618,-104.8916/us12/en
Read more »



Dec
14
Limited Support Availability 2018-12-14 2000-2259 MDT
Posted by Lindsay Schweitzer on 14 December 2018 07:48 PM
Limited Support Availability 2018-12-14 2000-2259 MDT

Our staff is attending a company event and support team responses may be delayed during this time.

Thank you; we appreciate you and your understanding!
Read more »



Dec
14
[Completed] Nimble SAN Firmware Upgrade - December 16, 2018
Posted by Jay Sudowski on 14 December 2018 11:22 AM
[Completed, 22:07 12-16-18]:  This maintenance is complete on all Nimble Storage Arrays.  As expected, no impact to workloads was observed.  



Date: 
December 16, 2018
Time: 9:00PM MT

Purpose of Work:
Nimble Storage has let us know that a couple of our storage arrays are running on a version of NimbleOS that has a bug that could result in an outage.  We will be upgrading our storage arrays to newer firmware that is not impacted by this bug.

Impact of Work:
Historically, we have performed many Nimble Storage upgrades which have been completely hitless and not at all noticeable to any workloads consuming storage from our arrays.  Accordingly, we do not anticipate any impact, but there is always a slim possibility that connectivity to the storage arrays will be disrupted, resulting in workloads relying on the storage to hard reboot.

Please contact us with any questions / comments / concerns.
Read more »