RSS Feed
Latest Updates
Jan
4
Network Maintenance Windows - January 6,7,8 & January 10,12,14
Posted by Jay Sudowski on 04 January 2017 04:17 PM

Dates: January 6, 7, 8 & January 10, 12, 14

Time:  Maintenance window will be open from 8:00pm mountain - 2:00am mountain time

Expected Impact:  Brief periods of latency and packet loss during the maintenance windows.

Purpose of Work:  We will be performing various network upgrades, including:

  • Bringing optical amplifiers online to increase the capacity on our dark fiber ring
  • Adding new line cards to our MX240 routers and grooming existing circuits over to them
  • Replacing our aging MX80s and EX4500 switches at 1801 California St with Juniper QFX5100 switches

During the maintenance window, we will be providing updates regarding specific activities that will be taking place during these windows.  Unfortunately, due to the scope of the work we need to perform, and the interrelatedness and dependencies of all of this work, we are bulk scheduling maintenance windows. 

 

UPDATES:

2017-01-06 21:10 MST: We are beginning the portion of our maintenance with the greatest risk for impact. If you experience any packet loss or delay it is very likely to be related.

2017-01-07 01:15 MST: We have concluded our maintenance activities for the evening.  All systems are operational.  

2017-01-07 20:34 MST: We are beginning our maintenance for the evening.  Tasks tonight include upgrading border1 to the latest JTAC recommended JunOS and then working to bring the replacements for our EX4500 switches online.

2017-01-07 23:50 MST: Our maintenance is concluded for the evening.  We encountered some odd issues with our border routers during the code upgrades, which took substantial amounts of time to investigate.  As a result, we will be working on the EX4500 replacement tomorrow evening.

2017-01-08 20:36 MST:  We will not be conducting any network maintenance this evening, and we will also be rescheduling the timing of the remaining tasks.  The schedule was not maintainable, and at this point most of our staff working on this project have gotten only 4 hours sleep over the past two nights.  A more official notice will follow, but the next maintenance events will be scheduled for Tuesday, Thursday, and Saturday this coming week, with slightly shorter windows (9PM - 1AM).  

2017-01-10 23:36 MST:  During our network maintenance for the evening, we were operating under reduced redundancy.  Unfortunately, as our luck would have it, one of our border routers decided to coredump, causing a network wide outage for approximately 25 minutes.  During these 25 minutes, we were rolling back our previous changes made to effect the maintenance window, which involved reverting both physical and logical networking configurations.  Connectivity was disrupted at 22:50 and restored at 23:15.  We are still actively working to determine more about the outage.  The maintenance window may be extended to address any issues that remain related to this unexpected problem.

2017-01-11 00:43 MST:  All systems are working as expected at this time.  All further maintenance is canceled, until we can work with our vendor to get to the bottom of the issues that occured this evening.


Read more »



Jan
2
[Complete] Handy Networks Shared SQL 2005 Server Maintenance
Posted by David Cunningham on 02 January 2017 06:34 PM

Final Update, Mon, Jan  2, 2017 11:56:14 PM: Maintenance has proceeded as planned, and the shared SQL 2005 server is now fully available and interfaces with Helm once again.

Update, Mon, Jan  2, 2017 11:01:51 PM: Maintenance is proceeding according to schedule, and may be finished in the next hour.  Updates to follow.

Update, Mon, Jan  2, 2017 10:00:17 PM: We have brought the SQL 2005 Service offline for the beginning of maintenance, with all going according to plan so far.  Updates will be provided on the hour, from here.

==

We will be bringing the SQL 2005 server utilized by customer sites on our Helm Control Panel cluster offline tonight, at 10PM MST, for urgent maintenace.

 

Impact: This will only affect the direct shared hosting customers utilizing our internal servers. 

Estimated downtime: 2-3 hours.  We will update you when we begin the maintenace, and if any new developments occur.


Read more »



Dec
22
Limited Support Availability - Thursday 2016-12-22
Posted by Anthony Kolka on 22 December 2016 08:14 AM

When: 2016-12-22 from 18:30 MST to 23:30 MST

Our staff will be attending a holiday gathering during this time and we will not have anyone onsite during these hours. Any non-critical requests will be delayed until after the event.

 

Thank you very much for your understanding and happy holidays to all!


Read more »



Nov
16
Resolved - Emergency Maintenance for row B4.
Posted by Anthony Kolka on 16 November 2016 09:19 AM

2016-11-16 08:42AM MST B-4 PDU unit failure:

At 08:42 MST the Power Distrobution unit for row B-4 failed. We are working to replace it and while that is happenning we are relocating hosts utilizing that PDU to other equipment in order to restore services as rapidly as possible.

 

2016-11-16 10:10AM: All affected servers have returned to service. We thank you for your patience and apologize for the inconvenience.


Read more »



Oct
19
Network Maintenance - Friday, October 21 9:00PM - 2:00AM
Posted by Jay Sudowski on 19 October 2016 11:08 AM

9:15PM - This maintenance window is closed.  The upgrade to 20Gbps was successful.  No noticeable customer impact was detected.

 

Date: October 21, 2016
Time: 9:00PM - 2:00AM

Description of Work:
We will be upgrading the connectivity between our downtown Denver data center and our Denver Tech Center data center from 10Gbps to 20Gbps.  Additionally, we will be making some modifications to lower the load on the routing engine of our core routers.

Impact of Work:
During this work, brief periods of packet loss and latency should be expected.


Read more »