RSS Feed
Latest Updates
Sep
15
H5 Data Center Connectivity Disruption
Posted by Jay Sudowski on 15 September 2016 05:52 PM

Date - 9/15/16
Time - 5:11PM Local Time / 23:11 GMT

Description:

One of our 10G waves that connects our downtown data center to our tech center data center went down at approximately 5:11PM local time. There was a short disruption to traffic (2-3 minutes), while traffic failed over to the backup route.  This is the second time this circuit has gone down in a week, depending on the information we get from Level(3) we may declare an emergency maintenance window and cut over our transport to a dark fiber circuit Zayo just delivered yesterday.


Read more »



Aug
24
Limited support availability 8/24/2016, 1900 - 2200 MDT
Posted by Tyler Molamphy on 24 August 2016 05:41 PM

Date: 8/24/2016

Time: 19:00 - 22:00 MDT

Details:

   Support access will be limited during the mentioned time period while our staff attend a monthly all hands meeting. Please keep this in mind when submitting tickets during this time period. As always, if you have an emergency please feel free to call us.


Read more »



Jun
9
Resolved - Connectivity issues
Posted by Anthony Francis on 09 June 2016 01:23 PM

2016-06-09 01:00PM MDT

   We have experienced a loss of connectivity due to a major outage with our carrier Telia, which they have confirmed. The issue was amplified by the fact that another provider, HE, also uses Telia. On our request after they recieved confirmation from Telia, HE de-peered with Telia until the issue is resolved.

We recieved these responses from Telia:

"Dear Customer,

  This is part of a larger outage that is currently being investigated.

 

Kind Regards,

 

Luis Nuñez

Customer Care, Data & Infra"

Then:

"We are currently experiencing issue with a backbone router in New York."

 

If you are still experiencing connectivity issues reaching our network, please run a traceroute and see if your connection is traversing Telia, if it is, please contact your ISP and ask them to shut down their session with Telia until the issue is resolved.

 Update 2016-06-09 01:56PM MDT

We have recieved an update from Telia:

"Dear Customer,

 

Our core team has resolved an issue on our backbone causing U.S. customers packet loss. The root cause analysis will follow and we expect no further packet loss due to this issue. "

Update 2016-06-10 10:35AM MDT

We have recieved an RFO from Telia:

"

Dear Customer, 

 

This is a Reason for Outage Report with details regarding the case you have opened with TeliaSonera International Carrier.

 

Country:

United States

TeliaSonera Case Reference:

00563796

Network Impact:

Packet loss on the Telia Carrier U.S. backbone.

Case Opened:

6/9/2016 7:00 PM (After the issue had begun)

Case Ready for Service:

6/9/2016 7:23 PM

 

Reason for Outage:

Incorrect ISIS metric and multiple commits while turning up new Telia Carrier backbone links in Dallas caused a loop of reconverging BGP and ISIS protocols. This put a very high CPU load on our U.S. routers and caused some trans-Atlantic congestion.

Actions Taken:

The nyk-bb1 inner-core router (New York) was the first router to show real problems when we received alarms indicating packet loss on transit-Atlantic traffic together with high CPU utilization.  The router was taken out of service.  Further investigation revealed that the root cause was too many commits by Implementation while turning up new backbone links on dls-b22 (Dallas), along with an incorrect metric.  The configuration in dls-b22 was rolled back to alleviate the problem and nyk-bb1 has been put back in service.  This resolution is permanent and the will be no further loss related to this issue

Additional Information:

Telia Implementation team is making significant changes to their way of working to mitigate this from happening in the future.

 

 

 

 

Please note that all the time stamps given above are in UTC unless otherwise stated.

 

We sincerely regret the inconvenience this service outage may have caused you."


Read more »



Mar
14
[Complete] Email service migration - Handy Networks SmarterMail servers
Posted by Jeff Shotnik on 14 March 2016 04:17 PM

Update 03/16 @ 10:45PM MDT:

The mail3 and mail4 migration has been completed successfully - all email services are online. If you experience any issues please submit a support ticket right away.

==========

We will be migrating our SmarterMail email servers to a new server running the latest Windows Server and latest SmarterMail version this Wednesday at 9PM MDT. Betwen 9PM and 11PM, users on our Mail3 and Mail4 email servers will experience an email outage -- we'll need to shutdown the mail service so that file changes can be synced to the new server. Incoming/outgoing email will be interrupted but compliant sending mail servers will try again once we've completed the migration.

 

Updates will be posted here as we progress.


Read more »



Feb
24
Emergency Network Maintenance Windows: Feb 24, 2016 @ 9:00PM - 1:00AM
Posted by Jay Sudowski on 24 February 2016 01:29 PM

Resolution: Our maintenance window has gone through without issue; we make another annoucement if future maintenance windows are necessary.

---

We will be conducting emergency network maintenance this evening from February 24 @ 9:00PM - 1:00AM to address the underlying condition that has caused the two periods of packet loss and latency that were experienced earlier today.  During this time, you can expect to have several other periods of packet loss and latency.


Read more »