RSS Feed
Latest Updates
Jun
9
Resolved - Connectivity issues
Posted by Anthony Francis on 09 June 2016 01:23 PM

2016-06-09 01:00PM MDT

   We have experienced a loss of connectivity due to a major outage with our carrier Telia, which they have confirmed. The issue was amplified by the fact that another provider, HE, also uses Telia. On our request after they recieved confirmation from Telia, HE de-peered with Telia until the issue is resolved.

We recieved these responses from Telia:

"Dear Customer,

  This is part of a larger outage that is currently being investigated.

 

Kind Regards,

 

Luis Nuñez

Customer Care, Data & Infra"

Then:

"We are currently experiencing issue with a backbone router in New York."

 

If you are still experiencing connectivity issues reaching our network, please run a traceroute and see if your connection is traversing Telia, if it is, please contact your ISP and ask them to shut down their session with Telia until the issue is resolved.

 Update 2016-06-09 01:56PM MDT

We have recieved an update from Telia:

"Dear Customer,

 

Our core team has resolved an issue on our backbone causing U.S. customers packet loss. The root cause analysis will follow and we expect no further packet loss due to this issue. "

Update 2016-06-10 10:35AM MDT

We have recieved an RFO from Telia:

"

Dear Customer, 

 

This is a Reason for Outage Report with details regarding the case you have opened with TeliaSonera International Carrier.

 

Country:

United States

TeliaSonera Case Reference:

00563796

Network Impact:

Packet loss on the Telia Carrier U.S. backbone.

Case Opened:

6/9/2016 7:00 PM (After the issue had begun)

Case Ready for Service:

6/9/2016 7:23 PM

 

Reason for Outage:

Incorrect ISIS metric and multiple commits while turning up new Telia Carrier backbone links in Dallas caused a loop of reconverging BGP and ISIS protocols. This put a very high CPU load on our U.S. routers and caused some trans-Atlantic congestion.

Actions Taken:

The nyk-bb1 inner-core router (New York) was the first router to show real problems when we received alarms indicating packet loss on transit-Atlantic traffic together with high CPU utilization.  The router was taken out of service.  Further investigation revealed that the root cause was too many commits by Implementation while turning up new backbone links on dls-b22 (Dallas), along with an incorrect metric.  The configuration in dls-b22 was rolled back to alleviate the problem and nyk-bb1 has been put back in service.  This resolution is permanent and the will be no further loss related to this issue

Additional Information:

Telia Implementation team is making significant changes to their way of working to mitigate this from happening in the future.

 

 

 

 

Please note that all the time stamps given above are in UTC unless otherwise stated.

 

We sincerely regret the inconvenience this service outage may have caused you."


Read more »



Mar
14
[Complete] Email service migration - Handy Networks SmarterMail servers
Posted by Jeff Shotnik on 14 March 2016 04:17 PM

Update 03/16 @ 10:45PM MDT:

The mail3 and mail4 migration has been completed successfully - all email services are online. If you experience any issues please submit a support ticket right away.

==========

We will be migrating our SmarterMail email servers to a new server running the latest Windows Server and latest SmarterMail version this Wednesday at 9PM MDT. Betwen 9PM and 11PM, users on our Mail3 and Mail4 email servers will experience an email outage -- we'll need to shutdown the mail service so that file changes can be synced to the new server. Incoming/outgoing email will be interrupted but compliant sending mail servers will try again once we've completed the migration.

 

Updates will be posted here as we progress.


Read more »



Feb
24
Emergency Network Maintenance Windows: Feb 24, 2016 @ 9:00PM - 1:00AM
Posted by Jay Sudowski on 24 February 2016 01:29 PM

Resolution: Our maintenance window has gone through without issue; we make another annoucement if future maintenance windows are necessary.

---

We will be conducting emergency network maintenance this evening from February 24 @ 9:00PM - 1:00AM to address the underlying condition that has caused the two periods of packet loss and latency that were experienced earlier today.  During this time, you can expect to have several other periods of packet loss and latency.


Read more »



Feb
12
Connectivity issues at DTC location
Posted by Jeff Shotnik on 12 February 2016 12:27 AM

Update - 02:06AM MDT: 

Connectivity to our DTC location has been restored, but we are still working with Level3 to ensure that the problem has been completely resolved. 

==========

Update  - 01:50AM MDT:

Level3 has identified an issue in the Denver Metro area, and is working to resolve it. We will continue to provide updates as we receive them. 

==========

Update - 12:41AM MDT:

We are in contact with our transit provider to identify any network connectivty problems. We are also on-site reviewing our network gear for issues.

==========

We have been alerted of connectivity issues at our Denver Tech Center location. We are working on the issue as quickly as possible and will update here.


Read more »



Jan
19
[Complete] SAN storage device software upgrade - Thursday Feb 4th
Posted by Jeff Shotnik on 19 January 2016 10:02 PM

Update 02/05/16 @ 12:18AM:

The SAN software update has been successfully completed, zero I/O disruption to connected servers.

 

 

==========

Update 02/04/16:

Our SAN software upgrade is underway. Updates to come.

==========

Update 01/21/16: 

We're going to defer this maintanence until Feb 4th. We have a few internal systems that need maintenance performed before we can proceed with a SAN storage software update.

==========

Our team will be upgrading the software on one of our SAN storage devices this Thursday, Jan 21st at 10:00PM MDT. We do not expect any issues/outages during the upgrade and we should be completed by 11:30PM MDT. The device we will be performing an upgrade on is used by our Hyper-V Cloud service and by a few other customers who we will reach out to directly.

We'll be updating this post during the upgrade. If you have any questions or concerns feel free to contact our support team.


Read more »