Date - 9/15/16
Time - 5:11PM Local Time / 23:11 GMT

Description:

One of our 10G waves that connects our downtown data center to our tech center data center went down at approximately 5:11PM local time. There was a short disruption to traffic (2-3 minutes), while traffic failed over to the backup route.  This is the second time this circuit has gone down in a week, depending on the information we get from Level(3) we may declare an emergency maintenance window and cut over our transport to a dark fiber circuit Zayo just delivered yesterday.

Date: 8/24/2016

Time: 19:00 - 22:00 MDT

Details:

   Support access will be limited during the mentioned time period while our staff attend a monthly all hands meeting. Please keep this in mind when submitting tickets during this time period. As always, if you have an emergency please feel free to call us.

2016-06-09 01:00PM MDT

   We have experienced a loss of connectivity due to a major outage with our carrier Telia, which they have confirmed. The issue was amplified by the fact that another provider, HE, also uses Telia. On our request after they recieved confirmation from Telia, HE de-peered with Telia until the issue is resolved.

We recieved these responses from Telia:

"Dear Customer,

  This is part of a larger outage that is currently being investigated.

Kind Regards,

Luis Nuñez

Customer Care, Data & Infra"

Then:

"We are currently experiencing issue with a backbone router in New York."

If you are still experiencing connectivity issues reaching our network, please run a traceroute and see if your connection is traversing Telia, if it is, please contact your ISP and ask them to shut down their session with Telia until the issue is resolved.

 Update 2016-06-09 01:56PM MDT

We have recieved an update from Telia:

"Dear Customer,

Our core team has resolved an issue on our backbone causing U.S. customers packet loss. The root cause analysis will follow and we expect no further packet loss due to this issue. "

Update 2016-06-10 10:35AM MDT

We have recieved an RFO from Telia:

"

Dear Customer, 

This is a Reason for Outage Report with details regarding the case you have opened with TeliaSonera International Carrier.

Please note that all the time stamps given above are in UTC unless otherwise stated.

We sincerely regret the inconvenience this service outage may have caused you."

Update 03/16 @ 10:45PM MDT:

The mail3 and mail4 migration has been completed successfully - all email services are online. If you experience any issues please submit a support ticket right away.

==========

We will be migrating our SmarterMail email servers to a new server running the latest Windows Server and latest SmarterMail version this Wednesday at 9PM MDT. Betwen 9PM and 11PM, users on our Mail3 and Mail4 email servers will experience an email outage -- we'll need to shutdown the mail service so that file changes can be synced to the new server. Incoming/outgoing email will be interrupted but compliant sending mail servers will try again once we've completed the migration.

Updates will be posted here as we progress.

Resolution: Our maintenance window has gone through without issue; we make another annoucement if future maintenance windows are necessary.

---

We will be conducting emergency network maintenance this evening from February 24 @ 9:00PM - 1:00AM to address the underlying condition that has caused the two periods of packet loss and latency that were experienced earlier today.  During this time, you can expect to have several other periods of packet loss and latency.