RSS Feed
News
Aug
23

[Completion, Sun Aug 26 23:53:05 DST 2018]

All servers affected have been returned to service, with the majority having been rebooted without issue by 11:08.

The maintenance has concluded successfully, for the servers our update platform could reach.  Servers that could not be reached by our update platform or servers that were listed as 'excluded' previously will have their updates scheduled separately, as needed.


[Update, Sun Aug 26 21:30:31 DST 2018]

We are now beginning this maintenance, and fully managed Centos 6+ servers (aside from those previously listed as excluded, and customer ID 2086) will be rebooted 5 at a time, starting now.

We'll keep you updated throughout the maintenance, and individual server owners will be updated as appropriate.

==

Date: Sunday, August 26, 2018 
Time: 9:30PM - 11:30PM


Purpose of Work: 

We will be updating kernels for, and rebooting all Centos 6 and Centos 7 servers under our management, excluding hypervisors, highly available clustered applications, and servers owned by customers with the following billing IDs:

1407
2171
2431
1001

This is to protect against the follow remote DOS exploit, that leverages vulnerabilities in the Linux kernel to deny services with any appropriately modified TCP packet: https://access.redhat.com/articles/3553061


Impact Of Work: 

Customers whose servers are being updated may notice 2-8 minutes of downtime, typically.  We will be monitoring for any server that takes longer than this to boot up and regain services.

Servers will be updated and rebooted 5 at a time.

We will let you know if the maintenance ends ahead of schedule, or else update you regarding the status of this maintenance at the end of the scheduled window, with further information being forwarded to specific customers, if required.


Rescheduling:

If you would prefer to schedule your server reboot to occur at a specific alternate time, and have confirmed that you have one of more Fully-Managed Centos 6 or 7 servers ( that do not meet the previously specified exclusion criteria ), please reach out to support@handynetworks.com, or submit a ticket via our helpdesk, and specify your desired alternate update time before the scheduled update time.


Read more »



Jul
12
[Complete] - Internal Exchange Maintenance
Posted by Jeff Shotnik on 12 July 2018 12:21 PM
Our team is taking our internal Exchange system down for emergency/critical maintenance. This will affect the deliverability of emails to @handynetworks.com accounts, including our support account. If you have a critical issue please call our support team at 303-414-6910 x2 or submit a ticket via our helpdesk web interface - https://helpdesk.handynetworks.com . 

Expected service outage: 30-60 minutes.
Read more »



Mar
29
Critical Drupal Vulnerabilities
Posted by Jay Sudowski on 29 March 2018 08:02 AM
Yesterday, Drupal disclosed critical security vulnerabilities impacting Drupal 6-8.  If you are running Drupal, update your applications immediately.

https://www.drupal.org/sa-core-2018-002

A remote code execution vulnerability exists within multiple subsystems of Drupal 7.x and 8.x. This potentially allows attackers to exploit multiple attack vectors on a Drupal site, which could result in the site being completely compromised.  This issue also affects Drupal 6. 


Read more »



Mar
26

[Completion, Wed, Mar 28, 2018 11:54:02 PM] The faulty module has been replaced, and we've confirmed that our server is recognizing the new module.  Maintenance is complete.

[Update, Wed, Mar 28, 2018 11:33:35 PM] We have identified a DIMM with issues, and will be replacing it shortly.  Maintenance will be extended slightly to carry this out.

[Update, Wed, Mar 28, 2018 10:31:29 PM] We are now beginning this maintenance; the management portal will be down shortly.  One thing I did not consider in the earlier announcement is that support staff may also have limited ability to get into fully-managed windows servers during this maintenance.  Barring emergencies, we will be waiting until the end of this maintenance window to proceed with windows troubleshooting.

Date: Wednesday, March 28, 2018 
Time: 10:30PM - 11:30PM


Purpose of Work: 

We will be temporarily shutting down the Database Server that our management portal at https://manage.handynetworks.com/ uses in order to diagnose and mitigate a memory issue we've encountered in the last hour.



Impact Of Work: 

Customers attempting to use https://manage.handynetworks.com/ will be unable to during the duration of the maintenance.  Some tie-in functions in the billing portal may also be unavailable, though core functionality should be fine.

Feel free to send in a ticket to support@handynetworks.com if you are encountering issues unrelated to this.


Read more »



Mar
15
[Complete ]Shared Mail Server maintenance, Sunday 2018-03-18, 21-00 DST
Posted by David Cunningham on 15 March 2018 12:41 AM
[Update, Sun, Mar 18, 2018  9:38:49 PM]  While faster than the previous attempt, the filesystem scan process was still not proceeding at an acceptable rate for a reasonably short maintenance window of 1-3 hours.

We will be making plans to perform this maintenance on an offline copy of the VM, copying all mail data over from the live instance once that is complete.

This should take significantly less downtime, and will be scheduled shortly after the copy is ready.

For now, maintenance is complete, and mail can come through as usual.



[Update, Sun, Mar 18, 2018  9:11:54 PM]  Maintenance has begun.  If we do not see significant progress in the next 20 minutes, we will be bringing the shared mailserver back up and planning a third method.


Date: Sunday, March 18, 2018 

Time: 9PM - 10:00PM


Purpose of Work: 

We will be attempting a second round filesystem maintenance on our shared mailserver to ensure continued stability and data integrity, with new methods.


Impact Of Work: 

Customers on our shared mail server will not be able to access their mail domains while the maintenance is in progress. 

Customers will also not be able to send tickets in via email directly, so tickets will have to be submitted via Helpdesk. 

In case of emergency please call us directly. +1-303-414-6910 x2. 



We will let you know if the maintenance ends ahead of schedule, or needs to be extended.
Read more »



Mar
7
[Complete] Shared Mail Server maintenance, Sunday 2018-03-11, 21-00 DST
Posted by David Cunningham on 07 March 2018 09:51 PM

[Post-completion note, Sun, Mar 11, 2018 10:18:55 PM]: Apparently the helpdesk itself does not use our shared mail server when sending outbound mail, but the news feature does. As such, the first update was likely just received by subscribers out of sequence.  You may disregard that: this maintenance is complete.

[Completed, Sun, Mar 11, 2018 10:01:03 PM MDT]:  The maintenance has not completed, and is not close enough to completion to merit extending this maintenance window at this time.  We are rolling back changes made, and may revisit this matter at a later date, if a more timely process or pressing need is discovered.   The shared mail server is back online.

[Update, Sun, Mar 11, 2018  9:14:18 PM MDT]:  The shared mail server is now down for maintenance.  We will keep tabs on this for the next 45 minutes, and provide an update at that time.   Remember to submit new tickets or replies directly through the helpdesk until maintenance is complete.


Date: Sunday, March 11, 2018
Time: 9PM - 10:00PM


Purpose of Work:

We will be performing filesystem maintenance on our shared mailserver to ensure continued stability and data integrity.


Impact Of Work:

Customers on our shared mail server will not be able to access their mail domains while the maintenance is in progress.

Customers will also not be able to send tickets in via email directly, so tickets will have to be submitted via Helpdesk.

In case of emergency please call us directly. +1-303-414-6910 x2.



We will let you know if the maintenance ends ahead of schedule, or needs to be extended.



Read more »