=Completion, Thu Nov  8 01:38:54 STD 2018

This maintenance is now finished.  Further reboots for HA VMs will not be required, and all have returned to service.


=Update, Wed Nov  7 23:36:22 STD 2018

This maintenance will be extended tonight, as workflow has been slower than predicted, but impact to individual VMs is still minimal.

I will update this post when it is complete, or if we'll be dividing it into separate nights. 

=Update, Wed Nov  7 21:18:28 STD 2018:

Some corrections and revisions:

This will actually impact all self-managed HA cloud servers as well.  If you are not on our Hyper-V cluster, you will not be impacted.


-Servers will be updated and rebooted at the same time, with the exception of those that are part of a highly available application cluster of their own.

Servers will be updated 10 at a time, with the latency of the underlying storage monitored closely during maintenance.  I still expect maintenance to be over within 1-2 hours, and individual impact will still be minimal.


We will begin, as scheduled, in 10 minutes.


===

Date: Wednesday, November 7, 2018
Time: 9:30PM - 10:30PM


Purpose of Work: 

We will be updated the VM configuration version for all HA cloud servers, tomorrow night.  

Impact Of Work: 

Procedure will involve a graceful shutdown, a 1 minute upgrade process, and the server will then be turned back on.

Customers whose servers are being updated may notice 2-3 minutes of downtime, typically.  We will be monitoring for any server that takes longer than this to boot up and regain services.

Servers will be updated and rebooted at the same time, with the exception of those that are part of a highly available application cluster of their own.



We will let you know if the maintenance ends ahead of schedule, or else update you regarding the status of this maintenance at the end of the scheduled window, with further information being forwarded to specific customers, if required.


Rescheduling:

If you would prefer to schedule your server reboot to occur at a specific alternate time, and have confirmed that you have one or more Fully-Managed Highly Available Cloud server ( that do not meet the previously specified exclusion criteria ), please reach out to support@handynetworks.com, or submit a ticket via our helpdesk, and specify your desired alternate update time before the scheduled update time.

[Completion, Sun Aug 26 23:53:05 DST 2018]

All servers affected have been returned to service, with the majority having been rebooted without issue by 11:08.

The maintenance has concluded successfully, for the servers our update platform could reach.  Servers that could not be reached by our update platform or servers that were listed as 'excluded' previously will have their updates scheduled separately, as needed.


[Update, Sun Aug 26 21:30:31 DST 2018]

We are now beginning this maintenance, and fully managed Centos 6+ servers (aside from those previously listed as excluded, and customer ID 2086) will be rebooted 5 at a time, starting now.

We'll keep you updated throughout the maintenance, and individual server owners will be updated as appropriate.

==

Date: Sunday, August 26, 2018 
Time: 9:30PM - 11:30PM


Purpose of Work: 

We will be updating kernels for, and rebooting all Centos 6 and Centos 7 servers under our management, excluding hypervisors, highly available clustered applications, and servers owned by customers with the following billing IDs:

1407
2171
2431
1001

This is to protect against the follow remote DOS exploit, that leverages vulnerabilities in the Linux kernel to deny services with any appropriately modified TCP packet: https://access.redhat.com/articles/3553061


Impact Of Work: 

Customers whose servers are being updated may notice 2-8 minutes of downtime, typically.  We will be monitoring for any server that takes longer than this to boot up and regain services.

Servers will be updated and rebooted 5 at a time.

We will let you know if the maintenance ends ahead of schedule, or else update you regarding the status of this maintenance at the end of the scheduled window, with further information being forwarded to specific customers, if required.


Rescheduling:

If you would prefer to schedule your server reboot to occur at a specific alternate time, and have confirmed that you have one of more Fully-Managed Centos 6 or 7 servers ( that do not meet the previously specified exclusion criteria ), please reach out to support@handynetworks.com, or submit a ticket via our helpdesk, and specify your desired alternate update time before the scheduled update time.

A remote code execution vulnerability exists within multiple subsystems of Drupal 7.x and 8.x. This potentially allows attackers to exploit multiple attack vectors on a Drupal site, which could result in the site being completely compromised.  This issue also affects Drupal 6. 

[Completion, Wed, Mar 28, 2018 11:54:02 PM] The faulty module has been replaced, and we've confirmed that our server is recognizing the new module.  Maintenance is complete.

[Update, Wed, Mar 28, 2018 11:33:35 PM] We have identified a DIMM with issues, and will be replacing it shortly.  Maintenance will be extended slightly to carry this out.

[Update, Wed, Mar 28, 2018 10:31:29 PM] We are now beginning this maintenance; the management portal will be down shortly.  One thing I did not consider in the earlier announcement is that support staff may also have limited ability to get into fully-managed windows servers during this maintenance.  Barring emergencies, we will be waiting until the end of this maintenance window to proceed with windows troubleshooting.

Date: Wednesday, March 28, 2018 
Time: 10:30PM - 11:30PM


Purpose of Work: 

We will be temporarily shutting down the Database Server that our management portal at https://manage.handynetworks.com/ uses in order to diagnose and mitigate a memory issue we've encountered in the last hour.



Impact Of Work: 

Customers attempting to use https://manage.handynetworks.com/ will be unable to during the duration of the maintenance.  Some tie-in functions in the billing portal may also be unavailable, though core functionality should be fine.

Feel free to send in a ticket to support@handynetworks.com if you are encountering issues unrelated to this.