RSS Feed
News
Oct
29

Hello,

Overnight staff is working remotely tonight due to the poor road conditions as cdot has advised avoiding travel until normal commuting hours. During this time non-emergency hands-on support (such as hardware upgrades or changeouts that do not involve a failing host) will be delayed until Wednesday morning.

Normal helpdesk support is not affected.


Read more »



Oct
23
Limited Support Availability 2019-10-23 18:15-22:00 MDT
Posted by David Cunningham on 23 October 2019 05:57 PM
Limited Support Availability 2019-10-23 18:15-22:00 MDT

Our staff is attending a company event and support team responses may be delayed during this time, with troubleshooting limited except in response to emergency matters.

Operations should resume as normal around 10PM Denver time.

Thank you; we appreciate you and your understanding!
Read more »



Oct
21
New cPanel Licensing Structure
Posted by Lindsay Schweitzer on 21 October 2019 03:31 PM

cPanel has recently announced a complete overhaul to the way they structure their licensing costs. Going forward, cPanel licenses will be billed based on number of accounts rather than a flat rate for the license. In order to ensure we’re billing our clients for the most accurate number of accounts, we will be charging for all licenses on the 13th of each month.

An explanation and FAQ on these changes from cPanel can be found here: https://forums.cpanel.net/pages/partners/

The new pricing structure is outlined below:

Cloud:

cPanel Admin Cloud - Up to 5 Accounts - $13.40/mo
cPanel Pro Cloud - Up to 30 Accounts - $18.73/mo
cPanel Plus Cloud - Up to 50 Accounts - $26.75/mo
cPanel Premier Cloud - Up to 100 Accounts - $34.25/mo
cPanel Premier Cloud Bulk Account - Each account over 100, per account - .10c/mo

Metal:

cPanel Premier Metal - Up to 100 Accounts - $34.25/mo
cPanel Premier Metal Bulk Account - Per Account - .10c/mo

We regret to have to pass along these increased license costs to our clients, and understand the impact these increased costs will have for your business. 

Your billing account has been updated to reflect these new license costs. You can review this information in the billing portal at https://billing.handynetworks.com

Please do not hesitate to reach out to us with any additional questions or concerns you may have regarding these licensing changes, and thank you for your understanding as we work through these changes together.




Read more »



Sep
23

[Update, Wed Sep 25 00:47:12 MDT 2019] - Uninstallation of the old revision of this update has been confirmed; updates will now begin, and should be done servers within the scope in 1-2 hours.  During this time your host may be rebooted.

We are also including GMT hosts in this update cycle, despite the time of day.  Reboots should be fast, from what we've observed.




[Update, Tue Sep 24 20:29:43 MDT 2019] -
We will be re-deploying this update with today's revision of these updates, rather than the revision deployed yesterday.

The impact of tonight's adjusted maintenance will be the same, except servers rebooted yesterday night will have one additional reboot before the main maintenance is resumed, as part of the re-installation of this update.  This should only be about 25% of the servers in the intended scope.

The scope of the work remains the same as defined yesterday; uninstallation of the older revision will begin shortly, with the main update event beginning in roughly 1-2 hours.



Purpose of Work:

Earlier today, Microsoft released an out-of-band (or earlier than usual) update to patch a vulnerability with the internet explorer scripting engine.  Said vulnerability already has working exploits that have been encountered 'in the wild', and thus must be responded to immediately.

This vulnerability allows for remote code execution running in the context of the user that accessed an infected webpage via IE; if an admin or privileged website user gets compromised in this way, the entire server could be effectively compromised.

As such, RDS servers and any server where admins routinely use IE to perform research while looking into server issues / download programs are the most at-risk. Certain programs that use the IE backend (one example being quickbooks) may also be at-risk in some situations.


You can read more about the exploit (and patches mitigating it), here: https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2019-1367

We will update you as maintenance begins.


Exceptions:

As discussed, only fully-managed windows servers will be targeted by this maintenance to begin with.  Self-managed hosts must be updated by the responsible parties, and I would recommend this happen soon.

Any standalone hypervisors that would incur downtime of VMs as part of this maintenance reboot will be skipped; tickets requesting scheduling have been sent out to server owners.

Any customers with a multitude of servers on their own active directory domains will be skipped; tickets have been sent out to them, as well.

Any customers included in a GMT timezone update group will instead be updated tomorrow at 3PM.  If this doesn't happen for you, but should, let us know.  We can change your update group.

2003 servers will not be affected, as there was not a patch for them ( through windows updates or otherwise ).  This is typical on EOL systems like 2003 hosts, but not always the case, so it bears repeating.


Impact of Work:
All affected hosts will be rebooted automatically / ASAP to propagate fixes, starting at 11PM MDT on Monday the 23rd.

Internal systems (such as the management portal) may be temporarily impacted in the time it takes to reboot them.

Hypervisors in a failover cluster will have rolling reboots done via CAU, in order to eliminate VPS downtime on said clusters.



Please contact us with any questions / comments / concerns.


Read more »



Aug
22
[RESOLVED] Outages of some VMs on shared hypervisors
Posted by Lindsay Schweitzer on 22 August 2019 02:15 PM

UPDATE: This issue is resolved and all VMs should be back online. If you're experiencing any lingering issues related to this outage, please open a ticket on our helpdesk so we can address. Thank you for your patience.

We’re currently addressing issues related to the DDoS attack of a VM on one of our shared hypervisors. We’re working to resolve the issues related to this attack and have the services of all affected clients up as quickly as possible.



Read more »



Aug
18

[Completion] All but one of the dedicated clusters are upgraded, all 2016+ non-HV servers are upgraded, and the upgrade process can be left unattended at this point.  

If you have your own fully managed hypervisors, or a fully managed server on its own domain, and would like us to manage your updates, please contact us for update scheduling as soon as is feasible, to ensure these vulnerabilities are patched.


[Update 3, Tue Aug 20 20:12:10 MDT 2019]
As announced yesterday, Fully managed windows server patching and automatic reboots for certain servers on our domain will resume for 2016 build 1703 to 2019 servers (and windows 10 hosts on those builds), starting now.

As before, this will exclude managed clients' standalone hypervisors.  Hypervisors in failover clusters will have rolling updates applied manually where not already done, tonight.

Fully managed windows servers not joined to any domain (including ours) will be updated (and manually configured to use our WSUS) on a case-by-case basis.


[Update 2, Tue Aug 20 00:46:33 MDT 2019]
All pending updates have been confirmed to have been applied or initiated for fully managed windows servers on our domain, with OS levels of baseline 2016 and below.

As covered in the previous update, newer servers will have these updates applied tomorrow night, once all pending updates have downloaded to the WSUS server.

[Update, Mon Aug 19 22:02:58 MDT 2019] Due to a required option to get WSUS to download the relevant updates for Server 2019 (and some newer builds of Windows 10) not having been set yet, said OS versions will likely not be updated until tomorrow evening.

Normal updates of all Windows versions from 2008 R2 up to 2016 (for non-hypervisors) are occurring now.  Reboots may occur shortly.


Purpose of Work:
Several pre-authentication vulnerabilities targeting Remote Desktop Protocol in servers running Windows Server 2008 R2 or newer have been discovered; all of which allow for Remote Code Execution.

Because the vulnerability requires absolutely no authentication, it could be spread rapidly within a network via use of 'Worm' style malware, at which point the exploiter would effectively have full control of all infected hosts.


Due to the ease of exploitation, and the impact of exploitation, we will be patching and rebooting all affected, fully-managed hosts overnight. 

Hypervisors would be a general exception to this, and customer-owned Windows HVs that host unmanaged VMs should have their maintenance scheduled with us, separately.


You can read more about the exploit (and patches mitigating it), here: https://msrc-blog.microsoft.com/2019/08/13/patch-new-wormable-vulnerabilities-in-remote-desktop-services-cve-2019-1181-1182/

The following patches are among those that will be applied:

https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2019-1226
https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2019-1222
https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2019-1182
https://portal.msrc.microsoft.com/en-US/security-guidance/advisory/CVE-2019-1181

We will update you as maintenance begins.

Impact of Work:
All affected hosts will be rebooted automatically / ASAP to propagate fixes, starting at 8PM MDT on Monday the 19th.  

Internal systems (such as the management portal) may be temporarily impacted in the time it takes to reboot them.

Hypervisors will be done last.  Hypervisors in a failover cluster will have rolling reboots done, in order to eliminate VPS downtime on said clusters.

Any hosts not on our fully-managed domain (usually because they have their own domain) will not be impacted; the controlling organizations will be notified separately.


Please contact us with any questions / comments / concerns.


Read more »