Webmail update post Atlanta shared server issues

Many of you may be aware that the Atlanta shared CPanel server involved in the latest network problems stored our old classic webmail client; this was our go to for many years, and through many iterations, but after this last server rebuild, we found that it simply cannot be secure enough to handle mail the way our outside mail provider wants it.  Therfore, we have defaulted to using CPanel's webmail clients on the host e-mail sever; this will be maintained and updated directly by the mail provider, and will be a smoother experience overall.

For applications on the shared server that USE outbound e-mail, we have switched to using SENDMAIL or PHP mail where applicable.  If you have any issues with outbound mail as part of forms or other applications, please let us know so we can investigate.

Thank you for bearing with us once again, and we hope that this is yet another permanent solution to a long standing problem.

Investigating downtime with Atlanta CPanel server

EDIT:  The issue is with the network and some related hardware, but our data center decided to spin up a new server anyway, and restore a backup from Sunday, August 23rd.   The new server/node contains updated hardware which should resolve the occasional connectivity problems previously experienced.  At this writing, the server is up, but data restoration is still in progress.

It appears sites on cpanel.ruggieroav.net are inaccessible at the moment, due to possible network issue.  We are trying to confirm with the provider, and if it is instead a hardware problem, they will hopefully have it solved quickly.  We won't make any further updates unless we need to; be assured we are on top of the situation.  As this server houses our helpdesk system, that too is offline.  

We are sorry for the inconvenience and thank you for your patience.

Infrastructure Update

UPDATE 7/11/2020:
The majority of the files and links relating to our infrastructure described below are in the process of being restored.  By the end of the month, everything should be back where it was, as certain links will be updated, and some may require intervention; this will all be handled on a case by case basis.  The original message stands below:

The data center where we were hosting many of our supplementary files, such as music links, ISO images for customers, and installation scripts, was recently hit with a catastrophic data encryption bug that left a good percentage of storage unavailable to us.  We have backups of recent data, but since the amount of data was in total over 5 terabytes, it will take a while to rebuild the infrastructure that we previously had.  Moreover, many of the links to files and data that were sent to customers are now invalid.   We want to make clear a few points:

- At no time was customer data itself stored on the disk[s] in question.

- Backups were out of date by several months but they do exist and it gives us a baseline for restoration.  We will make an effort to redistribute the data as best as possible to other servers, so as to avoid a major loss like this in the future.

- There is still a possibility that the data center will recover / decrypt the data on the offending disks, so we are hoping that we can restore the data directly, but it still won't be accessible the same way it was previously.  What's more, we will not be utilizing this data center for any other server deployments going forward.

Please bear with us while we handle the gargantuan task of rebuilding our data store.  In the meantime, we ask that you open a ticket if you require something to install on, or rebuild, a virtual server instance.

Shared CPanel hosting server downtime explanation

The data center in Atlanta where we had moved several client websites had a network routing issue that left several of those sites inaccessible for many hours today.  The sites are back up and running - actually they never really were offline - but the underlying network problem forced the data center to assign a new IP address to the host server.

Propagation - the time it takes all the other systems on the public internet to recognize this change - can take several hours.  We will be looking into appropriate compensation for any customers who were affected; as with any network issue, some may have not even noticed, due to caching.  We will keep monitoring and hope this is a one-off incident which can be mitigated faster if it does happen again.

Thank you for your understanding and patience.

Partial Network Outage Affecting AU Proxy

We are currently experiencing a partial outage in our Sydney location affecting VPN and Proxy services.. Our network engineers are actively working with our upstream providers to restore connectivity as quickly as possible.  There will be no further updates unless there are unforeseen problems.

Issues with forms processing

We have modified our forms processor to fix some long standing bugs, most notably those with e-mail delivery. 

Previously, messages generated from certain web forms on clients' websites were posting to the proper location and were viewable from the forms' admin panel, but NOT being delivered to e-mail; also users were not receiving e-mail confirmation of their submissions.  We believe we have fixed that issue, and will continue to monitor from now into the new year.

Scheduled Maintenance: Datacenter Migration in New York

Attention customers with servers in New York:

In 10 days the company INCERO will be migrating their servers from a legacy New York facility to a new NYC Tier 1 data center. 

Migration will take place on Monday evening, October 21, 2019 around 10pm EST and is expected to be completed within 6 hours.   Servers will obviously be offline during this process.   The DC is well prepared for this migration and anticipates that it goes smoothly.  Once this move is complete the following network improvements will be gained:

- IRP blend of Level 3, Hurricane Electric, NTT, Cogent, and Telia transit along with NYIIX and DE-CIX peering fabrics. Incero's legacy network in New York consisted of GTT transit only.
- DDoS protection


Maintenance Window:
———————————————
Monday 10/21/2019    10:00pm - 7:00am EST


We will not post any follow-up unless there are problems.  To avoid a false alarm, we reserve the right to PAUSE our downtime monitor for the affected servers, so as not to trigger an alert.  Please rest assured will will confirm everything is working once the migration has been verified complete.  Thank you for your understanding and patience.