Wednesday, March 3, 2021

Undetected Downtime on "host 2" server, now resolved

During the afternoon of March 2, 2021 one of our primary hosting servers experienced an unexpected downtime event due to the datacenter taking the underlying storage array offline.  We have mitigated the issue, however without advanced notice the migration of data took longer than anticipated.  We will be reaching out to the affected customers, and compensating appropriately.

Thank you for your understanding.

Sunday, November 29, 2020

Server consolidation, hosting changes

Over the weekend we consolidated some server space and file storage, due to the fact that we don't have as many clients on our shared hosting space as we once did.   Beginning in 2021 we are implementing a policy whereby all new shared hosting space will be provisioned manually.  We have also implemented backup e-mail services, as well as additional redundant backup space.

Also, take note that we will be adhering strictly to our VPS password policy in the new year as well.  If you have any questions, see our website, or contact us.

We apologize for any minor downtime this shift in resources may have caused.  We will continue to monitor server usage and shift resources to the best of our ability to fully utilize disk space and CPU power so that all customers get the amazing performance they expect.

Thursday, August 27, 2020

Webmail update post Atlanta shared server issues

Many of you may be aware that the Atlanta shared CPanel server involved in the latest network problems stored our old classic webmail client; this was our go to for many years, and through many iterations, but after this last server rebuild, we found that it simply cannot be secure enough to handle mail the way our outside mail provider wants it.  Therfore, we have defaulted to using CPanel's webmail clients on the host e-mail sever; this will be maintained and updated directly by the mail provider, and will be a smoother experience overall.

For applications on the shared server that USE outbound e-mail, we have switched to using SENDMAIL or PHP mail where applicable.  If you have any issues with outbound mail as part of forms or other applications, please let us know so we can investigate.

Thank you for bearing with us once again, and we hope that this is yet another permanent solution to a long standing problem.

Wednesday, August 26, 2020

Investigating downtime with Atlanta CPanel server

EDIT:  The issue is with the network and some related hardware, but our data center decided to spin up a new server anyway, and restore a backup from Sunday, August 23rd.   The new server/node contains updated hardware which should resolve the occasional connectivity problems previously experienced.  At this writing, the server is up, but data restoration is still in progress.

It appears sites on are inaccessible at the moment, due to possible network issue.  We are trying to confirm with the provider, and if it is instead a hardware problem, they will hopefully have it solved quickly.  We won't make any further updates unless we need to; be assured we are on top of the situation.  As this server houses our helpdesk system, that too is offline.  

We are sorry for the inconvenience and thank you for your patience.

Thursday, July 9, 2020

Infrastructure Update

UPDATE 7/11/2020:
The majority of the files and links relating to our infrastructure described below are in the process of being restored.  By the end of the month, everything should be back where it was, as certain links will be updated, and some may require intervention; this will all be handled on a case by case basis.  The original message stands below:

The data center where we were hosting many of our supplementary files, such as music links, ISO images for customers, and installation scripts, was recently hit with a catastrophic data encryption bug that left a good percentage of storage unavailable to us.  We have backups of recent data, but since the amount of data was in total over 5 terabytes, it will take a while to rebuild the infrastructure that we previously had.  Moreover, many of the links to files and data that were sent to customers are now invalid.   We want to make clear a few points:

- At no time was customer data itself stored on the disk[s] in question.

- Backups were out of date by several months but they do exist and it gives us a baseline for restoration.  We will make an effort to redistribute the data as best as possible to other servers, so as to avoid a major loss like this in the future.

- There is still a possibility that the data center will recover / decrypt the data on the offending disks, so we are hoping that we can restore the data directly, but it still won't be accessible the same way it was previously.  What's more, we will not be utilizing this data center for any other server deployments going forward.

Please bear with us while we handle the gargantuan task of rebuilding our data store.  In the meantime, we ask that you open a ticket if you require something to install on, or rebuild, a virtual server instance.

Saturday, December 28, 2019

Shared CPanel hosting server downtime explanation

The data center in Atlanta where we had moved several client websites had a network routing issue that left several of those sites inaccessible for many hours today.  The sites are back up and running - actually they never really were offline - but the underlying network problem forced the data center to assign a new IP address to the host server.

Propagation - the time it takes all the other systems on the public internet to recognize this change - can take several hours.  We will be looking into appropriate compensation for any customers who were affected; as with any network issue, some may have not even noticed, due to caching.  We will keep monitoring and hope this is a one-off incident which can be mitigated faster if it does happen again.

Thank you for your understanding and patience.

Thursday, December 12, 2019

Partial Network Outage Affecting AU Proxy

We are currently experiencing a partial outage in our Sydney location affecting VPN and Proxy services.. Our network engineers are actively working with our upstream providers to restore connectivity as quickly as possible.  There will be no further updates unless there are unforeseen problems.