Thursday, July 9, 2020

Infrastructure Update

UPDATE 7/11/2020:
The majority of the files and links relating to our infrastructure described below are in the process of being restored.  By the end of the month, everything should be back where it was, as certain links will be updated, and some may require intervention; this will all be handled on a case by case basis.  The original message stands below:

The data center where we were hosting many of our supplementary files, such as music links, ISO images for customers, and installation scripts, was recently hit with a catastrophic data encryption bug that left a good percentage of storage unavailable to us.  We have backups of recent data, but since the amount of data was in total over 5 terabytes, it will take a while to rebuild the infrastructure that we previously had.  Moreover, many of the links to files and data that were sent to customers are now invalid.   We want to make clear a few points:

- At no time was customer data itself stored on the disk[s] in question.

- Backups were out of date by several months but they do exist and it gives us a baseline for restoration.  We will make an effort to redistribute the data as best as possible to other servers, so as to avoid a major loss like this in the future.

- There is still a possibility that the data center will recover / decrypt the data on the offending disks, so we are hoping that we can restore the data directly, but it still won't be accessible the same way it was previously.  What's more, we will not be utilizing this data center for any other server deployments going forward.

Please bear with us while we handle the gargantuan task of rebuilding our data store.  In the meantime, we ask that you open a ticket if you require something to install on, or rebuild, a virtual server instance.

No comments: