Friday, August 18, 2017

Resolved: network issues (Atlanta)

As of 2:28 AM DST our Atlanta DC's network (more specifically the subnet that contained our servers) came back up.  We apologize for any previous inconvenience, and encourage customers to contact us if they experience any lingering issues.

Thursday, August 17, 2017

Packet loss and other network issues (Atlanta)

As of 4:38 PM DST, techs in our Atlanta data center are seeing major packet loss against services hosted there.  Engineers continue to investigate, and have stabilized some elements of the network, however not all functionality has returned.  Services in Atlanta are still affected and being investigated.  We will update if necessary when everything is back to normal.  

Friday, August 11, 2017

Mid-Day Downtime explained

This post serves to explain two downtime scenarios that occurred in distinctly different regions of our infrastructure. 



The first that occurred this morning concerns continued intermittent downtime in the Fibernetics datacenter in Canada (although the issue is more related to the tenant IN the data center where we have our equipment housed, not necessarily the datacenter itself).  We shall continue to monitor these situations and plan to MOVE the respective services to a new datacenter or hardware if the problem continues.  Customers should know that these specific servers never did contain mission-critical customer data, (though our main website was on one the servers at one point, and has since been moved).  In most cases this hardware and related services are used as a testbed prior to deploying production systems.



The second downtime scenario today revolved around hardware in a New York datacenter operated by Incero - services in that DC and on the hardware involved have been extremely stable, but the underlying software (Linux operating system)needed to be upgraded during a somewhat inopportune moment.  The software "kernel" is now running stable, and should be for the foreseeable future.


If you have questions about this or any other issue, feel free to e-mail us or open a support ticket.

Monday, July 24, 2017

Resolved: Power Issues (Atlanta)

Our server is coming back online, and all services should be available shortly.  Thank you again for your patience.

Power issues affecting one server (Atlanta)

One of our main servers in our Atlanta DC is down, affecting some clients, especially those who access webmail and our forms-based services.  Actual mail delivery is not affected, as mailboxes are on different servers.


The note from the DC is: "Racks L11-20 & M11-20 have lost power due to bus failure."


Services will be up again as soon as the appropriate power is restored.  Thank you for your patience.

Thursday, July 6, 2017

Server issues in one datacenter (Canada)

A power issue in the Fibernetics  Telecomm Datacenter in Canada where we host some ancillary services* is causing these systems to remain offline.  According to outage report here:
http://outages.kwhydro.ca/gridvu/
all systems should be back online by this afternoon, based on the Eastern Standard time zone.


* Affected services include:


Canada Proxy and Backup
POP3 to Webmail gateway
Remote office, virtual desktop


If you have any questions or concerns, please open a support ticket or send us an e-mail.  Apologies for any inconvenience.

Tuesday, March 21, 2017

Resolved: Packet loss at Atlanta Datacenter

Our primary web server has now reconnected and is operating properly; however, there is still one backup system which has not yet recovered.  We are in contact with the DC and will provide an update here only if we need to take additional steps to manage customer data or notify them.  Thank you as always for your patience.