Dear Client,

Over the past 48 hours, we have been battling server downtime issues worse than any that I have dealt with in my 10 years in the business.  What started as a simple hard drive failure (we plan for those) grew into a 3-drive failure in our one server node.  From there, we restored the server, only to have the hard drives fail again.

So, we made the call to migrate every one to an entirely new server node and stop forcing the issue with a setup that had clearly degraded into a fault-ridden setup.  This turned out to be the right call.

After migration, there were many issues with sites.  First off, many clients lost data that had been saved in the last 24 hours.  We backup everyone's websites daily at no cost to you.  This is better than nothing, but there was still some data loss.

Second, while our datacenter technicians were dealing to the larger issues of network connectivity, server configurations, and drive stability, I personally went through every one of our clients' websites and confirmed/repaired MySQL database connections, SSL certificate installations, FTP logins (where possible), site functionality, and file integrity.

In sum, this has been a 48 hour struggle for us, our datacenter technicians, and you, our clients.

Outages like this cascade through the tiers of clients.  We are but one client for our datacenter company.  However, through us, this outage affects hundreds of businesses that rely on us to keep their websites up and their storefronts open.  For this, we apologize.

It's not our fault (or anyone's fault) that hard drives failed - this happens every day, though not to the extent of this situation.  However, you put your business in our hands because you trust that we are better prepared for these situations than you would be on your own.  It's here that we let you down.

In our post-downtime assessment, this is what we've realized went right:

* Our backups worked and we were prepared for the restore process.
* Active server monitors made us aware of the issue instantly.
* Server technicians were on the issue within minutes.
* Our client databases gave us the configuration data we needed for each of your websites to be re-configured properly.
* Our Twitter feed (@elecfunction) stayed up to date with the latest news.

But, more importantly, here's where we will improve:

* We lost email contact with the loss of our DNS server.  We need to have a failover DNS server to keep our email alive so that we can stay in better touch.
* We should have immediately prepared another server node in the event that the first node could not be salvaged.  Instead we put our hope in recovery and not migration, and this cost us time.
* We did not have a backup client database to use for emergency contact in situations like this.
* Our backups had too much old data which slowed the restore process.  We should keep them fresh and agile.

That's our preparation plan for the future, and something that's being put into action right now.

On top of that, we are extending an offer for free assisted migration of your website to another hosting provider.  We hope that you will stick with us but if you have decided to move elsewhere, we'll help you move at no cost.

Of course, if you are having issues with your website right now, please email support@heroframework.com to create a ticket and we will fix the issue promptly.

Finally, if you would like to discuss this further with me, please give call 1-888-828-4186 or email brock@electricfunction.com and I will gladly talk to you (if available) or setup a time to discuss what went wrong, what will be fixed in the future, and your options right now.

Again, I apologize for the past two days of downtime and sincerely appreciate your business.  You put your trust in us because you know we care about your website and your business and I hope the lengths that we have gone in the past two days to bring things back online show that you were right in doing so.

Sincerely,
Brock Ferguson
President
Electric Function.com