Online Back-Up Company Loses Customer Data
By Anna Johnson on March 25th, 2009Imagine paying money to have a third party back-up your data – all your business records, customer databases, etc – and then have that third party LOSE it all.
Unfortunately, this is all too real. In 2007 online backup and storage provider Carbonite lost the data of 54 of its 7,500 customers, allegedly due to running defective software and hardware.
The situation has led Carbonite to sue two of its service providers, whose technology, Carbonite alleges, was defective and led to the loss of its customers’ data, and damage to its business and reputation.
There are a few lessons arising from this. Firstly, make sure you have local copies of your data before you back it up. Secondly, make sure you have watertight agreements in place with YOUR service providers to ensure that if anything goes wrong… they pay up!
March 26th, 2009 at 1:06 pm
I would like to make sure that your readers understand two points with regard to Carbonite’s lawsuit against Promise Technologies:
1) This event happened over a year ago. We do not say this to minimize the matter. But we do want to point out that this has not happened in a long time and is not an ongoing problem.
2) The total number of Carbonite customers who were unable to retrieve their data was 54, not 7,500.
Here is what happened: The Promise servers that we were purchasing in 2006 and 2007 use RAID technology to spread data redundantly across 15 disk drives so that if any one disk drive fails, you don\’t lose any data. The RAID software that makes all this work is embedded as \"firmware\" in the storage servers. In this case, we believe that the firmware on the servers had bugs that caused the servers to crash. Carbonite automatically restarted all 7,500 backups and more than 99% of these were completely restored without incident. Statistically, about 2 out of every 1,000 consumer hard drives will crash every week, so 54 of these customers had their PCs crash before their re-started backups were complete. Since they weren’t completely backed up when their PCs crashed, these customers were unable to restore all of their files from Carbonite. Most of the 54 got some or most of their data back. We took full responsibility for what happened and I did my best to call each of these customers personally to apologize.
As a result of our problems with the Promise servers, we switched to a popular Dell server that uses RAID6 – an improved RAID that allows for the loss of 3 of the 15 drives simultaneously before you lose any data. This configuration is in theory 36 million times more reliable than a single disk drive — the chances of 3 out of 15 drives failing at the same time are almost nil.
So far, Promise has refused to accept responsibility for their equipment’s failures, so now we are suing them to get our money back. The Dell RAID servers have been flawless and we\’re extremely happy with them.
Dave Friend, CEO
Carbonite, Inc.
March 26th, 2009 at 5:36 pm
Thanks for clarifying Dave. While a small percentage of total customers lost their data, it’s obviously a nightmare for those who were affected, including Carbonite.
March 27th, 2009 at 11:24 am
Carbonite should not lose customer’s data due to faulty equipment, however, if they have every right to go after an equipment supplier for faulty equipment. Amidst all this, I switched to MyOtherDrive.com and have been very happy.
March 27th, 2009 at 3:04 pm
It just goes to show that these companies aren’t infallible. While online backup is convenient, it is important to make sure your data is safe. I’d choose some of the big guys for backing up my data like http://www.myotherdrive.com, http://jungledisk and http://mozy
April 1st, 2009 at 4:45 am
Well, there is no surprise. I think people are too much into the cloud computing lately. This cannot be trusted 100% especially if we’re talking about backup. So, you always want to make sure you have a backup copy of what you are backing up online on a hard drive (better NAS appliance) and DVDs.
I also switched to Novosoft Remote Backup (www.remotedatabackup.net) since they offer a very decent bundle of online backup account of over 100Gb storage and free software very handy for managing my backups on-site.
April 6th, 2009 at 5:59 pm
After reading Mr. Friend’s comments, I think that his response itself may be misleading. It appears that in fact Carbonite had an equipment failure. The equipment which failed had ~7,500 user accounts. At the time the equipment failed, all of the data on the equipment itself was LOST.
After the failure, Carbonite’s servers were restarted/fixed/reformatted, or accounts moved to another server or whatever was required…at which time all of the customer PCs were instructed to reinitialize and re-send a full backup of all data up to Carbonite’s servers.
While statistically, Mr. Friend is probably correct..in that only a handful of customers probably actually had failures during the reload period (and hence permanently lost data), it is also correct to report that the data in 7,500 user accounts were lost for some period of time. Furthermore, it is also probably correct to say that any files that were deleted in the past 30 days from a user’s account would not have been recoverable from the customer’s PC – since they were deleted locally. Hence, these are permanently lost…but likely not needed anyway.
A quick read of the Carbonite T&C’s reveals that is is unlikely that Carbonite suffered significant loss of service revenue…except of course refunds to those “unlucky 54.” Reputation damage is of course real.
Given the way that technology is rushed out the door these days, it is highly likely that vendor equipment failure was indeed a major factor in the data loss. Or course, much like Carbonite, I would expect that Promise has similarly worded T&C’s which would limit its liability and permit it ample opportunity to fix any problems a customer complained about. Carbonite must truly believe it can reach a settlement with this vendor to recoup some of its capx investment….or it would have let this matter fade away. As it stands now, the lawsuit and associated headlines will remain memorialized in Google cache forever.