Small Office Disaster Recovery == Disaster (--Recovery)

The office is small. Really small. So small that you can answer the front door with one hand while you use the other hand to spray roach repellent around the back door. The mobile workforce that the office directly supports is equally small and I'm partially responsible for supporting both of them. I say "partially" because I work part time... on a volunteer basis... from 1,000 miles away. The remaining part of the time that I'm not helping the office the users create their own "solutions". By "solutions" I of course mean "catastrophes". Causing more pain is the fact that the laptops are used for both personal and business reasons so who knows what data ends up where. Oh, and this is a non-profit so resources are exceptionally tight. And did I already mention that I'm 1,000 miles away from the office? Okay, great. With that back story in mind...

The multi-tiered data protection scheme for mobile workers involves:
1. Making sure that you always have a firm grip on the laptop. Preferably using both hands.
2. Prayer (second on the list because God does not answer prayers for data integrity after using your netbook in a game of frisbee golf. Trust me.)
3. Acronis Home version 12
4. File copy backup of the files and folders in the user's profile folder.

Steps three and four are repeated at the user's whim. Fortunately, these particular users had a healthy paranoia about their data and made frequent backups. Apparently they used Acronis to image the laptops and would also open their user profile folders and copy / paste all data to an external drive. I was happy enough, but never checked to see if they had used Acronis's continuous protection features to backup files automatically each and every day.

One day, when both mobile users where on the road for a teaching event, I got a voicemail from one of them. Apparently her laptop was freezing just seconds after starting to load Windows. Of course, they're also a day away from embarking on a week long teaching event. Sweet! The only bright spot so far is that the two mobile workers were together and had copied their vital teaching files to eachother's laptops so they could share laptops in case of just such an event.

From the descriptions of the problems that I could get over the phone, the issue was some kind of corrupted system file. A repair installation wouldn't work because the Windows installer didn't even see a partitioned hard drive. Just a big unpartitioned block of unknown space. Double sweet! I had to wait until the end of the week for them to get back to the office where more resources would allow me to remotely look into the issue.

Once they were in the office, me being 1,000 miles away still made it virtually impossible to attempt any recovery of the hard drive with boot CD tools (like the System Rescue CD or the hard drive tools provided by the manufacturer). Yes, I have a KVM over IP switch in the office (Lantronix Spider). I could have had them connect the drive to a desktop and then boot off of a rescue CD and etc., but that was a last resort. One user took about 2 hours to figure out over the phone how to hook up and use an external SATA/IDE to USB device and I didn't want to attempt to coach them through setting up a KVM over IP device on top of it. And besides! I didn't need to do that since the user took an image of their hard drive just a day or two before leaving on their teaching tour!! Huzzah!

One problem: The Acronis image (a .TIB file) was unable to be restored due to some strange error that said "This is not the last volume of the backup" or something similar. Reports of this error abounded on the internets but no one had a clear solutions. Acronis's support forums were no help either. Nothing I could do would open it. I even tried VMWare Converter. In the end, it seemed to be a simple case of file corruption. Suh-weet!!

Next I had the remote user connect the ailing drive to a desktop via a SATA/IDE to USB adapter (mentioned briefly above) but I couldn't browse the drive. Opening My Computer after connecting the drive caused the explorer window to hang for several minutes and finally throw some kind of "semaphore timeout" error. While I puzzled over that mess I decided to look into the user's attempt at a file copy of everyhing in their profile folder.

The one problem with that backup "solution" was that hidden files and folders were not enabled in their profile settings and as a result the "Local Settings" folder was not copied. The Local Settings folder is famous for containing the Outlook personal folder files that user have grown so fond of. Thusly, one of the most crucial stores of information was missing. (Did I mention they still use POP3 mail accounts that download mail from off the server and don't leave copies?)

The final hope for any recent copies of data was the faltering hard drive itself. The user's son (himself a competent tech-head in his thirties; i.e not a pimply level 57 WoW freak) had told his parent about some kind of data recovery tool that he used with success in the past. It was called Zero Assumptions Recovery (ZAR for short). The son also provided a key to unlock the trialware. "Don't ask questions, don't ask questions..." I mumbled to my whimpering conscience.

Using ZAR I was able to see the file and folder structure of the drive, but was unable to attempt a file based recovery. After hours of progress it stalled and would not go further. No errors provided. Just a frozen progress bar. Within ZAR was an option to recover an entire disk to an .img file rather than a file based recovery. I chose that and several hours later was happy to see a complete 70+GB disk image had been recovered. I opened it with WinRAR and was able to browse directories and even recover files!

This saved the day, as there were several important files that were needed for a surprise early board meeting that had been called for the next day (the fun just doesn't stop!). However, to my dismay, the .PST files in the users Local Settings folder that were recovered from the newly created .img file did not behave the way that I had hoped. When connecting them to a fresh installation of Outlook (did I mention they were still using Outlook 2002?), an error message informed me that it was not a proper PST file.

Suh-WEET! ::fist pump::

To make a long story longer, I tried virtually every outlook recovery tool known to humankind and came to the conclusion that they all do the same things under the hood but just have different UIs. In fact, many of them don't even bother to change the UI. I think there's some PST recovery application sweat shop in Taiwan that pumps out these products and markets them to private consulting firms that then brand them and resell them for a massive markup.

In the end, the user had to revert back to a PST file that had been backed up via an Outlook export about a year prior. The laptop had to have a new hard drive (obviously), Windows was reinstalled as were all of the users applications and files were manually moved to the new user profile from what was recovered out of the .img file. On the bright side, the user got a bigger hard drive and a cleaner system out of the ordeal. Why all files seemed to be perfectly fine except the .PST files is beyond me.

What did I get out of it? Bad posture from sitting in my chair for 4 days straight, a regret that I ever lobbied to have an IP phone connected to their phone system so I was only a local extension away and a sharpened paranoia concerning backup integrity. The perils of working for a non profit, being a volunteer and working remotely converged into one major Tylenol moment. However, now that I think about it, they can't fire a volunteer... maybe that's the one bright spot in this story.

Now I'm looking into a better backup solution that could hopefully prevent such failures in the future and keep me from wasting a week with inane busywork trying to copy and paste the right files back to a fresh Windows installation. Backup Exec for Windows Small Business Servers and DriveImageXML seem to be decent products at the moment. Any suggestions would be appreciated.

And lest anyone uses words like "The Bat" and "Amanda" let me just say -- thanks, but no thanks.