Robust Backup with File System and Data Verification

2013-04-14 • SEND FEEDBACK •
Related: backup, Carbon Copy Cloner, cloning, data integrity, diglloydTools, IntegrityChecker

diglloydTools
(data integrity/validation)

A robust professional backup process includes these attributes:

• Physical redundancy: at least three separate backups stored safely away from the computer, preferably more backups as the value of the data rises, and preferably not all backups stored in the same place.
• Process: not wiping out all backups while backing up (hence at least three backups so one can cycle through them on a rotating basis).
• Verification of file system integrity on the backup drives.
• Verification of data integrity for all files on the backup drives after the backup or at any later time (IntegrityChecker).

Working example

Note: my system is a multi-redundant effort as shown here (multiple backups done at once). The simplest form is one original to one backup. See Cloning as a Backup Strategy for a simplified use case.

My data is essential to my living, so I cannot afford to risk losing it. Accordingly, I maintain multiple sets of backups (each with multiple drives). Shown below are volume icons on the desktop as part of a periodic (monthly) backup effort.

My process is made less onerous by first making clone backups to one transfer drive, the four 'xfer' volumes xfer.Boot, xfer.Master, xfer.Archive1, xfer.ArchiveP, which are clone backups of my master (original) volumes: Boot, Master, Archive1, ArchiveP. So xfer.* is itself a backup drive (with four volumes on it).

After the 'xfer' volumes are brought up to date, all volumes (partitions) on it are themselves perfect clone backups.

To avoid the nuisance of the full backup process interfering with my daily work, I move the 'xfer.*' drive to a 2nd machine which has all these other backup drives online (temporarily for the backup). That machine becomes a “backup mule” which I can ignore while I continue my work on my primary workstation.

Recommended external backup drive: OWC Mercury Elite Pro with USB3 + eSATA + Firewire ports, 3TB or 4TB.

Backup up 'xfer' partitions to three external clone backups

Process

Disable SpotLight on all backup drives, unless you prefer a 8-hour backup to a 2-hour backup (yet another case of poor Apple judgment—degrades workflow).

Basic process:

1. Clone xfer.Master to b1.Master.YYYY-MMDD.
2. Clone xfer.Boot to b1.Boot.YYYY-MMDD.
3. Clone xfer.Archive1 to b1.Archive1.YYYY-MMDD.
4. Clone xfer.ArchiveP to b1.ArchiveP.YYYY-MMDD.
5. Repeat for remaining volumes.
6. Verify file systems on all backup volumes (more on that below).
7. Verify data integrity for all files on all backup volumes (IntegrityChecker), can be run in parallel with one IntegrityChecker for each drive/volume.
8. Store backups in safe place way from computer.

Optimizing with multiple backups

Making this many backups can take a while. For big jobs like this, some ordering tricks can be done:

• After an 'xfer' volume is cloned to one backup drive, that clone can be cloned simultaneously to a different backup drive. Just avoid competing clones on the same drive or everything will slow way down due to hard drive contention.
• Use an N-way RAID-1 mirror to make N identical copies onto N drives. But with this approach, don’t put all your eggs into one basket: have other backups also.

Avoid any approach that causes contention for the hard drive as this will slow things down tremendously.

Data verification

diglloydTools

The use of IntegrityChecker to verify perfect data integrity (no corruption) is strongly advised for valuable data.

Since the integrity check “passes through” to an Nth-generation copy, it is sufficient to validate the last copy; it cannot be correct if an intermediate started out corrupted.

Note well that the file system can be intact even while particular files are corrupted (damaged). Or vice versa.

OWC ThunderBay 4
High capacity, high-performance fault-tolerant storage for photography and video.

Non-RAID or RAID-0/1/4/5/10.
Capacities up to 72 Terabytes!

More about OWC Thunderbay 4...

Hard drives or SSD.

File system sanity check

Do not assume that the file system is good on the backup drives.

For efficiency, I use a script to verify file system integrity all mounted volumes in one command, with errors being immediately obvious at a glance in the output. Verification both before and after making the clone backups is advised.

Verifying the file system (example)

Show below is output from running the script to verify all mounted volumes. It is trivially easy to see at a glance if there are any issue— none here.

llcMule:Desktop lloyd$ ./verifyMountedVolumes.sh 
Verifying the file system on all mounted volumes...
Volume: Mule
Started file system verification on disk4s2 Mule
Error: -69888: Couldn't unmount disk
Volume: a2
Started file system verification on disk5s2 a2
Checking file system
Checking Journaled HFS Plus volume
Checking extents overflow file
Checking catalog file
Checking multi-linked files
Checking catalog hierarchy
Checking extended attributes file
Checking volume bitmap
Checking volume information
The volume a2 appears to be OK
Finished file system verification on disk5s2 a2
Volume: b1.Archive1.2013-0408
Started file system verification on disk1s7 b1.Archive1.2013-0408
Checking file system
Checking Journaled HFS Plus volume
Checking extents overflow file
Checking catalog file
Checking multi-linked files
Checking catalog hierarchy
Checking extended attributes file
Checking volume bitmap
Checking volume information
The volume b1.Archive1.2013-0408 appears to be OK
Finished file system verification on disk1s7 b1.Archive1.2013-0408
Volume: b1.ArchiveP-2013-0408
Started file system verification on disk1s9 b1.ArchiveP-2013-0408
Checking file system
Checking Journaled HFS Plus volume
Checking extents overflow file
Checking catalog file
Checking multi-linked files
Checking catalog hierarchy
Checking extended attributes file
Checking volume bitmap
Checking volume information
The volume b1.ArchiveP-2013-0408 appears to be OK
Finished file system verification on disk1s9 b1.ArchiveP-2013-0408
Volume: b1.Boot.2013-0408
Started file system verification on disk1s3 b1.Boot.2013-0408
Checking file system
Checking Journaled HFS Plus volume
Checking extents overflow file
Checking catalog file
Checking multi-linked files
Checking catalog hierarchy
Checking extended attributes file
Checking volume bitmap
Checking volume information
The volume b1.Boot.2013-0408 appears to be OK
Finished file system verification on disk1s3 b1.Boot.2013-0408
Volume: b1.Master.2013-0408
Started file system verification on disk1s5 b1.Master.2013-0408
Checking file system
Performing live verification
Checking Journaled HFS Plus volume
Checking extents overflow file
Checking catalog file
Checking multi-linked files
Checking catalog hierarchy
Checking extended attributes file
Checking volume bitmap
Checking volume information
The volume b1.Master.2013-0408 appears to be OK
Finished file system verification on disk1s5 b1.Master.2013-0408
Volume: b2.Archive1.2013-0408
Started file system verification on disk0s7 b2.Archive1.2013-0408
Checking file system
Checking Journaled HFS Plus volume
Checking extents overflow file
Checking catalog file
Checking multi-linked files
Checking catalog hierarchy
Checking extended attributes file
Checking volume bitmap
Checking volume information
The volume b2.Archive1.2013-0408 appears to be OK
Finished file system verification on disk0s7 b2.Archive1.2013-0408
Volume: b2.ArchiveP-2013-0408
Started file system verification on disk0s9 b2.ArchiveP-2013-0408
Checking file system
Checking Journaled HFS Plus volume
Checking extents overflow file
Checking catalog file
Checking multi-linked files
Checking catalog hierarchy
Checking extended attributes file
Checking volume bitmap
Checking volume information
The volume b2.ArchiveP-2013-0408 appears to be OK
Finished file system verification on disk0s9 b2.ArchiveP-2013-0408
Volume: b2.Boot.2013-0408
Started file system verification on disk0s3 b2.Boot.2013-0408
Checking file system
Checking Journaled HFS Plus volume
Checking extents overflow file
Checking catalog file
Checking multi-linked files
Checking catalog hierarchy
Checking extended attributes file
Checking volume bitmap
Checking volume information
The volume b2.Boot.2013-0408 appears to be OK
Finished file system verification on disk0s3 b2.Boot.2013-0408

Volume: b2.Master.2013-0408
Started file system verification on disk0s5 b2.Master.2013-0408
Checking file system
Checking Journaled HFS Plus volume
Checking extents overflow file
Checking catalog file
Checking multi-linked files
Checking catalog hierarchy
Checking extended attributes file
Checking volume bitmap
Checking volume information
The volume b2.Master.2013-0408 appears to be OK
Finished file system verification on disk0s5 b2.Master.2013-0408
Volume: x.Archive1.2013-0408
Started file system verification on disk2s4 x.Archive1.2013-0408
Checking file system
Checking Journaled HFS Plus volume
Checking extents overflow file
Checking catalog file
Checking multi-linked files
Checking catalog hierarchy
Checking extended attributes file
Checking volume bitmap
Checking volume information
The volume x.Archive1.2013-0408 appears to be OK
Finished file system verification on disk2s4 x.Archive1.2013-0408
Volume: x.Boot
Started file system verification on disk2s2 x.Boot
Checking file system
Checking Journaled HFS Plus volume
Checking extents overflow file
Checking catalog file
Checking multi-linked files
Checking catalog hierarchy
Checking extended attributes file
Checking volume bitmap
Checking volume information
The volume x.Boot appears to be OK
Finished file system verification on disk2s2 x.Boot
Volume: x.Master-2013-0408
Started file system verification on disk2s3 x.Master-2013-0408
Checking file system
Checking Journaled HFS Plus volume
Checking extents overflow file
Checking catalog file
Checking multi-linked files
Checking catalog hierarchy
Checking extended attributes file
Checking volume bitmap
Checking volume information
The volume x.Master-2013-0408 appears to be OK
Finished file system verification on disk2s3 x.Master-2013-0408
Volume: xfer.Archive1
Started file system verification on disk3s7 xfer.Archive1
Checking file system
Checking Journaled HFS Plus volume
Checking extents overflow file
Checking catalog file
Checking multi-linked files
Checking catalog hierarchy
Checking extended attributes file
Checking volume bitmap
Checking volume information
The volume xfer.Archive1 appears to be OK
Finished file system verification on disk3s7 xfer.Archive1
Volume: xfer.ArchiveP
Started file system verification on disk3s9 xfer.ArchiveP
Checking file system
Checking Journaled HFS Plus volume
Checking extents overflow file
Checking catalog file
Checking multi-linked files
Checking catalog hierarchy
Checking extended attributes file
Checking volume bitmap
Checking volume information
The volume xfer.ArchiveP appears to be OK
Finished file system verification on disk3s9 xfer.ArchiveP
Volume: xfer.Boot
Started file system verification on disk3s3 xfer.Boot
Checking file system
Checking Journaled HFS Plus volume
Checking extents overflow file
Checking catalog file
Checking multi-linked files
Checking catalog hierarchy
Checking extended attributes file
Checking volume bitmap
Checking volume information
The volume xfer.Boot appears to be OK
Finished file system verification on disk3s3 xfer.Boot
Volume: xfer.Master
Started file system verification on disk3s5 xfer.Master
Checking file system
Checking Journaled HFS Plus volume
Checking extents overflow file
Checking catalog file
Checking multi-linked files
Checking catalog hierarchy
Checking extended attributes file
Checking volume bitmap
Checking volume information
The volume xfer.Master appears to be OK
Finished file system verification on disk3s5 xfer.Master