How to Verify Multiple Backups Simultaneously (IntegrityChecker)

2014-02-08 • SEND FEEDBACK •
Related: backup, command line, data integrity, diglloydTools, How-To, IntegrityChecker, sudo, Terminal.app

IntegrityChecker

See How to Safely Transfer Data or Verify Backups for the simple and common case.

This is an advanced page for professionals and advanced users looking to manage many backups and to check their integrity efficiently (in parallel).

First, realize that with hard drives, attempting to verify multiple partitions on the same physical drive is self-defeating: the drive heads will seek wildly back and forth between partitions (“drive head convention”), causing a massive slowdown (the same idea applies to SpotLight— disable Spotlight on all backups).

Simultaneous data verification on multiple volumes

Perhaps a system includes a 4-bay unit with multiple backups as four independent drives/volumes, and the wish is to verify the contents of each with IntegrityChecker.

Verifying one volume, then another, then another, etc takes a long time, so why not verify all volumes simultaneously?

Open a Terminal window for each volume and invoke IntegrityChecker on each*.

In each Terminal window, verify the volume by name. Volume names shown in bold, use quotes around the volume name if the name contains space(s):

sudo ic verify MasterClone-2014-0208 # in the 1st Terminal window

sudo ic verify Archive1Clone-2014-0208 # in the 2nd Terminal window

sudo ic verify ArchivePClone-2014-0208 # in the 3rd Terminal window

... etc ...

That’s all there is to it. The verification will run in parallel. However, it will not be a speedup if the volumes are partitions on the same physical drive (see discussion above), so apply this technique only if the volumes are on separate physical drives.

* Use of 'sudo' recommended to avoid OS X permissions errors, which can occur with certain types of volume setups.

Shown below are six 'ic' (IntegrityChecker) processes running on six different drives/volumes. The quad-core Retina MacBook Pro is doing around 740 MB/sec on average and the 4 real CPU cores are maxed out.

Six simultaneous IntegrityChecker 'ic' processes processing six different volumes/drives.

What about verifying several partitions on the same drive?

Sometimes the goal is to verify several partitions on the same physical drive. As discussed, this should not be done in parallel on a hard drive (head contention). But there is a way to start things going and walk away. Simply gang up several commands separated by the “;” character; these will execute one after the other.

Using the above volume names, this invocation will execute each one after the other in one Terminal window:

sudo ic verify MasterClone-2014-0208; sudo ic verify Archive1Clone-2014-0208; sudo ic verify ArchivePClone-2014-0208