IntegrityChecker Java: Simultaneous Data Verification on any Number of Hard Drives or SSDs

2021-10-08 • SEND FEEDBACK |
Related: backup, data integrity, diglloydTools, hard drive, How-To, IntegrityChecker, Java, SSD

See also: Checking for Data Corruption after macOS Hoses Your System

I let some of my tertiary backup drives go stale for some months, and finally brought them all up to day today. I like to do that after coming back from a trip.

With hard drives, multiple backups can take a loooooong time, with the 8TB drives and 14TB drives mostly full, though there are some tricks to also doing simultaneous backups*.

After backing up, verifying data integrity on all 8 drives can be done very efficiently with IntegrityChecker Java (icj). All eight drives can be verified simultaneously*, using one instance of icj (one per Terminal window).

Since each drive is independent of the others, the only gating factors are I/O bandwidth (no problem in an OWC Thunderbay), and CPU power (ample on the 8-core 2018 iMac 5K. The hard drives are slow enough that each drive takes only ~35% of one CPU core). In other words, the time it takes depends entirely on how fast (slow) the drives are. A 4-core CPU would see higher CPU utilization of course.

Avoid simultaneous backups or data verification if the data is on the same physical hard drive (such as two volume partitions); it will cause the drive to bang its drive heads continuously and run very slower. SSDs do not have this limitation, as there are no moving parts.

IntegrityChecker Java: Simultaneous Data Verification on any Number of Hard Drives at Once

Below, 8 backup volumes totaling 71TB were validated, showing perfect data integrity—not a single bit in error.

IntegrityChecker Java: 8 backup volumes totaling 71TB validated with perfect data integrity