SEARCH
diglloydTools™
Yet Again, IntegrityChecker Saves Me from Major Data Loss — Corrupted Files in Multiple Backups
Related: backup, data integrity, diglloydTools, IntegrityChecker, Java, laptop, MacBook
re: data integrity
re: IntegrityChecker Java
re: backup
Any photographer or similar who fails to use use a program like IntegrityChecker Java is making a very bad move—sooner or later you will be hit by bit rot, software bugs, user mistakes, etc. Don’t get caught with your pants down, either by poor backup hygiene, or the undetectable surprise of data corruption!
re: Major Data Loss Today — But Luckily I Had Just Made Several Backups
re: Data Loss Prevented: IntegrityChecker Saves my Bacon by Detecting Corrupted Files after a Clone
I was doing a major backup operation as I tore-down my 2019 Mac Pro (now sold). It ended up taking over four days (!) because of a failing (and critical) 8TB SSD that first became extremely slow and flaky and then rapidly degraded to the point of going into read-only mode. And two others performing badly (when it rains, it pours).
The chores including switching over to the 2021 MacBook Pro M1 Max as interrim machine, which turned out to be a major effort because of software and hardware differences.
UPDATE 2023-07-12: the MBP M1 Max proved to be the best machine for my work ever, at home or on the road... well, 64GB proved not quite enough once (very big job) but otherwise superb. See Apple 2021 MacBook Pro M1 Max: Continues to Impress as Faster than even my 28-core Mac Pro.
As the backups were finishing, I entered the data integrity verification phase. Are you verifying data integrity with your backups? If not, you are setting your self up for a lot of pain someday, guaranteed.
Corrupted backups
UPDATE 2023-06-19: this problem cropped up again on several backups, but this time with plain text files so I could directly look-at and diff the source and the backup copy—obviously different, but same file size and same file dates! Clearly CCC did not copy the file contents, but appears to have set the file date to match the original.
Therefore, I have concluded that it was not file corruption per se, but a failure of Carbon Copy Cloner to copy files that should have been copied, a VERY serious bug. As I have this seen this repeatedly now, my strongest possible advice is to assume that CCC may screw up. ALWAYS icj verify your backups! Whether this failure to copy a file is due to some error during copy that goes unreported, or some reliance on macOS’ changes list on APFS I do not know. But am now CERTAIN that this is a CCC bug. This is precisely the sort of thing that makes IntegrityChecker Java necessary and indeed mandatory for backup strategy.
Normally all is well and I have the peace of mind knowing that not a single bit is awry.
However, IntegrityChecker Java (for Mac, Windows, Linux) detected data corruption in a large number of DNG files in a 150GB folder. How or why they were corrupted I cannot say. But the originals verified as perfect, so I was able to remove the problem folder, replace it, and then verify data integrity perfection. Lacking that, I would have had half a dozen backups all with corrupted DNG files in that folder.
What good is a backup with corrupted data? It’s an awesomely bad workflow practice to not prove the data integrity of each and every backup. And have no way of proving data integrity when the day comes when that backup is needed for data restoration. What is poor practice for everyone could become a business-wrecking failure for professionals.
Consult with Lloyd on backup strategy and data verification.
And while some backup/cloning programs have a feature to verify copied files (slowly!), that does you no good an hour or a month or a year later to verify that the files remain non-corrupted. Worse, you are trusting the same program to verify what it might have screwed up in the first place—not a good protocol anytime anywhere—all software has bugs.
All good
Shown below, IntegrityChecker Java has just verified that ~14TiB of data in 1.25 million files is bit-for-bit identical to the original files (verification speed is limited by the speed of the SSD or hard drive).
MPB:DIGLLOYD $ icj verify Clones.Primary # icj 3.0fc24 2023-06-16 11:00 ... Hashing 1254861 files totaling 13867.8 GiB in 202297 folders... 0%: 37 files 1470 MiB @ 367 MiB/sec, 00:04.004 ... 100%: 1254861 files 13867.8 GiB @ 512 MiB/sec, 19:06:29 DONE ============================== OVERALL STATUS SUMMARY ============================== 202297 folders totaling 13867.8 GiB in /Volumes/Clones.Primary # With hash: 1254861 files # Without hash: 2 # Hashed: 1254861 # Empty: 1003 # Missing Files: 0 # Missing Folders: 0 # New Folders: 33 # Changed size: 0 # Changed date: 0 # Changed content + date, size unchanged: 0 # Total files differing: 0 # Num ignored folders: 2376 # Num ignored files: 4628 # Attributes: 1225781, 14 stale, 29082 lacking # SUSPICIOUS files: 0
