SEARCH
diglloydTools™
Validating Data Transfer/Copy Before Deleting the Source/Original Data: diglloydTools IntegrityChecker
Related: backup, data integrity, data transfer, diglloydTools, iMac, IntegrityChecker, SSD
See also Data Integrity Over Time, and with OS Changes and Data Loss Prevented: IntegrityChecker Saves my Bacon by Detecting Corrupted Files after a Clone.
For background see: Detecting Corruption / Validating Data Integrity Over Time and Across Drives and Backup/Restore.
Every time I take a photography trip, I come back with something between 100GB and 800GB of new image files. These files typically reside on the internal SSD of my 2019 iMac 5K, which I use in the field in my Mercedes Sprinter photography adventure van.
When I return home, the first thing I do is to offload the trip photos to my primary working drive (an 8TB OWC Thunderblade). Recent years have shown me that the macOS Finder is prone to some serious bugs with file copying, so there is just no way I am going to assume that it did the copy correctly.
Accordingly, I want to verify that the transferred items are absolutely identical before I delete the items off the internal SSD.
Procedure for validating a data transfer
Here is my procedure, which presumes a backup has already been made:
- Run IntegrityChecker 'update' on the files to be copied*.
- Copy the files to their new home.
- Run IntegrityChecker verify on the copies and the originals*.
- Compare the results—they should be identical with no warnings of any kind.
- Assuming things are identical, delete the originals.
This procedure guarantees that no matter what might happen, any issue in the copying process (missing/skipped files, bit flips, etc) are caught. Or for that matter, anything else nefarious that might possibly happen.
* Use the Java version, as it can also flag folders that have been deleted since the last update (a missing folder by definition has no files and no hashes).
Example
On the source drive (internal SSD):
Do an 'update' first as noted above. The verify will then show "Without hash: 0", because all files will have a cryptographic hash.
The elegance of this approach is terrific: at any future time, this copy or the 100th copy of this copy on any media can be verified again—no original needed, just verify the copy. A single command verifies the folder and all subfolders, whether 1MB or 500 terabytes.
diglloyd-iMac:2019-1204-TripPhotos lloyd$ icj verify /Master/2019-1204-TripPhotos/ ... ================================================================================= FILE STATUS SUMMARY for 105 folders 2019-12-18 14:48:34 ================================================================================= # With hash: 14029 # Without hash: 0 # Missing : 0 # Hashed: 14029 # Changed size: 0 # Changed date: 0 # Changed content + date, size unchanged: 0 # Total changed content: 0 # SUSPICIOUS: 0
On the destination drive (external SSD):
The files have been copied. Verify them. The number of files "With hash" should match and there should be zero files with changes of any kind.
diglloyd-iMac:DIGLLOYD lloyd$ icj verify /Work/2019-1204-TripPhotos/ ..... ================================================================================== FILE STATUS SUMMARY for 105 folders 2019-12-18 16:12:23 ================================================================================== # With hash: 14029 # Without hash: 0 # Missing : 0 # Hashed: 14029 # Changed size: 0 # Changed date: 0 # Changed content + date, size unchanged: 0 # Total changed content: 0 # SUSPICIOUS: 0
