All Posts by Date or last 15, 30, 90 or 180 days.
also by Lloyd: diglloyd.com photography and WindInMyFace.com

Thank you for buying via links and ads on this site,
which earn me advertising fees or commissions.
As an Amazon Associate I earn from qualifying purchases.

Other World Computing...
B&H Photo...
Amazon
As an Amazon Associate I earn from qualifying purchases.
877-865-7002
Today’s Deal Zone Items... Handpicked deals...
$155 $100
SAVE $55

$2198 $1998
SAVE $200

$1799 $1599
SAVE $200

$999 $779
SAVE $220

$2397 $1897
SAVE $500

$1299 $949
SAVE $350

$799 $549
SAVE $250

$1499 $1099
SAVE $400

$799 $549
SAVE $250

$2797 $2497
SAVE $300

$1997 $1797
SAVE $200

$549 $499
SAVE $50

$1699 $949
SAVE $750

$240 $175
SAVE $65

$3399 $2699
SAVE $700

$1100 $880
SAVE $220

Memory Upgrades for 2019 Mac Pro - Save Up to 65% vs Factory Costs

Validating Data Transfer/Copy Before Deleting the Source/Original Data: diglloydTools IntegrityChecker

See also Data Integrity Over Time, and with OS Changes and Data Loss Prevented: IntegrityChecker Saves my Bacon by Detecting Corrupted Files after a Clone.

For background see: Detecting Corruption / Validating Data Integrity Over Time and Across Drives and Backup/Restore.

Every time I take a photography trip, I come back with something between 100GB and 800GB of new image files. These files typically reside on the internal SSD of my 2019 iMac 5K, which I use in the field in my Mercedes Sprinter photography adventure van.

When I return home, the first thing I do is to offload the trip photos to my primary working drive (an 8TB OWC Thunderblade). Recent years have shown me that the macOS Finder is prone to some serious bugs with file copying, so there is just no way I am going to assume that it did the copy correctly.

Accordingly, I want to verify that the transferred items are absolutely identical before I delete the items off the internal SSD.

Procedure for validating a data transfer

Here is my procedure, which presumes a backup has already been made:

  1. Run IntegrityChecker 'update' on the files to be copied*.
  2. Copy the files to their new home.
  3. Run IntegrityChecker verify on the copies and the originals*.
  4. Compare the results—they should be identical with no warnings of any kind.
  5. Assuming things are identical, delete the originals.

This procedure guarantees that no matter what might happen, any issue in the copying process (missing/skipped files, bit flips, etc) are caught. Or for that matter, anything else nefarious that might possibly happen.

* Use the Java version, as it can also flag folders that have been deleted since the last update (a missing folder by definition has no files and no hashes).

Example

On the source drive (internal SSD):

Do an 'update' first as noted above. The verify will then show "Without hash: 0", because all files will have a cryptographic hash.

The elegance of this approach is terrific: at any future time, this copy or the 100th copy of this copy on any media can be verified again—no original needed, just verify the copy. A single command verifies the folder and all subfolders, whether 1MB or 500 terabytes.

diglloyd-iMac:2019-1204-TripPhotos lloyd$ icj verify /Master/2019-1204-TripPhotos/
...
=================================================================================
FILE STATUS SUMMARY for 105 folders 2019-12-18 14:48:34
=================================================================================
# With hash: 14029
# Without hash: 0
# Missing : 0
# Hashed: 14029
# Changed size: 0
# Changed date: 0
# Changed content + date, size unchanged: 0
# Total changed content: 0
# SUSPICIOUS: 0

On the destination drive (external SSD):

The files have been copied. Verify them. The number of files "With hash" should match and there should be zero files with changes of any kind.

diglloyd-iMac:DIGLLOYD lloyd$ icj verify /Work/2019-1204-TripPhotos/
.....
==================================================================================
FILE STATUS SUMMARY for 105 folders 2019-12-18 16:12:23
==================================================================================
# With hash: 14029
# Without hash: 0
# Missing : 0
# Hashed: 14029
# Changed size: 0
# Changed date: 0
# Changed content + date, size unchanged: 0
# Total changed content: 0
# SUSPICIOUS: 0

Save the tax, we pay you back, instantly!
View all handpicked deals...

Sony FE 24-70mm f/2.8 GM Lens
$2198 $1998
SAVE $200

diglloyd.com | Terms of Use | PRIVACY POLICY
Contact | About Lloyd Chambers | Consulting | Photo Tours
Mailing Lists | RSS Feeds | Twitter
Copyright © 2020 diglloyd Inc, all rights reserved.
Display info: __RETINA_INFO_STATUS__