All Posts by Date or last 15, 30, 90 or 180 days.
also by Lloyd: photography and

Links on this site earn me fees or commissions.
As an Amazon Associate I earn from qualifying purchases @AMAZON

Consult with Lloyd: cameras, computers, backup, etc...
Lloyd’s Patreon
Designed for the most demanding needs of photographers and videographers.
The fastest, toughest, and most compatible portable SSD ever with speeds up to 2800MB/s.

diglloydTools IntegrityChecker Java v2.0 fc4 Available for Download

See data integrity, including recent post Data Loss Prevented: IntegrityChecker Saves my Bacon by Detecting Corrupted Files after a Clone bit rot.

IntegrityChecker java runs on any platform with a JVM. I’m looking for Windows, Linux, and NAS users to test IntegrityChecker java. Contact Lloyd.

A major update to diglloydTools IntegrityChecker is now available. Key things are noted below.

This final candidate (fc) is production-ready, or so I believe. Click through the license and download page to download:

Download IntegrityChecker Java 2.0 fc4

Documentation is largely in place but may fall a bit short in a few areas.

Buy diglloydTools (includes IntegrityChecker Java).


New feature: folder hash hierarchies

Folder hash hierarchies record all hash values for an entire folder hierarchy or volume in a single ".icjh" file. These case can be applied to any folder and can be “nested”, and are intelligently updated. All top level folders specified automatically get their own hierarchy file.

Hierarchy files can also be used in combination with traditional every-folder “.icj” files. This may be useful when rearranging lots of folders, in that the ".icj" file travels with every folder. But for most backups and data verification purposes, the ".icj" files are not superfluous.

For folders like my Apple Mail folder with 113784 folder and growing, that’s one (1) ".icjh" file instead of 113784 ".icj" files which eliminates the need to read/write 113784 ".icjh" files—which is also a bonus for minimizing backup activity. And there is absolutely no value in having those files in each folder, since the mail folder is never rearranged and always backed up in it entirety.
Hash data for 113784 folders containing 387296 files loaded in 12109 ms.

Hierarchy files also mean that one can use icj on ".git" hierarchies without git complaining about unwanted files in its 'objects' and 'pack' folders. Ditto for any program which dislikes unknown files in its folder hierarchy, library, etc.

A folder hash hierarchy also makes it possible to hash read-only media and store that hash file elsewhere. This can be implemented if interest in it is seen.

New feature: in-place update but with full compatibility

IntegrityChecker java maintains full compatibility with prior hash formats, include the native version ".ic" format (SHA1). Backward 'verify' compatibility is essential, since many pros burn their work to read-only media—read-only media cannot be updated or changed.

A revised hash format (chained SHA512) is introduced with the update command automatically verifying the existing hash and computing the new and faster SHA512 hash. In this way, updating is seamless and results in the best possible speed once updated.


Performance is terrific — shown below is icj on a 2.5 GHz 28-core Mac Pro doing 12.5 GB/sec hashing (12500 MB/sec). That’s about 446 MB/sec per real core which are running at a relatively slow 2.5 Ghz each.

File globbing

Not yet implemented but planned soon: file globbing, which will allow precise control of which files are hashed and which are not, folders to be included or excluded, etc.

56 virtual CPU cores in use (28 real cores) — diglloydTools IntegrityChecker doing SHA512 hash at 12.5 GB/sec
View all handpicked deals...

Samsung 2TB T9 Portable SSD
$250 $150
SAVE $100 | Terms of Use | PRIVACY POLICY
Contact | About Lloyd Chambers | Consulting | Photo Tours
Mailing Lists | RSS Feeds |
Copyright © 2020 diglloyd Inc, all rights reserved.
Display info: __RETINA_INFO_STATUS__