All Posts by Date or last 15, 30, 90 or 180 days.
also by Lloyd: diglloyd.com photography and WindInMyFace.com

Thank you for buying via links and ads on this site,
which earn me advertising fees or commissions.
As an Amazon Associate I earn from qualifying purchases.

Other World Computing...
B&H Photo...
Amazon
As an Amazon Associate I earn from qualifying purchases.
Capacities up to 56TB and speeds up to 1527MB/s
877-865-7002
Today’s Deal Zone Items... Handpicked deals...
$1199 $779
SAVE $420

$249 $219
SAVE $30

$1699 $1299
SAVE $400

$1499 $1149
SAVE $350

$200 $140
SAVE $60

$89 $69
SAVE $20

$120 $35
SAVE $85

$209 $179
SAVE $30

$3498 $3498
SAVE $click

$3699 $1299
SAVE $2400

$1000 $700
SAVE $300

$3899 $1499
SAVE $2400

$748 $648
SAVE $100

$2999 $1999
SAVE $1000

$1699 $999
SAVE $700

$1698 $1398
SAVE $300

$4499 $3499
SAVE $1000

$1399 $1049
SAVE $350

$1149 $799
SAVE $350

$1399 $1049
SAVE $350

$1498 $1198
SAVE $300

$2797 $2597
SAVE $200

$280 $250
SAVE $30

$1699 $1299
SAVE $400

$8399 $7599
SAVE $800

$899 $799
SAVE $100

$1499 $999
SAVE $500

$1699 $1299
SAVE $400

For Mac/Windows/Linux/etc — diglloydTools IntegrityChecker Java version 1.3b5 Now Available — Includes Automated Install Script for macOS users

diglloydTools IntegrityChecker Java version on any computer with Java—Mac, Windows, Linux, etc. More about IntegrityChecker and why every professional should be using it.

Buy diglloydTools

IntegrityChecker Java version (icj) 1.3b5

This version is the culmination of three weeks of coding for 14 hours a day even over the holidays. A tremendous amount of effort has gone into this version of icj, including researching how to deliver performance that very few native-code programs can match. icj makes use of CPU cores and disk I/O speed like very few programs can.

Just posted is diglloydTools IntegrityChecker Java version 1.3b5 (if downloaded before 16:00 Pacific time Jan 19, please download it again a minor error was in the jdk download script):

License and download page...

This is a beta version but well tested (and IntegrityChecker only reads your files anyway!). It will likely become the official release version soon.

Installation

Version 1.3b5 builds includes an install script for macOS that both installs IntegrityChecker Java (icj) and also optionally installs Java itself. All you have to do is answer "yes" or "no" when prompted.

Documentation/help

Documentation is currently inadequate and is coming soon. For now please see the overview page.

If you need support, please copy/paste the text from the Terminal window—don’t send screen shots as they are very large and frequently contain too little information.

Improvements/additions

  • Automated installation script for macOS.
  • Even higher performance, particularly on hard drives. On a 16-core Mac Pro and suitably fast drives, total hashing throughput can exceed 7.5 gigabytes per second. The “fast” 3.2 GB/sec internal Apple SSDs are too slow for full icj performance on 8 CPU core machines and reasonably recent 6 core machines.
  • Auto-detects whether the drive is an SSD or hard drive, an optimizes accordingly. For technical reasons this cannot always be known; use --optimize HDD if icj incorrectly decides the drive is an SSD.
  • Improved handling of special files, permissions issues, etc.
  • Greatly reduced per-file memory usage (however, due to the extreme performance, memory usage may still rise quickly when the Java virtual machine has no time to garbage collect).
  • icj saved my 700 gigabytes of wasted space via its "dupes" command, which finds duplicate files (I had inadvertently downloaded several of the same large photo shoots this year). It determines which files are most likely the original/best ones to keep by dates and by existence of sidecar files, etc. Super-cool is that it emits commands that can either clone the duplicates (in effect leaving them there as before but eliminating 100% of the duplicate space usage) or commands that remove the duplicates. Cloning does require APFS on macOS however.
  • icj can compare two folder trees, so for example it is possible to see if a folder one one drive is the same as on another.

Quick intro

Most people are not comfortable with the command line. It is simple to use.

  1. Open a Terminal window (/Applications/Utilities/Terminal).
  2. Type "icj" followed by a space followed by the command (e.g. verify or update).
  3. Type a space after the command, then drag anything into the Terminal window from the Finder (a folder or multiple folders, or a volume or even multiple volumes).
  4. Press the RETURN or ENTER key.

In step #3, you can also type in the desired item(s) instead.

Usage tips

  • To stop (forcibly kill) icj in a Terminal window, type control-C (control, not cmd).
  • To temporarily suspend icj in a Terminal window, type control-Z. To resume, type "fg" (foreground).
  • The macOS file cache can steadily degrade performance as it caches absolutely everything that icj reads. There is currently no technical way to tell macOS to not cache when reading files from a Java program. To enable icj to flush this cache, run icj using "sudo" as in:
    sudo icj verify MasterData
  • Spaces matter in file, folder and volume names. So you want to verify the volume "My Stuff", the volume name must be quoted, like this (include the straight quotes):
    icj verify "My Stuff"
    This is a nuisance, so it is better just to not use spaces in volume or folder names that you use a lot, e.g., “MyStuff”, not “My Stuff”.
  • You can run icj in more than one Terminal window. So if you want to simultaneously operate on three different backup drives (such as to verify), just open three Terminal windows and start icj in each window, something like:
    icj verify Volume1
    icj verify Volume2
    icj verify Volume3
  • You can "detune" icj if it is desirable for it to use less CPU time buy specifying few threads and buffers. For example, adding these options will limit icj to just two hashing threads and four I/O buffers:
    icj verify --threads 2 --large-buffers 4 MyData
  • If you want to verify things one after another to restrict CPU and/or memory usage, the best way is separate invocations. This command works fine, but it does the three items as one job:
    icj verify MyStuff Work MasterData
    Instead, use a ";" to separate the commands, like this:
    icj verify MyStuff; icj verify Work; icj verify MasterData

Overview of commands

Typing "icj" or "icj help" in Terminal will show this summary.

diglloyd-MacPro:MPG lloyd$ icj
# icj version 1.3b5 @ 2020-01-19 13:00
# Copyright 2018-2020 DIGLLOYD INC. All Rights Reserved
# Use of this software requires a license. http://macperformanceguide.com/Software-License.html
# Sun Jan 19 13:23:05 PST 2020
Available commands:
verify      verify hash values
status      summarize files that are new, or of changed size or date
compare     compare: compares two folders for equality
update      update new and date/size changed files, forget missing items
update-all  update hash values for all files, whether or not they already have hashes
update-new  update only files lacking hash values
clean       remove all hash data files
dupes       show duplicate files: options --size  --types type[,type]* --emit <rm|clone|nop>
empty       show empty files
sha         test hashing speed: options --size 1M --sha SHA-512 
version     display the version and other information
help        show help, specify which command such as 'help verify'
Manual at https://diglloydtools.com/manual/IntegrityCheckerJava.html    

Example commands

Lines that start with "#" are comments. These examples assume a folder called MyStuff and a volume (entire drive) called Work.

# ensure that hashes exist for all files on volume Work
icj update Work # ensure that hashes exist for all files in folder MyStuff icj update MyStuff # ensure that hashes exist for all files in folder /Volumes/Work/Photos icj update /Volumes/Work/Photos
OWC Envoy Pro EX SSD
Blazingly fast Thunderbolt 3 SSD!

Up to 4TB capacity, USB-C compatible.

USB-C model also available


Great for travel or for desktop!
OWC Thunderblade Thunderbolt 3 SSD

Blazing fast, up to 16TB.

YEE HAH!



√ No more slow and noisy hard drives!
OWC Envoy Express

World’s first Thunderbolt 3 M.2 NVME SSD enclosure.

Super fast, bus-powered, 3.3 oz, DIY easy, rugged and compact!


See also OWC Express 4M2
√ No more slow and noisy hard drives!
OWC Accelsior 4M2 PCIe SSD
6000 MB/sec!
Mac or PC.


Ideal for Lightroom, Photoshop, video.
Capacity up to 16TB!

Save the tax, we pay you back, instantly!
View all handpicked deals...

LG UM7370PUA 70" Class HDR 4K UHD Smart LED TV
$1199 $779
SAVE $420

diglloyd.com | Terms of Use | PRIVACY POLICY
Contact | About Lloyd Chambers | Consulting | Photo Tours
Mailing Lists | RSS Feeds | Twitter
Copyright © 2020 diglloyd Inc, all rights reserved.
Display info: __RETINA_INFO_STATUS__