2023-11-27: IntegrityChecker java version 3.0 fc35
(includes icj, dtj, img, tj, jattr tools)

See the change history.

Download: download icj 3.0fc35 (3.0 final candidate)

MD5 hash of zip file: f69ec3040633ef501834fa93fb5f9d8b
SHA-512 hash of zip file: f1d347f9a0b14b4925d488247cc6f0dcf8ffb3cc5c00182b1e2ed1a732dedb6972c1ce72b1c7da6425ab46a3f0bb32a93f8c2b22adc77c79af78b45cd4721999 2023-0923-icj3_0fc35.zip

DiskTester, MemoryTester, IntegrityChecker (DEPRECATED)...

The traditional native-code diglloydTools DMG (DiskTester, MemoryTester) are now officially deprecated. All future releases are Java only.

2023-01-22: IntegrityChecker java version 3.0 fc17

Download: download icj 3.0fc17 (3.0 final candidate)

MD5 hash of zip file: 7e8af19b095762067ec8afba05ea3ec8
SHA-512 hash of zip file: c61f03145f4eeee9fa852e71fc32dbda1f8dfe0101a848b85d6dae57972c589533c7b25c96e5b26db1a6138c0430ee05caab4f5dccdeae01207a12fe760cf57a

...

The traditional native-code diglloydTools DMG can be downloaded separately here (no changes):

2022-09-07: IntegrityChecker java v 2.0.5fc3

This is likely to be the last update for icj version 2.0, with icj version 3.0 going into testing shortly.

• 2022-09-07 Minor changes to retain compatibility with upcoming version 3.0
• 2022-09-06 Bug fix for error when processing filenames with characters interpreted by java as a "/", such as ≠ character.

Download of icj and its script file: download icj 2.0.5fc3

MD5 hash of zip file: 2ac288ec92b7da27c88f1a0eca2060e5
SHA-512 hash of zip file: cfa25f42a4b0814f60549af92a26ea72d9b266ced016fce974560d35e1efe059476f07f702d3ee13dd74d97f1482599981d28db73b85c437cc41270d38a815ff

The traditional native-code diglloydTools DMG can be downloaded separately here (no changes):

2022-04-29: IntegrityChecker java v 2.0.3b25

A wide variety of small fixes and changes.

• 2022-04-29 Added various warning flags to icj_prefs controlling which things are flagged:
  warn.ILLEGAL_FILENAME=true
  warn.FILENAME_COMPATIBILITY=false
  warn.DEAD_SYMLINKS=true
  warn.FILE_CHANGED_IN_FLIGHT=false
  warn.MISSING_SUBFOLDERS=false
• 2022-04-28 'clean' command now always removes .icj_temp files, which accrue if icj is killed while writing hierarchy files.
• 2022-04-28 Added .icj_temp files to ignore list
• 2022-04-27 fixed thread-waste issue where each hierarchy file was retaining its ThreadPoolExecutor after parsing. Executor is now freed after parsing, avoiding limits on thread count and reducing memory usage.
• 2022-04-27 fixed race condition whereby .icjh files at a higher level could be written sooner than ones at a lower level, leading to erroneous albeit harmless authoritative hierarchy file locating. Fixed by setting all hierarchy files written as a group to all share the same lastModified file date.
• 2022-04-27 improved messaging when there are no hierarchy files in top level folder but there is one in an ancestor folder.
• 2022-04-25 installer overwrites icj_prefs.txt, but saves the prior version. Installer does not overwrite icj_prefs_customized.txt.
• 2022-04-25 Unified the MATCHED/RENAMED code. Also enhanced for multiple matches.
• 2022-04-25 icj_prefs.txt always look for icj_prefs_customized.txt, loading it if present.
• 2022-04-24 Added FLAG_DEAD_SYMLINKS flag for dead symlinks in icj script. Not yet available as preference.
• 2022-04-24 Miscellaneous output added to verbose mode (--output=verbose)
• 2022-04-24 Fixed a sporadic (as in never seen before!) race condition while reading hierarchy files that could sporadically result in a java.util.ConcurrentModificationException.
• 2022-04-24 Warn of file names that have cross-platform compatibility issues; Edit the NOTE_FILENAME_COMPATIBILITY flag in icj/icj.bat to toggle behavior. Not yet available as a preference.
• 2022-04-24 Detect and flag and continue with illegal filenames on Windows that might be present in macOS systems.
• 2022-04-23 Fixed a few output cases to use the correct name separator for the platform eg "/" vs "\"
• 2022-04-23 All the MATCHES for missing files are now shown, not just the first one.
• 2022-04-23 Missing/renamed file output is improved for clarity.
• 2022-04-22 improved the missing files support: (1) missing files that are found are separated from those that cannot be found, (2) the summary number reflects found files, (3) the user manual discusses how the update command gives the best information on missing/renamed files. https://diglloydtools.com/manual/IntegrityCheckerJava-update.html
• 2022-04-22 Added --save option for update command. If not specified, saving is done as it always has been. But sometimes it is desirable to examine files moved or renamed before saving changes. The verify and status command are read only, but those commands cannot deduce as much because they operated only on files already having hashes, which moved/renamed files might not yet have.
  icj update --save=prompt ...
  icj update --save=no ...
  icj update --save=yes ...
• 2022-04-22 "#include" directive in icj_prefs.txt changed to "@include" to avoid confusion with comment lines. Users making use of it should change the "#" to "@".
• 2022-04-22 Fixed bug whereby empty file count would include missing files.
• 2022-04-22 Java installer uses openjdk-18.

Download of icj and its script file: download icj 2.0.3b25

MD5 hash of zip file: 11c92e74536b3509d4f0d3c2b251b1e2
SHA-512 hash of zip file: 3fc99f375ae5d340086cf792bfe7dc82033cb11f5e777b81c7422162503c397463fe0d987dce4fcbe261d896c9f0dbe45a170215e6c46fb1749a7d0cdc5a1eb1

The traditional native-code diglloydTools DMG can be downloaded separately here (no changes):

2022-04-13: IntegrityChecker java v 2.0.3b7

Minor fixes.

• 2022-04-13 Various tweaks/fixes to support running on Windows 10 including instructions for installation.
• 2022-04-13 Preference files are now moved to ".icj_config" folder in user home directory. Preference file names are now "icj_prefs.txt" and "icj_prefs_customized.txt" (no longer "dot" files and now in the subfolder). Changes made to support Windows.

Download of icj and its script file: download icj 2.0.3b7

MD5 hash of zip file: 619ecba10198f318e559f27c6ce6ed3f
SHA-512 hash of zip file: 692851c1afd7064acd53f3e57c716daf5eaa2ce8039aea95c4afc67612b144f4631af403e9e9858c22d9df87c8cbe913bbd27e483d4b7be698e02cb67c694b2d

The traditional native-code diglloydTools DMG can be downloaded separately here (no changes):

2022-01-02: IntegrityChecker java v 2.0.2

Minor fixes.

• 2022-01-02 Updated installJava.sh to install JDK 17.0.1, including choosing the Intel or Apple Silicon variant automatically.
• 2022-01-01 Changed the 'sha' command to test hashing speed for a variety of thread counts, emitting a summary table when done. Can be used to characterize performance on different machines and/or tune performance expectations.
• 2021-12-20 Added SHOW_RECENT_THROUGHPUT flag, which governs whether to show throughput over the past 5000 I/O requests, typically 3-5 seconds on fast SSD.
• 2021-10-26 Changed the purge code (if invoked with 'sudo') to purge *at least* every N seconds rather than waiting N seconds after the prior purge. Better steady-state purging results. Added milliseconds to the "P" suffix when purges occur.

Download of icj and its script file: download icj 2.0.2

MD5 hash of zip file: 0c7f0b4ce76581225aa0292dfc0fa5bc
SHA-512 hash of zip file: 17ff7561a2277634d1f592a1868b369417deb65b2aed92565395b30a104669dd191e4d05325d7f14a5b07d45cdb9bab132dbde5e1df152f7e3de80cfcdd030c6

The traditional native-code diglloydTools DMG can be downloaded separately here (no changes):

2020-08-04: IntegrityChecker java v 2.0

Final 2.0 version, with some small fixes.

• 2021-03-06 fixed bug whereby a change in case for a folder name on a case-insensitive file system would add a duplicate entry and see the folder as missing
• 2021-03-06 improved help to show available options
• 2020-10-01 minor changes to installer to work on Linux
• 2020-10-01 minor changes in detecting operating system specific behavior for Linux ("id -urgg")
• 2020-10-01 minor change to address internal redundant folder scanning warning

Download of icj and its script file: download icj 2.0.0

MD5 hash of zip file: 483bc743920f6bef7c94c5d183b6521e
SHA-512 hash of zip file: dcaa3e32c4edb8a8bb1111a7b456f84330dfd601ce81865ff4ad634c2151d586099f8f8c571a9aea61855a6ade2621fccb0a35391980ef7fef16268653adbf28

The traditional native-code diglloydTools DMG can be downloaded separately here (no changes):

2020-08-04: IntegrityChecker java v 2.0 fc4

Major release.

Download of icj and its script file: download icj 2.0 fc4

MD5 hash of zip file: 1689db89263d93453cc22349f6c10edd
SHA-512 hash of zip file: eb3bb2412682010b336c539a05b5b020b48ccb4eb6be04f5dcf35cc1402a4bc630d2f12ad5755b1f16241ac0cd1b7e668761a869322a17629ce84e2c94d54900

The traditional native diglloydTools DMG can be downloaded separately here (no changes):

2020-01-21: IntegrityChecker java v1.3b11

About the only issue that is coming up is permissions and security issues—files and folders that cannot be read because of permissions, ownership, being locked, etc along with other oddball cases like problem files being in legacy ".ic" files.

Accordingly, this releases now uses java.nio instead of java.io for iterating over folders and files, which exposes certain types of special cases more clearly. Defensive code with better reporting for problem files/folders is included.

Download of icj and its script file: download icj 1.3b11

MD5 hash of zip file: b095e79779cddf13b1eac90c50d7a2ee
SHA-512 hash of zip file: cfbf4a18c396be78a50fdc2b6ff3ca44d871bfa9bf437ad0593289eedb323c80251a938712e382232bde69de6e189b0736cf6334459bff727931ea9e1ec1b5ca

The diglloydTools DMG can be downloaded separately, here:

2020-01-20: IntegrityChecker java v1.3b7

See previous notes also.

This release incorporates help for every command and integrates it with web site pages, opening them automatically. For example, “icj help verify” will open the web page for online help.

Separate download of icj and its script file: download icj 1.3b7

MD5 hash of zip file: 6c57f9f35e0c8cfe826cb21f34939c3c
SHA-512 hash of zip file: ad9995d85d415521f7d7f25616ba5bfc8cf12f7c2458b832337d074ac282260bae9357e2e1f47a58525aca98f31f4f3f80979929415556b8a51a53a1acbe0489

2020-01-20: IntegrityChecker java v1.3b6

See previous notes. This minor update fixes two things:

• The install script install-icj.sh is fixed to be runnable from any directory (no need to 'cd' to directory containing the script).
• the help summary for "clean" now shows the "--ic" option.

----- icj download removed, use most current version -----

NOTE: if desired, download the diglloydTools dmg disk image from the Dec 26 entry below, but replace its icj with the version here.

2020-01-19: IntegrityChecker java v1.3b5
More performance, better handling of special files, automated installation on macOS

Standalone release of IntegrityChecker java version 1.3b5. Download the full Dec 26 release of diglloydTools first if the DiskTester and MemoryTester tools are desired.

This is a beta version which will likely be a formal non-beta release shortly.

NOTE: for full speed benefit (use of SHA-512), the following procedure is advised:

1. First verify your data on your various folders or volumes:

icj update; icj verify

Then assuming all looks good, do a 'clean' (removes hash files) and then 'update', which will hash with SHA-512:

icj clean; icj update

----- icj download removed, use most current version -----

NOTE: if desired, download the diglloydTools dmg disk image from the Dec 26 entry below, but replace its icj with the version here.

2020-01-06: IntegrityChecker java v1.2b2
Major Performance Improvement with SHA-512

Standalone release of IntegrityChecker java version 1.1b10. Download the full Dec 26 release first if the DiskTester and MemoryTester tools are desired.

No change to native version. New features:

1) Backwards compatible with older versions that used SHA1

2) Various small tweaks

2) fixes issue with symbolic links.

3) Version 1.2b2 builds on the major performance improvements of IntegrityChecker version 1.1b10 by adding support for SHA-512 hashing. [Speed is now so fast that IntegrityChecker Java will be I/O limited on recent Macs, even by the extremely fast internal SSDs. There is some actual speed benefit, but given the icj is now I/O bound, the is an about 80% reduction in CPU usage.

This is a beta version. Its is backward compatible with prior versions but prior versions are not forward-compatible with updated folders due to thchange to SHA-512 as the default hashing algorithm.

NOTE: for full speed benefit (use of SHA-512), the following procedure is advised:

1. First verify your data on your various folders or volumes:

icj update; icj verify

Then assuming all looks good, do a 'clean' (removes hash files) and then 'update', which will hash with SHA-512:

icj clean; icj update

----- icj download removed, use most current version -----

NOTE: if desired, download the diglloydTools dmg disk image from the Dec 26 entry below, but replace its icj with the version here.

2020-01-03: Major performance improvements to IntegrityChecker java

Standalone release of IntegrityChecker java version 1.1b10. Download the full Dec 26 release first if the DiskTester and MemoryTester tools are desired.

No change to native version.

New features:

1) Numerous small enhancedments.

2) Performance has been further improved, including faster concurrent loading of stored hash files.

3) the 'dupes' command supports options for files over a specified size and for a list of file types eg:
icj dupes --size 32K
icj dupes --types RAW # all raw file types
icj dupes --types DNG,ARW,CR2,CR3,NEF

4) Hashing speed can be directly checked for your particular machine. This is a prelude to an update which will offer higher performance hashing and is otherwise of no particular use for end users.
icj sha
icj sha --size 1M --iterations 1000 -sha sha1

This is a beta version. Its is backward compatible with prior versions but prior versions are not forward-compatible with updated folders due to the addition of missing folder support. In other words, do not run older versions once this version updates folders.

----- icj download removed, use most current version -----

NOTE: if desired, download the diglloydTools dmg disk image from the Dec 26 entry below, but replace its icj with the version here.

Version 2.2.15, December 26, 2019

IntegrityChecker java version 1.1b1

No change to native version.

New features:

- Folders are now tracked, so that if an entire folder anywhere in the hieararchy goes missing, a warning is emitted.

- the 'dupes' command is now much more useful. It emits 'rm' commands that can be copied/pasted in Terminal to remove unwanted duplicate files. Use these commands with care, since icj cannot know for sure which duplicate should be considered the primary. However, an apparent primary is selected by date and intelligent use of the name.

Java version of IntegrityChecker (icj) has been optimized for extreme performance:

- Total throughput hits nearly 3 gigabytes per second on fast 8-core Macs such on an 2019 iMac 5K or 2019 MacBook Pro (assuming a very fast SSD such as the internal SSD). It is now fast enough that CPU power on 12 core or more machines should outrun SSD speed, that is, icj is I/O bound unless an SSD can deliver 4GB/sec or more.

- Throughput both for mixed-size files and large files is now very close to the limits of SSD speed (as tested on 2018 iMac 5K 8-core on its fast internal SSD)

- A performance bug is now fixed that resulted in unpredictable 2X to 3X performance losses on the same workload (instantiation of threads in thread pool was not being triggered reliably).

- Performance on hard drives has been improved by up to 3X. Due to API limitations, hard drives still do not perform as well as hoped. The NUM_IO_BUFFERS configuration parameter in the icj script file should be reduced to 4 buffers for better performance (more than a small number of buffers degrade performance on hard drives, but are esential for SSD performance). Change -DNUM_IO_BUFFERS=0 to -DNUM_IO_BUFFERS=4 for improved performance on hard drives.

This is a beta version. Its is backward compatible with prior versions but prior versions are not forward-compatible with updated folders due to the addition of missing folder support. In other words, do not run older versions once this version updates folders.

----- icj download removed, use most current version -----

...

Note: Install-cmd-line-tools.app does not work, use this in Terminal:
sudo /Applications/diglloydTools/spt/make_links.sh

Version 2.2.15, December 30, 2017

Same as Dec 24 release except that icj (Java version of IntegrityChecker) has been optimized for much higher performance.

----- icj download removed, use most current version -----

Version 2.2.15, December 24, 2017

Rebuilt on macOS 10.13.2 and optimized IntegrityChecker for extremely fast SSDs. See release notes.

MD5 hash for dmg.zip: b7998121eed299e1341bc57a4fc1170d

Version 2.2.14, August 16, 2016

No change to native versions, but macOS DMG now includes icj.

MD5 hash for dmg.zip: c0599fcaf654931f8b052eac993006c7

----- icj download removed, use most current version -----

Version 2.2.14, August 4, 2016

MD5 hash: e250c2c441b45f266da7260d3829029b

----- icj download removed, use most current version -----

Version 2.2.13, Jan 24, 2016

MD5 hash: 0cb5fcdf10011a34d695b61a68bf7495

Version 2.2.12, Dec 5, 2015

MD5 hash: 9245f176a88dfd5bea286f12f8c9e85d

Version 2.2.11, Feb 28, 2015

MD5 hash: 1348f5e7e1fc7381cca641fb2b42a1ec

Version 2.2.10, Dec 4, 2014

MD5 hash: a88c74d5171c0fb8a016e08b491d3efa

Version 2.2.9, Nov 21, 2014

MD5 hash: 16998cd16b466e534db3efee0fd3c886

Version 2.2.8b, Nov 15, 2014

MD5 hash: 0e3f5c1fa5f0fc05527275dfc10ff987

Version 2.2.7b, August 29, 2014

MD5 hash: c5df928aa807807b53ce056c1fc93d61

Version 2.2.7a, July 15, 2014

MD5 hash: 0724f4e47738c5dc9fbbd957a32d6ad9

Version 2.2.6, May 6, 2014

MD5 hash: 0dc691a1795ed6885bed642bc7f8c328

Version 2.2.5, May 5, 2014

MD5 hash: a25b8627feb78fd0101920f0a70d5ca4

Version 2.2.4, May 3, 2014

MD5 hash: 0abbcbfc9f6a99194c82cd9cdbbc752d

Version 2.2.3, April 1, 2014

MD5 hash: ec9d860e8117752ae3628e21319f386a

Version 2.2.2, October 26, 2013

Older versions

Version 2.0.1b2, May 30, 2010 (runs on OS X 10.6)