IntegrityChecker (icj) Java Release 3.0 fc34

2023-09-22 • SEND FEEDBACK |
Related: Apple, Apple Silicon, backup, data integrity, diglloydTools, DiskTester, IntegrityChecker, Java

re: IntegrityChecker Java
re: DiskTester Java
re: data integrity

Consult with Lloyd to design a storage and/or backup system and/or high performance workflow.

Change history below...

Who needs it?

You do not have a real backup until you have a verified backup.

That means all the data can be read (a physical check on the storage medium) and is cryptographicallly proven to be intact/unchanged with no files or folders missing.

No photographer or videographer or other professional should be operating without data integrity validation. Whether bit rot or malware or software bugs or hardware problems, can you afford to remain unaware of data corruption, or to not know if your backups are intact and undamaged?

Compatibility

IntegrityChecker Java (icj) supports Mac, Windows, Linux, etc—anything with Java, an unrivalled cross-platform data integrity solution.

Get Data Integrity Assurance now

Download page for existing customers.

Get IntegrityChecker Java

Or keep guessing about whether your backups suck.

What’s changed

If you are using an Apple Silicon M1/M2 Mac, run installJava.sh to install Java 21—you will see 4X the hashing speed vs earlier JDK version— twice the speed of Intel. Even the slowest M1 Mac will blaze through things as fast as the drive can go.

This release offers early acces to DiskTester Java (dtj) and Tools (tj), img (imaging tools), jattr (extended attributes). Sorry, no documentation yet on the new tools, but there is some built-in help.

• 3.0fc34
  2023-09-22 updated installJava.sh to use Java 21 final GA release
  2023-09-14 updated documentation for 'matches' command
  2023-09-14 Fixed 'matches' command to work for all documented flags
  2023-09-14 added 'caffeinate' command to invocation, so that icj will not allow system sleep while running.
  2023-09-06 fixed installer bug that failed to copy-over 'dtj', 'img', and 'tj'.
  2023-08-27 'matches' command now shows [missingfolders.ignore] matches.
  2023-08-26 Added missingfolders.ignore feature.
  2023-08-23 2023-08-23 install-icj.sh now preserves (does not replace) icj_prefs.txt
• 3.0fc32
  2023-08-03 icj: added report.MATCHED_HASH_MIN_SIZE option to prefs. It applies a minimum size cutoff as to whether missing files with matching hashes are reported. Values can be of the form "1234" or "32K" or "2M", etc eg "report.MATCHED_HASH_MIN_SIZE=128K"
  2023-08-01 icj: fixed bug where missing folders would report incorrect file count, subfolder counter, byte count for missing folders in the intermediate listing.
  2023-08-01 icj: orphaned files (files whose parent folder is also missing) are now NOT listed by default.
  The --orphans option must be used to list the files.
  2023-07-29 dtj: fixed bug where --numbuf option was not being used
• 3.0fc31
  2023-07-28 The installJava.sh script now installed JDK 21ea on Apple Silicon Macs, delivering a 4X hashing speed increase on Apple Silicon Macs.
  2023-07-25 In addition to 'jattr', new tools now include 'img' (Imaging Java), 'tj' (Tools Java), 'dtj' (DiskTester Java). Available commands include wipefiles, wipefree, fixdates (tj) fixdates from EXIF (img), aseries, stacks. These new tools are usable, but exploratory, and most lack documentation (yet).
  2023-07-23 Given the new considerations of super-fast hashing and super-fast SSDs over 20GB/sec, the --numbuf and --bufsize options have been restored for command line options.
  2023-07-22 Added optimization for super-fast SSDs on Apple Silicon via larger buffers up to 32MB on M2 Ultra (up from only 1MB). Raises peak throughput from ~17GB/sec to ~21GB/sec on M2 Ultra and from ~7GiB/sec to ~9GiB/sec on M1 Max. Optimization only applies to JDK 21 or later, which has 4X faster SHA512 hashing speed.
  023-07-22 Improved performance of initial scan by about 5-8% by using JNA native function mapping.
  2023-07-20 If a corrupted hexadecimal hash string is detected, a CorruptedHashEntryException is thrown which makes the issue more understandalone. This arose in a case where a SAN was corrupted hash entries (and other data).
• 3.0fc30
  2023-07-19 Fixed ICJ_CLASSPATH path in icj.bat
  2023-07-18 'sha' command: added per cpu figure to command output.
  2023-07-18 'sha' command: changed to do all CPU counts up to maximum real CPU count, not every-other-past-10.
  2023-07-18 'sha' command: Increased maximum --size value to 128M, added existing --threads option to user manual
  2023-07-02 bug fix: the 'clean' command without the --kind argument was cleaning only icj_temp. Now it cleans {ic,icj,icj_temp} by default (without the --kind argument), doing what the user manual states.
  2023-07-01 Improved message behavior with non writeable folders.
  2023-07-01 Added **/private/{var,etc,tmp,tftpboot}/** to [folders.ignore] to better deal with files owned by 'root' within backups of boot drive
  2023-07-01 Removed assertion that was barfing for folders owned by root.
  2023-06-30 Improved error handling when a regular folder is replaced by a symbolic link.
• 3.0fc28
  2023-06-30 Fixed bug where icj would fail to finish any legacy 'ic' file of 16K or smaller in size, losing an I/O buffer, thus quickly running out of buffers, halting progress and waiting forever for hashing to finish.
• 3.0fc27
  2023-06-27 FYI early access JDK 21 offers 4X the hashing speed on Apple Silicon. For example, a single CPU can now do ~1500MB/sec on an M1 Max.
  2023-06-19 Fixed a race condition that resulted in "... java.lang.AssertionError "finalizeHashesNew: mData_SHA512_64K_CHAINED not finished...". When this happened, icj would not complete at the very end ("waiting for...").
  2023-06-19 Inserted warning when buffers are unobtainable for 15 seconds or more to help detect hanging I/O system.
  2023-06-16 Fixed bug where DATE_CHANGED was issued when date really had not changed, causing files to be unnecessarily hashed. This stemmed from mismatched milliseconds portions of file modification date.
  2023-06-16 A new utility 'jattr' is now included, a more useful/powerful tool than Apple’s xattr command, and runs on both macOS and Linux.
• 3.0fc23
  2023-04-04 Fixed bug where prefs files were being ignored for the various flags, introduced around fc18.
  2023-04-04 File status messages are now aligned visually with mixed-length tag like RENAMED vs MOVED*
  2023-04-04 SYSTEM_SLEEP advisory rejiggered to SYSTEM_UNRESPONSIVE with exact times and the exact unexpected delay.
• 3.0fc20
  2023-03-30 All outstanding issues with normalized file paths now resolved. No more missing files if normalization varies, no warnings should appear, cross compatibility across JDK20 vs earlier JDKs as well as APFS/HFS, and in all combinations thereof. The changes should apply across operating system foibles as well.
• 2023-03-30 It is recommended that all macOS users move to JDK20 by running the java installer. That’s because JDK20 is the first version to not forcibly normalize file paths to NDF, a behavior that could cause issues across file system. With JDK20, file paths that icj sees are normalized the same as in the native file system.