MAJOR Performance Improvements to diglloydTools IntegrityChecker Java version 1.1b10
Just posted is diglloydTools IntegrityChecker Java version 1.1b10.
Runs on any computer with Java—Mac, Windows, Linux, etc. More about IntegrityChecker and why every professional should be using it.
This is a beta version but well tested (and IntegrityChecker only reads your files anyway!).
Version 1.1b10 is backward compatible with prior versions but older versions are not forward-compatible with updated folders of 1.1b10 due to the addition of missing folder support (see below). In other words, do not run older versions once this version updates folders.
- Folders are now tracked, so that if an entire folder anywhere in the hierarchy goes missing, a warning is emitted.
- The 'dupes' command is now much more useful. It emits 'rm' commands that can be copied/pasted in Terminal to remove unwanted duplicate files. Use these commands with care, since icj cannot know for sure which duplicate should be considered the primary. However, an apparent primary is selected by date and intelligent use of the name. Saved me 600GB! (I had somehow had multiple downloads of some very large raw file shoots from my trips).
Optimized for extreme performance.
- Major performance enhancements (up to 5X faster, depending on CPU cores and speed of the drive). Performance is highly optimized for SSD, but should be considerably better for most hard drives too.
- Concurrent loading of folder hash files is up to 3X faster.
- Throughput can hit 4.6GB/second on the OWC Accelsior 4M2 PCIe SSD, and about 2.9 GB/sec on most Apple internal SSDs (8 core systems).
- Total throughput hits nearly 3 gigabytes per second on fast 8-core Macs such on an 2019 iMac 5K or 2019 MacBook Pro (assuming a very fast SSD such as the internal SSD). It is now fast enough that CPU power on 12 core or more machines should outrun SSD speed, that is, icj is I/O bound unless an SSD can deliver 4GB/sec or more.
- Throughput both for mixed-size files and large files is now very close to the limits of SSD speed (as tested on 2019 iMac 5K 8-core on its fast internal SSD)
- A performance bug is now fixed that resulted in unpredictable 2X to 3X performance losses on the same workload (instantiation of threads in thread pool was not being triggered reliably).
- Performance on hard drives has been improved by up to 3X. Due to API limitations, hard drives still do not perform as well as hoped.