All Posts by Date or last 15, 30, 90 or 180 days.

As an Amazon Associate I earn from qualifying purchases @AMAZON

Designed for the most demanding needs of photographers and videographers.
Connect and charge all of your devices through a single Thunderbolt or USB-C port.

IntegrityChecker Java: How Fast Can Hashing Go on the 13GB/sec OWC Accelsior 8M2 PCIe SSD?

re: How to Purge the File System Disk Cache: 'sudo purge' in Terminal.app
re: data integrity
re: IntegrityChecker Java

Consult with Lloyd on your photo or video workstation, backup and data integrity, etc.

See the disk speed tests of the OWC Accelsior 8M2 PCIe SSD, the fastest drive MPG has ever tested, by a factor of 2X!

CLICK TO VIEW: OWC Accelsior 4M2 and 8M2 PCIe SSD

Cross-platform data integrity validation

IntegrityChecker Java is the fastest and easiest way to validate the integrity of your data on Mac or Windows or Linux or any platform that can run Java. On any file system or device supported by Java, read-write or read-only, hard disk or SSD or WORM or optical, everything/anything with a file system.

And today’s IntegrityChecker should run fine a decade from how, so long as you have Java, because unlike Apple removing/changing APIs on a whim, Java APIs have been compatible forever (excepting a few rare thread-unsafe APIs), and will remain so.

But a downside of cross-platform future-proof Java code (and even POSIX APIs) is that they have no access to special disk I/O flags that tell the operating system to not cache disk reads. And as it turns out, macOS has a nasty caching performance bug that cuts disk speed in half on very fast drives.

Certain native-code programs can utilize a special API flag to avoid this bug, but many do not. Adobe Photoshop for example not only has quite slow I/O speeds for opening big files, but makes it worse by allowing caching.

Java-based or POSIX-based programs have no access to defeating unwanted caching. I’ve programmed IntegrityChecker Java to include a partial mitigation so long as it is run with sudo. Results are variable—this approach continuously chews up an entire CPU core, unable to keep up on super-fast SSDs such as the OWC Accelsior 8M2. In other words, caching happens faster than the cache can be flushed, which shows just how inefficient Apple’s caching implementation is once it very quickly uses up available memory.

diglloydTools IntegrityChecker Java: Optimizing Performance by Working Around macOS Caching Performance Bugs

Hashing speed

This first graph is from a smallish test, using only 34.6GB of data on a 2019 Mac Pro with 384GB of memory, ample memory to cache the data in full many times over.

With pre-cached data* (“cached”), data is read directly out of the cache and hashed at ~12.7 GB/sec.

With caching enabled but the data not yet read*, no meaningful performance difference is seen between the two PCIe SSDs (ignoring a small variation run-to-run). Even though the OWC Accelsior 8M2 is capable of ~12600MB/sec vs ~6500MB/sec for the 4M2. That’s the overhead of macOS caching at work.

Cached or uncached*, the two PCIe SSDs are throttled to the limitations of macOS caching.

Caching is supposed to be a performance-enhancer, but when the data only needs to be read and processed once (the most common situation!), it has become a major performance hit.

As shown in the 2nd graph, the loss of performance gets much worse on large jobs when the available memory is fully used for caching.

* Uncached times used sudo purge prior; cached results were after running a first time.

IntegrityChecker Java hashing speed: cached vs uncached on OWC Accelsior 4M2/8M2

Below, the potential speed without the macOS caching performance bug is shown in gray. Actual results are in green.

This 8TB test causes macOS to quickly use up all available memory for cached files, whereupon Apple’s grossly inefficient algorithm throttles maximum I/O speed to about 3.2 GB/sec. This is pathetic software engineering from Apple that was passable 3 or 4 years ago, but here in 2021 it’s a bad joke.

As shown, macOS caching throttles the ~12.7 GB/sec OWC Accelsior 8M2 down to about 3.1 GB/sec—a 75% loss of performance. With the constant sudo purge via running IntegrityChecker Java (icj) with sudo, that performance loss is reduced to 8.2GB/sec, a loss of “only” 35%.

The OWC Accelsior 4M2 sees similar losses in being throttled down to the ~3.2GB/sec range, but the relative loss vs maximum speed is a performance loss of “only” about 50%. Using sudo purge on the OWC Accelsior 4M2 yields performance reasonably close to what the SSD is capable of.

IntegrityChecker Java hashing speed: constantly purging the buffer cache vs allowing unfettered caching, Accelsior 4M2/8M2
View all handpicked deals...

Voigtlander MACRO APO-LANTHAR 65mm f/2 Aspherical Lens for Sony E
$999 $849
SAVE $150

diglloyd.com | Terms of Use | PRIVACY POLICY
Contact | About Lloyd Chambers | Consulting | Photo Tours
Mailing Lists | RSS Feeds | X.com/diglloyd
Copyright © 2020 diglloyd Inc, all rights reserved.
Display info: __RETINA_INFO_STATUS__