See the disk speed tests of the OWC Accelsior 8M2 PCIe SSD, the fastest drive MPG has ever tested, by a factor of 2X!
Cross-platform data integrity validation
IntegrityChecker Java is the fastest and easiest way to validate the integrity of your data on Mac or Windows or Linux or any platform that can run Java. On any file system or device supported by Java, read-write or read-only, hard disk or SSD or WORM or optical, everything/anything with a file system.
And today’s IntegrityChecker should run fine a decade from how, so long as you have Java, because unlike Apple removing/changing APIs on a whim, Java APIs have been compatible forever (excepting a few rare thread-unsafe APIs), and will remain so.
But a downside of cross-platform future-proof Java code (and even POSIX APIs) is that they have no access to special disk I/O flags that tell the operating system to not cache disk reads. And as it turns out, macOS has a nasty caching performance bug that cuts disk speed in half on very fast drives.
Certain native-code programs can utilize a special API flag to avoid this bug, but many do not. Adobe Photoshop for example not only has quite slow I/O speeds for opening big files, but makes it worse by allowing caching.
Java-based or POSIX-based programs have no access to defeating unwanted caching. I’ve programmed IntegrityChecker Java to include a partial mitigation so long as it is run with sudo. Results are variable—this approach continuously chews up an entire CPU core, unable to keep up on super-fast SSDs such as the OWC Accelsior 8M2. In other words, caching happens faster than the cache can be flushed, which shows just how inefficient Apple’s caching implementation is once it very quickly uses up available memory.
This first graph is from a smallish test, using only 34.6GB of data on a 2019 Mac Pro with 384GB of memory, ample memory to cache the data in full many times over.
With pre-cached data* (“cached”), data is read directly out of the cache and hashed at ~12.7 GB/sec.
With caching enabled but the data not yet read*, no meaningful performance difference is seen between the two PCIe SSDs (ignoring a small variation run-to-run). Even though the OWC Accelsior 8M2 is capable of ~12600MB/sec vs ~6500MB/sec for the 4M2. That’s the overhead of macOS caching at work.
Cached or uncached*, the two PCIe SSDs are throttled to the limitations of macOS caching.
Caching is supposed to be a performance-enhancer, but when the data only needs to be read and processed once (the most common situation!), it has become a major performance hit.
As shown in the 2nd graph, the loss of performance gets much worse on large jobs when the available memory is fully used for caching.
* Uncached times used sudo purge prior; cached results were after running a first time.
Below, the potential speed without the macOS caching performance bug is shown in gray. Actual results are in green.
This 8TB test causes macOS to quickly use up all available memory for cached files, whereupon Apple’s grossly inefficient algorithm throttles maximum I/O speed to about 3.2 GB/sec. This is pathetic software engineering from Apple that was passable 3 or 4 years ago, but here in 2021 it’s a bad joke.
As shown, macOS caching throttles the ~12.7 GB/sec OWC Accelsior 8M2 down to about 3.1 GB/sec—a 75% loss of performance. With the constant sudo purge via running IntegrityChecker Java (icj) with sudo, that performance loss is reduced to 8.2GB/sec, a loss of “only” 35%.
The OWC Accelsior 4M2 sees similar losses in being throttled down to the ~3.2GB/sec range, but the relative loss vs maximum speed is a performance loss of “only” about 50%. Using sudo purge on the OWC Accelsior 4M2 yields performance reasonably close to what the SSD is capable of.