diglloydTools IntegrityChecker Java: Optimizing Performance by Working Around macOS Caching Performance Bugs
re: data integrity and diglloydTools and IntegrityChecker Java
re: diglloydTools IntegrityChecker Java: Optimizing Performance on Hard Drives
re: Optimizing Performance by Working Around macOS Caching Performance Bugs
MacOS utilizes unused main memory for file system caching via the “unified buffer cache”.
On systems with very large amounts of memory, hundreds of gigabytes can be used for this caching. For example, I have seen 330GB of caching on my 384GB 2019 Mac Pro, see “Cached Files”:
Caching is good, right? Yes, if/when done well, but with Apple and for years now, it is an inefficient implementation on macOS that cuts speed of fast SSDs in half or more.
UPDATE Dec 2021: the performance loss is greater for faster SSDs. All SSDs are effectively throttled to about 3.2GB/sec once all available memory is used for caching. This is a 75% performance loss for fast SSDs like the OWC Accelsior 8M2. Some native apps might be smart enough to set a special I/O flag to disable caching, but most are not.
I thought at first that it was a case of “more memory, worse performance”, but it looks to be much worse, a brokenly-slow algorithm likely involving inefficient memory allocation and copying. Because it’s just as bad a hit with modest amounts of memory (eg 64GB), as with 384GB.
The worst situations are those where large amounts of data are read, because even if it’s reading it once and only once, macOS caches it all, then cycles through its now-full cache for new reads, exerting a huge performance drag. Examples include:
- Playing-back or processing video stream.
- Backups or cloning or file copying.
- Data integrity verification as with IntegrityChecker Java.
- Sluggish application performance and spinning rainbow beachball problems. Weird delays of 1/2/10/30 seconds at unpredictable times.
- Minutes to launch some applications instead of seconds. For example, my web server can take 3 minutes instead of 12 seconds to start up, then is glacially slow as it is I/O bottlenecked.
There are two key implications to this macOS performance bug:
- On my OWC Mercury Accelsior M42 PCIe SSD and my 2019 Mac Pro with 384GB memory, the caching hit is so bad that performance rapidly drops from 6.5 GB/sec to under 2GB/sec, and stays there until the cache is cleared. What’s the point of
- It can take MINUTES of a CPU pegged out at 100% to clear the cache (e.g., 'sudo purge'). That also means application performance can suffer strange long pauses as the cache is cleared when an application needs memory.
Working around the bug for IntegrityChecker Java
IntegrityChecker Java builds in a special facility to clear-out the cache every 5 seconds. But it can only do so if run with 'sudo', as in:
sudo icj verify ...
sudo icj update ...
When invoked this way, you will see the 'purge' process running steadily while icj does its thing, and icj will annotate its progress with a "P" each time a purge is done. Even so, macOS caching is so pathologically inefficient that purges tend to take 10-40 seconds each! And the battle is still lost—in macOS Big Sur the purging code is so slow that often caching never drops below the 100GB level. In spite of constant invocations of 'purge'. Still, it manages to keep the performance in the 5.3 GB/sec range, which is way better than below 2GB/sec.
Hashing 412056 files totaling 12585.9 GiB in 28445 folders...
0%: 435 files 16.1 GiB @ 5478 MiB/sec, 00:03.013
0%: 861 files 33.3 GiB @ 5669 MiB/sec, 00:06.019
0%: 1316 files 50.2 GiB @ 5695 MiB/sec, 00:09.029
0%: 1873 files 68.2 GiB @ 5805 MiB/sec, 00:12.0
0%: 2425 files 86.5 GiB @ 5891 MiB/sec, 00:15.0 P
0%: 2786 files 104.9 GiB @ 5956 MiB/sec, 00:18.0
0%: 3069 files 119.2 GiB @ 5806 MiB/sec, 00:21.0
1%: 3356 files 136.0 GiB @ 5795 MiB/sec, 00:24.0
1%: 3692 files 153.8 GiB @ 5825 MiB/sec, 00:27.0
1%: 3990 files 170.5 GiB @ 5814 MiB/sec, 00:30.0
1%: 4290 files 187.9 GiB @ 5824 MiB/sec, 00:33.0
1%: 4575 files 205.4 GiB @ 5836 MiB/sec, 00:36.0 P
1%: 4892 files 223.8 GiB @ 5871 MiB/sec, 00:39.0
1%: 5164 files 241.2 GiB @ 5875 MiB/sec, 00:42.0
2%: 5496 files 257.7 GiB @ 5859 MiB/sec, 00:45.0
2%: 5926 files 275.5 GiB @ 5873 MiB/sec, 00:48.0
2%: 6558 files 292.2 GiB @ 5862 MiB/sec, 00:51.0
2%: 7015 files 309.3 GiB @ 5860 MiB/sec, 00:54.0
2%: 7243 files 325.2 GiB @ 5837 MiB/sec, 00:57.0
2%: 7557 files 341.6 GiB @ 5825 MiB/sec, 01:00
2%: 7854 files 358.3 GiB @ 5820 MiB/sec, 01:03
2%: 8252 files 375.1 GiB @ 5815 MiB/sec, 01:06
3%: 8534 files 390.5 GiB @ 5791 MiB/sec, 01:09
3%: 8541 files 403.2 GiB @ 5730 MiB/sec, 01:12 P