All Posts by Date or last 15, 30, 90 or 180 days.
also by Lloyd: diglloyd.com photography and WindInMyFace.com

Thank you for buying via links and ads on this site,
which earn me advertising fees or commissions.
As an Amazon Associate I earn from qualifying purchases.

Other World Computing...
B&H Photo...
Amazon
As an Amazon Associate I earn from qualifying purchases.
Upgrade the memory of your 2018 Mac mini up to 64GB
877-865-7002
Today’s Deal Zone Items... Handpicked deals...
$399 $199
SAVE $200

$2797 $2497
SAVE $300

$1398 $998
SAVE $400

$1499 $999
SAVE $500

$549 $499
SAVE $50

$799 $549
SAVE $250

$1699 $949
SAVE $750

$3899 $1499
SAVE $2400

$999 $849
SAVE $150

$1398 $998
SAVE $400

$1798 $1198
SAVE $600

$240 $175
SAVE $65

$240 $175
SAVE $65

$1399 $799
SAVE $600

$1798 $1198
SAVE $600

$1997 $1797
SAVE $200

$3399 $2699
SAVE $700

$400 $300
SAVE $100

$1699 $949
SAVE $750

$1199 $1099
SAVE $100

$1199 $1099
SAVE $100

$999 $949
SAVE $50

$1699 $949
SAVE $750

diglloydTools IntegrityChecker Java: Optimizing Performance by Working Around macOS Caching Performance Bugs

re: data integrity and diglloydTools and IntegrityChecker Java
re: diglloydTools IntegrityChecker Java: Optimizing Performance on Hard Drives
re: Optimizing Performance by Working Around macOS Caching Performance Bugs

MacOS utilizes unused main memory for file system caching via the “unified buffer cache”.

On systems with very large amounts of memory, hundreds of gigabytes can be used for this caching. For example, I have seen 330GB of caching on my 384GB 2019 Mac Pro, see “Cached Files”:

Massive performance drag in macOS caused by pathological file system caching

Caching is good, right? Yes, when done right, but it’s an incompetent implementation whose performance drag escalates with more memory. The more memory you have, the worse the I/O performance gets. In technical terms, it doesn’t 'scale'. The bug exists to this day in all recent macOS versions.

Worst cases

The worst situations are those where large amounts of data are read, because even if it’s reading it once and only once, macOS caches it all, then cycles through its now-full cache for new reads, exerting a huge performance drag. Examples include:

  • Playing-back or processing video stream.
  • Backups or cloning or file copying.
  • Data integrity verification as with IntegrityChecker Java.
  • Sluggish application performance and spinning rainbow beachball problems. Weird delays of 1/2/10/30 seconds at unpredictable times.
  • Minutes to launch some applications instead of seconds. For example, my web server can take 3 minutes instead of 12 seconds to start up, then is glacially slow as it is I/O bottlenecked.

Implications

There are two key implications to this macOS performance bug:

  • On my OWC Mercury Accelsior M42 PCIe SSD and my 2019 Mac Pro with 384GB memory, the caching hit is so bad that performance rapidly drops from 6.5 GB/sec to under 2GB/sec, and stays there until the cache is cleared. What’s the point of
  • It can take MINUTES of a CPU pegged out at 100% to clear the cache (e.g., 'sudo purge'). That also means application performance can suffer strange long pauses as the cache is cleared when an application needs memory.

Working around the bug for IntegrityChecker Java

IntegrityChecker Java builds in a special facility to clear-out the cache every 5 seconds. But it can only do so if run with 'sudo', as in:

sudo icj verify ...
sudo icj update ...

When invoked this way, you will see the 'purge' process running steadily while icj does its thing, and icj will annotate its progress with a "P" each time a purge is done. Even so, macOS caching is so pathologically inefficient that purges tend to take 10-40 seconds each! And the battle is still lost—in macOS Big Sur the purging code is so slow that often caching never drops below the 100GB level. In spite of constant invocations of 'purge'. Still, it manages to keep the performance in the 5.3 GB/sec range, which is way better than below 2GB/sec.

Hashing 412056 files totaling 12585.9 GiB in 28445 folders... 
0%: 435 files 16.1 GiB @ 5478 MiB/sec, 00:03.013
0%: 861 files 33.3 GiB @ 5669 MiB/sec, 00:06.019
0%: 1316 files 50.2 GiB @ 5695 MiB/sec, 00:09.029
0%: 1873 files 68.2 GiB @ 5805 MiB/sec, 00:12.0
0%: 2425 files 86.5 GiB @ 5891 MiB/sec, 00:15.0 P
0%: 2786 files 104.9 GiB @ 5956 MiB/sec, 00:18.0
0%: 3069 files 119.2 GiB @ 5806 MiB/sec, 00:21.0
1%: 3356 files 136.0 GiB @ 5795 MiB/sec, 00:24.0
1%: 3692 files 153.8 GiB @ 5825 MiB/sec, 00:27.0
1%: 3990 files 170.5 GiB @ 5814 MiB/sec, 00:30.0
1%: 4290 files 187.9 GiB @ 5824 MiB/sec, 00:33.0
1%: 4575 files 205.4 GiB @ 5836 MiB/sec, 00:36.0 P
1%: 4892 files 223.8 GiB @ 5871 MiB/sec, 00:39.0
1%: 5164 files 241.2 GiB @ 5875 MiB/sec, 00:42.0
2%: 5496 files 257.7 GiB @ 5859 MiB/sec, 00:45.0
2%: 5926 files 275.5 GiB @ 5873 MiB/sec, 00:48.0
2%: 6558 files 292.2 GiB @ 5862 MiB/sec, 00:51.0
2%: 7015 files 309.3 GiB @ 5860 MiB/sec, 00:54.0
2%: 7243 files 325.2 GiB @ 5837 MiB/sec, 00:57.0
2%: 7557 files 341.6 GiB @ 5825 MiB/sec, 01:00
2%: 7854 files 358.3 GiB @ 5820 MiB/sec, 01:03
2%: 8252 files 375.1 GiB @ 5815 MiB/sec, 01:06
3%: 8534 files 390.5 GiB @ 5791 MiB/sec, 01:09
3%: 8541 files 403.2 GiB @ 5730 MiB/sec, 01:12 P
Massive performance drag in macOS caused by pathological file system caching
Thunderbolt 4 Dock

Thunderbolt 4 hub and ports!

Any Mac with Thunderbolt 3.


OWC Accelsior 4M2 PCIe SSD
6000 MB/sec!
Mac or PC.


Ideal for Lightroom, Photoshop, video.
Capacity up to 16TB!

Save the tax, we pay you back, instantly!

diglloyd.com | Terms of Use | PRIVACY POLICY
Contact | About Lloyd Chambers | Consulting | Photo Tours
Mailing Lists | RSS Feeds | Twitter
Copyright © 2020 diglloyd Inc, all rights reserved.
Display info: __RETINA_INFO_STATUS__