All Posts by Date or last 15, 30, 90 or 180 days.
also by Lloyd: diglloyd.com photography and WindInMyFace.com

Thank you for buying via links and ads on this site,
which earn me advertising fees or commissions.
As an Amazon Associate I earn from qualifying purchases.

Other World Computing...
B&H Photo...
Amazon
As an Amazon Associate I earn from qualifying purchases.
Upgrade the memory of your 2018 Mac mini up to 64GB
877-865-7002
Today’s Deal Zone Items... Handpicked deals...
$1100 $880
SAVE $220

$999 $779
SAVE $220

$1299 $949
SAVE $350

$799 $549
SAVE $250

$1499 $999
SAVE $500

$799 $549
SAVE $250

$2797 $2497
SAVE $300

$1997 $1797
SAVE $200

$549 $499
SAVE $50

$1699 $949
SAVE $750

$240 $175
SAVE $65

$3399 $2699
SAVE $700

Years Later, macOS Big Sur leaves Major Disk I/O Performance Bug Unfixed

Several years ago, I noticed that for reading large amount of data (such as for data validation with IntegrityChecker, 8K video processing, etc), macOS had a nasty bug in which it eventually cuts I/O speed in half or worse. At that time, I saw speed losses of up to 80%.

The speed losses stem from the macOS “unified buffer cache” maintenance. Essentially, all unused memory is put to good use by caching files read from disk. If and when those files are needed again, speeds hit up to 12GB/sec when read from the cache. This works great for smaller tasks.

But if a large data set is processed (approaching or exceeding available memory), then it becomes pure overhead of constantly clearing out the oldest cached items to make room for new ones, none of which will ever be used, since new stuff just replaces old stuff and the old stuff is never needed again (or is long since out of the cache).

The net result is an overhead that cuts the speed of fast SSDs down to about 3.3GB/sec.

Maybe 3.3 GB/sec does not sound so bad, but if the SSD can do 6.5GB/sec, that's cutting performance in half, doubling the time required for things like data validation.

There are other issues: it can cause mysterious long pauses (minutes!) if an application needs memory that has to be reclaimed from the unified buffer cache, even on my Mac Pro with only 384GB memory. On a machine with 768GB or 1536GB memory it could take up to 8X longer!

Example

Other examples might include video processing, particularly 4/6/8K video streams.

Below, IntegrityChecker Java (icj) is invoked on the volume Work, a 16TB PCIe SSD (the superlative OWC Mercury Accelsior).

My implementation of icj is beautifully optimized for threaded computation with asynchronous (overlapping) I/O. If you have 200 CPU cores and a volume capable of 200 GB/sec, icj will make use of it insofar as the operating system can multitask efficiently.

But as can be seen, speed starts out at maximum of 6.5GB/sec, then quickly and steadily degrades to ~3.3 GB/sec.

diglloydMP:MPG lloyd$ icj verify Work
# icj 2.0 fc6 2020-10-01 08:20
...
Hashing 154267 files totaling 7451.3 GiB in 11774 folders... 
0%: 515 files 19.1 GiB @ 6526 MiB/sec, 00:03.000
0%: 987 files 38.6 GiB @ 6594 MiB/sec, 00:06.000
0%: 1548 files 58.3 GiB @ 6631 MiB/sec, 00:09.000
1%: 2121 files 77.9 GiB @ 6648 MiB/sec, 00:12.0
1%: 2688 files 97.5 GiB @ 6658 MiB/sec, 00:15.0
1%: 3029 files 117.1 GiB @ 6663 MiB/sec, 00:18.0
1%: 3374 files 136.9 GiB @ 6670 MiB/sec, 00:21.0
2%: 3743 files 156.6 GiB @ 6678 MiB/sec, 00:24.0
2%: 4122 files 176.4 GiB @ 6685 MiB/sec, 00:27.0
2%: 4433 files 196.0 GiB @ 6685 MiB/sec, 00:30.0
2%: 4737 files 215.7 GiB @ 6688 MiB/sec, 00:33.0
3%: 5069 files 235.3 GiB @ 6688 MiB/sec, 00:36.0
3%: 5449 files 254.9 GiB @ 6689 MiB/sec, 00:39.0
3%: 5861 files 274.8 GiB @ 6697 MiB/sec, 00:42.0
3%: 6599 files 294.3 GiB @ 6694 MiB/sec, 00:45.0
4%: 7073 files 314.0 GiB @ 6696 MiB/sec, 00:48.0
...
5%: 8470 files 383.7 GiB @ 5454 MiB/sec, 01:12
...
6%: 9179 files 509.1 GiB @ 5263 MiB/sec, 01:39
..
9%: 15297 files 737.2 GiB @ 4120 MiB/sec, 03:03	<=== major speed loss at 9% done
...
14%: 23388 files 1051.7 GiB @ 3733 MiB/sec, 04:48
...
14%: 24721 files 1106.6 GiB @ 3697 MiB/sec, 05:06
...
66%: 80848 files 4946.5 GiB @ 3367 MiB/sec, 25:04
...
72%: 103218 files 5436.0 GiB @ 3340 MiB/sec, 27:46
...
99%: 154172 files 7448.9 GiB @ 3260 MiB/sec, 39:00
Waiting for 32 of 154267 files to finish...
100%: 154267 files 7451.3 GiB @ 3260 MiB/sec, 39:00
Checking overall status for 11774 folders... done.

Sure would be nice if instead of more emojis and new bugs, Apple would make the core operating system facilities performant and reliable.


Save the tax, we pay you back, instantly!
View all handpicked deals...

Apple 13.3" MacBook Air with Retina Display (Early 2020, Gold)
$999 $779
SAVE $220

diglloyd.com | Terms of Use | PRIVACY POLICY
Contact | About Lloyd Chambers | Consulting | Photo Tours
Mailing Lists | RSS Feeds | Twitter
Copyright © 2020 diglloyd Inc, all rights reserved.
Display info: __RETINA_INFO_STATUS__