macOS: 3+ CPU Cores for up to several minutes after Rebooting

At one time, it used to be that rebooting meant a few seconds of system activity, then your Mac was yours to use.

My Mac no longer belongs to me for several minutes after rebooting.

Apple now runs so many crapware daemons at system startup that my work is impeded for up to several minutes after reboot. And with sleep mode now broken on my iMac 5K, whether the machine is put to sleep or booted up you the user lose.

As shown below, 3+ CPU cores are in use doing absolutely nothing to serve my needs. It’s not just CPU usage, there is a lot of disk I/O going on that slows other things down, as well as many megabytes of network access—a serious headache in the field when connected using a weak cell phone signal (local hot spot).

As far as I can tell:

• 'lsd' daemon process is a launch service process. Why it needs to run for minutes...?
• 'tccd' daemon process is what?
• syspolicyd daemon process issome kind of security thing.
• mdworker and mds and similar are all Spotlight cruft.

But it has been a lot worse not long ago. For a while, every time I ran a Photoshop javascript for image processing, Apple’s 'sandboxd' would chew up an entire CPU core continuously and greatly slow the script. That might now be fixed, finally, at least on macOS Big Sur.

Not that Adobe is much better—numerous daemon processes even when not using Adobe software, all consuming memory and generally crapping-up the environment. I 'get it' that Adobe background processes are needed while using Adobe software. The rest of the time I’d like the 8 or 10 Adobe daemon processes to quit and leave things alone.

3+ CPU cores chewed up for up to several minutes after reboot

Latest macOS Mojave Update on my iMac 5K Wrecks Sleep Mode; iMac Wakes up Dozens of times a day

I run macOS Mojave on my 2019 iMac 5K (I will never “upgrade” the OS).

This iMac 5K has been the most reliable Mac I’ve ever used, excepting the 2012 MacBook Pro when used as a server. I know this particular iMac 5K of mine very well, exactly how it has performed for 2.5 years now.

I installed the latest macOS Mojave around September 30. Since then, my iMac 5K wakes up frequently out of sleep, sometimes as often as every 5 minutes, day or night. This is driving me crazy.

Checking the system logs, I am trying to determine what is causing the wakeups, but so far I am not sure.

Trust Apple to screw things up—every time there is an update, new bugs appear.

Apple ARM M1X CPU Macs eg MacBook Pro Next Week... Time to Consider?

The first iteration of the Apple ARM technology with the Apple M1 chip was/is fairly impressive, but suffered from a maximum 16GB memory limit—fine for web and email but a non-starter for many photographers and professionals.

Next week (Oct 18), Apple will announce new 14-inch and 16-inch MacBook Pro modes featuring the Apple M1X chip, said to offer more CPU cores and higher memory capacity.

Details are sketchy, but rumors suggest 10 CPU cores (8 high performance, 2 efficiency) and up to 64GB memory, as well as more GPU cores. Assuming such specifications, the new models might outperform just about everything else in the Mac lineup so long as there is enough memory. Maybe and maybe not—the only thing that matters is how fast real-world tasks run, not some silly benchmark metric.

Rumors also suggest that Apple might finally un-stupid the design*, which made practical usage a total PITA. Rumors suggest return of an SD camera card slot, MagSafe charger, and more ports. If all this works out, it might finally be time to look at the MacBook Pro as a worthy offering within the Mac lineup.

Rumors do not suggest fixing the non-upgradeable design approach; it is likely that memory and CPU and SSD will all remain soldered onto the logic board, meaning if you buy 'wrong', you have to re-buy an up-spec'd model. But at least with SSD, a fast external SSD can be added, a bit of a nuisance but a viable solution. For photographers, anything less than a 1TB SSD and 32GB memory is a really bad idea.

Supply chain woes might make supply and delivery of these new MacBook Pros pretty sluggish. Already Apple apparently has had to cut back Apple iPhone 13 production, and it would not be surprising to see similar limitations with MacBook Pros.

As for an M1X iMac 5K and a new Mac Pro: it looks like they will be many months off. And the M1X chip while presumably sufficient for an iMac 5K is not likely to be fit for a Mac Pro design (too limited).

* For some years now, Apple has made a strong impression of allowing the latest “dog” to piss on the design fire hydrant—nonsensical asinine design changes. Examples included the hated keyboard changes, the worthless and annoying touch bar design (instead of real keys), removal off important ports (SD slot, USB-A, DisplayPort, charging asinineness).

Portable SSDs for Travel Storage and Especially Backup

As a photographer, I travel 4-5 months of the year in my Sprinter van for photography. My travel machine is the 2019 iMac 5K*, safely stowed in the Gator Cases Creative Pro 27" iMac Carry Tote when driving, and powered by a 10kW battery lithium iron phosphate battery pack off a 2000W DC/AC inverter. The 2019 iMac 5K is astonishingly power efficient given its included display and I can run for 3-4 days on the power packs without recharging.

Anyway, a key thing I do after every day’s shoot is to download my images and make several backups. Ditto after working in the van some days, after I have organized and published stuff.

* Laptops are a very poor and hopelessly inefficient solution for the work I need to do—very small screens, terrible ergonomics to ruin your posture and nag your neck, godawful keyboard and a mouse is essential too.

Best SSDs for travel and backup

Over the past few years I’ve worked with 4 or 5 different SSDs while traveling. There are multiple considerations:

• SSD speed is important, though only to a point. Still, if it’s a big shoot and it was an all day hike and I want to get to sleep, backup speed can be an irritant if not fast enough.
• I like to take one SSD with me on hikes as a precaution against break-in/theft. A larger/heavier SSD is a nuisance and poorly suited for my hikes.
• Durability—over time things can fail.
• Compatibility: it is very helpful to be able to plug a drive into my laptop (older one), which requires USB-C. A Thunderbolt SSD is incompatible with my older Thunderbolt-2 laptop.

Samsung T5 I have used for several years (not a fan of the "dog" Samsung T7) . One of my four T5’s a bad connector now, but still works. I like them for their compact size and light weight, but they are half the speed of the slowest of the OWC offerings. OTOH, they offer hardware encryption, which has its uses if theft while traveling is a concern.

OWC Envoy Pro Elektron (240GB to 2TB capacities) is my all-around favorite for travel: extremely solid built, twice the speed of the Samsung T5, and still very compact. Up to 2TB capacities make it hard to beat.

OWC Envoy Pro SX and OWC Envoy Pro FX are both superb—the very best performance. These are the drives to get for speed. But being solidly-built enough for a truck to run over them, they feel like solid hunks of metal and they do not fit into a pants pocket, unlike the OWC Envoy Pro Elektron.

Four portable SSDs, USB-C and Thunderbolt.
Car keys to show size/scale

diglloydTools MemoryTester Finds Bad Memory in Server Today

diglloydTools includes IntegrityChecker for data integrity (including a cross-platform Java version that works on Mac, Windows, Linux, etc), and MemoryTester and DiskTester.

I have new respect for my own testing tool! My diglloydTools MemoryTester has always been 'boring' with memory free issues, in that it is a good tool for proving real-world memory bandwidth. But today it confirmed bad memory modules were in fact the problem.

Today’s afternoon in a data center fixing my web server turned out to be bad memory modules, which diglloydTools MemoryTester found instantly. Having run 24X7 for some years, they started failing a month or so ago, with sporadic warning signs including:

• macOS El Capitan sporadically crashing with a panic twice a week or so.
• Java Virtual Machine (JVM) randomly once a day, from an internal memory check.
• Source code versioning software 'git' failing 'git gc' and 'git fsck' with corruption/hashing errors. And the apparently corrupted index file from last week.
• Random reports of file corruption by IntegrityChecker Java (files randomly fail to validate, not consistent).

OWC memory modules
failed after ~9 years of use including ~4 years of 24X7 server use
OWC replaced the modules — lifetime warranty

Instant and correct diagnosis

But diglloydTools MemoryTester can be run at any time without special protocols. But all eight test threads found the bad memory problems within a few seconds.

diglloydWeb:WIND lloyd$ mt stress
MemoryTester 1.6.1 64-bit, diglloydTools 2.2.15, 2017-12-24 18:12
Copyright 2006-2017 DIGLLOYD INC. All Rights Reserved
Use of this software requires a license. See http://macperformanceguide.com/Software-License.html
OS X 10.11.6, 8 CPU cores @ 2300 MHz, 16384 MiB memory
Monday, October 11, 2021 at 2:26:38 PM Pacific Daylight Time
********************************** Stress Test *********************************
Test duration: 8h
CPU percent: 100% per cpu core
Num CPU threads: 8
Memory per thread (requested): 1.79 GiB
...
Speeds denote memory bandwidth: copy and compare speed (memcpy + 64-bit compare)
Allow several minutes for averages to stabilize, especially with large memory sizes.
Task EatCPU-5 exited with error: -1, msg ="BAD MEMORY, changed after copy!"
6s:  7160  1432 1432 1432 1432 1432    0    0    0 (best)
10s: 10492  1749 1749 1749 1749 1749    0 1749    0 (best)
Task EatCPU-3 exited with error: -1, msg ="BAD MEMORY, changed after copy!"
16s: 11586  2207 2207 1655 1655 2207    0 1655    0 (best)
20s: 12398  2214 2214 1771 1328 2214    0 2214  443 (best)
Task EatCPU-4 exited with error: -1, msg ="BAD MEMORY, changed after copy!"
Task EatCPU-2 exited with error: -1, msg ="BAD MEMORY, changed after copy!"
26s: 13320  2391 2391 2049 1025 2049    0 2391 1025 (best)
30s: 13338  2668 2371 1778  889 1778    0 2371 1482 (best)
36s: 13849  2721 2721 1484  742 1484    0 2721 1979 (best)
40s: 14033  2896 2896 1337  668 1337    0 2896 2005 (best)
Task EatCPU-6 exited with error: -1, msg ="BAD MEMORY, changed after copy!"
46s: 14344  3102 3102 1163  582 1163    0 2714 2520 (best)
50s: 14451  3390 3212 1071  535 1071    0 2498 2676 (best)
Task EatCPU-1 exited with error: -1, msg ="BAD MEMORY, changed after copy!"
56s: 14662  3825 3028  956  478  956    0 2231 3188 (best)
60s: 14582  4018 2827  893  446  893    0 2083 3422
Task EatCPU-7 exited with error: -1, msg ="BAD MEMORY, changed after copy!"
Task EatCPU-0 exited with error: -1, msg ="BAD MEMORY, changed after copy!"
66s: 14073  4060 2571  812  406  812    0 1894 3518

Verification of smooth operation after memory swapped-out

No problem with the swapped-out memory.

diglloydWeb:WIND lloyd$ mt stress
diglloydWeb:MPG lloyd$ mt stress -m 500M
MemoryTester 1.6.1 64-bit, diglloydTools 2.2.15, 2017-12-24 18:12
Copyright 2006-2017 DIGLLOYD INC. All Rights Reserved
Use of this software requires a license. See http://macperformanceguide.com/Software-License.html
OS X 10.11.6, 8 CPU cores @ 2300 MHz, 16384 MiB memory
Monday, October 11, 2021 at 3:18:44 PM Pacific Daylight Time
********************************** Stress Test *********************************
Test duration: 8h
CPU percent: 100% per cpu core
Num CPU threads: 8
Memory per thread (requested): 500 MiB
...
=====> Use ctrl-C to stop <=====
Speeds denote memory bandwidth: copy and compare speed (memcpy + 64-bit compare)
Allow several minutes for averages to stabilize, especially with large memory sizes.
6s:  9588   400  400  400  400  400 2797 2397 2397 (best)
10s:  9991   731  731  731  731  731 2437 1950 1950 (best)
16s:  9993   922  922  922  922  922 1999 1691 1691 (best)
20s:  9994   987  987  987  987  987 1851 1604 1604 (best)
26s: 10276  1047 1047 1047 1047 1047 1808 1618 1618 (best)
30s: 10239  1073 1073 1073 1073 1073 1734 1569 1569
36s: 10541  1171 1171 1171 1171 1171 1654 1516 1516 (best)
40s: 10487  1179 1179 1179 1179 1179 1613 1489 1489
46s: 10531  1188 1188 1188 1188 1188 1620 1458 1512
50s: 10539  1193 1193 1193 1193 1193 1591 1491 1491
56s: 10481  1199 1199 1199 1199 1199 1554 1466 1466
60s: 10614  1202 1244 1244 1244 1244 1534 1451 1451 (best)
66s: 10671  1244 1244 1244 1244 1244 1508 1471 1471 (best)
70s: 10632  1245 1245 1245 1245 1245 1494 1458 1458
76s: 10746  1278 1278 1278 1278 1245 1507 1442 1442 (best)
80s: 10740  1276 1276 1276 1276 1276 1494 1432 1432
86s: 10745  1274 1274 1274 1274 1274 1477 1448 1448
90s: 10851  1301 1273 1301 1301 1301 1495 1439 1439 (best)
96s: 10849  1298 1298 1298 1298 1298 1479 1454 1428
100s: 10815  1296 1296 1296 1296 1296 1470 1445 1420
106s: 10839  1317 1293 1293 1293 1293 1481 1434 1434
110s: 10854  1314 1292 1314 1314 1292 1473 1428 1428 (best)
116s: 10853  1311 1289 1311 1311 1311 1461 1440 1418
120s: 10844  1309 1309 1309 1309 1309 1454 1433 1413
126s: 10804  1306 1306 1306 1306 1306 1445 1425 1405
130s: 10779  1304 1304 1304 1304 1304 1439 1419 1400
136s: 10781  1302 1302 1320 1302 1302 1430 1412 1412
140s: 10794  1318 1300 1318 1318 1300 1425 1407 1407
146s: 10795  1315 1298 1315 1315 1315 1435 1401 1401
150s: 10774  1314 1297 1314 1314 1314 1430 1397 1397
...

Memory Fails in Web Server, with Sporadic Problems Making the Cause Mysterious, Until Today

I spent most of my afternoon (4 hours) in a Sunnyvale data center in a 100 decibel environment (shout to be heard). It would have been intolerable and impossible without my Sony WH-1000XM4 noise-canceling headphones.

I was down there thinking I needed an hour or less to fix a few corrupted git pack files. Turns out that there was no file corruption, but the problem was seemingly much worse—bad memory in the server. Symptoms included:

• macOS El Capitan sporadically crashing with a panic twice a week or so.
• Java Virtual Machine (JVM) crashing randomly at startup once a day, an internal memory check failure.
• Source code versioning software 'git' failing 'git gc' and 'git fsck' with corruption/hashing errors, problems appearing and disappearing and changing. And the apparently corrupted index file from last week.
• Random reports of file corruption by IntegrityChecker Java (files randomly fail to validate, then validate, not consistent).

The apparent file corruption at first made me think of a 'git' version mismatch problem causing real data corruption. But I quickly found that even known-good freshly-cloned data was being flagged as corrupted. But which files were flagged as corrupted changed—good, then bad, then good, then bad, and different files.

It finally 'clicked' that bad memory might be the issue.

As it turns out, files were not corrupted. What both 'git' and IntegrityChecker were reporting as bad hashes turned out to be a stored file hash with the correct value not matching a just-computed hash, which was not different because of bad memory causing bit errors that produced a different hash.

OWC memory modules
failed after ~9 years of use including ~4 years of 24X7 server use
OWC replaced the modules — lifetime warranty

Replacing the bad memory

By good luck (planning for the worst), I had brought along a spare server machine, slower, but fully capable of replacing the main one—and also with the same kind of memory modules. Once I confirmed the bad memory in the server, I opened it up and swapped-out its memory for the modules in the backup machine.

Once the memory was swapped-out in the server, all the issues went away, with no further problems in spite of aggressively stress-testing it.

The 2012 MacBook Pro — classic that still sells

You can still buy the 2012 MacBook Pro used at MacSales.com. Not a speed demon by any metric, but they last forever. Good web servers, music servers, etc. The best model is that "late 2012" 15-inch, with a 4-core CPU (4 real, 8 virtual cores).

My web server has been a late-2012 MacBook Pro 4-core running 24X7 for some years now. Not the fastest machine around, but incredibly reliable. Prior to that, 2-core models ran for most of a decade without any issues whatsoever. Any downtime until now had been purely software issues (aside from the rare data center glitches), and those were few and far between and generally my goofs.

Apple 2012 MacBook Pro 4-core 16GB

To its credit, Apple once long ago built hyper-reliable and durable machines (upgradeable too!). The 2012 MacBook Pros have been more reliable over the past ~13 years than any data center yet (Facebook and Google outages anyone?). No subsequent Apple MacBook Pro has been nearly as solid, and we’ve left the path up upgradeability some years ago. Apple now builds flaky, non-upgradeable MacBook Semi-Pro machines—shiny throw-away models built to the dilettante design-of-the-day ideas (no SD card slot? no USB-A ports? No ethernet port?).

With Apple, You are the Product Being Sold — Software Intent on Sales Ignores Your Interests

Apple does not build its apps like Apple Music for your convenience. Rather you are the product being sold, to those selling you stuff (Apple and all the creators from which they take 30% commission).

Apple Music is an advertisment app with disfunctional but just good just enough user interface to sucker you into using it.

Witness the screen below—all I want to do is to listen to my audiobook in peace. But after pausing and returning to it (every day and several times a day), Apple each time displays this crapware screen (which I have literally NEVER used), forcing me to take action just to return to where I was before.

Unwanted Apple crapware ads

IntegrityChecker Java: Simultaneous Data Verification on any Number of Hard Drives or SSDs

See also: Checking for Data Corruption after macOS Hoses Your System

I let some of my tertiary backup drives go stale for some months, and finally brought them all up to day today. I like to do that after coming back from a trip.

With hard drives, multiple backups can take a loooooong time, with the 8TB drives and 14TB drives mostly full, though there are some tricks to also doing simultaneous backups*.

After backing up, verifying data integrity on all 8 drives can be done very efficiently with IntegrityChecker Java (icj). All eight drives can be verified simultaneously*, using one instance of icj (one per Terminal window).

Since each drive is independent of the others, the only gating factors are I/O bandwidth (no problem in an OWC Thunderbay), and CPU power (ample on the 8-core 2018 iMac 5K. The hard drives are slow enough that each drive takes only ~35% of one CPU core). In other words, the time it takes depends entirely on how fast (slow) the drives are. A 4-core CPU would see higher CPU utilization of course.

Avoid simultaneous backups or data verification if the data is on the same physical hard drive (such as two volume partitions); it will cause the drive to bang its drive heads continuously and run very slower. SSDs do not have this limitation, as there are no moving parts.

IntegrityChecker Java: Simultaneous Data Verification on any Number of Hard Drives at Once

Below, 8 backup volumes totaling 71TB were validated, showing perfect data integrity—not a single bit in error.

IntegrityChecker Java: 8 backup volumes totaling 71TB validated with perfect data integrity

OWC Announces Thunderbolt Dual DisplayPort 1.4 Adapter supporting up to dual 8K displays

Slick! Now what I want is an 8K display with professional color management, at least 34 inches in size. But that presumes a video card than can support 8K also. And with supply chains strained to the breaking point, 8K displays are as rare as hen’s teeth, with very little selection. Well, no selection at all, even at B&H Photo.

But even with a 34-inch display, the pixel density is considerably more dense than a 27-inch 5K display. See Too-High Pixel Density on 5K and 8K Displays Impedes Image Assessment.

Get OWC Thunderbolt to Dual DisplayPort Adapter at macsales.com.

Practical issue: an 8K display will consume most of the write bandwidth on the Thunderbolt bus it is used on. Two 8K displays will leave almost zero bandwidthfor writing data. So on machines with a single Thunderbolt bus (iMac 5K and some of the smaller laptops), you cannot really use both 8K video and data, as a practical matter.

OWC Announces Thunderbolt Dual DisplayPort Adapter

Easily add up to two 4K or 8K displays for stunning images, increased productivity enhanced gaming

OWC®,the premier zero-emissions Mac and PC technology company,and a respected provider of Memory, External Drives, SSDs, Mac & PC docking solutions, and performance upgrade kits, announces the OWC Thunderbolt Dual DisplayPort Adapter.

With the new OWC Thunderbolt Dual DisplayPort Adapter, you’ll be amazed when your eyes see up to 8K resolution delivered by the OWC Thunderbolt Dual DisplayPort Adapter. By utilizing the latest DisplayPort 1.4 technology, this high-performance adapter delivers up to 4x higher resolution than DisplayPort 1.21. All you need is a single Thunderbolt port to plug it in, and this little wonder’s compact size and bus-powered convenience will open a world of possibilities.

...

OWC Thunderbolt Dual DisplayPort 1.4 Adapter

The OWC Thunderbolt Dual DisplayPort Adapter opens a world of possibilities

Professionals: Use it to stream high-resolution, life-like content on multiple monitors. Keep active projects open on one display while viewing members of an online meeting on another display. Keep an eye on chat boxes while simultaneously connecting with your followers on another screen.

Students: Improve online learning with multiple screens. Read text or watch the instructor on one display as you take notes on the other.

Gaming: Stretch that racing and flight sim game across two screens for an unbelievable view. Dedicate one screen as a map or voice chat display to eliminate the hassle of minimizing game windows.

OWC Thunderbolt Dual DisplayPort Adapter Highlights:

• Single display up to 8K @ 30Hz or 8K @ greater than 30Hz with DSC
• Dual display up to 4K @ 60Hz, 4K @ 144Hz with DSC, or 8K with DSC
• See it All: Watch videos, play games, create crisp digital signage walls, and make eye-catching presentations on a DisplayPort display or projector
• Full Display Potential: DSC (Display Stream Compression) compatible for higher refresh rates
• Stunning Visuals: HDR for sharper images, brighter colors, and greater contrast
• Hear More: Supports multichannel, high-definition digital audio formats
• Energy Efficient: Achieve high resolutions on two displays without draining system resources
• Protected: 1 Year OWC Limited Warranty

Apple Safari Tab Bar Colorization: How to Disable

Someone at Apple must have time on their hands, inventing new ways of continuously degrading good visual design. Apple Mail is a usability mess, and now the same mindset is moving to Apple Safari with the latest update.

I refer to the massive color smearing header being applied to some web sites by default.

Toggle to compare. Read on for how to disable the colorization.

Years ago, Apple had well-conceived and highly respected user interface guidelines that led the industry. These days, the operating principle is change for change’s sake with “shiny stuff” all the rage. A pox on elegance or usability. Perhaps there is some argument to be made for the heavy-handed colorization, but I consider it crapware that serves to confuse and surprise, to demand attention.

Apple Safari: colorized vs non-colorized browser header

Below, in Safaric Preferences => Tabs, uncheck Show color in tab bar.

Apple Safari: uncheck “Show color in tab bar” to clean up the tab bar colorization

Years Later, macOS Big Sur leaves Major Disk I/O Performance Bug Unfixed

Several years ago, I noticed that for reading large amount of data (such as for data validation with IntegrityChecker, 8K video processing, etc), macOS had a nasty bug in which it eventually cuts I/O speed in half or worse. At that time, I saw speed losses of up to 80%.

The speed losses stem from the macOS “unified buffer cache” maintenance. Essentially, all unused memory is put to good use by caching files read from disk. If and when those files are needed again, speeds hit up to 12GB/sec when read from the cache. This works great for smaller tasks.

But if a large data set is processed (approaching or exceeding available memory), then it becomes pure overhead of constantly clearing out the oldest cached items to make room for new ones, none of which will ever be used, since new stuff just replaces old stuff and the old stuff is never needed again (or is long since out of the cache).

The net result is an overhead that cuts the speed of fast SSDs down to about 3.3GB/sec.

Maybe 3.3 GB/sec does not sound so bad, but if the SSD can do 6.5GB/sec, that's cutting performance in half, doubling the time required for things like data validation.

There are other issues: it can cause mysterious long pauses (minutes!) if an application needs memory that has to be reclaimed from the unified buffer cache, even on my Mac Pro with only 384GB memory. On a machine with 768GB or 1536GB memory it could take up to 8X longer!

Example

Other examples might include video processing, particularly 4/6/8K video streams.

Below, IntegrityChecker Java (icj) is invoked on the volume Work, a 16TB PCIe SSD (the superlative OWC Mercury Accelsior).

My implementation of icj is beautifully optimized for threaded computation with asynchronous (overlapping) I/O. If you have 200 CPU cores and a volume capable of 200 GB/sec, icj will make use of it insofar as the operating system can multitask efficiently.

But as can be seen, speed starts out at maximum of 6.5GB/sec, then quickly and steadily degrades to ~3.3 GB/sec.

diglloydMP:MPG lloyd$ icj verify Work
# icj 2.0 fc6 2020-10-01 08:20
...
Hashing 154267 files totaling 7451.3 GiB in 11774 folders... 
0%: 515 files 19.1 GiB @ 6526 MiB/sec, 00:03.000
0%: 987 files 38.6 GiB @ 6594 MiB/sec, 00:06.000
0%: 1548 files 58.3 GiB @ 6631 MiB/sec, 00:09.000
1%: 2121 files 77.9 GiB @ 6648 MiB/sec, 00:12.0
1%: 2688 files 97.5 GiB @ 6658 MiB/sec, 00:15.0
1%: 3029 files 117.1 GiB @ 6663 MiB/sec, 00:18.0
1%: 3374 files 136.9 GiB @ 6670 MiB/sec, 00:21.0
2%: 3743 files 156.6 GiB @ 6678 MiB/sec, 00:24.0
2%: 4122 files 176.4 GiB @ 6685 MiB/sec, 00:27.0
2%: 4433 files 196.0 GiB @ 6685 MiB/sec, 00:30.0
2%: 4737 files 215.7 GiB @ 6688 MiB/sec, 00:33.0
3%: 5069 files 235.3 GiB @ 6688 MiB/sec, 00:36.0
3%: 5449 files 254.9 GiB @ 6689 MiB/sec, 00:39.0
3%: 5861 files 274.8 GiB @ 6697 MiB/sec, 00:42.0
3%: 6599 files 294.3 GiB @ 6694 MiB/sec, 00:45.0
4%: 7073 files 314.0 GiB @ 6696 MiB/sec, 00:48.0
...
5%: 8470 files 383.7 GiB @ 5454 MiB/sec, 01:12
...
6%: 9179 files 509.1 GiB @ 5263 MiB/sec, 01:39
..
9%: 15297 files 737.2 GiB @ 4120 MiB/sec, 03:03	<=== major speed loss at 9% done
...
14%: 23388 files 1051.7 GiB @ 3733 MiB/sec, 04:48
...
14%: 24721 files 1106.6 GiB @ 3697 MiB/sec, 05:06
...
66%: 80848 files 4946.5 GiB @ 3367 MiB/sec, 25:04
...
72%: 103218 files 5436.0 GiB @ 3340 MiB/sec, 27:46
...
99%: 154172 files 7448.9 GiB @ 3260 MiB/sec, 39:00
Waiting for 32 of 154267 files to finish...
100%: 154267 files 7451.3 GiB @ 3260 MiB/sec, 39:00
Checking overall status for 11774 folders... done.

Sure would be nice if instead of more emojis and new bugs, Apple would make the core operating system facilities performant and reliable.

Checking for Data Corruption after macOS Hoses Your System

A kernel panic should be a “never happens” or at least a rare event (once a year) on a competently implemented operating system. But macOS Big Sur does not come close to meeting that standard.

Instead we get superficial crapware and decreased reliability with every new macOS release—iterative buzzing manure piles heaped over macOS Big Sur—delivery soon with macOS Monterey.

After a macOS Big Sur kernel panic hosed my login account, I spent much of my day dealing with the aftermath.

A key recovery step is to check that data has not been corrupted. Using IntegrityChecker, I was able to validate two of my primary volumes, and discover a corrupted file on a 3rd. I got lucky given that the crash locked me out of my login account.

Checking for data corruption

If macOS crashes so hard that it hoses your login account, how in the world would you know that your data is intact, without using IntegrityChecker to check data integrity? Nothing else will serve.

The post-crash peace of mind that comes from using IntegrityChecker (java version, "icj") is priceless for any professional.

Primary SSD work volume — 'Master'

All OK on Master.

diglloydMP:MPG lloyd$ nice icj verify Master
# icj 2.0 fc6 2020-10-01 08:20
# ©2020 DIGLLOYD INC. All Rights Reserved. Valid license required.  https://diglloydtools.com
...
Process folder: /Volumes/Master
========================================================================================================================
2021-10-04 14:19:35 : 12325 folders totaling 1377.1 GiB
/Volumes/Master
========================================================================================================================
# With hash: 355360
# Without hash: 93
# Hashed: 355169
# Empty: 184
# Missing Files: 0
# Missing Folders: 0
# New Folders: 64
# Changed size: 10
# Changed date: 3
# Changed content + date, size unchanged: 3
# Total files differing: 13
# Num ignored folders: 9
# Num ignored files: 860
# SUSPICIOUS files: 0

Secondary SSD storage volume — 'Work'

A backup made just prior to my trip and thus a day prior to the crash shows a corrupted file. I was able to restore it from the backup and verify it as good. That’s not proof that the crash did it, but it sure looks like that, because the backed-up file verifies as intact. I replaced the damaged file with the intact backup file.

diglloydMP:MPG lloyd$ nice icj verify Work
# icj 2.0 fc6 2020-10-01 08:20
Hashing 154267 files totaling 7451.3 GiB in 11774 folders... 
0%: 502 files 18.6 GiB @ 6332 MiB/sec, 00:03.004
...
35%: 50153 files 2654.6 GiB @ 3898 MiB/sec, 11:37
/Volumes/Work/Archived/2019-0922-TripPhotos/NikonD850m/2019-1007-NikonD850m-DeathValleyMine/_DBW1715-Red090_BW.dng SUSPICIOUS
35%: 50444 files 2673.8 GiB @ 3910 MiB/sec, 11:40
...
100%: 154267 files 7451.3 GiB @ 3495 MiB/sec, 36:22
Checking overall status for 11774 folders... done.
Folders with changes: 1:
/Volumes/Work/Archived/2019-0922-TripPhotos/NikonD850m/2019-1007-NikonD850m-DeathValleyMine
Missing folders:
none
-- begin detailed file changes --
SUSPICIOUS_FILES  1	/Volumes/Work/Archived/2019-0922-TripPhotos/NikonD850m/2019-1007-NikonD850m-DeathValleyMine/_DBW1715-Red090_BW.dng
-- end detailed file changes --
========================================================================================================================
2021-10-04 15:00:27 : 11774 folders totaling 7451.3 GiB
/Volumes/Work
========================================================================================================================
# With hash: 154267
# Without hash: 0
# Hashed: 154267
# Missing Files: 0
# Missing Folders: 0
# Changed size: 0
# Changed date: 0
# Changed content + date, size unchanged: 0
# Total files differing: 0
# Num ignored folders: 5
# Num ignored files: 436
# SUSPICIOUS files: 1 (!) same size and date, but content changed (!)
(!) POSSIBLE DATA CORRUPTION (!)
A few ill-behaved programs change files without changing the modification date.
The following file contents have changed, but file dates and size have not changed.
/Volumes/Work/Archived/2019-0922-TripPhotos/NikonD850m/2019-1007-NikonD850m-DeathValleyMine/_DBW1715-Red090_BW.dng

Tertiary SSD work volume — 'Archived'

All OK on Archived.

diglloydMP:MPG lloyd$ nice icj verify 'Archived'
# icj 2.0 fc6 2020-10-01 08:20
# ©2020 DIGLLOYD INC. All Rights Reserved. Valid license required.  https://diglloydtools.com
...
Process folder: /Volumes/Archived
...
100%: 260935 files 5195.9 GiB @ 2671 MiB/sec, 33:11
Checking overall status for 16675 folders... done.
Missing folders: none
========================================================================================================================
2021-10-04 15:33:57 : 16675 folders totaling 5195.9 GiB
/Volumes/Archive
========================================================================================================================
# With hash: 260935
# Without hash: 0
# Hashed: 260935
# Missing Files: 0
# Missing Folders: 0
# Changed size: 0
# Changed date: 0
# Changed content + date, size unchanged: 0
# Total files differing: 0
# Num ignored folders: 7
# Num ignored files: 566
# SUSPICIOUS files: 0

macOS Big Sur Locks Me Out of my 2019 Mac Pro after Kernel Panic, Repudiating my Login Password

macOS Big Sur locks user out after kernel panic
“if you forgot your password, you can reset it using your Recovery Key”
===> “If a macOS kernel panic destroys your login...”

Here’s a crash experience you gotta love as one of the new “features” of macOS. In 36 years of using a Mac (since the original 128K Mac), I have never had this happen.

I was doing backups and data verification, with lots of I/O. Unless you love losing your stuff, that’s what you do when you do not trust unreliable software, e.g., macOS. But you don’t expect to not even be able to make backups without crashes!

I had the usual sinking feeling in my gut when the mouse cursor disappeared. Usual because it happens all too often. Thirty second later, the 2019 Mac Pro rebooted and presented its usual kernel-panic crash screen.

Rebooted fine, but when the login dialog appeared, my password no longer is accepted. I tried numerous times to no avail (and its not caps-lock or some such).

UPDATE: this is insanity: macOS locked me out of my computer (claimed invalid password), but its “reset password” recovery process accepts the password it locked me out-of. Huh?
Details below.

Kernel panics are a regular occurrence, not anomalies

In my experience, macOS Big Sur (and macOS Catalina) remain garbage in terms of reliability, at least if you try to use a powerful machine with computation and lots of I/O (users diddling with email/web and/or a drive or two are an entirely different situation). An embarrassment to Apple not perceived by Apple and so there is no feedback loop to fix anything. The last decent operating system version was macOS Mojave, which I still run on my 2019 iMac 5K, and which I greatly prefer.

My $20K 28-core Mac Pro with 384GB memory and Vega II video card kernel panics twice a week or so anytime it gets pushed hard. Which is what a Mac Pro is supposed to f*ing be made for—heavy duty use.

panic(cpu 10 caller 0xffffff800b9fea25): userspace watchdog timeout: no successful checkins from com.apple.WindowServer in 120 seconds
service: com.apple.logd, total successful checkins since wake (7130 seconds ago): 714, last successful checkin: 0 seconds ago
service: com.apple.WindowServer, total successful checkins since wake (7130 seconds ago): 702, last successful checkin: 120 seconds ago
service: com.apple.remoted, total successful checkins since wake (7130 seconds ago): 714, last successful checkin: 0 seconds ago
...
Kernel Extensions in backtrace:
com.apple.driver.watchdog(1.0)[4CFADD2A-613E-320D-AD14-9C9379E87CB7]@0xffffff800b9fc000->0xffffff800b9fefff
Process name corresponding to current thread: watchdogd
Mac OS version: 20G165
Kernel version: Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:21 PDT 2021; root:xnu-7195.141.6~3/RELEASE_X86_64
...

Disk I/O is a high-risk activity if you run 4 or 5 backups or similar. God help you if you also use too many CPU cores. The 2019 Mac Pro with macOS Big Sur is the biggest piece of sh*t ever produced by Apple, at least that I have used. My 2019 iMac 5K with macOS Mojave has been bulletproof, though it does not see the I/O loads as on the Mac Pro (too few Thunderbolt ports), so maybe it would turn sour if used as hard.

But never has it locked me out after a crash.

Is this some new bug from the most recent macOS Big Sur update? Instead of fixing the numerous serious reliability bugs in macOS Big Sur, we will soon enjoy a locust plague of even more bugs in macOS Monterey.

See Checking for Data Corruption after macOS Hoses Your System.

macOS Big Sur locks me out after kernel panic

Recovery... using the same password that macOS repudiates!

As discussed above, macOS would not let me log in with my password after a kernel panic.

But a process to recover from this macOS-caused disaster can be utilized by booting into Recovery Mode.

The recovery process accepts the same password that was repudiated at the login dialog. Makes no sense, but it seems that macOS not only kernel panics under load, but manages to destroy the user login password or database. I invested $22K in a high-end 2019 Mac Pro, and I have a computer incapable of not crashing when I try to utilize its 28-core CPU with heavy I/O loads (5 to 10 GB/sec). This macOS *cannot* do; it’s an unreliable POS. I have had to detune my expectations and the way I use the computer. Worst rip-off piece of crap from Apple in my lifetime.

The good news for me is that this inane process (should never have happened or be needed), fixed my login account and I can login again. The bad news is that if it happens once, it can happen over and over. I’m afraid to use more than a fraction of the 2019 Mac Pro capabilities given the crash (not the first one!).

macOS Big Sur Reset Password:unlock boot volume

macOS Big Sur Reset Password: new password

macOS Big Sur Reset Password: completed

Corrupted .git/index File: “git fatal: unordered stage entries in index”

This is a post for geeks like me using the git source code repository.

UDPATE 2021-10-11: all this trouble was really the result of bad memory in the server.

...

Today my web server somehow corrupted its .git/index file, rendering it impossible to pull any updates:

git fatal: unordered stage entries in index

I think I might have hit control-C at the wong time, and git corrupted the index file.

Because the repo is ~364GB, it is not an option to just pull it again! Yes, I know I am abusing git for a massive website—it was never designed for a repo that will soon approach half a terabyte. But as a testament to how well written it is, it has worked great.

I was shitting bricks that this meant a 300 mile drive home, then a visit to the Tier 1 server room in order to manually copy over a fresh/good git repo.

As it turns out, this simple approach worked just fine to rebuild .git/index:

rm .git/index
git reset

After 30 minutes or so, the job was done; git has to read the entire set of pack files totaling ~364GB, so it took a while even on an SSD.

How To Prevent System Sleep while a Terminal Command is Running: caffeinate

As part of my work, I frequently work remotely, relying on a cellular data via a personal hot spot on my iPhone.

Even with a cell phone booster, internet speed can be quite limited (think 10K/sec at times!), and sometimes flaky too—remote areas don’t exactly have great cell phone coverage, if at allm and an hours-long driving round trip is a huge time-waster.

Here’s a nasty little behavior of macOS: it doesn’t much care what you are doing in something like Terminal—computation, network access, disk access, etc are all ignored. So you take a short break, only to find that macOS has put the machine to sleep and effectively aborted the job.

Example

I use a source code management system for my web servers called git, and I frequently do a git push to sync my local development system to the server:

git push

But if macOS sleeps the system, the process goes to sleep, and when it wakes up the network connection is aborted. All that cellular data gone to waste, and job not done.

The solution is to prevent system sleep until the push is done using 'caffeinate':

caffeinate git push

Since I do this a lot, I like to make a shortcut (alias) for it:

alias push="caffeinate git push"

Example 2

Suppose you want to verify a large volume 'Photos' using IntegrityChecker. You want the system to sleep only after the verify has finished.

caffeinate icj verify Photos

Should you Upgrade to macOS Monterey?

It won’t be long before macOS Monterey is out.

See Apple’s preview of macOS Monterey, showing the usual “shiny stuff” with all those great new bugs you gotta have and all sorts of things pointlessly redesigned, so as to confuse you and make things harder to use, to keep your mind sharp relearning how to do something simple you used to understand.

OK, maybe there might be one useful thing in there somewhere, but all I see is crapware I’ll never need or use.

For the average user, upgrade if you like whatever new features are offered (nothing of value is typical). You might get a few headaches too, and a slew of new bugs and security vulnerabilities. Enjoy.

For the professional user relying on their Mac for work, MPG’s advice remains the same as usual—be cautious and reluctant about a major release of macOS.

1. Never upgrade to a major new release of macOS until at least 6 months have elapsed.

2. For workflows that work, never upgrade the OS to a major new release unless concrete and important benefits exist.

Basically, never upgrade to a major new release of macOS unless there are compelling benefits that exceed the downsides (which are hard to know until the installed).

Don H writes:

One item you might want to add to your list of concerns when upgrading (and you’ve been bitten by this yourself) is the often one-way migration of Mail to the new OS. Dabbling in a full system upgrade is one thing; committing your mail store to its caprices is another

When I eventually make the jump myself (and I’m still running Mojave here) the first thing I do is import *some* of my mail to see how that works. In the meantime I keep the bulk of my messages on my old system where they are safe. Only after I’m sure the new mail program works to my satisfaction will I import those messages, in batches, to the new system.

(I have meticulously saved every mail message I have sent or received, with only a few losses over the years, since the late 80s. For me that is my greatest ‘presence’ on my computer, more than all the files I have generated. I know for you it would be all your photos, but compared to mail messages those are easier to migrate or recover if lost. Mail is a total dealbreaker for me if the new OS doesn’t handle them correctly.)

MPG: yep, while one might reinstall an older macOS if macOS Monterey displeases, there is no going back on local mail repositories—a longstanding Apple arrogance of irrevocably changing (“upgrading”) mail formats—no respect for user data.

Apple is slow to fix critical security bugs, leaving you vulnerable to disaster

Hot on the heels of a zero-click security bug that could compromise your Mac or iPad/iPhone, a security researcher state that three zero-day bugs reported to Apple remain unfixed. That’s just one researcher; the are probably dozens of severe vulnerabities out in the wild.

Apparently, total system compromise bugs are not a top priority for Apple.

Disclosure of three 0-day iOS vulnerabilities and critique of Apple Security Bounty program

 I want to share my frustrating experience participating in Apple Security Bounty program. I've reported four 0-day vulnerabilities this year between March 10 and May 4, as of now three of them are still present in the latest iOS version (15.0) and one was fixed in 14.7, but Apple decided to cover it up and not list it on the security content page. When I confronted them, they apologized, assured me it happened due to a processing issue and promised to list it on the security content page of the next update. There were three releases since then and they broke their promise each time.

Ten days ago I asked for an explanation and warned then that I would make my research public if I don't receive an explanation. My request was ignored so I'm doing what I said I would. My actions are in accordance with responsible disclosure guidelines (Google Project Zero discloses vulnerabilities in 90 days after reporting them to vendor, ZDI - in 120). I have waited much longer, up to half a year in one case. 

I'm not the first person that is unhappy with Apple Security Bounty program. Here are some other reports and opinions:

...

Here are links to GitHub repositories that contain PoC source code that I've sent to Apple. Each repository contains an app that gathers sensitive information and presents it in the UI...

WIND: Apple software quality is an oxymoron. Apple Core Rot is getting worse, but that’s just for starters.

When it comes to security, your entire life and life savings could be impacted. Think of having your Mac or iPad/iPHone compromised, which then results in all your financial accounts being drained. It can and will happen to some unlucky people.

So we have a company whose propaganda is all about security and privacy that now wants to insert spyware infrastructure into iOS and macOS, while being slow to act or not acting at all on critical security bugs.