IntegrityChecker Java (icj): How to Eliminate the WARN_IO_STALL Messages

2022-01-03 • SEND FEEDBACK |
Related: Apple, Apple Core Rot, data integrity, diglloydTools, hard drive, How-To, IntegrityChecker, Java, SSD

re: IntegrityChecker Java
re: data integrity

The macOS file system has a weird bug for some years now (one of many Apple Core Rot problems).

On slow drives (hard drives and similar), macOS barfs with a deep I/O queue as used by IntegrityChecker Java (many outstanding I/O requests, up to 96 at a time). This works great on SSDs to improve throughput, but something internal to macOS cannot handle it on slow drives such as hard drives.

Locking up the I/O queue

Symptoms: at regular intervals, macOS locks up all I/O activity for many seconds (10 to 30 seconds). This lockup is not java-specific since all other I/O requests in other applications also stall.

Since icj queues up many requests at a time, every one of those I/O request stalls, and you’ll see a slew of warnings, something like this:

WARN_IO_STALL: delay of 11634ms buf24-33554432 started 1641220236703 now 1641220248337. This delay is a behavioral problem with macOS.
WARN_IO_STALL: delay of 11834ms buf30-33554432 started 1641220236741 now 1641220248575. This delay is a behavioral problem with macOS.
WARN_IO_STALL: delay of 11925ms buf31-33554432 started 1641220236742 now 1641220248667. This delay is a behavioral problem with macOS.
WARN_IO_STALL: delay of 11883ms buf5-33554432 started 1641220236902 now 1641220248785. This delay is a behavioral problem with macOS.
WARN_IO_STALL: delay of 12383ms buf8-33554432 started 1641220237092 now 1641220249475. This delay is a behavioral problem with macOS.
...

Workaround #1 — tell icj to optimize for hard drives

When operating on slow drives (hard drives and similar), cut the threads way down by telling icj to optimize for slow drives using "--optimize HDD" , like this (for verify):

icj verify --optimize HDD

or alternately one can tailor threads and buffers explicitly:

icj verify --threads 2 --large-buffers 4 --small-buffers 4

Edit default settings

This option is not recommended because it affects operations on all drives, even SSDs, cutting performance drastically on fast SSDs. But it could be useful in some scenarios.

Edit the icj file using a plain-text editor and configure these three things to suit:

export NUM_THREADS=2
export LARGE_BUFFERS=4
export SMALL_BUFFERS=4