Performance Tip for diglloydTools IntegrityChecker, java version

2019-02-22 • SEND FEEDBACK |
Related: data integrity, diglloydTools, IntegrityChecker, Java, memory

Apple macOS keeps changing. When I last tuned diglloydTools IntegrityChecker java ("icj"), certain parameters provided the fastest performance. Those parameters as of 10.13.6 and 10.14 now result in a substantial performance loss.

The native version of IntegityChecker ("ic" of the GUI that drives it) are not affected.

Recommended JDK for use with diglloydTools IntegrityChecker is Java 11, or OpenJDK 11.0.2.

The download page is not yet updated, but it is simple to make the performance change;

1. Open /Applications/diglloydTools/icj in any plain-text editor (e.g., TextEdit).

2. Look for the line with PERF_PARAMS. It will look something like this; change IO_BUFFERS_PER_THREAD to 2 and change IO_BUFFER_SIZE to 1024, as shown:

export PERF_PARAMS="-DNUM_THREADS=0 -DIO_BUFFERS_PER_THREAD=2 -DIO_BUFFER_SIZE=1024 -DSMALL_FILE_SIZE_CUTOFF=1 $ENCODING_TEST $ENCODING"

These parameters can be experimented with, but as of February 2019, they provide the best performance no both single hard drives and fast SSDs.

UPDATE March 5: behavior is very strange. On a fast SSD, I’m able to nearly double the speed by using 16 I/O buffers per thread, as follows:

export PERF_PARAMS="-DNUM_THREADS=0 -DIO_BUFFERS_PER_THREAD=16 -DIO_BUFFER_SIZE=1024 -DSMALL_FILE_SIZE_CUTOFF=1 $ENCODING_TEST $ENCODING"

Make sure that the number of virtual CPU cores X the number of memory buffers X memory buffer size is not too large. But even 8 threads using 16 buffers 1MB each is only 128MB.

I have long experience in performance tuning, but I cannot make sense of the behavior. I suspect that macOS has some squirrely memory management issues.

Testing

Here is how to run a test to check performance (make a change and re-test) each time, where "folderToVerify" is a test folder:

sudo purge; icj verify folderToVerify