diglloydTools MemoryTester Finds Bad Memory in Server Today, Prompted by IntegrityChecker Flagging Corrupt Files

diglloydTools includes IntegrityChecker for data integrity (including a cross-platform Java version that works on Mac, Windows, Linux, etc), and MemoryTester and DiskTester.

I have new respect for my own testing tool! My diglloydTools MemoryTester has always been 'boring' with memory free issues, in that it is a good tool for proving real-world memory bandwidth. But today it confirmed bad memory modules were in fact the problem.

It was diglloydTools IntegrityChecker Java that first flagged the issue by finding corrupted files that had verified only an hour or two earlier, a point to be taken seriously: data corruption may be much more likely from bad memory than from bit rot.

Today’s afternoon in a data center fixing my web server turned out to be bad memory modules, which diglloydTools MemoryTester found instantly. Having run 24X7 for some years, they started failing a month or so ago, with sporadic warning signs including:

• macOS El Capitan sporadically crashing with a panic twice a week or so.
• Java Virtual Machine (JVM) randomly once a day, from an internal memory check.
• Source code versioning software 'git' failing 'git gc' and 'git fsck' with corruption/hashing errors. And the apparently corrupted index file from last week.
• Random reports of file corruption by IntegrityChecker Java (files randomly fail to validate, not consistent).

OWC memory modules
failed after ~9 years of use including ~4 years of 24X7 server use
OWC replaced the modules — lifetime warranty

Instant and correct diagnosis

But diglloydTools MemoryTester can be run at any time without special protocols. But all eight test threads found the bad memory problems within a few seconds.

diglloydWeb:WIND lloyd$ mt stress
MemoryTester 1.6.1 64-bit, diglloydTools 2.2.15, 2017-12-24 18:12
Copyright 2006-2017 DIGLLOYD INC. All Rights Reserved
Use of this software requires a license. See http://macperformanceguide.com/Software-License.html
OS X 10.11.6, 8 CPU cores @ 2300 MHz, 16384 MiB memory
Monday, October 11, 2021 at 2:26:38 PM Pacific Daylight Time
********************************** Stress Test *********************************
Test duration: 8h
CPU percent: 100% per cpu core
Num CPU threads: 8
Memory per thread (requested): 1.79 GiB
...
Speeds denote memory bandwidth: copy and compare speed (memcpy + 64-bit compare)
Allow several minutes for averages to stabilize, especially with large memory sizes.
Task EatCPU-5 exited with error: -1, msg ="BAD MEMORY, changed after copy!"
6s:  7160  1432 1432 1432 1432 1432    0    0    0 (best)
10s: 10492  1749 1749 1749 1749 1749    0 1749    0 (best)
Task EatCPU-3 exited with error: -1, msg ="BAD MEMORY, changed after copy!"
16s: 11586  2207 2207 1655 1655 2207    0 1655    0 (best)
20s: 12398  2214 2214 1771 1328 2214    0 2214  443 (best)
Task EatCPU-4 exited with error: -1, msg ="BAD MEMORY, changed after copy!"
Task EatCPU-2 exited with error: -1, msg ="BAD MEMORY, changed after copy!"
26s: 13320  2391 2391 2049 1025 2049    0 2391 1025 (best)
30s: 13338  2668 2371 1778  889 1778    0 2371 1482 (best)
36s: 13849  2721 2721 1484  742 1484    0 2721 1979 (best)
40s: 14033  2896 2896 1337  668 1337    0 2896 2005 (best)
Task EatCPU-6 exited with error: -1, msg ="BAD MEMORY, changed after copy!"
46s: 14344  3102 3102 1163  582 1163    0 2714 2520 (best)
50s: 14451  3390 3212 1071  535 1071    0 2498 2676 (best)
Task EatCPU-1 exited with error: -1, msg ="BAD MEMORY, changed after copy!"
56s: 14662  3825 3028  956  478  956    0 2231 3188 (best)
60s: 14582  4018 2827  893  446  893    0 2083 3422
Task EatCPU-7 exited with error: -1, msg ="BAD MEMORY, changed after copy!"
Task EatCPU-0 exited with error: -1, msg ="BAD MEMORY, changed after copy!"
66s: 14073  4060 2571  812  406  812    0 1894 3518

Verification of smooth operation after memory swapped-out

No problem with the swapped-out memory.

diglloydWeb:WIND lloyd$ mt stress
diglloydWeb:MPG lloyd$ mt stress -m 500M
MemoryTester 1.6.1 64-bit, diglloydTools 2.2.15, 2017-12-24 18:12
Copyright 2006-2017 DIGLLOYD INC. All Rights Reserved
Use of this software requires a license. See http://macperformanceguide.com/Software-License.html
OS X 10.11.6, 8 CPU cores @ 2300 MHz, 16384 MiB memory
Monday, October 11, 2021 at 3:18:44 PM Pacific Daylight Time
********************************** Stress Test *********************************
Test duration: 8h
CPU percent: 100% per cpu core
Num CPU threads: 8
Memory per thread (requested): 500 MiB
...
=====> Use ctrl-C to stop <=====
Speeds denote memory bandwidth: copy and compare speed (memcpy + 64-bit compare)
Allow several minutes for averages to stabilize, especially with large memory sizes.
6s:  9588   400  400  400  400  400 2797 2397 2397 (best)
10s:  9991   731  731  731  731  731 2437 1950 1950 (best)
16s:  9993   922  922  922  922  922 1999 1691 1691 (best)
20s:  9994   987  987  987  987  987 1851 1604 1604 (best)
26s: 10276  1047 1047 1047 1047 1047 1808 1618 1618 (best)
30s: 10239  1073 1073 1073 1073 1073 1734 1569 1569
36s: 10541  1171 1171 1171 1171 1171 1654 1516 1516 (best)
40s: 10487  1179 1179 1179 1179 1179 1613 1489 1489
46s: 10531  1188 1188 1188 1188 1188 1620 1458 1512
50s: 10539  1193 1193 1193 1193 1193 1591 1491 1491
56s: 10481  1199 1199 1199 1199 1199 1554 1466 1466
60s: 10614  1202 1244 1244 1244 1244 1534 1451 1451 (best)
66s: 10671  1244 1244 1244 1244 1244 1508 1471 1471 (best)
70s: 10632  1245 1245 1245 1245 1245 1494 1458 1458
76s: 10746  1278 1278 1278 1278 1245 1507 1442 1442 (best)
80s: 10740  1276 1276 1276 1276 1276 1494 1432 1432
86s: 10745  1274 1274 1274 1274 1274 1477 1448 1448
90s: 10851  1301 1273 1301 1301 1301 1495 1439 1439 (best)
96s: 10849  1298 1298 1298 1298 1298 1479 1454 1428
100s: 10815  1296 1296 1296 1296 1296 1470 1445 1420
106s: 10839  1317 1293 1293 1293 1293 1481 1434 1434
110s: 10854  1314 1292 1314 1314 1292 1473 1428 1428 (best)
116s: 10853  1311 1289 1311 1311 1311 1461 1440 1418
120s: 10844  1309 1309 1309 1309 1309 1454 1433 1413
126s: 10804  1306 1306 1306 1306 1306 1445 1425 1405
130s: 10779  1304 1304 1304 1304 1304 1439 1419 1400
136s: 10781  1302 1302 1320 1302 1302 1430 1412 1412
140s: 10794  1318 1300 1318 1318 1300 1425 1407 1407
146s: 10795  1315 1298 1315 1315 1315 1435 1401 1401
150s: 10774  1314 1297 1314 1314 1314 1430 1397 1397
...