diglloydTools MemoryTester Finds Bad Memory in Server Today, Prompted by IntegrityChecker Flagging Corrupt Files
diglloydTools includes IntegrityChecker for data integrity (including a cross-platform Java version that works on Mac, Windows, Linux, etc), and MemoryTester and DiskTester.
I have new respect for my own testing tool! My diglloydTools MemoryTester has always been 'boring' with memory free issues, in that it is a good tool for proving real-world memory bandwidth. But today it confirmed bad memory modules were in fact the problem.
It was diglloydTools IntegrityChecker Java that first flagged the issue by finding corrupted files that had verified only an hour or two earlier, a point to be taken seriously: data corruption may be much more likely from bad memory than from bit rot.
Today’s afternoon in a data center fixing my web server turned out to be bad memory modules, which diglloydTools MemoryTester found instantly. Having run 24X7 for some years, they started failing a month or so ago, with sporadic warning signs including:
- macOS El Capitan sporadically crashing with a panic twice a week or so.
- Java Virtual Machine (JVM) randomly once a day, from an internal memory check.
- Source code versioning software 'git' failing 'git gc' and 'git fsck' with corruption/hashing errors. And the apparently corrupted index file from last week.
- Random reports of file corruption by IntegrityChecker Java (files randomly fail to validate, not consistent).

failed after ~9 years of use including ~4 years of 24X7 server use
OWC replaced the modules — lifetime warranty
Instant and correct diagnosis
But diglloydTools MemoryTester can be run at any time without special protocols. But all eight test threads found the bad memory problems within a few seconds.
diglloydWeb:WIND lloyd$ mt stress MemoryTester 1.6.1 64-bit, diglloydTools 2.2.15, 2017-12-24 18:12 Copyright 2006-2017 DIGLLOYD INC. All Rights Reserved Use of this software requires a license. See http://macperformanceguide.com/Software-License.html OS X 10.11.6, 8 CPU cores @ 2300 MHz, 16384 MiB memory Monday, October 11, 2021 at 2:26:38 PM Pacific Daylight Time ********************************** Stress Test ********************************* Test duration: 8h CPU percent: 100% per cpu core Num CPU threads: 8 Memory per thread (requested): 1.79 GiB ... Speeds denote memory bandwidth: copy and compare speed (memcpy + 64-bit compare) Allow several minutes for averages to stabilize, especially with large memory sizes. Task EatCPU-5 exited with error: -1, msg ="BAD MEMORY, changed after copy!" 6s: 7160 1432 1432 1432 1432 1432 0 0 0 (best) 10s: 10492 1749 1749 1749 1749 1749 0 1749 0 (best) Task EatCPU-3 exited with error: -1, msg ="BAD MEMORY, changed after copy!" 16s: 11586 2207 2207 1655 1655 2207 0 1655 0 (best) 20s: 12398 2214 2214 1771 1328 2214 0 2214 443 (best) Task EatCPU-4 exited with error: -1, msg ="BAD MEMORY, changed after copy!" Task EatCPU-2 exited with error: -1, msg ="BAD MEMORY, changed after copy!" 26s: 13320 2391 2391 2049 1025 2049 0 2391 1025 (best) 30s: 13338 2668 2371 1778 889 1778 0 2371 1482 (best) 36s: 13849 2721 2721 1484 742 1484 0 2721 1979 (best) 40s: 14033 2896 2896 1337 668 1337 0 2896 2005 (best) Task EatCPU-6 exited with error: -1, msg ="BAD MEMORY, changed after copy!" 46s: 14344 3102 3102 1163 582 1163 0 2714 2520 (best) 50s: 14451 3390 3212 1071 535 1071 0 2498 2676 (best) Task EatCPU-1 exited with error: -1, msg ="BAD MEMORY, changed after copy!" 56s: 14662 3825 3028 956 478 956 0 2231 3188 (best) 60s: 14582 4018 2827 893 446 893 0 2083 3422 Task EatCPU-7 exited with error: -1, msg ="BAD MEMORY, changed after copy!" Task EatCPU-0 exited with error: -1, msg ="BAD MEMORY, changed after copy!" 66s: 14073 4060 2571 812 406 812 0 1894 3518
Verification of smooth operation after memory swapped-out
No problem with the swapped-out memory.
diglloydWeb:WIND lloyd$ mt stress
diglloydWeb:MPG lloyd$ mt stress -m 500M
MemoryTester 1.6.1 64-bit, diglloydTools 2.2.15, 2017-12-24 18:12
Copyright 2006-2017 DIGLLOYD INC. All Rights Reserved
Use of this software requires a license. See http://macperformanceguide.com/Software-License.html
OS X 10.11.6, 8 CPU cores @ 2300 MHz, 16384 MiB memory
Monday, October 11, 2021 at 3:18:44 PM Pacific Daylight Time
********************************** Stress Test *********************************
Test duration: 8h
CPU percent: 100% per cpu core
Num CPU threads: 8
Memory per thread (requested): 500 MiB
...
=====> Use ctrl-C to stop <=====
Speeds denote memory bandwidth: copy and compare speed (memcpy + 64-bit compare)
Allow several minutes for averages to stabilize, especially with large memory sizes.
6s: 9588 400 400 400 400 400 2797 2397 2397 (best)
10s: 9991 731 731 731 731 731 2437 1950 1950 (best)
16s: 9993 922 922 922 922 922 1999 1691 1691 (best)
20s: 9994 987 987 987 987 987 1851 1604 1604 (best)
26s: 10276 1047 1047 1047 1047 1047 1808 1618 1618 (best)
30s: 10239 1073 1073 1073 1073 1073 1734 1569 1569
36s: 10541 1171 1171 1171 1171 1171 1654 1516 1516 (best)
40s: 10487 1179 1179 1179 1179 1179 1613 1489 1489
46s: 10531 1188 1188 1188 1188 1188 1620 1458 1512
50s: 10539 1193 1193 1193 1193 1193 1591 1491 1491
56s: 10481 1199 1199 1199 1199 1199 1554 1466 1466
60s: 10614 1202 1244 1244 1244 1244 1534 1451 1451 (best)
66s: 10671 1244 1244 1244 1244 1244 1508 1471 1471 (best)
70s: 10632 1245 1245 1245 1245 1245 1494 1458 1458
76s: 10746 1278 1278 1278 1278 1245 1507 1442 1442 (best)
80s: 10740 1276 1276 1276 1276 1276 1494 1432 1432
86s: 10745 1274 1274 1274 1274 1274 1477 1448 1448
90s: 10851 1301 1273 1301 1301 1301 1495 1439 1439 (best)
96s: 10849 1298 1298 1298 1298 1298 1479 1454 1428
100s: 10815 1296 1296 1296 1296 1296 1470 1445 1420
106s: 10839 1317 1293 1293 1293 1293 1481 1434 1434
110s: 10854 1314 1292 1314 1314 1292 1473 1428 1428 (best)
116s: 10853 1311 1289 1311 1311 1311 1461 1440 1418
120s: 10844 1309 1309 1309 1309 1309 1454 1433 1413
126s: 10804 1306 1306 1306 1306 1306 1445 1425 1405
130s: 10779 1304 1304 1304 1304 1304 1439 1419 1400
136s: 10781 1302 1302 1320 1302 1302 1430 1412 1412
140s: 10794 1318 1300 1318 1318 1300 1425 1407 1407
146s: 10795 1315 1298 1315 1315 1315 1435 1401 1401
150s: 10774 1314 1297 1314 1314 1314 1430 1397 1397
...