Newly In Stock: SEE ALL...
Sony Mirrorless wish list • Deals on Sony
$2997 Nikon Z7 II Mirrorless Camera OUT OF STOCK in Cameras: Mirrorless	$2297 Nikon D780 DSLR OUT OF STOCK in Cameras: DSLR	$2997 Nikon D850 DSLR OUT OF STOCK in Cameras: DSLR	$720 ZEISS 32mm f/1.8 Touit Lens for FUJIFILM X OUT OF STOCK in Lenses: Mirrorless	$497 Pentax 70mm f/2.4 HD Pentax DA Limited IN STOCK in Lenses: DSLR

Highly Recommended!
External SSD wish list • Deals on OWC
$220 SAVE $130 = 37.0% Western Digital 16.0TB Western Digital Ultrastar DC HC550 3.5-in… in Storage: Hard Drives	$500 SAVE $75 = 13.0% OWC 1.0TB OWC Atlas Pro SDXC V60 UHS-II Memory Card (2-Pack) in All Other Categories	$680 OWC 2.0TB OWC Atlas Ultra CFexpress 4.0 Type B Memory Card in All Other Categories	$580 OWC 4.0TB OWC Envoy Pro Elektron USB-C Portable NVMe SSD in All Other Categories	$630 OWC 4.0TB OWC Express 1M2 USB4 (40Gb/s) Bus-Powered Portable NVM… in All Other Categories	$2380 OWC 72.0TB OWC ThunderBay 4 Four-Drive Thunderbolt External Stor… in All Other Categories

diglloydTools IntegrityChecker Java Version: Finds Duplicates, Saved me 100GB!

2016-08-13 • SEND FEEDBACK |

Related: data integrity, diglloydTools, IntegrityChecker, Java

diglloydTools

A cool new feature of diglloydTools IntegrityChecker 'icj' (Java version) is finding duplicate files; it saved me a whopping 100GB! Over the past few years, I had inadvertently made copies when intending to move files due to a silent Finder bug: if files are marked as locked, the Finder makes copies—yikes!

By using the SHA-1 hashes, exact matches can be found unerringly and all but instantly (once the hashes are done via the 'update' command).

This example is after the 100GB duplications were found and deleted, so there is only 2.4 GiB left of duplicates.

Note: this listing is not displayed fully or properly due to bugs in web browsers.

# find all duplicates on volume Archive
diglloydMP:DIGLLOYD lloyd$ icj dupes Archive  
# icj version 1.0 beta 5 @ 2016-08-13 17:38
# Copyright 2016 DIGLLOYD INC. All Rights Reserved
# Use of this software requires a license. https://macperformanceguide.com/Software-License.html
# Sat Aug 13 20:33:49 PDT 2016
[1]={/Volumes/Archive}

Finding folders...4240.6920.9043.12381.14350.16676.16676
12003 ms to find 16676 folders

Loading hash data concurrently for 16676 folders...
0.100.200.300.400.500.600.700.800.900.1000.1100.1200....16500.16600....waiting for I/O to finish...15652.16676
Loaded hash data for 16676 folders in 23822 ms

995 ms to find duplicates

1 duplicate of "/Volumes/Archive/WebSiteOriginals/Infrared-S3-D70-5D/2006-0829-IR-camera-compare-all-orig/S3/091/DSCF0472.RAF" waste = 24 MiB
    /Volumes/Archive/WebSiteOriginals/Infrared-S3-D70-5D/2006-0829-IR-camera-compare/S3/091/DSCF0472.RAF
1 duplicate of "/Volumes/Archive/ArticlesAndReviews/D2XColor/materials/2005-0423 ManyTests/1/color-balance-correction/LCDPhotos/IMG_0044.JPG" waste = 1644 KiB
    /Volumes/Archive/ArticlesAndReviews/D2XColor/LCDPhotos/IMG_0044.JPG

... lines omitted for brevity ...

4 duplicates of "/Volumes/Archive/ArticlesAndReviews/D2X-vs-1DsMII/publish/logos/diglloyd36.gif" waste = 2644 bytes
    /Volumes/Archive/ArticlesAndReviews/ShiftLensStitching/publish/logos/diglloyd36.gif
    /Volumes/Archive/ArticlesAndReviews/D200-vs-D2X/publish/logos/diglloyd36.gif
    /Volumes/Archive/ArticlesAndReviews/TheSharpestImage/original/publish/logos/diglloyd36.gif
    /Volumes/Archive/ArticlesAndReviews/CoastalOptics60f4/CoastalOptics_60f4/publish/logos/diglloyd36.gif
1 duplicate of "/Volumes/Archive/WebSiteOriginals/Infrared-S3-D70-5D/2006-0829-IR-camera-compare/D70/no-filter/_DSC5742.xmp" waste = 4380 bytes
    /Volumes/Archive/WebSiteOriginals/Infrared-S3-D70-5D/2006-0829-IR-camera-compare-all-orig/D70/no-filter/_DSC5742.xmp
....

WASTED SPACE from duplicate files: 2388 MiB