SEARCH
diglloydTools™
Time Machine does not show all the items that are excluded!
Get fast high-capacity and high quality backup drives at OWC.
Update: mystery solved; see end of post. To make a long story short (and wasted afternoon), it turns out that Time Machine => Preferences => Privacy will not show excluded files or folders that are "." files or otherwise hidden files, such as git repositories.
....
Gah....! Skip to the explanation at the end. If Time Machine simply showed what is excluded, all of this could have been avoided. As it stands, there is no trivial way to see what is or is not excluded.
What’s worse than having no backup? Thinking you have a backup, but Apple Core Rot screws the pooch.
Just yesterday a reader wrote MPG to say that Apple Time Machine was not backing up his stuff automatically, at all. Nothing was excluded and Time Machine was enabled. This is a user who knows what he is doing. Update: this user reports that OS X 10.11.2 fixes the issue—it’s shows a dismal state of affairs that such bugs would occur in critical areas like this—Apple has had many years to have proper automated testing protocols in place. It speaks to slipshod software development, and it shows in every OS X releaes.
Today, with freshly-erased volumes (!) for Time Machine, it’s clear that something doesn’t add up. It’s scary: is the Finder wrong or is Time Machine just badly broken*? These are fresh Time Machine backups on freshly-erased backup volumes.
How does 328GB on Master PLUS 73GB on Boot = 401GB get backed up as 283GB?
resulting backup = 283GB
The last time I checked, 328 + 73 = 401GB. But there is only 286GB in the backup. That’s scary. I’ve triple checked the Time Machine exclusion list (well, OK, 5 or 6 times). Then I deleted it all, and re-excluded—no dice.
Time Machine is missing 114G or so for the volume Master (329GB on Master, but only 215GB of backup). Picking does not improve matters.
The machine was rebooted. No dice. The backup volume was erased and tried again from scratch—same problem.
Presumably Machine runs into some kind of bug and fails to backup about 118GB of data. If that’s the case, it also fails to report the issue.
MPG has long advised clone backups as the primary backup, with Time Machine a good adjunct (and IntegrityChecker strongly advised too). Now it looks like Time Machine is one more victim of Apple Core Rot on top of the Time Machine crashes when restoring files.
This is not the first time Time Machine has had problems. See Time Machine BUG: Silent Exclusion of Volumes.
For a look at what a competent and thoughtful developer does to approach the question of data integrity in the face of a barrage of destabilizing arbitrary Apple changes, read Coping with Apple's pace of innovation in an application that can delete files. Apple could use a few thoughtful adults to supervise the juvenile approach to OS X development.
* Disk Utility agrees with the Finder that there is 283GB on both tmSSD and mtm volumes.
Bug found: Time Machine does not show all excluded items
It turns out that three git repositories were excluded. Presumably I did this years ago, but had completely forgotten about it.
The issue is that excluded folders totaling 100+GB are simply not listed in the Time Machine exclusion list. They were/are not listed, because Time Machine will not show hidden files or folders in its exclusion list.
Had they been listed, it would have saved me hours of tracking down the problem and writing this blog post! Oh well.
The only way to find out if a hidden file or folder is excluded is at the command line:
diglloydMP:DIGLLOYD lloyd$ tmutil isexcluded .git
[Excluded] /Volumes/Master/diglloyd/DOMAINS/DIGLLOYD/.git
Worse, tmutil has no recursive or summary way to see what is or is not excluded. There are ways to do it using 'ls' and a script, invoking tmutil on every hidden file or folder, but what a hassle.
MPG reaffirms the use of clone backups with a program like Carbon Copy Cloner for a snapshot of everything. Simple, fast, directly accessible, bootable, hidden files visible.
