Advanced impact protection against drops and impact!
Optimizing Adobe Lightroom
This article was written for Lightroom 2.x. Most of the principles still apply, the main difference being the emergence of much faster quad-core laptops.
See also Optimizing Photoshop— the same hardware upgrades that speed up Photoshop also apply to Lightroom.
Adobe’s Lightroom 2.3 has garnered favor among photographers for good reason— an interface that makes sense without some of the headaches found with Apple’s iPhoto and Aperture.
The Lightroom interface is nicely done and quite functional, but performance is an area that needs some work, in spite of Lightroom being a 64-bit program. While a 64-bit program can use all the available memory, that’s of only modest benefit when the program is not engineered to make full use of available CPU cores (and memory and disk speed).
Let’s explore performance considerations with Adobe Lightroom 2.3, starting by looking at how effectively Lightroom uses multiple CPU cores. You can also skip ahead for a neat performance trick— click on the graph for more. Performance in Lightroom 4.x is slightly better, but the same under-utilization of CPU power remains.
CPU utilization PERMALINK
A cataloguing and RAW-file processing program like Lightroom ought to make full and efficient use of all available CPU cores.
Full use means that all the cores would be put to work, and efficient use (scalability) means that twice as many cores should run almost twice as fast: 16 cores should run 7X - 8X as fast as 2 cores (there is some overhead to more cores, so even 6X would be satisfactory).
Basic tasks like importing and exporting images are key to an efficient photographic workflow, so at the very least, those tasks, being bottlenecks, ought to run fast. Unfortunately, this is not the case with Lightroom 2.3.
These tests involve 128 Canon EOS 1Ds Mark III RAW files, averaging 24.7MB each. That’s not a particularly big job, and should give pause to anyone considering Lightroom for shoots involving hundreds of files at a time, even on a top-end Mac Pro.
While a Mac Pro tests substantially faster than a MacBook Pro, Lightroom makes very modest use of the awesome processing power of the 2.93GHz Mac Pro Nehalem, wasting 70 - 90% of the available computing power, on average, for these tests.
|Test||Mac Pro Nehalem 2.93GHz dual-cpu, March 2009 model 32GB memory
||MacBook Pro 2.93GHz, March 2009 model 8GB memory||Comments|
|Import* 128 CR2 files, 1:1 previews||603||932||Poor CPU utilization; the dual-core MacBook Pro takes only 50% longer than the 16-virtual-core Mac Pro.
Even the MacBook Pro has about 1/3 of the CPU cycles unused.
These times translate to 1.3 hours to import 1,000 files. on the fastest Mac Pro available as of May, 2009. That’s a problem for photographers on a deadline, or if you just want to view your images before bed after a long day in the field. Had all cores been used efficiently, we could expect processing time of about two minutes.
|Export 128 max-quality JPEGs||351||810||Still very poor CPU utilization, though the MacBook Pro does take 2.3X as long, indicating some increased use of the CPU cores on the Mac Pro.
Had all cores been used efficiently, we could expect processing time under two minutes.
* Importing referenced existing files on disk eg “Add photos to catalog without moving”, Initial Previews 1:1
Scalability — not there
Not only do most of the CPU cores stay idle, the scalability (efficiency) for the cores that are actually used is low. While it does appear that Lightroom allocates “threads” for all the cores, there exists some internal bottleneck (bug) which prevents them from working efficiently.
The performance is very similar to the poor scalability seen with Photoshop; apparently the same or similar approach is employed in Lightroom as in Photoshop CS4. Working efficiently with 2 cores is easy; doing so with 4/8/16 cores requires some engineering which apparently has been omitted. Even the utilization of two cores on the MacBook Pro disappoints for these tests, with its two cores going unused about 30% of the time.
In this test, the 16-virtual-core 2.93GHz Mac Pro should turn in astonishingly fast processing times compared to the 2-core MacBook Pro 2.93Ghz 17". But the 16-virtual-core Mac Pro offers only 1.5X - 2.3X speedup over the dual-core MacBook Pro. Considering the advanced caching and much higher memory bandwidth of the Mac Pro Nehalem, this is disappointing indeed.
Viewing CPU usage in Activity Monitor How, observe the black areas, which represent idle CPU cores. It should be obvious that the majority of the processing power is wasted—about 80%.
For the tests, both the Mac Pro and the MacBook Pro used the same drive, the ultra-fast Intel X25-E solid state drive (SSD), which reads at 260MB/sec and writes at 170MB/sec (as actually tested). See the reviews of its sibling, the X25-M.
The disk I/O was monitored during the test, and it was observed that Lightroom demands on disk speed for this test were very low, on the order of only 10MB/sec. Furthermore, a 4-way striped RAID was also tested, no difference.
So it’s not a disk speed issue for importing files in Lightroom, at least not when referencing the original files on disk. The story is similar for exporting JPEGs.
However, if you’re exporting huge 16-bit TIF files, drive speed does matter. And it will likely matter for any type of really large file (importing or exporting).
Performance tips (Export) PERMALINK
The core idea is that Lightroom is not smart enough to run efficiently on its own, so you have to load it up with more than one job to force more of the available CPU cores to be used. Lightroom should do this automatically! It’s not just Adobe though, the problem is rampant in the industry. See the report card.
As a software engineer with decades of experience, obvious problems like this still astound me— the ideas here should be obvious to any competent engineer. All that’s needed is the simple act of watching Activity Monitor and seeing most of the CPU cores, memory, and disk at low utilization.
Speeding up Export
Theoperation can be sped up substantially using multiple jobs; Lightroom calls these “operations”.
According to reader John L, this tip also works on Microsoft Windows, even running inside virtualization software (such as Parallels) on a Mac.
How to use two (or more) jobs
The basic approach follows that discussed in Optimizing DPP: manually create more than one processing task (job) to cajole the program into using more of the available CPU cores.
1. Select half the images as shown below, then start the.
2. Select the remaining files (immediately) and repeat the.
You can extend this idea to groups of 3 or 4 or more, but Lightroom scales so poorly that splitting things into more than three JPEG jobs is self-defeating. It all depends on the task; 12 jobs can work well for exporting compressed TIF (on the dual-CPU Mac Pro Nehalem).
On dual-core machines, two jobs is usually the maximum; in general do not start more jobs than there are CPU cores, because this will usually slow things down. When in doubt test it.
Performance tip — exporting 16-bit TIF PERMALINK
See the general tips above.
Exporting compressed TIF is much slower than uncompressed TIF— up to 10X slower. Even exporting JPEG files is much slower than uncompressed TIF. For the fastest possiblespeed, stick with uncompressed TIF.
Compressed TIF is single-threaded in Lightroom (as with Photoshop CS4), so avoid it unless you have lots of patience, or be sure to use as many jobs as there are CPU cores. I found that 8 simultaneous jobs (operations) used 8-10 CPU cores fully on my 16-virtual-core Mac Pro Nehalem for compressed TIF processing. Probably 12-14 jobs would max-out the CPUs, but it’s tedious and error prone to carve up the into that many sub-tasks.
Drive speed matters trivially when exporting JPEG files, even max-quality ones, because they are relatively small. But when exporting 16-bit uncompressed TIF files, a 21-megapixel, 25-megabyte RAW image balloons to a 120MB TIF, and disk speed suddenly becomes a factor, with files 10X larger than equivalent JPEGs. Exporting 8-bit TIF cuts the disk speed demands in half, but it will still be worth paying attention.
I tested exporting 128 16-bit TIF files from Lightroom, and found that using a 4-drive striped RAID improved performance dramatically. On the 4-drive striped RAID, files were processed in 1.95 seconds each, writing to a single drive capable of (only) 75MB/sec slowed the process to 3.39 seconds each, taking 75% longer!
Drive speed becomes even more important when more than one worker (job) is used, since that increases the demands on disk speed substantially, up to about 90MB/sec with 3 jobs running at once in this example; a volume capable of 80MB/sec will take 1.5 seconds to write the TIF, a 4-drive stripe will cut that time down to about 0.4 seconds, saving 1.1 seconds per file, a big deal if the files are taking only ~1.3 seconds each as they are here (128 files in 170 seconds with 3 jobs).
Performance comparison — exporting JPEGs PERMALINK
See the how-to steps above on how to start multiple jobs (operations) for .
Improvements using multiple jobs were seen with both the Mac Pro and MacBook Pro, with the Mac Pro showing a 33% reduction in processing time (50% longer with one job), and the MacBook Pro showing a 20% reduction in processing time (25% longer with one job). In short, it’s worth the effort for more than 20 files or so.
Note that using two jobs for importing slowed things down in my testing, indicating internal design problems in Lightroom. Even for exporting, the improvement is much less than it ought to be; we should expect double the speed on the 16-virtual-core Mac Pro, but we see only a 33% reduction in processing time, indicating internal program bottlenecks, since most of the CPU power goes unused.
The 2.93GHz MacBook Pro with 8GB OWC memory (below) does a credible job, and shows a nice improvement with two jobs, but still takes 2.3X to 3.4X longer, even though most of the Mac Pro horsepower is wasted. Two cores is inadequate for many tasks in many programs, the Mac Pro is the only way to go for a serious workstation. Unless your time is of little value, it’s worth paying for the Mac Pro, which isn’t a lot more expensive in terms of total system cost.
I ran into several bugs while working with Lightroom 2.3.
Memory usage bug PERMALINK
This bug is no longer present in Lightroom 3.x and 4.x.
While importing the 128 RAW file on the Mac Pro (32GB memory), memory usage (real memory) half-way through the import soared to an astounding 6GB , then stabilized and stayed there (Lightroom is a 64-bit program so it can access as much memory is available). I was able to repeat this several times, quitting and re-launching each time.
On the Macbook Pro (8GB memory), memory usage stabilized at about 1GB using the same files, same settings, and starting with an empty catalog on both machines.
Of course, if you have a 4GB machine and Lightroom wants 6GB, you can count on extremely slow performance as memory is paged to and from the disk—good luck.
This bug is no longer present in Lightroom 3.x and 4.x.
The weirdest bug was grabbing an image to drag it (scroll it), and having the image go very dark and red, the more movement the worse it gets, and with artifacts appearing at the edges. It happened with every image I tried, but went away when I quit and re-launched Lightroom.
I found the Adobe Lightroom interface well-designed and easy to use for what I wanted to do. It has a strong feature set, and has become the de-factor standard.
Even though Lightroom under utilizes much of the Mac Pro’s processing power, a Mac Pro remains the best choice for CPU power, storage, backup, etc.
Contact me for personalized consulting on your Mac setup, or optimizing your workflow.