Thank you for purchasing through links and ads on this site.
OWC / MacSales.com...
diglloyd Deal Finder...
Buy other stuff at Amazon.com...
Speed To Create, Capacity To Dream
Storage Wishlist…

Thunderbolt 10-Gigabit Networking: Real World Performance

Last updated 2015-11-16 - Send Feedback
Related: How To, Networking, Thunderbolt, Optimization

OWC offers high quality Thunderbolt cables in red/green/blue/black colors from 0.5 to 3.0 meters, as well as Thunderbolt optical cables from 10.0 to 30.0 meters in length.

UPDATE: Thunderbolt networking works at high speeds for a short duration only. After than, its speed drops to sub-gigabit speeds; see the sustained heavy load tests that follow.

The following pages discuss how to set up 10 Gigabit Thunderbolt networking between Macs. But how does it actually perform for real-world usage?

 MB = 1000 * 1000,  GB = 1000^3
MiB = 1024 * 1024, GiB = 1024^3

Test details

The file server was an 8-core 2013 Mac Pro with files on a striped pair of OWC Mercury Accelsior PCIe SSD cards in an OWC Helios enclosure, delivering about 1100MB/sec read performance, important since Thunderbolt networking in theory offers speeds in that range. An even faster SSD would have been preferable, since the SSD itself speed could have some influence (data has to be read from disk).

The client was a late 2013 MacBook Retina. The Thunderbolt cable was plugged into an OWC Thunderbolt 2 Dock. Though the Thunderbolt cable could have been plugged in directly to the MBP, this is a more realistic setup (ports are in short supply on the MBP).

The cable was a 3 meter OWC Thunderbolt 2 cable, color red (helpful in distinguishing cables when there are 6 of them coming out of a Mac Pro!).

SSD Upgrade for MacBook Pro Retina
Internal SSD Wishlist…

Thunderbolt 10 Gbps network performance tests

Photoshop open PSB over Thunderbolt networking

Opening a multi-gigabyte PSB file is about as fast as opening it directly on the Mac Pro on the local SSD volume—outstanding!

21.48 GB test folder

Finder copy speed over Thunderbolt networking

To test performance, a 21.48 GB / 20.0 GiB folder of multi-gigabyte files was used to test transfer speed.

Copy 21.48GB / 20.0 GiB: 32 seconds = 671 MB/sec / 625 MiB/sec

Command line copy over Thunderbolt networking

Unix 'cp' takes nearly 50% longer than a Finder copy. Probably the Finder is smarter about interleaving reads and writes (a copy also has to write to the local SSD). Speed was even worse at times (up to 51 seconds), so Unix 'cp' is not a good way to enjoy full speed.

time cp -r /Volumes/Scratch/TEMP ~/Desktop/
real	0m45.509s
user	0m0.019s
sys	0m18.455s
= 472 MB/sec / 494 MiB/sec
Deals Updated Daily at B&H Photo
View BEST Deals Right Now

DiskTester read-files over Thunderbolt networking

The DiskTester read-files command reads files (no writing) so it takes writing out of the process—thus a true measure of read performance.

DiskTester read-files is far more efficient than either 'cp' or Finder copy, delivering a whopping 1054 MB/sec / 1006 MiB/sec.

10 Gigabit Thunderbolt Bridge networking vs Gigabit Ethernet
llcT:DIGLLOYD lloyd$ disktester read-files --xfer 4M -b double /Volumes/Scratch/TEMP
DiskTester 2.2.11 64-bit, diglloydTools 2.2.11, 2015-02-28 15:08
Copyright 2006-2014 DIGLLOYD INC. All Rights Reserved
Use of this software requires a license. See http://macperformanceguide.com/Software-License.html
OS X 10.11.1, 8 CPU cores, 16384MB memory
Monday, November 16, 2015 at 13:59:01  Pacific Standard Time
disktester read-files --xfer 4M -b double /Volumes/Scratch/TEMP
read-files params:
Transfer size:        2048KB per buffer
Iterations:           1
Async double buffers: true
Scanning "/Volumes/Scratch/TEMP"...62
62 files in 4 folders.
---------------------------------- Iteration 1 ---------------------------------
Reading 62 files totaling 20.0 GiB...
IO method: dual asynchronous read buffers each of size 2 MiB
# Files   Amt Read  MiB/sec(all)    MiB/sec(1)   MiB/sec(10)   MiB/sec(30)
9     5.02GB     1051           1030         1031         1031    
20    10.13GB     1048           1049         1046         1039    
38     14.8GB     1058           1050          959         1023    
56     19.6GB     1010           1054          966          959    
62     20.0GB     1006           20.3          930          978    
Data forks read: 62
Resource forks ignored.
Total files read: 62
done.
Read 62 files totaling 20.0 GiB in 20.36 seconds @ 1006 MiB/sec
Command "read-files" executed in 20.54 seconds on Monday, November 16, 2015 at 13:59:22  Pacific Standard Time

IntegrityChecker verify over Thunderbolt networking

The IntegrityChecker verify command reads files and computes SHA1 hashes. It is thus a real-world computational task (not just a file copy).

A credit it to its highly optimized architecture, IntegrityChecker can read and verify files faster than the Finder can copy them, delivering a whopping 868 MB/sec / 808 MiB/sec. This outstanding performance might come in handy for users with very large networked data sets.

llcT:DIGLLOYD lloyd$ ic verify -b 32M /Volumes/Scratch/TEMP
IntegrityChecker(tm) v1.2.3 64-bit, diglloydTools 2.2.11, 2015-02-28 15:08
Copyright 2006-2014 DIGLLOYD INC. All Rights Reserved
Use of this software requires a license. See http://macperformanceguide.com/Software-License.html
OS X 10.11.1, 8 CPU cores, 16384MB memory
Monday, November 16, 2015 at 13:45:55  Pacific Standard Time
ic verify -b 32M /Volumes/Scratch/TEMP
Using threads = 4, read buffer size = 32768K, num buffers = 24
Looking for files in "/Volumes/Scratch/TEMP"...
Seconds to scan files: 0.00
62 files found in 0.00 seconds.
Scanning 3 folders for .ic files...3 hash-data files read in 0.00 seconds.
Selecting files for hashing...
Preparing to hash 56 files...
Files prepared, hashing 56 files...
0%:      0 files @ 0.0MB/sec, processed 0B
0%:      0 files @ 0.0MB/sec, processed 0B
14%:      2 files @ 883.5MB/sec, processed 2.87GB
19%:      4 files @ 863.6MB/sec, processed 3.78GB
25%:      7 files @ 875.3MB/sec, processed 4.92GB
31%:     11 files @ 902.9MB/sec, processed 6.27GB
37%:     14 files @ 905.5MB/sec, processed 7.41GB
41%:     16 files @ 823.7MB/sec, processed 8.16GB
50%:     18 files @ 902.8MB/sec, processed 10.05GB
54%:     20 files @ 877.9MB/sec, processed 10.81GB
59%:     24 files @ 873.6MB/sec, processed 11.8GB
63%:     28 files @ 868.7MB/sec, processed 12.7GB
67%:     31 files @ 857.0MB/sec, processed 13.4GB
72%:     35 files @ 857.8MB/sec, processed 14.3GB
72%:     35 files @ 810.3MB/sec, processed 14.3GB
80%:     40 files @ 848.4MB/sec, processed 16.0GB
85%:     44 files @ 844.6MB/sec, processed 16.9GB
89%:     47 files @ 835.8MB/sec, processed 17.7GB
95%:     51 files @ 843.0MB/sec, processed 19.1GB
Finished reading 56 files of 56
100%:     55 files @ 840.8MB/sec, processed 20.0GB
100%:     56 files @ 807.6MB/sec, processed 20.0GB
Processed 20.0 GiB in 25.4 seconds @ 807.6 MiB/sec
========================================================================================================================
ic verify -b 32M /Volumes/Scratch/TEMP
Monday, November 16, 2015 at 13:46:20  Pacific Standard Time
========================================================================================================================
# Files with stored hash: 56
# Files missing: 0
# Files hashed: 56
# Files without hashes: 0
# Files whose size has changed: 0
# Files whose date changed: 0
# Files whose content changed (same size): 0
# Suspicious files: 0

CPU usage

As with ethernet (which sucks up an entire CPU under full load), Thunderbolt networking consumes some CPU time. At full bandwidth, nearly a full CPU core is utilized (“kernel_task”).

21.48 GB test folder

Thunderbolt 10 Gbps network performance tests — sustained load

What fooled MPG was testing with large files. Here are the results reading one 101 GiB file ten times in DiskTester. In this specific case, there are huge benefits over gigabit ethenet, about 5X faster than possible with gigabit ethernet:

disktester read-files --iterations 10 pack-1414bd2927c21ad2c0eee754b9a2b9b13f0651d0.pack
# result summary:
Read 1 files totaling 100.7 GiB in 124.64 seconds @ 828 MiB/sec
Read 1 files totaling 100.7 GiB in 150.39 seconds @ 686 MiB/sec
Read 1 files totaling 100.7 GiB in 154.45 seconds @ 668 MiB/sec
Read 1 files totaling 100.7 GiB in 153.77 seconds @ 671 MiB/sec
Read 1 files totaling 100.7 GiB in 158.70 seconds @ 650 MiB/sec
Read 1 files totaling 100.7 GiB in 159.10 seconds @ 648 MiB/sec
Read 1 files totaling 100.7 GiB in 164.84 seconds @ 626 MiB/sec
Read 1 files totaling 100.7 GiB in 155.45 seconds @ 664 MiB/sec
Read 1 files totaling 100.7 GiB in 151.77 seconds @ 680 MiB/sec

However, reading a group of very large files has severe performance problems. Observe (below) how read speed drops quickly from over 1000 MB/sec down to ~96 MB/second. Stopping the test and restarting it shows a brief high performance burst followed by the pattern of quickly-declining performance. The performance pattern is actually more complicated, best described as FLAKY: testing shows highly erratic speeds: fast, very slow, medium fast, fast, very slow, etc. But mainly, it’s just slow. See the 2nd test invocation (disktester read-files) below for an example of that pattern in the MiB/sec(1) column.

Bottom line: Thunderbolt networking has benefits for sporadic usage with large files (such as opening multi-gigabyte Photoshop files). That is borne out by a few months of such use at MPG. However, Thunderbolt networking is unsuitable for any sustained usage such as uncompressed video editing or workgroup use with sustained activity.

diglloydIMAC:DIGLLOYD lloyd$ disktester read-files -b double -i 10 /Volumes/Scratch/TEMP
DiskTester 2.2.12 64-bit, diglloydTools 2.2.13, 2016-01-24 19:20
Copyright 2006-2016 DIGLLOYD INC. All Rights Reserved
Use of this software requires a license. See http://macperformanceguide.com/Software-License.html
OS X 10.11.3, 8 CPU cores, 65536MiB memory
2016-02-04 at 09:06:16469
disktester read-files -b double -i 10 /Volumes/Scratch/TEMP
read-files params:
Transfer size:        65536 KiB per buffer
Iterations:           10
Async double buffers: true
Scanning "/Volumes/Scratch/TEMP"...327
327 files in 24 folders.
---------------------------------- Iteration 1 ---------------------------------
Reading 327 files totaling 205.1 GiB...
IO method: dual asynchronous read buffers each of size 64 MiB
# Files     Amt Read MiB/sec(all)   MiB/sec(1)  MiB/sec(10)  MiB/sec(30)
8     4.92 GiB     1102         1028         1018         1018    
18     9.04 GiB     1031         1001          840          931    
18     9.66 GiB     1102         1001          840          931    
20    10.17 GiB      521         95.2          334          515  <== 90% decline in speed    
21    10.68 GiB      461          104          279          450    
22     11.2 GiB      401         80.2          225          387    
24     11.7 GiB      325         98.0          169          322    
25     12.2 GiB      305         94.2          151          299    
27     12.8 GiB      274          137          128          271    
28     13.3 GiB      267          122          120          262    
29     13.8 GiB      255         88.7         98.2          248    
31     14.4 GiB      232         99.1         98.0          230    
32     14.9 GiB      224         89.0         99.2          221    
34     15.6 GiB      215          127          104          179    
35     16.1 GiB      211          104          105          169    
36     16.6 GiB      208         99.5          105          162    
37     17.1 GiB      191         91.5          100          150    
38     17.6 GiB      188         84.8         97.2          143    
39     18.0 GiB      183         82.5         96.5          137    
41     18.6 GiB      176          115         96.7          129    
42     19.1 GiB      175          110         98.5          125    
43     19.6 GiB      172         85.0         96.2          121    
45     20.1 GiB      164         97.6         91.8          113    
46     20.8 GiB      164          104         92.2          110    
48     21.4 GiB      160          122         95.4          104    
48     21.9 GiB      164          122         95.4          104    
48     22.4 GiB      168          122         95.4          104    
48     22.9 GiB      171          122         95.4          104    
48     23.4 GiB      175          122         95.4          104    
48     23.9 GiB      179          122         95.4          104    
48     24.4 GiB      183          122         95.4          104    
49     24.9 GiB      147         95.3         96.2         96.8    
51     25.4 GiB      144          114         96.9         97.1    
52     25.9 GiB      143         77.6         94.9         97.0    
53     26.4 GiB      143         98.7         95.8         97.2    
56     27.2 GiB      140          143         99.0         98.5    
58     27.8 GiB      140         95.7         99.6         97.8    
59     28.3 GiB      140          129          106         98.6    
61     28.8 GiB      138         91.2          103         98.2    
63     29.4 GiB      137          165          111         99.0    
65     30.1 GiB      137          149          115         99.2    
65     30.6 GiB      139          149          115         99.2    
65     31.1 GiB      142          149          115         99.2    
65     31.6 GiB      144          149          115         99.2    
66     32.1 GiB      134         96.4          105         98.8    
66     32.6 GiB      136         96.4          105         98.8    
66     33.1 GiB      138         96.4          105         98.8    
66     33.6 GiB      141         96.4          105         98.8    
66     34.1 GiB      143         96.4          105         98.8    
66     34.6 GiB      145         96.4          105         98.8    
66     34.9 GiB      146         96.4          105         98.8    
66     35.4 GiB      148         96.4          105         98.8    
66     35.9 GiB      150         96.4          105         98.8    
66     36.3 GiB      152         96.4          105         98.8    
66     36.8 GiB      154         96.4          105         98.8    
66     37.3 GiB      156         96.4          105         98.8    
68     38.0 GiB      125          109         93.8         95.9    
69     38.6 GiB      124          111         93.9         96.6    
69     39.1 GiB      126          111         93.9         96.6    
69     39.6 GiB      127          111         93.9         96.6    
69     40.0 GiB      129          111         93.9         96.6    
70     40.5 GiB      122         93.8         94.0         96.5    
70     41.0 GiB      124         93.8         94.0         96.5    
70     41.5 GiB      125         93.8         94.0         96.5    
70     42.0 GiB      127         93.8         94.0         96.5    
71     42.7 GiB      121         89.1         93.3         95.7    
71     43.3 GiB      123         89.1         93.3         95.7    
71     43.8 GiB      124         89.1         93.3         95.7    
72     44.3 GiB      119          103         94.5         96.0    
73     44.9 GiB      119          112         94.1         96.5    
74     45.5 GiB      120          106         94.1         96.9    
75     46.0 GiB      119         99.4         93.6         97.0    
75     46.5 GiB      120         99.4         93.6         97.0    
75     47.0 GiB      121         99.4         93.6         97.0    
75     47.7 GiB      123         99.4         93.6         97.0    
75     48.2 GiB      124         99.4         93.6         97.0    
75     48.7 GiB      126         99.4         93.6         97.0    
75     49.2 GiB      127         99.4         93.6         97.0    
76     49.8 GiB      117         90.8         92.8         96.1    
76     50.3 GiB      118         90.8         92.8         96.1    
76     50.8 GiB      119         90.8         92.8         96.1    
76     51.4 GiB      121         90.8         92.8         96.1    
76     51.9 GiB      122         90.8         92.8         96.1    
76     52.3 GiB      123         90.8         92.8         96.1    
76     52.8 GiB      124         90.8         92.8         96.1    
79     53.3 GiB      113          116         93.8         95.2    
81     54.1 GiB      114          126         96.2         95.6    
82     54.7 GiB      114          117         95.5         96.1    
84     55.4 GiB      114          120         96.4         96.9    
85     55.9 GiB      113         93.6         96.1         96.4    
^C
Caught signal 2 (SIGINT)
Program should respond within a few seconds.
FileReaderTask::ReadAll: exiting
Task FileReaderTask exited with error: user cancelled [-128]
Read 85 files totaling 56.3 GiB in 510.09 seconds @ 113 MiB/sec
Command "read-files" executed in 510.49 seconds on 2016-02-04 at 09:14:46959
diglloydIMAC:DIGLLOYD lloyd$ sudo purge
Password:
diglloydIMAC:DIGLLOYD lloyd$ time disktester read-files -b double -i 10 /Volumes/Scratch/TEMP
DiskTester 2.2.12 64-bit, diglloydTools 2.2.13, 2016-01-24 19:20
Copyright 2006-2016 DIGLLOYD INC. All Rights Reserved
Use of this software requires a license. See http://macperformanceguide.com/Software-License.html
OS X 10.11.3, 8 CPU cores, 65536MiB memory
2016-02-04 at 09:32:33963
disktester read-files -b double -i 10 /Volumes/Scratch/TEMP
read-files params:
Transfer size:        65536 KiB per buffer
Iterations:           10
Async double buffers: true
Scanning "/Volumes/Scratch/TEMP"...327
327 files in 24 folders.
---------------------------------- Iteration 1 ---------------------------------
Reading 327 files totaling 205.1 GiB...
IO method: dual asynchronous read buffers each of size 64 MiB
# Files     Amt Read MiB/sec(all)   MiB/sec(1)  MiB/sec(10)  MiB/sec(30)
3     2.45 GiB      520          494          493          493    
5     3.53 GiB      403          168          388          388    
6     4.03 GiB      324          103          304          304    
7     4.54 GiB      280          101          256          256    
15     7.15 GiB      285          381          222          280    
18     9.41 GiB      328          463          476          285    
27     13.1 GiB      370          815          651          359    
37     17.2 GiB      424          680          824          526    
47     20.9 GiB      456          991          728          723    
51     25.7 GiB      508         1017          840          850    
65     30.2 GiB      544          574          928          851    
66     32.2 GiB      561          946          930          868    
66     32.7 GiB      569          946          930          868    
66     33.2 GiB      578          946          930          868    
66     33.7 GiB      587          946          930          868    
66     34.2 GiB      595          946          930          868    
66     35.7 GiB      622          946          930          868    
66     36.1 GiB      628          946          930          868    
66     36.6 GiB      637          946          930          868    
66     37.1 GiB      645          946          930          868    
67     37.6 GiB      339          105          172          291    
68     38.3 GiB      341          158          168          285    
69     38.9 GiB      332          123          160          272    
69     39.4 GiB      336          123          160          272    
69     39.9 GiB      340          123          160          272    
70     40.4 GiB      298          104          144          236    
70     40.9 GiB      302          104          144          236    
70     41.4 GiB      306          104          144          236    
70     41.9 GiB      309          104          144          236    
71     42.4 GiB      272         90.8          131          207    
71     42.9 GiB      275         90.8          131          207    
71     43.4 GiB      278         90.8          131          207    
71     43.9 GiB      281         90.8          131          207    
74     45.2 GiB      247          748          120          183    
75     48.5 GiB      264          983          123          185 <== blips to fast again 
76     51.9 GiB      274          626          132          200    
84     55.4 GiB      280          577          703          198    
86     56.7 GiB      285          641          724          200    
87     57.4 GiB      281          101          382          194    
89     58.0 GiB      274          114          269          188    
90     58.5 GiB      269          101          217          183    
92     59.1 GiB      264          109          174          177    
94     59.7 GiB      260          128          137          172    
96     60.4 GiB      258          139          111          161    
96     60.8 GiB      259          139          111          161    
98     61.2 GiB      251         88.8          103          178 <== far slower than gigabit ethernet 
100     61.8 GiB      248         89.5          102          188    
101     62.4 GiB      246          115          105          206    
103     63.0 GiB      242          111          105          234    
105     63.7 GiB      240          110          103          216    
107     64.2 GiB      236         96.2          106          144    
108     64.7 GiB      233         94.9          106          140    
109     65.4 GiB      232          121          108          137    
110     66.0 GiB      230          115          110          133    
112     66.6 GiB      227          105          108          125    
113     67.2 GiB      227          104          107          120    
114     67.7 GiB      225          104          106          115    
115     68.1 GiB      222         95.0          105          110    
117     68.8 GiB      218          130          104          105    
118     69.4 GiB      216          110          106          105    
120     70.1 GiB      215          126          106          106    
121     70.6 GiB      215          143          109          107    
123     71.3 GiB      211         99.7          105          106    
125     71.8 GiB      209         97.0          105          104    
126     72.3 GiB      208         99.0          107          103    
128     73.0 GiB      206          139          109          107    
130     73.6 GiB      205          112          108          108    
132     74.2 GiB      204          138          114          108    
134     74.9 GiB      202          103          116          108    
136     75.5 GiB      201          104          119          108    
138     76.2 GiB      200          128          117          110    
140     76.8 GiB      199          124          117          110    
142     77.4 GiB      197          121          114          110    
142     77.9 GiB      199          121          114          110    
142     78.4 GiB      200          121          114          110    
143     79.1 GiB      194         97.5          107          109    
143     79.8 GiB      195         97.5          107          109    
^C
Caught signal 2 (SIGINT)
Program should respond within a few seconds.
FileReaderTask::ReadAll: exiting
Task FileReaderTask exited with error: user cancelled [-128]
Read 143 files totaling 80.0 GiB in 428.40 seconds @ 191 MiB/sec
Command "read-files" executed in 430.54 seconds on 2016-02-04 at 09:39:44502
Blazing-fast PCIe storage for Mac Pro Tower

diglloyd.com | Terms of Use | PRIVACY POLICY
Contact | About Lloyd Chambers | Consulting | Photo Tours
Mailing Lists | RSS Feeds | Twitter
Copyright © 2008-2017 diglloyd Inc, all rights reserved.
Display info: __RETINA_INFO_STATUS__