SEARCH
diglloydTools™As an Amazon Associate I earn from qualifying purchases @AMAZON
Thunderbolt 10-Gigabit Networking: Real World Performance
Related: Apple macOS, CPU cores, data integrity, diglloydTools, DiskTester, ethernet, how-to, IntegrityChecker, Mac Pro, memory, networking, optimization, SSD, Thunderbolt
OWC offers high quality Thunderbolt cables in red/green/blue/black colors from 0.5 to 3.0 meters, as well as Thunderbolt optical cables from 10.0 to 30.0 meters in length.
UPDATE: Thunderbolt networking works at high speeds for a short duration only. After than, its speed drops to sub-gigabit speeds; see the sustained heavy load tests that follow.
The following pages discuss how to set up 10 Gigabit Thunderbolt networking between Macs. But how does it actually perform for real-world usage?
MB = 1000 * 1000, GB = 1000^3
MiB = 1024 * 1024, GiB = 1024^3
Test details
The file server was an 8-core 2013 Mac Pro with files on a striped pair of OWC Mercury Accelsior PCIe SSD cards in an OWC Helios enclosure, delivering about 1100MB/sec read performance, important since Thunderbolt networking in theory offers speeds in that range. An even faster SSD would have been preferable, since the SSD itself speed could have some influence (data has to be read from disk).
The client was a late 2013 MacBook Retina. The Thunderbolt cable was plugged into an OWC Thunderbolt 2 Dock. Though the Thunderbolt cable could have been plugged in directly to the MBP, this is a more realistic setup (ports are in short supply on the MBP).
The cable was a 3 meter OWC Thunderbolt 2 cable, color red (helpful in distinguishing cables when there are 6 of them coming out of a Mac Pro!).
Thunderbolt 10 Gbps network performance tests
Photoshop open PSB over Thunderbolt networking
Opening a multi-gigabyte PSB file is about as fast as opening it directly on the Mac Pro on the local SSD volume—outstanding!
Finder copy speed over Thunderbolt networking
To test performance, a 21.48 GB / 20.0 GiB folder of multi-gigabyte files was used to test transfer speed.
Copy 21.48GB / 20.0 GiB: 32 seconds = 671 MB/sec / 625 MiB/sec
Command line copy over Thunderbolt networking
Unix 'cp' takes nearly 50% longer than a Finder copy. Probably the Finder is smarter about interleaving reads and writes (a copy also has to write to the local SSD). Speed was even worse at times (up to 51 seconds), so Unix 'cp' is not a good way to enjoy full speed.
time cp -r /Volumes/Scratch/TEMP ~/Desktop/
real 0m45.509s
user 0m0.019s
sys 0m18.455s
= 472 MB/sec / 494 MiB/sec
DiskTester read-files over Thunderbolt networking
The DiskTester read-files command reads files (no writing) so it takes writing out of the process—thus a true measure of read performance.
DiskTester read-files is far more efficient than either 'cp' or Finder copy, delivering a whopping 1054 MB/sec / 1006 MiB/sec.
llcT:DIGLLOYD lloyd$ disktester read-files --xfer 4M -b double /Volumes/Scratch/TEMP
DiskTester 2.2.11 64-bit, diglloydTools 2.2.11, 2015-02-28 15:08
Copyright 2006-2014 DIGLLOYD INC. All Rights Reserved
Use of this software requires a license. See https://macperformanceguide.com/Software-License.html
OS X 10.11.1, 8 CPU cores, 16384MB memory
Monday, November 16, 2015 at 13:59:01 Pacific Standard Time
disktester read-files --xfer 4M -b double /Volumes/Scratch/TEMP
read-files params:
Transfer size: 2048KB per buffer
Iterations: 1
Async double buffers: true
Scanning "/Volumes/Scratch/TEMP"...62
62 files in 4 folders.
---------------------------------- Iteration 1 ---------------------------------
Reading 62 files totaling 20.0 GiB...
IO method: dual asynchronous read buffers each of size 2 MiB
# Files Amt Read MiB/sec(all) MiB/sec(1) MiB/sec(10) MiB/sec(30)
9 5.02GB 1051 1030 1031 1031
20 10.13GB 1048 1049 1046 1039
38 14.8GB 1058 1050 959 1023
56 19.6GB 1010 1054 966 959
62 20.0GB 1006 20.3 930 978
Data forks read: 62
Resource forks ignored.
Total files read: 62
done.
Read 62 files totaling 20.0 GiB in 20.36 seconds @ 1006 MiB/sec
Command "read-files" executed in 20.54 seconds on Monday, November 16, 2015 at 13:59:22 Pacific Standard Time
IntegrityChecker verify over Thunderbolt networking
The IntegrityChecker verify command reads files and computes SHA1 hashes. It is thus a real-world computational task (not just a file copy).
A credit it to its highly optimized architecture, IntegrityChecker can read and verify files faster than the Finder can copy them, delivering a whopping 868 MB/sec / 808 MiB/sec. This outstanding performance might come in handy for users with very large networked data sets.
llcT:DIGLLOYD lloyd$ ic verify -b 32M /Volumes/Scratch/TEMP
IntegrityChecker(tm) v1.2.3 64-bit, diglloydTools 2.2.11, 2015-02-28 15:08
Copyright 2006-2014 DIGLLOYD INC. All Rights Reserved
Use of this software requires a license. See https://macperformanceguide.com/Software-License.html
OS X 10.11.1, 8 CPU cores, 16384MB memory
Monday, November 16, 2015 at 13:45:55 Pacific Standard Time
ic verify -b 32M /Volumes/Scratch/TEMP
Using threads = 4, read buffer size = 32768K, num buffers = 24
Looking for files in "/Volumes/Scratch/TEMP"...
Seconds to scan files: 0.00
62 files found in 0.00 seconds.
Scanning 3 folders for .ic files...3 hash-data files read in 0.00 seconds.
Selecting files for hashing...
Preparing to hash 56 files...
Files prepared, hashing 56 files...
0%: 0 files @ 0.0MB/sec, processed 0B
0%: 0 files @ 0.0MB/sec, processed 0B
14%: 2 files @ 883.5MB/sec, processed 2.87GB
19%: 4 files @ 863.6MB/sec, processed 3.78GB
25%: 7 files @ 875.3MB/sec, processed 4.92GB
31%: 11 files @ 902.9MB/sec, processed 6.27GB
37%: 14 files @ 905.5MB/sec, processed 7.41GB
41%: 16 files @ 823.7MB/sec, processed 8.16GB
50%: 18 files @ 902.8MB/sec, processed 10.05GB
54%: 20 files @ 877.9MB/sec, processed 10.81GB
59%: 24 files @ 873.6MB/sec, processed 11.8GB
63%: 28 files @ 868.7MB/sec, processed 12.7GB
67%: 31 files @ 857.0MB/sec, processed 13.4GB
72%: 35 files @ 857.8MB/sec, processed 14.3GB
72%: 35 files @ 810.3MB/sec, processed 14.3GB
80%: 40 files @ 848.4MB/sec, processed 16.0GB
85%: 44 files @ 844.6MB/sec, processed 16.9GB
89%: 47 files @ 835.8MB/sec, processed 17.7GB
95%: 51 files @ 843.0MB/sec, processed 19.1GB
Finished reading 56 files of 56
100%: 55 files @ 840.8MB/sec, processed 20.0GB
100%: 56 files @ 807.6MB/sec, processed 20.0GB
Processed 20.0 GiB in 25.4 seconds @ 807.6 MiB/sec
========================================================================================================================
ic verify -b 32M /Volumes/Scratch/TEMP
Monday, November 16, 2015 at 13:46:20 Pacific Standard Time
========================================================================================================================
# Files with stored hash: 56
# Files missing: 0
# Files hashed: 56
# Files without hashes: 0
# Files whose size has changed: 0
# Files whose date changed: 0
# Files whose content changed (same size): 0
# Suspicious files: 0
CPU usage
As with ethernet (which sucks up an entire CPU under full load), Thunderbolt networking consumes some CPU time. At full bandwidth, nearly a full CPU core is utilized (“kernel_task”).
Thunderbolt 10 Gbps network performance tests — sustained load
What fooled MPG was testing with large files. Here are the results reading one 101 GiB file ten times in DiskTester. In this specific case, there are huge benefits over gigabit ethenet, about 5X faster than possible with gigabit ethernet:
disktester read-files --iterations 10 pack-1414bd2927c21ad2c0eee754b9a2b9b13f0651d0.pack # result summary: Read 1 files totaling 100.7 GiB in 124.64 seconds @ 828 MiB/sec Read 1 files totaling 100.7 GiB in 150.39 seconds @ 686 MiB/sec Read 1 files totaling 100.7 GiB in 154.45 seconds @ 668 MiB/sec Read 1 files totaling 100.7 GiB in 153.77 seconds @ 671 MiB/sec Read 1 files totaling 100.7 GiB in 158.70 seconds @ 650 MiB/sec Read 1 files totaling 100.7 GiB in 159.10 seconds @ 648 MiB/sec Read 1 files totaling 100.7 GiB in 164.84 seconds @ 626 MiB/sec Read 1 files totaling 100.7 GiB in 155.45 seconds @ 664 MiB/sec Read 1 files totaling 100.7 GiB in 151.77 seconds @ 680 MiB/sec
However, reading a group of very large files has severe performance problems. Observe (below) how read speed drops quickly from over 1000 MB/sec down to ~96 MB/second. Stopping the test and restarting it shows a brief high performance burst followed by the pattern of quickly-declining performance. The performance pattern is actually more complicated, best described as FLAKY: testing shows highly erratic speeds: fast, very slow, medium fast, fast, very slow, etc. But mainly, it’s just slow. See the 2nd test invocation (disktester read-files) below for an example of that pattern in the MiB/sec(1) column.
Bottom line: Thunderbolt networking has benefits for sporadic usage with large files (such as opening multi-gigabyte Photoshop files). That is borne out by a few months of such use at MPG. However, Thunderbolt networking is unsuitable for any sustained usage such as uncompressed video editing or workgroup use with sustained activity.
diglloydIMAC:DIGLLOYD lloyd$ disktester read-files -b double -i 10 /Volumes/Scratch/TEMP DiskTester 2.2.12 64-bit, diglloydTools 2.2.13, 2016-01-24 19:20 Copyright 2006-2016 DIGLLOYD INC. All Rights Reserved Use of this software requires a license. See https://macperformanceguide.com/Software-License.html OS X 10.11.3, 8 CPU cores, 65536MiB memory 2016-02-04 at 09:06:16469 disktester read-files -b double -i 10 /Volumes/Scratch/TEMP read-files params: Transfer size: 65536 KiB per buffer Iterations: 10 Async double buffers: true Scanning "/Volumes/Scratch/TEMP"...327 327 files in 24 folders. ---------------------------------- Iteration 1 --------------------------------- Reading 327 files totaling 205.1 GiB... IO method: dual asynchronous read buffers each of size 64 MiB # Files Amt Read MiB/sec(all) MiB/sec(1) MiB/sec(10) MiB/sec(30) 8 4.92 GiB 1102 1028 1018 1018 18 9.04 GiB 1031 1001 840 931 18 9.66 GiB 1102 1001 840 931 20 10.17 GiB 521 95.2 334 515 <== 90% decline in speed 21 10.68 GiB 461 104 279 450 22 11.2 GiB 401 80.2 225 387 24 11.7 GiB 325 98.0 169 322 25 12.2 GiB 305 94.2 151 299 27 12.8 GiB 274 137 128 271 28 13.3 GiB 267 122 120 262 29 13.8 GiB 255 88.7 98.2 248 31 14.4 GiB 232 99.1 98.0 230 32 14.9 GiB 224 89.0 99.2 221 34 15.6 GiB 215 127 104 179 35 16.1 GiB 211 104 105 169 36 16.6 GiB 208 99.5 105 162 37 17.1 GiB 191 91.5 100 150 38 17.6 GiB 188 84.8 97.2 143 39 18.0 GiB 183 82.5 96.5 137 41 18.6 GiB 176 115 96.7 129 42 19.1 GiB 175 110 98.5 125 43 19.6 GiB 172 85.0 96.2 121 45 20.1 GiB 164 97.6 91.8 113 46 20.8 GiB 164 104 92.2 110 48 21.4 GiB 160 122 95.4 104 48 21.9 GiB 164 122 95.4 104 48 22.4 GiB 168 122 95.4 104 48 22.9 GiB 171 122 95.4 104 48 23.4 GiB 175 122 95.4 104 48 23.9 GiB 179 122 95.4 104 48 24.4 GiB 183 122 95.4 104 49 24.9 GiB 147 95.3 96.2 96.8 51 25.4 GiB 144 114 96.9 97.1 52 25.9 GiB 143 77.6 94.9 97.0 53 26.4 GiB 143 98.7 95.8 97.2 56 27.2 GiB 140 143 99.0 98.5 58 27.8 GiB 140 95.7 99.6 97.8 59 28.3 GiB 140 129 106 98.6 61 28.8 GiB 138 91.2 103 98.2 63 29.4 GiB 137 165 111 99.0 65 30.1 GiB 137 149 115 99.2 65 30.6 GiB 139 149 115 99.2 65 31.1 GiB 142 149 115 99.2 65 31.6 GiB 144 149 115 99.2 66 32.1 GiB 134 96.4 105 98.8 66 32.6 GiB 136 96.4 105 98.8 66 33.1 GiB 138 96.4 105 98.8 66 33.6 GiB 141 96.4 105 98.8 66 34.1 GiB 143 96.4 105 98.8 66 34.6 GiB 145 96.4 105 98.8 66 34.9 GiB 146 96.4 105 98.8 66 35.4 GiB 148 96.4 105 98.8 66 35.9 GiB 150 96.4 105 98.8 66 36.3 GiB 152 96.4 105 98.8 66 36.8 GiB 154 96.4 105 98.8 66 37.3 GiB 156 96.4 105 98.8 68 38.0 GiB 125 109 93.8 95.9 69 38.6 GiB 124 111 93.9 96.6 69 39.1 GiB 126 111 93.9 96.6 69 39.6 GiB 127 111 93.9 96.6 69 40.0 GiB 129 111 93.9 96.6 70 40.5 GiB 122 93.8 94.0 96.5 70 41.0 GiB 124 93.8 94.0 96.5 70 41.5 GiB 125 93.8 94.0 96.5 70 42.0 GiB 127 93.8 94.0 96.5 71 42.7 GiB 121 89.1 93.3 95.7 71 43.3 GiB 123 89.1 93.3 95.7 71 43.8 GiB 124 89.1 93.3 95.7 72 44.3 GiB 119 103 94.5 96.0 73 44.9 GiB 119 112 94.1 96.5 74 45.5 GiB 120 106 94.1 96.9 75 46.0 GiB 119 99.4 93.6 97.0 75 46.5 GiB 120 99.4 93.6 97.0 75 47.0 GiB 121 99.4 93.6 97.0 75 47.7 GiB 123 99.4 93.6 97.0 75 48.2 GiB 124 99.4 93.6 97.0 75 48.7 GiB 126 99.4 93.6 97.0 75 49.2 GiB 127 99.4 93.6 97.0 76 49.8 GiB 117 90.8 92.8 96.1 76 50.3 GiB 118 90.8 92.8 96.1 76 50.8 GiB 119 90.8 92.8 96.1 76 51.4 GiB 121 90.8 92.8 96.1 76 51.9 GiB 122 90.8 92.8 96.1 76 52.3 GiB 123 90.8 92.8 96.1 76 52.8 GiB 124 90.8 92.8 96.1 79 53.3 GiB 113 116 93.8 95.2 81 54.1 GiB 114 126 96.2 95.6 82 54.7 GiB 114 117 95.5 96.1 84 55.4 GiB 114 120 96.4 96.9 85 55.9 GiB 113 93.6 96.1 96.4 ^C Caught signal 2 (SIGINT) Program should respond within a few seconds. FileReaderTask::ReadAll: exiting Task FileReaderTask exited with error: user cancelled [-128] Read 85 files totaling 56.3 GiB in 510.09 seconds @ 113 MiB/sec Command "read-files" executed in 510.49 seconds on 2016-02-04 at 09:14:46959 diglloydIMAC:DIGLLOYD lloyd$ sudo purge Password: diglloydIMAC:DIGLLOYD lloyd$ time disktester read-files -b double -i 10 /Volumes/Scratch/TEMP DiskTester 2.2.12 64-bit, diglloydTools 2.2.13, 2016-01-24 19:20 Copyright 2006-2016 DIGLLOYD INC. All Rights Reserved Use of this software requires a license. See https://macperformanceguide.com/Software-License.html OS X 10.11.3, 8 CPU cores, 65536MiB memory 2016-02-04 at 09:32:33963 disktester read-files -b double -i 10 /Volumes/Scratch/TEMP read-files params: Transfer size: 65536 KiB per buffer Iterations: 10 Async double buffers: true Scanning "/Volumes/Scratch/TEMP"...327 327 files in 24 folders. ---------------------------------- Iteration 1 --------------------------------- Reading 327 files totaling 205.1 GiB... IO method: dual asynchronous read buffers each of size 64 MiB # Files Amt Read MiB/sec(all) MiB/sec(1) MiB/sec(10) MiB/sec(30) 3 2.45 GiB 520 494 493 493 5 3.53 GiB 403 168 388 388 6 4.03 GiB 324 103 304 304 7 4.54 GiB 280 101 256 256 15 7.15 GiB 285 381 222 280 18 9.41 GiB 328 463 476 285 27 13.1 GiB 370 815 651 359 37 17.2 GiB 424 680 824 526 47 20.9 GiB 456 991 728 723 51 25.7 GiB 508 1017 840 850 65 30.2 GiB 544 574 928 851 66 32.2 GiB 561 946 930 868 66 32.7 GiB 569 946 930 868 66 33.2 GiB 578 946 930 868 66 33.7 GiB 587 946 930 868 66 34.2 GiB 595 946 930 868 66 35.7 GiB 622 946 930 868 66 36.1 GiB 628 946 930 868 66 36.6 GiB 637 946 930 868 66 37.1 GiB 645 946 930 868 67 37.6 GiB 339 105 172 291 68 38.3 GiB 341 158 168 285 69 38.9 GiB 332 123 160 272 69 39.4 GiB 336 123 160 272 69 39.9 GiB 340 123 160 272 70 40.4 GiB 298 104 144 236 70 40.9 GiB 302 104 144 236 70 41.4 GiB 306 104 144 236 70 41.9 GiB 309 104 144 236 71 42.4 GiB 272 90.8 131 207 71 42.9 GiB 275 90.8 131 207 71 43.4 GiB 278 90.8 131 207 71 43.9 GiB 281 90.8 131 207 74 45.2 GiB 247 748 120 183 75 48.5 GiB 264 983 123 185 <== blips to fast again 76 51.9 GiB 274 626 132 200 84 55.4 GiB 280 577 703 198 86 56.7 GiB 285 641 724 200 87 57.4 GiB 281 101 382 194 89 58.0 GiB 274 114 269 188 90 58.5 GiB 269 101 217 183 92 59.1 GiB 264 109 174 177 94 59.7 GiB 260 128 137 172 96 60.4 GiB 258 139 111 161 96 60.8 GiB 259 139 111 161 98 61.2 GiB 251 88.8 103 178 <== far slower than gigabit ethernet 100 61.8 GiB 248 89.5 102 188 101 62.4 GiB 246 115 105 206 103 63.0 GiB 242 111 105 234 105 63.7 GiB 240 110 103 216 107 64.2 GiB 236 96.2 106 144 108 64.7 GiB 233 94.9 106 140 109 65.4 GiB 232 121 108 137 110 66.0 GiB 230 115 110 133 112 66.6 GiB 227 105 108 125 113 67.2 GiB 227 104 107 120 114 67.7 GiB 225 104 106 115 115 68.1 GiB 222 95.0 105 110 117 68.8 GiB 218 130 104 105 118 69.4 GiB 216 110 106 105 120 70.1 GiB 215 126 106 106 121 70.6 GiB 215 143 109 107 123 71.3 GiB 211 99.7 105 106 125 71.8 GiB 209 97.0 105 104 126 72.3 GiB 208 99.0 107 103 128 73.0 GiB 206 139 109 107 130 73.6 GiB 205 112 108 108 132 74.2 GiB 204 138 114 108 134 74.9 GiB 202 103 116 108 136 75.5 GiB 201 104 119 108 138 76.2 GiB 200 128 117 110 140 76.8 GiB 199 124 117 110 142 77.4 GiB 197 121 114 110 142 77.9 GiB 199 121 114 110 142 78.4 GiB 200 121 114 110 143 79.1 GiB 194 97.5 107 109 143 79.8 GiB 195 97.5 107 109 ^C Caught signal 2 (SIGINT) Program should respond within a few seconds. FileReaderTask::ReadAll: exiting Task FileReaderTask exited with error: user cancelled [-128] Read 143 files totaling 80.0 GiB in 428.40 seconds @ 191 MiB/sec Command "read-files" executed in 430.54 seconds on 2016-02-04 at 09:39:44502
