Coding Horror: Gigabit Ethernet and Back of the Envelope Calculations

July 12, 2005

Gigabit Ethernet and Back of the Envelope Calculations

At work today, we had a problem with a particular workstation. Although it was connected to a gigabit ethernet hub, network file transfers were "too slow". How do you quantify "too slow"?

a gigabit ethernet LAN connection

I was reminded of chapter seven of Programming Pearls -- The Back of the Envelope:

It was in the middle of a fascinating conversation on software engineering that Bob Martin asked me, "How much water flows out of the Mississippi River in a day?" Because I had found his comments up to that point deeply insightful, I politely stifled my true response and said, "Pardon me?" When he asked again I realized that I had no choice but to humor the poor fellow, who had obviously cracked under the pressures of running a large software shop.
My response went something like this. I figured that near its mouth the river was about a mile wide and maybe twenty feet deep (or about one two-hundred-and-fiftieth of a mile). I guessed that the rate of flow was five miles an hour, or a hundred and twenty miles per day. Multiplying
1 mile x 1/250 mile x 120 miles/day ~ 1/2 mile³/day
showed that the river discharged about half a cubic mile of water per day, to within an order of magnitude. But so what?
At that point Martin picked up from his desk a proposal for the communication system that his organization was building for the Summer Olympic games, and went through a similar sequence of calculations. He estimated one key parameter as we spoke by measuring the time required to send himself a one-character piece of mail. The rest of his numbers were straight from the proposal and therefore quite precise. His calculations were just as simple as those about the Mississippi River and much more revealing. They showed that, under generous assumptions, the proposed system could work only if there were at least a hundred and twenty seconds in each minute. He had sent the design back to the drawing board the previous day. (The conversation took place about a year before the event, and the final system was used during the Olympics without a hitch.)
That was Bob Martin's wonderful (if eccentric) way of introducing the engineering technique of "back-of-the-envelope" calculations. The idea is standard fare in engineering schools and is bread and butter for most practicing engineers. Unfortunately, it is too often neglected in computing.

To diagnose the network throughput issues, I busted out my copy of pcattcp and started doing some baseline network speed measurements. It's a great utility, and quite simple to use; just run one instance on a remote machine using the -R flag, then run another instance on the client with -t (remotename) and you're off to the races.

But even before that, I started with a loopback test:

C:\Program Files\ttcp>pcattcp -t -f M localhost

PCAUSA Test TCP Utility V2.01.01.08
TCP Transmit Test
  Transmit    : TCP -> 127.0.0.1:5001
  Buffer Size : 8192; Alignment: 16384/0
  TCP_NODELAY : DISABLED (0)
  Connect     : Connected to 127.0.0.1:5001
  Send Mode   : Send Pattern; Number of Buffers: 2048
  Statistics  : TCP -> 127.0.0.1:5001
16777216 bytes in 0.17 real seconds = 93.02 MB/sec +++
numCalls: 2048; msec/call: 0.09; calls/sec: 11906.98

This is helpful because it establishes an absolute upper bound on network performance. Even with an infinitely fast network, I won't achieve more than 93 megabytes per second throughput-- at least not on my PC. And this is a completely in-memory test; real world network operations may depend on hard disk reads and writes, which will be far slower.

A good rule of thumb for real-world throughput is:

10baseT = 1 megabyte/sec
100baseT = 10 megabytes/sec
1000baseT = 30 megabytes/sec

All my ttcp testing over the last couple years has confirmed these numbers, plus or minus ten percent. I don't have as much experience with gigabit throughput, since I just got my first gigabit router, but you definitely shouldn't expect the perfect scaling we achieved moving from 10baseT to 100baseT. Without any major tweaking, you'll get only a fraction of the tenfold bandwidth improvement you might expect:

I noticed a significant improvement in multicast performance, measured by the time required to send a 690MB disk image to 18 multicast clients in one session. The HP NetServer LT6000r served as the multicast server, and the clients were using Fast Ethernet links to the desktop switch. On the Fast Ethernet network, the task took 19 minutes. On the Gigabit Ethernet network, the time was reduced to 9 minutes.
I measured the transfer of a large (1GB) file between the same hosts over Fast Ethernet and Gigabit Ethernet links with sustained network traffic (streaming media to multiple unicast clients). The file transfer took 230 seconds on Fast Ethernet and 88 seconds on Gigabit Ethernet.
Overall, my tests showed that Gigabit Ethernet provided a tangible performance improvement, but bottlenecks elsewhere kept the overall throughput lower than I had hoped. I was satisfied with Gigabit Ethernet performance relative to Fast Ethernet, and I was particularly impressed that general network responsiveness remained acceptable even during peak network loads. But I was disappointed not to be able to reach much beyond 450Mbps on the Lab's most capable server.

To be fair, that article is from 2002. A typical new desktop PC probably has more bandwidth and power than the author's fastest server. Even with those real world caveats, gigabit ethernet still offers 2 to 3 times the performance of 100baseT, which isn't exactly chopped liver, either.

In the end, our issue at work had nothing to do with the "problem" desktop. After a bit of ad-hoc ttcp testing, we found that nobody could achieve more than about 11 megabytes per second throughput to the server, even when directly connected to the gigabit switch. Download pcattcp and try for yourself. Some other interesting experiments you can run with ttcp are UDP (-u) versus TCP/IP, and varying the packet size (-l 4096).

Posted by Jeff Atwood View blog reactions

« VNC vs. Remote Desktop

Code Colorizing and Readability »

Comments

I finally unpacked my server, which has a gigabit 64-bit PCI card (it's a dual Athlon MP system). I'm using the d-link gigabit router described above, and the integrated nForce 4 gigabit on my desktop. The tray icons for both connections show 1.0 Gbps.

I get anywhere from ** 33-35 megabytes/sec ** when running a TTCP test from my desktop to the server. That's in the ballpark...

D:\Storage\ttcp>pcattcp -t -f M homeserver
PCAUSA Test TCP Utility V2.01.01.08
TCP Transmit Test
Transmit : TCP -> 192.168.0.10:5001
Buffer Size : 8192; Alignment: 16384/0
TCP_NODELAY : DISABLED (0)
Connect : Connected to 192.168.0.10:5001
Send Mode : Send Pattern; Number of Buffers: 2048
Statistics : TCP -> 192.168.0.10:5001
16777216 bytes in 0.48 real seconds = 33.06 MB/sec +++
numCalls: 2048; msec/call: 0.24; calls/sec: 4231.40

---

Jeff Atwood on July 24, 2005 8:22 PM

Experimenting with larger packet sizes (the default is 8192 bytes):

pcattcp -t -l 16384 -f M homeserver
51.20 MB/sec

pcattcp -t -l 32768 -f M homeserver
69.41 MB/sec

pcattcp -t -l 65536 -f M homeserver
71.23 MB/sec

pcattcp -t -l 131072 -f M homeserver
69.72 MB/sec

Gigabit ethernet tops out at ~70 megabytes per second, at least for pure memory read/writes.

Jeff Atwood on July 24, 2005 8:27 PM

The old 64-bit PCI gig-E card went kaput. The new card is a single chip design and it does *substantially* better in the same server on the same exact network-- nothing else has changed:

pcattcp -t -f M homeserver
34.12 MB/sec

pcattcp -t -l 16384 -f M homeserver
64.00 MB/sec

pcattcp -t -l 32768 -f M homeserver
93.16 MB/sec

pcattcp -t -l 65536 -f M homeserver
105.00 MB/sec

pcattcp -t -l 131072 -f M homeserver
103.73 MB/sec

Er.. wow. About the same with small 8kb packets, but WAYYY better with larger ones! Which is a good thing, because all this donkey porn isn't going to copy itself from the server. That's all I'm saying..

Jeff Atwood on October 15, 2005 2:25 AM

According to the Windows documentation, the TCP/IP default window size is 17,520 bytes. That's a reasonable setting for gig-e, but not optimal.

http://support.microsoft.com/kb/q224829/

It auto-negotiates up to 65kb. This can be improved, though, per this article which looks quite consistent with the kb doc above.

http://www.enterpriseitplanet.com/networking/features/article.php/3497796

Here's how to enable large frames in the TCP/IP stack in Windows XP, Windows 2000, and Windows Server 2003:

Edit the registry via regedit.exe and navigate to

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters

Increase the TCP/IP window size to 132k and enable rfc1323 scaling and timestamps:

- Add a registry DWORD named TcpWindowSize, and enter 131400 (decimal)
- Add a registry DWORD named Tcp1323Opts, and set the value to 3.

increase the TCP buffer sizes:

- Add a registry DWORD named ForwardBufferMemory, and set the value to 80000
- Add a registry DWORD named NumForwardPackets, and set the value to 60000

I put these four tweaks in a standalone registry file for your convenience:
http://www.codinghorror.com/blog/files/gigabit_large_window.zip

Jeff Atwood on October 15, 2005 2:54 AM

NumForwardPackets should NOT be set to more than ForwardBufferMemory / 256!

http://www.microsoft.com/resources/documentation/Windows/2000/server/reskit/en-us/regentry/58793.asp

Also, do a google on Numforwardpackets.

curious on June 10, 2006 8:58 AM

So, set ForwardBufferMemory to 79872, divide by 256 and that would give NumForwardPackets a max of 312? Does that sound about right?

Kidego on July 18, 2006 8:10 AM

- ForwardBufferMemory, and set the value to 80000
- NumForwardPackets, and set the value to 60000

NumForwardPackets should NOT be set to more than ForwardBufferMemory / 256!

if my calculations are correct, 80000 /256 > 60000 !!
Is this a typo ?

Pat357 on September 13, 2006 1:36 PM

I looked in http://www.codinghorror.com/blog/files/gigabit_large_window.zip and it's not a typo. I'd like to add that ForwardBufferMemory must be a multiple of 256 and the default value is 74,240.

pLu on January 9, 2007 3:52 AM

Remember that if the program is copying the data to HDD, there will your bottleneck be. The system will probably not transfer faster than it needs to, so your HDD write speed will be the determining factor. (I think).

Samuel Warren on July 9, 2007 2:17 PM

Well your hard drive may seem to be bottlenecking as it is the slowest component of the computer, but many modern desktop hard drives can read/write on average 50 MB/s (that's Big M Big B for megabytes a second). Personally, my 500 gig sata2 samsung 7200rpm can read/write from 60 to 80 MB/s bursting up to 100 MB/s or more. Even my older IDE 7200 rpm can transfer at least 30 MB/s. Speaking of bottlenecks, USB hard drives are being bottlenecked to 25-30 MB/s as a result of USB HI-speed's low sustained bandwidth(theoretical burst speeds up to 480 Mb/s but sustains around 300 Mb/s). This is one reason why I'd prefer a file server with gigabit connections rather than a JBOD configuration of USB hard drives.

Justin Chen on October 24, 2007 2:31 PM

Jan 2008 now...

Just tested with OS X 10.5.1 between a PowerMac G5 and a PowerBook G4 both connected via Gigabit (standard mtu of 1500) to a NETGEAR GS105. I'm getting 70MB/sec which is much faster than the rule of thumb posted above (this is between two machines not in memory on a single machine).

ttcp -r -s -f M
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001 tcp
ttcp-r: socket
ttcp-r: accept from 192.168.0.190
ttcp-r: 1677721600 bytes in 22.36 real seconds = 71.57 MB/sec +++
ttcp-r: 234073 I/O calls, msec/call = 0.10, calls/sec = 10470.32
ttcp-r: 0.2user 5.0sys 0:22real 23% 0i+0d 0maxrss 0+0pf 223322+0csw

My in memory tests report 455 MB/sec and 128 MB/sec.

Computer 1 (PowerMac G5)
ttcp -r -s -f M &
ttcp -t -s -n 710400 -f M localhost
ttcp-t: buflen=8192, nbuf=710400, align=16384/0, port=5001 tcp -> localhost
ttcp-t: socket
ttcp-t: connect
ttcp-t: 5819596800 bytes in 12.18 real seconds = 455.55 MB/sec +++
ttcp-t: 710400 I/O calls, msec/call = 0.02, calls/sec = 58310.97
ttcp-t: 0.5user 11.6sys 0:12real 99% 0i+0d 0maxrss 0+0pf 15+0csw

Computer 2 (PowerBook G4)
ttcp -r -s -f M &
ttcp -t -s -n 305200 -f M localhost
ttcp-t: buflen=8192, nbuf=305200, align=16384/0, port=5001 tcp -> localhost
ttcp-t: socket
ttcp-t: connect
ttcp-t: 2500198400 bytes in 18.57 real seconds = 128.39 MB/sec +++
ttcp-t: 305200 I/O calls, msec/call = 0.06, calls/sec = 16433.41
ttcp-t: 0.5user 5.5sys 0:18real 32% 0i+0d 0maxrss 0+0pf 813+0csw

Sean Roehnelt on January 27, 2008 4:07 PM

Great article, very useful. Am actually looking at an industrial application that requires about 50MB/s - 80 MB/s data rates and I'm investigating whether gigabit ethernet could be a viable (cheap) solution.

However I have one query, when you refer to changing the packet size, I presume you don't actual mean the packet size but rather the amount of data transmitted? If this is the case the speed difference is probably to due to efficiencies of scale.

I thought the packet size (MTU) defaulted to 1500, or 9kB (jumbo frames).

Tom Martin on March 5, 2008 6:38 AM

I just tried benchmarking my home network. All my pc is connected to a gigabit ethernet router. Gigabit ethernet is really a lot fast.

Using pcattcp -t -l 131072 -f M MyPC, I can get about 108-109 MB/s which is about 864-872 Mbps. For file transfer from PC1->PC2, i can get about 38MB/s.

Alex on March 20, 2008 11:10 PM

The PCATTCP download has a virus in it.

...picked up by Avast.

Trojan on December 18, 2008 1:41 PM

I just wrote 70MB/s across gigabit, so I'm incredibly happy.

Andrew Szeto on May 26, 2009 7:32 AM