programming and human factors
by Jeff Atwood
OSNews published a nine-language performance roundup in early 2004. The results are summarized here:
| int | long | double | trig | I/O | ||
| Visual C++ | 9.6 | 18.8 | 6.4 | 3.5 | 10.5 | 48.8 |
| Visual C# | 9.7 | 23.9 | 17.7 | 4.1 | 9.9 | 65.3 |
| gcc C | 9.8 | 28.8 | 9.5 | 14.9 | 10.0 | 73.0 |
| Visual Basic | 9.8 | 23.7 | 17.7 | 4.1 | 30.7 | 85.9 |
| Visual J# | 9.6 | 23.9 | 17.5 | 4.2 | 35.1 | 90.4 |
| Java 1.3.1 | 14.5 | 29.6 | 19.0 | 22.1 | 12.3 | 97.6 |
| Java 1.4.2 | 9.3 | 20.2 | 6.5 | 57.1 | 10.1 | 103.1 |
| Python/Psyco | 29.7 | 615.4 | 100.4 | 13.1 | 10.5 | 769.1 |
| Python | 322.4 | 891.9 | 405.7 | 47.1 | 11.9 | 1679.0 |
It's not a very practical benchmark, but it does tell us a few things. It's no surprise that C++ is at the head of the pack. But the others aren't terribly far behind. What I find really interesting, though, is how most of the languages clump together in the middle. There's no significant performance difference between Java and .NET if you throw out the weirdly anomalous trig results.
However, there is one language definitely bringing up the rear-- Python. That's because it's an interpreted language. This is explained in Code Complete:
Interpreted languages tend to exact significant performance penalties because they must process each programming-language instruction before creating and executing machine code. In the performance benchmarking I performed for this chapter and chapter 26, I observed these approximate relationships in performance among different languages:
Language Type of Language Execution Time Relative to C++ C++ Compiled 1:1 Visual Basic Compiled 1:1 C# Compiled 1:1 Java Byte code 1.5:1 PHP Interpreted > 100:1 Python Interpreted > 100:1
Clearly, the performance penalty for interpreted languages is extreme. How extreme? If you have to ask, you probably can't afford it.
One of the biggest stigmas for early Visual Basic developers was that our code wasn't compiled. It was interpreted. Interpreted executables were yet another reason so-called "professional" developers didn't take VB seriously. It was too slow. This finally changed when we got compiled executables in 1997 with VB 5.
The most commonly used interpreted language today, however, is JavaScript. And JavaScript is the very backbone of Web 2.0. How is this feasible if JavaScript is a hundred times slower than Java? Consider this ancient 1996 JavaScript benchmark page :
| 1996 | 2006 | ||
| primes | 0.15 | 0.02 | 8x |
| pgap | 3.13 | 0.06 | 52x |
| sieve | 5.05 | 0.02 | 252x |
| fib(20) | 2.15 | 0.03 | 72x |
| tak | 10.44 | 0.08 | 131x |
| mb100 | 8.4 | 0.2 | 42x |
In ten years, JavaScript performance has improved a hundredfold. But so what, right? Computers get faster every year. Well, our computers are now so fast that-- with very few exceptions-- we don't care how much interpreted code costs any more.
What many pundits don't realize is that the viability of interpreted JavaScript for mainstream applications is a relatively recent development. Consider this JavaScript benchmark of "ridiculously long algorithms and looping statements". The top three results are all of 2003 vintage:
| AMD 1900+ | 1.6 GHz | 12.25 sec |
| P4 Mobile | 2.2 GHz | 15.48 sec |
| P4 Celeron | 1.4 GHz | 17.43 sec |
The slowest computer I own, a 1.2 GHz Pentium M laptop purchased in 2003, completes this test in 13.64 seconds. The one I'm currently typing on completes it in around 7 seconds. So even in the last three years, we've almost doubled the speed of JavaScript.
I don't expect this trend of doubling performance to continue. I think JavaScript is about as fast as it can get now without some kind of really advanced dynamic compilation scheme. If you browse the results of BenchJS, a more recent JavaScript test suite, I think you'll agree that they've plateaued. We might reduce that from 6 seconds to 4 seconds over the next two years, but that's minor compared to the 100x speedup we've already had.
David nails it. I'd venture to say that 80% to 90% of perf problems today are a result of badly written code, not the language the code is written in.
There are common perf problems that can occur in any language. Like when you see someone load a dataset of thousands of IDs and then iterate through each one loading the record from the database one at a time instead of doing a bulk load of all the data in one query. That won't magically be quick because you used C++.
Haacked on February 8, 2006 1:29 AMcomputers are fast enough today that most problems are not going to be limited by the language you choose
That's right. And I'd say this has only really become true in the last 3 years-- we have enough brute force computing power to ignore the massive 100:1 perf penalty of interpreted code.
I remember a web app we wrote in 2003 that had hundreds of kilobytes of heavy-duty client JavaScript. Simply sorting a column of data with 50 rows took many seconds on the then-typical client P3-800 machines.
On today's 2.5+ gigahertz machines, that same operation would probably be instantaneous.
Jeff Atwood on February 8, 2006 3:02 AMGreat post as always, but reagarding this:
Well, our computers are now so fast that-- with very few exceptions-- we don't care how much interpreted code costs any more.
Doesn't it depend on where the code is interpreted? JavaScript is interpreted on the client-side so performance isn't a problem. But in a web app, Python code (correct me if I am wrong) is interpreted on the server. When the code is executed on a server with thousands of simultaneous users, poor performance can definitely have an impact.
Dave Delay on February 8, 2006 3:48 AMBut in a web app, Python code (correct me if I am wrong) is interpreted on the server. When the code is executed on a server with thousands of simultaneous users, poor performance can definitely have an impact.
What's happening, as others have pointed out, is that the overall percentage of CPU time spent on the language is very low now relative to the time spent on I/O. CPUs keep getting dramatically faster, but the disk, memory, and network interfaces aren't!
Given the pace of CPU speed innovation, eventually all languages-- even the 100:1 perf interpreted type-- will perform roughly the same. It's basically already happened as of a year ago.
Would a web app written in C++ scale better? Probably. But it's also pretty trivial to buy another inexpensive white box PC, build a web farm, and double the number of users your web app supports.
Consider the numbers. A web server built in 2006 will support 2x the number of users compared to a 2003 server -- and 100x the number of users compared to a 1996 server!
Jeff Atwood on February 8, 2006 4:12 AMI would be interested to see how IronPython ranks, since it is a dotNet language, and should, in theory at least, be compiled.
I notice that Delphi is STILL missing as if it wasn't a real language. That gets annoying.
As far as VB 5 being compiled, I believe that was just the ocx controls. Applications even to the last non dotnet vb version continues to be bytecode if I understand things correctly.
VB.Net, however, is a real language at last and I can respect it finally.
Xepol on February 8, 2006 4:29 AMThe Python result is a bit surprising, because Python code is also bytecode-compiled, so one has to wonder: what's the big difference with Java, for example? Maybe it's this: Python has a number of properties that are not the best choices if speed were the #1 concern -
(a) the __call__ interface is said to be slow, however I can't really find good explanations on this so I don't really know why. I think it's this because when you define a function object:
def foo(): return "bar"
you get a function object which has an object __call__ of type 'method-wrapper'. So when you call foo(), the interpreter calls foo.__call__() and somewhere along the way constructs a stack frame object.
(b) iterator objects throw a StopIteration exception when the iteration is to be terminated; this affects every for loop. So when you really need speed, you might need a while loop which increments the index into the sequence.
(c) variable lookup is faster when done in the stack frame than in an instance:
class C:
``myList = []
``def A(self):
````for a in range(1000000):
``````myList.append(a)
``def B(self):
````localList = myList
````for a in range(1000000):
``````localList.append(a)
Here, C.B() will run faster than C.A(), because looking up the object referenced by C.myList is quicker if it's done from the stack frame.
There are more of these little things. It's just a language that wasn't created with performance in mind.
Joost on February 8, 2006 7:52 AMJoost, there may be a misunderstanding of terminology.
.NET, Java, and (presumably) Python are first compiled into byte-code, that is true.
However, the byte-code generated for a Python program is then _interpreted_. That means every byte-code instruction is re-evaluated whenever it is executed.
The byte-code generated for .NET and Java, on the other hand, is _compiled_ to machine language, by a so-called "just-in-time compiler". That means, during the first run through a loop Java will be as slow as Python, but during all subsequent runs it will be much faster since machine code is being executed.
(C# and Java use the same JIT compilation, by the way; one shouldn't be listed as "compiled" and the other as "byte code".)
What's really surprising is how poorly Python fares even with the Psyco JIT compiler. I guess one guy can't be reasonably expected to do as well as the research labs at Sun and Microsoft! However, my own benchmarks absolutely confirm the posted benchmarks: Python really is 100x slower than C#, and Psyco is still ~10x slower.
That's for algorithms that are actually written in Python, of course... many so-called "Python programs" are really just stubs of Python scripts that call C DLLs for all performance-critical code.
Chris Nahr on February 8, 2006 8:05 AMNice as always, I smell a good discussion coming up. :)
Anyway, why is Visual J# and Visual C# performance not equal, I thought they where compiled to the same? But again I have no experience with J#.
I assume Viusal C# and Visual VB performs equally?
I am old java asp - C# programmer, but recently at my current job position I am developing VB.Net 2003. But I am finding several issues like Viusal Studio add-in functions that are only available in C#. This is not a request for another C# vs. VB.Net discussion, but does Microsoft priorities both languages equally? It is more because we had a discussion regarding a new project where I was hoping it would be implemented in C#, but since the company already has so much experience with VB6 they valued the switch to VB.Net would be more straightforward. But again I get the chance to develop VB.net. #61514;
Regarding Javascript, it is actually scary that not more focus is upon that language since it is being used more and more. Microsoft is shipping a new browser and all people talk about is tabbed browsing and html standards. Not that I want Microsoft to make more of their own Javascript and DOM extensions, they did enough damage on that front already. But as far as I know with my little knowledge is that Netscape and Sun made javascript, but are they still following up?
I think the biggest difference with Javascript/EMCAScript and scripting languages is that you don’t really know what they are doing. Normal programming languages can be debugged and disassembled; you can measure down to little pieces what is happening. And on the same time you have different browsers that run it differently. For instance IE6.0 has some bad memory leaks or stupid way of handle memory when it comes to JS where you have to restart the browser in order to get a clean run of your script. Ahh don’t get me started on that browser.. :(
On the other hand you also have the worst form of copy/paste among coders when it comes to Javascript. I hate watching ok developers copy/paste some Dreamweaver-Image-Mouseover script just because the language is “too messy”, “too clumsy” or whatever other excuses they come up with.
As a general comment, this attitude that we no longer need to care about efficiency since computers are so amazingly fast today has been around basically ever since there were computers. And there has always been an opposite attitude that every cycle counts and we always need new features that require better efficiency.
The truth is really somewhere in the middle. Even on a 4.77 MHz PC interpreted batch files were "fast enough"... for SOME tasks! On the other hand, even on a multicore next-generation game console C/C++ will still be mandatory since the games are supposed to look even better than the previous generation.
As computers got faster, the number of tasks that could be done with less efficient (but more convenient) tools increased. But usually this also requires a better infrastructure with more and more powerful libraries that are written in C/C++.
The Python/C example above is paradigmatic: how fast do you think your JavaScript would be if the browser commands you're using were also implemented in JavaScript? JIT compilation is now so good that most of the .NET/Java libraries are themselves written in C#/Java but interpreted languages still can't do that today.
Chris Nahr on February 8, 2006 8:14 AMSorry the language, I am not much of a english writer. Anyway with Visual VB i ment Visual VB.Net.
peter Palludan on February 8, 2006 8:14 AMGreat stuff! I'm also very surprised by the Python results. That would definitely make me think twice before using it.
In the old days I was always amazed at just how fast PowerBuilder used to run. It is a byte-code/interpreted language as well. Yet you could write pretty nice applications on machines as low as 486's and still get great performance. I wonder if you took Powerbuilder and put it up against the likes of Java what kind of difference you would see?
matt on February 8, 2006 8:55 AMC# looks good:)
gaech on February 8, 2006 8:59 AMIIRC and FWIW, Basic compilers just stuffed the interpreter and the p-code in a file with .exe on the end. don't remember how long that went on.
w/regard to the JS limits: my javakiddies decided to pull the sort capability from the jsp pages we built. we're "side grading" to Spring/iBATIS/etc but the reality is that our customers won't go out and buy new machines to run our stuff. sorting 1,000s of rows in IE leads to hangs, or so the users think. AJAX style does it faster.
As a recent Python convert (coming from C#) I am neither surprised nor concerned about the results. There are so many other factors that _should_ go into choosing a language. What about learning curve, tools, libraries, community support, readability, etc?
As for performance, what kind of app are you writing? A web app where database access and transmission of the HTML are going to be a couple of orders of magnitude slower then any of these languages? A tool for parsing documents where native regex support is of primary concern? A monte-carlo simulation where you are floating-point bound?
Looking at performance in a vacuum is a fun discussion exercise but without a context to evaluate all of the criteria I think its not that useful.
I agree with Jeff's main thesis: computers are fast enough today that most problems are not going to be limited by the language you choose. Rather the architecture, design and quality of code will have a much more dramatic effect.
While its true that the relative performance between scripting languages and C will remain large for a long time, so many apps are limited by UI input, network performance, I/O and other components that have _not_ followed the dramatic increases in speed of the CPU, that focusing on this one issue is a little silly.
David Avraamides on February 8, 2006 12:59 PMthe choice of a language involves many factors, and performance is just one, and probably pretty low on the list
And as of the last 2-3 years, performance hardly matters any more in practical terms. That's why I cited the benchmarks above. So if it was low on the list of priorities before, it should have practically fallen off the bottom by now!
Jeff Atwood on February 9, 2006 2:08 AMI notice that Delphi is STILL missing as if it wasn't a real language. That gets annoying.
That would have been a good point a few years ago but by now, I think we can consider Delphi a dead language, thanks to Borland's tireless efforts to kill it in the marketplace...
Chris Nahr on February 9, 2006 2:38 AMAs with so many benchmarks, there's value in looking at the details. The benchmark at kazdan.com (a href="http://kazdan.com/current/technology/javabenchmark.html)"http://kazdan.com/current/technology/javabenchmark.html)/a is really measuring the performance of the script to repeatedly update the values of fields in a form (hidden fields in this case) of a web-browser document.
The inner loops of this benchmark update the hidden form fields 'cycle' and 'count' of the 'benchmark' form. If those field updates are changed to updates of regular global variables, the benchmark reduces to below 1 second performance - about 1/50 of the the speed if the benchmark is updating fields.
Now, benchmarking field updates may well be of interest to someone, but it should be clear that this benchmark is not measuring looping, math, or algorithms.
Unfortunately, we don't have the historical data to see how well the older machines ran the script when modified to update non-document-field variables.
Regarding Server Scalability, I think the real measurement you want to look at is something like cost per 1000 users.
For example, writing something in C++ instead of Python might mean you need one less server to scale out, so you save the cost of a server and associated licenses.
But the cost to write that C++, with equivalent security, robustness and feature list, will WAY more than outweigh the savings of that solitary server.
Haacked on February 9, 2006 4:06 AMEric Lippert has written about performance at various times, and he notes that performance isn't interesting in the abstract; it's interesting as a measure of how close you are to your particular goal. He also notes that if you're going to optimize -- ie, worry about speed -- not only should you know what you're shooting for ("how slow is good enough?"), but you should also start by measuring, and then fixing the things that are the slowest. As has been pointed, the execution speed of any given lines of code is probably nothing in comparison to I/O speed, wire speed for Web pages, database speed, interacting with OMs/DOMs, and the drag effect of one's dumb algorithms. :-)
It's also been noted, albeit less directly, that the choice of a language involves many factors, and performance is just one, and probably pretty low on the list. Ease of development, deployment, maintenance, and appropriateness to the problem (no one is programming DHTML in C++, for example) probably count for a lot more. It's hard to imagine a situation in which a group of programmers sits down with a chart like this one and uses that to decide what language to code in. I suspect the real benefit of these types of comparisons is to enable people to say "Whew, I'm glad to see that my language isn't the slowest one."
mike on February 9, 2006 5:18 AMit should have practically fallen off the bottom by now!
Yet people are still weirdly fascinated by comparative performance, and some folks occasionally publish benchmarks! :-)
mike on February 10, 2006 10:59 AMNot all the devices accessing the Internet are high-end powerful desktops. Many of them are also PDAs, Smartphones, Set-top boxes, etc.. with a limited memory and 200MHz to 400MHz processors.
On the other hand, it is true that hardware performance improves constantly but requirements also are constantly evolving. Initially JavaScript was used for user input validation and DHTML, but nowadays, we are talking about real-time 3D JavaScript engines and JavaScript web servers :-0 .
So, IMHO I think performance and efficiency always matters.
mac on May 23, 2007 1:44 PMIt is amazing what happens when you leave your old projects on the web for a few years.
Eventually it seems that even the most remote sites are seen by a few people.
I wrote that code the only way I knew how at the time. I must say, it sure is painful to see now, but at the same time, I remember being so proud that I was able to finish that project.
So why did I make it?
I was trying to help my mom out with a problem.
In 2003 she had a work laptop from 1999 that was not going to be replaced. It is the 333MHz that is listed in the chart. Now a 4 year old computer is normally not “that” bad. However, as the world moved on to Windows XP, a 333 MHz machine was not going to cut it.
For reference as to the speed change in 4 years: A year earlier I had bought a P4-M 2.2 GHz and only a few months after I was running a P4 3.2GHz machine.
My goal was to show how bad a Celeron 333 MHz was in comparison to current machines. JavaScript was pretty much the only language that I knew well enough to write this type of program with an interface. I wasn’t going to code some C/C++ stuff up and hand my mom a program run at the command line, and I didn’t own a Visual Studio License to make one in Visual Basic.
In the end before she had the chance to use it to prove her case she got some side work that required a laptop right away, so a Pentium M was bought.
None the less, the program was finished and I chose to leave it be.
If nothing else, it is sometimes fun to look at an antique and wonder why someone did it that way once upon a time.
Since then I have managed several web based corporate solutions and written a few apps here or there.
Only a few have ever been polished up, my most recent one was: http://errormessage.kazdan.com
Before you judge it, understand that the goal is to give someone who is not a computer expert a fighting chance to find the solution to an error code. It is about 95% complete at this point.
Oh, just for fun I ran a 2.4GHz Core 2 Duo with 2GB ram running Windows Vista Ultimate on the benchmark test. It was completed in 12.146 seconds in IE7. I suppose I should have probably closed the other 15 or so apps I have open… oh well.
Well this is all interesting but see again it matters what you do, let us teake the other extreme example: For me speed is often paramount. Im mainly dealing simulation and rendering. And yes it does not matter much if one frame takes 1 hour to render, because as you say i can buy more boxes. Howevr remember that the 1 hour represents the quite optimized.
Now consider me taking a 10 times hit. me render time that already is in 24h/machine per second of output. Now a feature film has 60*60*120 seconds roughly. So we are talking of 1183 computer years versus 11830 computer years.
On the other hand theres a lot of extra time for many other tasks it doesn't really matter if i waste one second here and there to draw nice load metrics or roboustity checks that can well be even 200X speed hit stuff if its easy to do since it don't matter in the whole picture.
the J on February 14, 2008 6:28 AMI prefer php. Easy, powerful and fast.
Buy metformin on May 4, 2008 11:32 AMAs far as Psycho JIT compiler for Python goes, I imagine he hasnt been able to implement all the crazy tricks that most JIT engines have in them. Steve Yegge covers some of them in one of his blog posts. (See the five slides on JIT Compilation)
a href=http://steve-yegge.blogspot.com/2008/05/dynamic-languages-strike-back.htmlhttp://steve-yegge.blogspot.com/2008/05/dynamic-languages-strike-back.html/a">http://steve-yegge.blogspot.com/2008/05/dynamic-languages-strike-back.html/a">http://steve-yegge.blogspot.com/2008/05/dynamic-languages-strike-back.htmlhttp://steve-yegge.blogspot.com/2008/05/dynamic-languages-strike-back.html/a
Basically, with a JIT compiler going, you can do real-time analysis of how the program actually runs, and change the program to take advantage of optimizations based on that. This ends up being more effective than the compiled language optimizations, since those are based on guesses. If Psycho JIT could do that in-line optimizations that the JVM can do, then we'd probably see Python have drastic speed increases.
JMJ on August 14, 2008 11:35 AMIf we take out the 'long' test, we see a rather different picture: the performance of non-Python languages varies by more than a factor of 3 (instead of just under 2), and with Psyco, we now catch up to within a factor of 2 of Java.
However, I'm not just fiddling with the numbers. There's solid justification for getting rid of this test from the data: it is unfairly biased against Python because *in Python, 'long' is an arbitrary-precision data type* (equivalent to the library BigInt class in Java, for example) whereas in the others, it is simply a larger integer primitive. The authors claim to test 64-bit integer arithmetic, but there is no such type in Python.
It occurs to me that, as the test is described, the other languages would not actually calculate the same result. When the code gets to the first multiplication, it should evaluate 10,000,000,002 * 10,000,000,003, which won't fit in 64 bits (since each operand - deliberately, mind - doesn't fit in 32).
Karl Knechtel on August 19, 2008 1:51 PMInteresting read. The staggering performance loss associated with Python intrigues me, in particular. The reason: I've been closely following the development process of one CCP hf, an Icelandic gaming company which owns a multiplayer online game which hosts up to fourty-five thousand players at peak time.... And its server is written entirely in Python O_O;;
Five years old, now, the game has scaled admirably in light of all the hardware improvements that've occurred. However, it's really struggling to keep it's head above water since Python's stackless implementation has really poor (read: non-existent) support for system-level multi-threading and multi-core process support. CCP hf has been beating their heads against the glass for about a year now, and I would be surprised if they haven't come to regret their decision to deploy an interpreted language solution back in 2002. They can only expand vertically so far - they've already got a tremendously powerful server farm; and Python limitations are preventing them from expanding horizontally with multi-core support. CCP has ended up looking at alternative solutions like infiniband to fill in for Python's shortcomings.
again. would you be willing to run this test again.
Sexy costumes on June 30, 2009 10:20 AMI think performance still matters, we shouldn't use new hardware as a crutch for producing sloppy, poor performing code.
I think one thing to be considered is the cost of maintaining code. C++ will be more difficult to maintain than VB.NET code, so there is a trade off for the speed. Also, each language can be considered a tool, pick the right tool for the application. The good news is whether you pick VB.NET (VB), Java, or C#, performance should be relatively the same. For most business applications, any of these languages should suffice. As the processing demands increase, you want to get closer to the hardware, so that's when C++, C, or even assembly come into play. These will be specialized applications that demand the highest of performance.
I think it boils down to what you know. If you have been using VB for a long time, no need to learn Java, just develop in VB.NET. So, the path you choose for the next application should be with your strengths with previous applications, because performance doesn't matter between C#, Java, and VB.NET.
Jon Raynor on February 6, 2010 9:46 PMWould you be willing to run this test again and include Google Chrome?
Jason on February 6, 2010 9:46 PMThe comments to this entry are closed.
Subscribe in a reader
Subscribe via email
|
|
Traffic Stats |