programming and human factors
by Jeff Atwood
In The Day Performance Didn't Matter Any More, I found that the performance of JavaScript improved a hundredfold between 1996 and 2006. If Web 2.0 is built on a backbone of JavaScript, it's largely possible only because of those crucial Moore's Law performance improvements.
But have we hit a performance wall? Is it possible for browsers to run JavaScript significantly faster than they do today? I've always thought that just-in-time optimizing (or even compiling) JavaScript was an unexplored frontier in browser technology. And now the landscape has shifted:
Perhaps browser teams will begin to consider JavaScript performance a competitive advantage. The last time I looked for common JavaScript benchmarks, I came away deeply disappointed. That's why I'm particularly excited by the SunSpider benchmark: it's remarkably well thought out, easy to run, and comprehensive.
It's based on real code that does interesting things; both things that the web apps of today are doing, and more advanced code of the sorts we can expect as web apps become more advanced. Very few of the tests could be classed as microbenchmarks.It's balanced between different aspects of the JavaScript language -- not dominated by just a small handful of different things. In fact, we collected test cases from all over the web, including from other benchmarks. But at the same time, we avoided DOM tests and stuck to the core JavaScript language itself.
It's super easy to run in the browser or from the command line, so you can test both pure engine performance, and the results you actually get in the browser.
We included statistical analysis so you can see how stable the results you're getting really are.
Maciej Stachowiak, a member of Apple's WebKit team, graciously explained what each subsection of the benchmarks do in the comments:
| 3d | Pure JavaScript computations of the kind you might use to do 3d rendering, but without the rendering. This ends up mostly hitting floating point math and array access. |
| access | Array, object property and variable access. |
| bitops | Bitwise operations, these can be useful for various things including games, mathematical computations, and various kinds of encoding/decoding. It's also the only kind of math in JavaScript that is done as integer, not floating point. |
| controlflow | Control flow constructs (looping, recursion, conditionals). Right now it mostly covers recursion, as the others are pretty well covered by other tests. |
| crypto | Real cryptography code, mostly covers bitwise operations and string operations. |
| date | Performance of JavaScript's "date" objects. |
| math | Various mathematical type computations. |
| regexp | Regular expressions. Pretty self-explanatory. |
| string | String processing, including code to generate a giant "tagcloud", extracting compressed JS code, etc. |
SunSpider is the best JavaScript benchmark I've seen, something we desperately need in an era where JavaScript is the Lingua Franca of the web. I was so excited, in fact, that I ran some quick benchmarks to compare the four major players in the browser market:
What surprised me here is that Firefox is substantially slower than IE, once you factor out that wildly anomalous string result. I had to use a beta version of Opera to get something other than invalid (NaN) results for this benchmark, which coincidentally summarizes my opinion of Opera. Great when it works! I expected Opera to do well; it was handily winning JavaScript benchmarks way back in 2005. The new kid on the block, Safari, shows extremely well particularly considering that it is running outside its native OS X environment. Kudos to Apple. Well, except for that whole font thing.
If you're curious how each browser stacked up in each benchmark area, I broke that down, too:
If you need greater detail-- including variances-- you can download my complete set of SunSpider 0.9 results as a text file.
If I've learned anything from the computer industry, it's that competition benefits everyone. Here's hoping that a great JavaScript browser performance showdown spurs the browser teams on to better performance in this increasingly crucial area.
Nice work, you've presented the data well.
I work on a Javascript-heavy application and would definitely like to see performance improve. My applications deal with a ton of data and are struggling a bit on the common computer at my workplace.
The IE string result is curious. Any idea why it's happening?
Eam on December 19, 2007 7:21 AMJeff,
Can you better define what each of the benchmarks means? For example is 'access' proper I/O or parsing the DOM tree?
If 'string' means 'parsing any stringified data' that means that its probably involved in building the DOM tree from the [X]HTML which means that it is one of the more important aspects of the benchmark.
What does '3d' mean? z-positions? Proper 3d rendering? Making buttons look roundy??
The point is that these tests appear to cover a significant part of the javascript libraries but fails to identify which ones are used more often on the web. I would hope that most browsers implementations of javascript are optimized for the more common web use-cases.
A benchmark set up to measure the performance of javascript "as it is commonly used on the web" might show us more useful results. Properly identifying the benchmark terms would be a good step at the minimum.
Freiheit on December 19, 2007 7:24 AMNow we know why code refactoring was one of the primary concerns for Firefox 3
Jim Robert on December 19, 2007 7:33 AMHow come you didn't mention Tamarin? It's been making huge news lately.
FYI, Adobe donated a JIT compiler for ActionScript (a very close relative to JavaScript and ActionScript) to Mozilla. It'll be a major feature of Mozilla 2 and JavaScript 2 support.
http://en.wikipedia.org/wiki/Tamarin_%28JIT%29
James Justin Harrell on December 19, 2007 7:39 AM@James Justin Harrell: At this point Tamarin isn't relevant as it cannot run the majority of the JavaScript available on the Internet. It may interesting in the future, but it's worth waiting until it is viable as technology before making noise about it.
@Freiheit: The benchmarks are purely about JavaScript. There is no DOM access, rendering or network access.
Mark Rowe on December 19, 2007 7:46 AMFreiheit, I too wish there was more documentation and explanation of each test.
Here's a complete list of the tests:
var tests = [ "3d-cube", "3d-morph", "3d-raytrace", "access-binary-trees", "access-fannkuch", "access-nbody", "access-nsieve", "bitops-3bit-bits-in-byte", "bitops-bits-in-byte", "bitops-bitwise-and", "bitops-nsieve-bits", "controlflow-recursive", "crypto-aes", "crypto-md5", "crypto-sha1", "date-format-tofte", "date-format-xparb", "math-cordic", "math-partial-sums", "math-spectral-norm", "regexp-dna", "string-base64", "string-fasta", "string-tagcloud", "string-unpack-code", "string-validate-input" ];
To load each one, add it to the URL like so:
http://webkit.org/perf/sunspider-0.9/3d-cube.html
Then simply view source; each test is contained in an embedded script tags in the page.
Jeff Atwood on December 19, 2007 7:54 AMI'd love it if you could run the code on Firefox 3 beta 2 and add that to your graphics.
David on December 19, 2007 8:11 AMAnyway, do performance in such small part (but yet important) of the browser matter? In the end it's how many seconds it takes from the point that I press enter to actually render the stuff on the screen. In that respect, I believe that the rendering process is actually what needs more attention.
Why measure just javascript? Measure everthing that it takes to put a webpage in my monitor. You are seeing with the eyes of a developer, as you said programmer should see their software as the user would see.
Hoffmann on December 19, 2007 8:11 AM@Mark Rowe:
If I follow you right this is disconnected from how Javascript is used in the browser. For example if I have javascript change the style on a particular element it is just fiddling with some variable that the browser provided for it. That kind of access is no different from setting some arbitrary value; Javascript is just doing some processing and sticking the result somewhere.
Thanks for the clarification, I abhor JavaScript but I'm kind of interested in this discussion because it shows how better coding of the JS engine can make a browser better.
Freiheit on December 19, 2007 8:22 AMHoffmann, performance of js matters because whole web 2.0 is about huge js libraries which perform still quite slow to be widely used.
lubos on December 19, 2007 8:23 AM
The results more or less bear out my own experiences. I've found Opera to be the fastest gun in the West with regard to both rendering and script processing. Sadly, Firefox is the worst. Hitting sites like the ExtJS libraries, or scriptaculous's site, etc. Firefox on both Linux and Windows stutters and struggles. Even with html/css it is terrible. I have a page that uses a UL/LI vertical menu with a little opacity applied and position:fixed. I get "tearing" when the page is scrolled on a machine with 4GB of RAM and a Quad-core CPU! Other browsers are smooth as butter.
I tested Firefox 3 beta 1 the other week and it was noticeably faster but still stuttered and teared. I hope they can improve. As it stands, I use FF for tools like Firebug but Opera for browsing.
Foo Kung on December 19, 2007 8:31 AMI second the other David's request to have FF3b2 benchmarked, so far it seems much faster for me.
David on December 19, 2007 8:50 AMI second the other David's request to have FF3b2 benchmarked
OK, I'm running some benchmarks here on my home PC, so why not. It's Vista x64, 3.2 GHz Core 2 Duo.
IE7 32-bit -- 17100 ms
IE7 64-bit -- 15909 ms
Firefox 2.0.11 -- 10768 ms
Firefox 3 b2 -- 8260 ms
The specific improvements from FF2 - FF3 in each area:
3d -- 12% faster
access -- 21% slower (!)
bitops -- 47% faster
controlflow -- 11% faster
crypto -- 7% faster
date -- 65% faster
math -- 5% faster
regexp -- 14% faster
string -- 6% faster
http://www.codinghorror.com/blog/files/sunspider-benchmark-results-ff2-vs-ff3b2.txt
I will never understand why Firefox doesn't offer a 64-bit version. I'm sure it'd be even speedier!
Jeff Atwood on December 19, 2007 9:57 AMGreat article, one question though, what did you use to generate those beautiful graphs?
Fred on December 19, 2007 10:00 AMJeff: If you're taking requests, I'd be curious to see how the latest WebKit nightly build (http://nightly.webkit.org/) stacks up against the other browsers on Windows.
Hoffmann: This benchmark focusses on one part of the functionality of the web browser. There are other benchmarks that cover DOM access, page loading and rendering. A benchmark that covers every piece of functionality in a web browser provides information that is incredibly complex to analyse and thus very difficult to use in a practical fashion to optimise the browser.
Freiheit: Setting a variable and setting a property of a DOM object appear syntactically similar but are vastly different in terms of implementation. Setting a property on a DOM object typically results in the rendering engine being required to update the layout, repaint a portion of the screen, etc. The performance of these operations is obviously of great interest to developers, both of web applications and browsers, but are not the focus of SunSpider. There are other benchmarks that cover these areas.
"I will never understand why Firefox doesn't offer a 64-bit version. I'm sure it'd be even speedier!"
Well, it is open source..\
Just like Fred, I have seen some visually appealing graphs in your posts. Can you please share what you use to generate these graphs? Is it a charting tool from a spreadsheet, perhaps Excel?
jaywalker on December 19, 2007 11:45 AMActually anomalous is exactly the correct word for it: anomalous -- inconsistent with or deviating from what is usual, normal, or expected.
Looking at the other 8 sections, all the browsers are in the same range of time. The string test _was_ incosistent with all of the other tests.
billy on December 20, 2007 1:00 AMIn my testing of Firefox 2.x and IE 6.x on the AJAX applications I have developed for my client, I have found that Firefox is 2 to 5 times faster for real world operations that load XML data, transform it via XSLT, and render lots of DOM elements (DIVs, IMGs, TABLEs, etc.) as a result. My client uses IE only but I develop using Firefox and when I test under both environments I really notice the difference in performance. IE 6 sucks. I have not tested under IE 7 since my client has yet to upgrade to it.
Mark Carson on December 20, 2007 1:04 AMIf you could run this on Vista SP1 when it is released, that'd be great. SP1 improves on JavaScript performance in IE7.
Ed on December 20, 2007 1:07 AMI fail to see how this benchmark is useful for anything but "intellectual curiosity". Of what use is it to leave out DOM manipulation and events, the one thing that almost all code is about unless you write a scientific application that's more busy "on the inside" compared to any other (web) applications, that by far spend ALMOST ALL their time manipulating the interface - i.e. CSS, DOM events???
Thanks for the work, I'll keep a bookmark to your site - I may have no right to be so negative (not having paid a dime or done anything else for you), but I decided to post because few others did and I really fear a lot of people, especially the less technical but more marketing ones, might misinterpret those results.
Michael Hasenstein on December 20, 2007 1:12 AMHere are the results I got on my MacBook (1.83 Ghz Intel Core 2 Duo - 2 GB 667 Mhz DDR2 SDRAM Mac OS X 10.5.1)
IE7 (Parallels - XP Pro SP 2)
94 s
Safari 3.0.4
22.3 s
WebKit 5523.10.6
10.0 s
Firefox 2.0.0.11
36.7 s
Firefox 3.0b1
22.2 s
Opera 9.25
NaN s
That said, I have heard of people working around it by shoving all their strings into an array and joining them all at the same time, but that's quite a pain when you have to go to such lengths just for one browser.
I've been doing this for a bit now, and I feel the code is actually easier to read then doing string concats. I also fail to see how it's a pain. maybe 1 extra line, regardless of the number of concatenations.
That aside, I haven't encountered a real-world scenario where the string bug would cause any major side-effect; perhaps only the slightest pause on a reasonable machine. I mean really... who's doing tens of thousands of concats?!?! The access benchmarks were much more alarming to me.
Another good example would be Redfin which is actually AJAX based real estate portal. If you search a locality, it shows a map with location of all the results and below the map it shows details of each result. This table it constructs by appending several hundreds of strings. - MSDN JScript blog
Several hundred times for said functionality? Quite minimal compared to 20-30k times.
However, I'm curious: does this mean that when you are doing DOM manipulation (createElement, appendChild, etc) that you are indeed only creating strings and using concatenation to append them to some node in the DOM? that seems rather ridiculous really.
I think the graphs you made are great. The reasons people have listed as to why they dislike them seem to lack any thought in my perspective. Thanks for the article! Kudos.
eNtropicChild on December 20, 2007 1:17 AMI think it's absolutely typical for our web period to ignore all kind of performance issues: Use big image files, link to thousands of lines of javascript code and make a website as bulky as possible. Let's start to remind real world example. If this is a high-end machine, don't even think of what a normal machine would need. Maybe it's time again to change minds?
Stefan on December 20, 2007 1:19 AMHi Jeff,
I tried out the benchmark on my system on different browsers including Flock and Firefox 3 beta 2, and found them to be much faster than Firefox 2 (barring IE7 due to the string test). The interesting thing is that Flock 1.0.3 is based on Firefox 2, but is still over 10% faster overall. I have recorded my results on my blog (http://abaditya.wordpress.com/2007/12/20/firefox-3-beta-2-javascript-benchmark-plus-why-flock-is-faster-than-firefox-2/) and also published a Google spreadsheet with my results (http://spreadsheets.google.com/pub?key=pCUHgnruBnILDFJ7TXTEMQg).
Michael Hasenstein: The point is to help optimize browser javascript engines. Utility to the public (if there is any) is a side benefit.
As a direct result of the existence of this benchmark, the current nightly builds of WebKit have immensely faster javascript execution. If the data had been cluttered with other parts of the browser it would have been much less clear where optimizations were needed.
Other benchmarks can (and have!) been created to test rendering, layout, DOM mutation, etc...
David Smith on December 20, 2007 1:21 AMDavid Smith: Je suis d'accord (Agreed). I said it my last paragraph why I was posting. Had there been no Digg-/Slashdot-effect, attracting the masses, I had kept quiet (following my last but one sentence). I'm all for what is being done here for development! But with all the attention... okay, no reason to repeat myself.
Michael Hasenstein on December 20, 2007 1:50 AMAMD Athlon 64 3200+ Laptop Windows XP Pro x64 Edition 2 GB Ram
Firefox 2.0.0.11 x64 15542.6 ms
Firefox 2.0.0.11 Official 26503.6 ms
IE 6.0 32-bit 47474.0 ms
IE 6.0 64-bit Crashed in controlflow-recursive
Opera 9.25 13537.8 ms *
Opera 9.50 Beta 11470.2 ms
Safari 3 Beta 13295.4 ms
* Opera 9.25 had problems with fasta, dna and format-tofte so I added up the
other times and came to 11134.6 ms. I added in the numbers from Opera 9.50
for the three failure cases on the guess that 9.50 should be faster than 9.25
and came up with the result as shown.
It should be interesting to see what a Firefox 3.0 x64 build can do. I haven't
had the time to setup the build environment for 3.0 but maybe sometime next
year.
Quite amazing that this was even published. A very quick look at the string function routines will quickly show that they are using non-standard calls, and IE doesn't support them. Once the javascript is corrected, IE outperforms firefox.
A javascript string is not a simple character array, and should not be accessed as such. Many calls in the suspect script (Written by Mozilla), attempts to do so with calls like "data[i]", when it really should be calling the function charCodeAt. Once you change all the invalid statements like "data[i]" to "data.charCodeAt(i)", the code runs fine in IE.
Robert McKee on December 20, 2007 2:05 AMrlk (December 20, 2007 10:45 AM)
Out of curiosity, are you doing all changes in the DOM live? Because that will be tend to be ridiculously slow as each change is going to cause the rendering engine to kick in and reflow the document. If you are doing them live, try doing something like take the root node of where you are going to make the changes, pull it out of the DOM (removeChild()), make all your changes, then put it back in.
zoips on December 20, 2007 2:06 AMYou will be assimilated.
yoyoma on December 20, 2007 2:12 AMThe bitops-bitwise-and test appears to really be a test of JavaScript global variable access speed. The same test code wrapped in a function with all local variables is blindingly fast in Firefox on a Mac in comparison. I was getting 53ms instead of around 2900ms. So the test does show a performance issue but not what it claims to be testing.
Tim on December 20, 2007 2:15 AMSlightly off topic...
I heard a podcast on conversations network by one of the main guys at Google, talking about their toolkit that allows you to write your JS app in Java (strong types, debugging etc. etc.) and then cross compile it into optimised JS. I think that if I were going to write a large JS app I'd give this a good look, much richer and easier to develop a large app without so much overhead, and the toolkit handles the browser incompatibilities and JS optimisation without you having to worry about it.
Also, JavaScript slows things down in another way: simply loading it as text over and over again. The new version of Rails has some configuration options that gather up the Scriptacolous and other stuff into one zip file automatically. David Heinemar Hansen says that this has given significant performance improvement at 37 signals, almost for free. (cf another podcast on conversations network where he talks about this).
Francis Fish on December 20, 2007 2:17 AMZoips, I will need to take a look at it. That could account for the problem.
rlk on December 20, 2007 2:24 AMAs some other posters observed, the stacking chart format obscures some details - it gives the overall performance, but you can't see the key areas where there are anomalies between the browsers.
A quick charting exercise in Excel reveals:
1) IE - strings[*], base64, validate-input, and tagcloud are the weaknesses
2) Firefox - 3d, bitops, bitwise-and, date are the weaknesses
3) Safari - no obvious areas of weakness - on par with the average, and occasionally better than the others.
4) Opera - similar to Safari, except tends to be better more often than on par.
The only 'weakness' for Opera is unpack-code, but the (relative) poor performance here is far outweighed in the grand scheme of things.
[*] See Robert McKee's observation above about the test validity
Running the test on Leopard, Firefox 3b2 is slightly faster than Safari 3, and both are substantially faster than Opera; admittedly I'm using Opera 9.23 so may not get optimal results, but both FF and Safari come out ~1.5 times faster. Safari, strangely, took longer to complete the test -- there was an inexplicable delay between finishing one section and starting the next, but the individual sections ran quickly...
Mat Hall on December 20, 2007 2:47 AM@Robert McKee
Good catch. In a future version of the benchmark, we'll avoid this construct (and also add checking for correct output from the tests). However, I think this affects onlly the string-base64 test, not the other string tests. Please let me know if you find other bugs in the tests.
Maciej Stachowiak on December 20, 2007 2:47 AM"If Web 2.0 is built on a backbone of JavaScript, it's largely possible only because of those crucial Moore's Law performance improvements."
Or maybe we're seeing these huge improvements precisely *because* so much of the web uses javascript? Seems a lot more likely to me.
Pete on December 20, 2007 3:50 AMJeff said, "I had to use a beta version of Opera to get something other than invalid (NaN) results for this benchmark, which coincidentally summarizes my opinion of Opera. Great when it works!"
I would have to agree with that - every time I've tried Opera, I've been disappointed - and I WANT it to work! However, that isn't the reason I think Opera is going to fail. The reason Opera is going to fail is amply demonstrated when I try to use gmail, Google Notebook or Google Spreadsheets from Opera (I am running on Linux - haven't tried lately from Windows) and am told it is unsupported. THAT is what makes Opera irrelevant - if Google is explicitly writing their apps to not support a browser, then that browser is for all intents and purposes dead, even if it doesn't realize it yet.
Jim on December 20, 2007 3:59 AM32 sec and 40 sec in Firefox 3 beta 2 on an old Celeron 2.2 GHz with 1.24gb RAM, vs 60-65 sec in Firefox 2.0.0.11 on the same box.
ANgus S-F on December 20, 2007 4:02 AM@Maciej, Robert:
"charCodeAt(i)" != "[i]"
"charAt(i)" == "[i]"
The second implies string alloc where the first returns an integer. I expect it'll be a wash.
Regards
Alex Russell on December 20, 2007 4:04 AM@Alex:
Thanks for pointing that out. However, I do think it will make some difference in IE, because string[i] returns undefined, so they're not actually executing the intended benchmark and probably something weird and bogus is happening. I will publish 0.9.1 soon with some fixes to the tests.
Maciej Stachowiak on December 20, 2007 4:25 AMSince people have asked, here's what the test categories are about:
3d: Pure JavaScript computations of the kind you might use to do 3d rendering, but without the rendering. This ends up mostly hitting floating point math and array access.
access: Array, object property and variable access.
bitops: Bitwise operations, these can be useful for various things including games, mathematical computations, and various kinds of encoding/decoding. It's also the only kind of math in JavaScript that is done as integer, not floating point.
controlflow: Control flow constructs (looping, recursion, conditionals). Right now it mostly covers recursion, as the others are pretty well covered by other tests.
crypto: Real cryptography code, mostly covers bitwise operations and string operations.
date: Performance of JavaScript's "date" objects.
math: Various mathematical type computations.
regexp: Regular expressions. Pretty self-explanatory.
string: String processing, including code to generate a giant "tagcloud", extracting compressed JS code, etc.
What's not in the test: It's true that for a lot of JavaScript code on web sites, the bottleneck is more in DOM operations and layout than JS per se. There are interesting benchmarks available for that sort of thing, but we wanted to keep this one focused on core JS processing, which does matter for many operations on the web. Bottom line page load time benchmarks are also interesting, but do not substitute for this kind of testing. In many cases, the performance of JS hits you when doing various operations on a Web 2.0/AJAX type site after the initial page load. In any case, we may come out with other benchmarks in the future.
The second chart is horrible. I don't think Tufte would approve. It makes it kind of hard to compare the different browsers
Which is exactly the point of the FIRST graph, directly above it. Sheesh.
I think you missed the point, Jeff. The information shown might be correct, but the style of graph is inappropriate. A regular bar graph (different browsers side by side) would be better than a stacked bar. Nobody really cares what the total effect of running a string process in all of the browsers sequentially would be, because nobody would do it.
Otherwise, interesting post.
Rob
Rob on December 20, 2007 4:31 AMI don't like the way you've stacked your graphs. Makes it really hard to compare visually the different individual tests. And does that axis jump from 6 to 16 seconds? nice!
cris on December 20, 2007 4:44 AM@Jeff
"2) It is useful to know what general class of activities in JavaScript are slow, and that's what the bottom graph shows -- string operations and bitwise operations tend to be the slowest activities in JavaScript. Just scan the relative colors to see which browser was good (or bad) at a particular class of operations in JavaScript."
I don't think this benchmark is suitable for figuring out what classes of activities in JavaScript are slower or faster than others. I tried to roughly balance the individual tests, but some categories have more tests than others, by design. So, for example, you can't conclude that "regexp" stuff is faster than "access" stuff; there are more separate "access" subtests, by design, since object access is somewhat more common than regexp processing.
Maciej Stachowiak on December 20, 2007 4:44 AMI have use Array.join when concatenating strings. I've used more than one platform that exhibits the IE style slowdown when repeatedly concatenating strings with operator+, whereas Array.join has always performed well enough, with fairly linear performance degradation (although it is not the fastest in some cases).
@eNtropicChild
I with you on the syntax front... arr.push wins over str+=otherStr for me
spender on December 20, 2007 4:48 AMI did a quick test in IE... concatenating a 7 char string 20000 times:
with string concatenation (+=) :1340ms
with Array.push/Array.join :62ms
From your article: "The browser market is more competitive than it has been in years, with Opera 9.5, Firefox 3, Safari 3, and IE 8 all vying for the coveted default browser position."
I just wonder how come you\re saying IE8 is competing to anything while it's not even out? Actually, it's not finished, i haven't even seen any betas yet. I'd argue that as long as it's not an actual product in peoples hands, it's not competing with anything.
I love this blog but you're saying some "odd" stuff lately. I hope it was just a typo.
Kris on December 20, 2007 4:58 AMWhy hate on the font thing? What do fonts have to do with performance? It's a good article except for that comment.
sapphirecat on December 20, 2007 4:58 AMI think you missed the point, Jeff. The information shown might be correct, but the style of graph is inappropriate.
OK, fair enough. Valid points, all. I changed the chart.
Jeff Atwood on December 20, 2007 5:01 AM1) The top graph and bottom graph are mirrors. So if you don't like the bottom graph, look at the top one.
They are both terrible graphs, but at least the top one gives you a visual indication of the overall performance of each browser in the total set of tests.
The bottom graph may as well be a table of numbers with '70s-ish colors splattered on it. The visual cues are at best misleading, and generally don't help a bit. After studying the numbers a while you can figure out what it's trying to tell you, but it misleads you in almost every way possible (for instance, the numbers on the bottom of the graph don't help because the browsers are stacked one after another with no relation to time). My initial impression of the first bar on the graph was that opera was the slowest and firefox the fastest on the string test, in opposition to the first graph. Only be reading the numbers and taking in the rest of the graph did I realize that the position of the browsers was the same across bars and had nothing to do with the result of the test in question.
It would have been better if the bar's full length was that of the longest time and the browsers ordered from fastest to slowest, essentially merging the more standard display of 4 bars in a group (one next to the other) into 1 multi-colored bar that still had a relationship to the scale at the bottom of the graph, rather than stacking them end to end in the same order on each line.
Vizeroth on December 20, 2007 5:06 AMbah, figures you'd change it while I was typing (I was slow). Too bad I don't really find much of interest in the actual benchmarks, though, other than the usual cautions that come along for the ride when someone releases a benchmark for their own products. At least, for once, Apple actually gives you the tools to run the benchmarks for yourself. They used to be pretty terrible about posting benchmark results with little or no information to make them repeatable.
Vizeroth on December 20, 2007 5:09 AMI just did the test in Konqueror. Too bad I can't actually compare them to anything, since my Firefox just complains that some script is taking too long to execute and then it wants to kill them. Is that how it's supposed to work?
Andreas on December 20, 2007 5:36 AMMaybe it's because I've been reading Edward Tufte lately, but I don't think the second graph makes much sense at all. Browsers are stacked together, yet add up to a total time? Just in case I wanted to see what would happen if I gave the same task to all browsers and see their total? And the scale jumps from 6s to 16s, corrupting the data view - if a comparative graphic is not going to use a consistent scale, the data would be better presented as a table.
At minimum I would stack the bars for each browser. As it data displays now, I don't know what I'm supposed to get out of it.
Chris on December 20, 2007 5:58 AMSlashdotted.... :-)
Jesse on December 20, 2007 6:17 AMYou've misspelled "Xaprb" in the test name.
Tim McCormack on December 20, 2007 6:43 AM@Francis Fish
Are you talking about the GWT (Google Web Toolkit)?
We are in the process of building an app using the GWT, and it is going well. Basically, Google provides several widgets for Java to replace the standard swing components.. they provide a pretty complete set of controls, and as long as you use those controls in your app, you can convert the project to javascript.
Wayne on December 20, 2007 6:44 AMInteresting...
I just ran it on a few machines here and was intrigued by the results. When I get more time I'll do a write-up on my blog, but until then...
Out of a Sun v250 (UltraSPARC w/Solaris), HP server (P-III w/ Win2k3), and an Apple Powerbook (PowerPC), and a whitebox x64 running Solaris (all very different boxes indeed!):
1) Safari on the Mac
2) Firefox on x64 Solaris
3) Mozilla on SPARC Solaris
4) Mozilla on x64 Solaris
5) Firefox on SPARC Solaris (threw errors during the test)
6) Firefox on Windows (threw errors during the test)
7) IE on Windows
Zoips, thanks, that helped a good bit. Even so, 1400 rows takes about 8-12 seconds to sort. The Perl code that generates these reports takes a fraction of a second to sort and print out (including all formatting) that many rows.
rlk on December 20, 2007 7:05 AMThere is a browser war going on, but it's not the front end, it's the rendering engines. The contenders are really Microsoft (Silverlight and .NET), Adobe (Flash), and WebKit (JavaScript/Ajax).
In a few years, most of the "webpages" will be displayed on handheld devices that don't even seem to be running a browser (much like the iTunes store). The idea that the contest is between IE and FireFox is much like thinking that the battle for supremacy in the American luxury car market is between Lincoln and Cadillac.
Right now, Microsoft is having difficulties getting its act together. .NET was never really a rendering engine technology and Silverlight has the same problems as Flash: You need a proprietary client in order to see its content. That may be possible when you are only talking about x86 based chips and only two major operating systems with one having 90% of the market. When you have thousands of devices, each with its own chipset and maybe its own OS, you simply cannot depend upon proprietary clients.
Apple/Google are banking on WebKit and AJAX. WebKit is already in Mac OS X, the iPhone, and is on Android. Google is also banking on it for all of its stuff. KDE has now adopted WebKit over its own KHTML (which WebKit was based), and the GTK+ project is also adopting it too. That puts WebKit firmly into Linux which means it'll be in all Linux based devices like setup boxes, phones, and PDAs.
I am not too sure what Microsoft is going to do. Next year will be very tumultuous for them. As the price of hardware drops, it gets harder and harder to convince PC makers to buy Windows licenses on all PCs they manufacture. Linux desktops are becoming more consumer oriented, and you can't beat the cost of a Linux license. Combine that with OpenOffice, and you can offer a PC for about $100 less than a competitor. When you're talking about $200 PCs, that's quite a bit of money.
Microsoft also hasn't done very well with Windows Mobile. Windows has been difficult to scale down to handheld devices, and they've pretty much lost the Setup box market. Scientific Atlanta, which has a near monopoly on the market, has gone solidly behind Linux. The iPhone, which hasn't even been out for the whole year yet, has outsold all Windows Mobile smartphones combined.
Microsoft isn't going to go away, and I am sure they have plenty of things they're working on. They're a smart bunch of people, and I'm sure they see the same things I mentioned. Bill Gates has always said that Microsoft is just an innovation away from irrelevancy which is why he always fought so hard for Windows's monopoly. My guess is that the OM license price for Vista Home Basic will drop to less than a dollar by the end of next year. That will encourage PC manufactures to stick to Windows over Linux. I suspect that a Microsoft Home Basic Edition is also in the works too and will have a similar OM license price.
The big question is how Microsoft will handle the non-Desktop market. Right now, they're way behind, but Microsoft has been in this position before with Netscape, and came out on top stronger than ever before. Can Microsoft do that again?
David on December 20, 2007 7:14 AMThis is a fun excersize, but I dispute the usefulness of the results to the real world.
What do we use Javascript for in our web applications? Making 3D cubes? Doing crypto work? No. We use it for the following tasks that are not tested in the "benchmark":
1) Ansyncronous and syncronous HTML GET calls.
2) DOM calls to the page structure for getElementByID() and setting style properties
3) DOM updates like new Image().
4) Responding to events like onClick() and onLoad()
The string benchmarks are the most disparate, and other than the examples I mentioned, they are where most of the processing time will be spent. Regular expressions (input validation) and validating input are the only really useful javascript metrics that I see here.
Chris Chubb on December 20, 2007 7:31 AM@David: I don't think you really understand what rendering engines are. please read this article http://en.wikipedia.org/wiki/Layout_engine before commenting on something you're not competent about.
@Jeff: thank you for the interesting analysis.
Matas on December 20, 2007 7:33 AMI think Tufte might like the first chart and the second... you are looking at two distinct angles on the same data set.
The first is the relative performance of each benchmark with the suite by each browser—you get to see where each team needs to spend more time to maximize performance... the Webkit team would do well to work on "access" and "bittops" while any effort "controlflow" would yield only a marginal impact on improving the overall score.
The second chart does a pretty darn good job at comparing which browser does the best/worst—the relative performance of a benchmark between browsers.
Nice work Jeff! I'd love to the first chart for my browser when I run SunSpider!
George Mckinlay on December 20, 2007 7:46 AM@Chris Chubb
Actually, people do crypto in JavaScript for real. Here's an example: http://www.clipperz.com/. Many of the other benchmark test cases come from real code that people reported as slow, either in Safari or other browsers, or from real JS libraries. And yes, as JavaScript gets faster, it will become more commonplace to use it for games and 3D code.
You are right that on many web sites, DOM/layout performance matters more than core JS. Maybe we need better DOM benchmarks, too. But it is useful to be able to analyze performance of different engine components separately.
Maciej Stachowiak on December 20, 2007 7:51 AMPlease take into account how long it takes to load the browser...
A cheap browser like Konqueror can load and display a page while firefox is still opening and loading as an app..
Jeff,
Little off the topic question. Can you tell me how you generate those graphs? Any specific tool or macros etc. I am a big fan of how you present information in your blog and would love some tips.
The second chart is horrible. I don't think Tufte would approve. It makes it kind of hard to compare the different browsers.
njkayaker on December 20, 2007 8:07 AMCanFDG'ssuperlonglinkspleasebefixed?Thanks.
transcriber on December 20, 2007 8:44 AMprepare to get slashdotted ;)
@Aditya Banerjee: did you try Firefox with a clean profile? It's entirely possible that errant extensions that you don't have in your Flock install would be causing a 10% drop.
Robin on December 20, 2007 9:46 AMWhile I commend you for taking the time to do these tests, the graphs at the top are a poor way to present the data. I don't think people really care to compare how long one benchmark compares against another, they care how the various browsers compare.
I suggest you swap the browser and test data sources in your bar graphs so that people can more easily compare how well browsers fared against each other.
Cheers
Rob on December 20, 2007 10:01 AMi'm looking forward to better js debugging tool. as of now, it's seriously lacking.
jin on December 20, 2007 10:05 AMAlthough Safari is based on KHTML could you run some benchmarks
with Konqueror to see how well it compares against its Apple raised cousin?
@Mat Hall: Opera 9.25 and 9.50 use different scripting engines, and the results are much better for 9.50:
Linear_b is used in Opera versions up to and including 9.25: http://en.wikipedia.org/wiki/Linear_b_(script_engine)
Futhark is used in Opera 9.50: http://en.wikipedia.org/wiki/Futhark_(script_engine)
Arve on December 20, 2007 10:19 AMWhy hasn't anyone complained about IE7 having one of it's tests thrown out? Obviously it's a test that the other browsers can handle and should not be thrown out for IE. Has anyone actually looked into why IE7 does so bad on the string test?
David C on December 20, 2007 10:24 AMScatter shot comments follow:
As someone who has done a lot of javascript, I agree that the string result for IE is likely very typical.
And with the bitops, I'm surprised that the other browsers were able to do so much better than Firefox since bitops in JavaScript are inherently inefficient.
And I agree with some other posters: the second graph is pretty but remarkably bad, making it very easy to compare the runtime of different benchmarks (apples and oranges--useless!) as totaled across the browsers (double useless!) and very difficult to compare the browsers to each other.
Leaving off the strings (unfairly, since one does an awful lot of string manipulation in many apps), Firefox is about 30% slower than IE. That really ain't much, and as soon as you do some string work that takes almost 9 times as long in IE, that 30% vanishes fast.
Most important, Firefox has the most complete and correct implementation of JavaScript of the browsers, and although I'd like it to be faster, on balance I'll take the power of the most complete JavaScript implementation over the 30% speed.
@jin, Firebug?
JavaScript is also good for artificial intelligence.
Mentifex on December 20, 2007 10:32 AM@Robin My tests were not under a controlled scenario as I have noted in my blog post. Also, Mike Shaver (http://abaditya.wordpress.com/2007/12/20/firefox-3-beta-2-javascript-benchmark-plus-why-flock-is-faster-than-firefox-2/#comments) has pointed out in my blog comments that the Flock JS engine is not any different from that of Firefox 2. In fact I tried out the tests on another PC (Athlon 64 3200+, 1 GB RAM) both in the safe mode and with extensions enabled for Firefox 2 and Flock, and found the results to be similar (FF2 was slightly faster this time). I have updated my post (http://abaditya.wordpress.com/2007/12/20/firefox-3-beta-2-javascript-benchmark-plus-why-flock-is-faster-than-firefox-2/) with this note.
Aditya Banerjee on December 20, 2007 10:41 AMI'm using JavaScript to do client-side manipulation of large tables (filtering, sorting, and hiding columns). Performance is good up to maybe 100 rows with 20 columns, and it's usable up to a few hundred rows, but it really bogs down beyond that. This is an internal application, so I can't actually post a link to it.
Some of the reports are as many as 9000 (9 *thousand*) rows, at which point things really bog down badly (we're talking minutes for sorting here). The actual sort still takes less than half the total time required to sort on a column; the rest of it is mostly related to actually reordering the rows in the DOM and rendering the result.
Filtering isn't as bad, but all the string manipulations (I'm using Firefox on either a 2.2 GHz P4 running Linux or a 1.8 GHz UltraSPARC-IV+ running Solaris; the difference in performance isn't going to affect the usability to any great degree) still add up.
The reason for this client-side data manipulation is to allow users to generate customized reports.
rlk on December 20, 2007 10:45 AMAs the main author of the SunSpider benchmark, I don't think it makes sense to discount IE's results on the string tests. As far as I can tell, what the string tests do is sane and realistic, and they are showing a genuine performance problem in IE, in types of code that are pretty common on the web. As you can see in the detailed results, more than one of the string tests does much worse than other browsers.
After reading various complaints about the Opera browser, I have to insist that I have very few issues running Opera today. This includes all contemporary websites such as: Gmail, Goog. Reader, Digg, Hotmail, Y!Mail, etc. NO ISSUES and no complaints. The price of greatness is the occasional stumble. I have been using Opera as my main browser since 6.x, and it has improved in compatibility across the web ever so much. Granted, there are times Firefox is necessary - if something doesn't appear right in the O, use FF. and if all else fails, try IE.
Additionally, I successfully ran the sunspider test on Opera (v9.24) in Linux without getting the NaN errors Jeff mentioned. [ result is 10.7 s ] (Macbook Pro 3rdGen SR 2.2C2D, 2GB, Ubunutu 7.10)
Nate on December 20, 2007 10:57 AMAnd I agree with some other posters: the second graph is pretty but remarkably bad, making it very easy to compare the runtime of different benchmarks (apples and oranges--useless!) as totaled across the browsers (double useless!) and very difficult to compare the browsers to each other.
Disagree.
1) The top graph and bottom graph are mirrors. So if you don't like the bottom graph, look at the top one.
2) It is useful to know what general class of activities in JavaScript are slow, and that's what the bottom graph shows -- string operations and bitwise operations tend to be the slowest activities in JavaScript. Just scan the relative colors to see which browser was good (or bad) at a particular class of operations in JavaScript.
Also, please do NOT post ultra-long URLs in the comments. I'll be removing those. If you want to post your results, use TinyURL or another shortening service.
Jeff Atwood on December 20, 2007 11:21 AMJeff Atwood on December 19, 2007 09:57 PM wrote:
I will never understand why Firefox doesn't offer a 64-bit
version. I'm sure it'd be even speedier!
This is a Windows x64 Firefox build for XP. Some run it on Vista too. Perhaps you could give it a try. Official Mozilla Windows x64 support will arive with Firefox 3.0 as there apparently is a customer requirement for this.
Michael Moy on December 20, 2007 11:32 AMThere was an article about IE's string performance in the Jscript team blog on MSDN a while back : http://blogs.msdn.com/jscript/archive/2007/10/17/performance-issues-with-string-concatenation-in-jscript.aspx
Aaargh! on December 20, 2007 11:34 AMIt would be very interesting to do the same tests Jeff did but comparing just IE 6 and IE 7.
Adiel on December 20, 2007 11:44 AMThe second chart is horrible. I don't think Tufte would approve. It makes it kind of hard to compare the different browsers
Which is exactly the point of the FIRST graph, directly above it. Sheesh.
Jeff Atwood on December 20, 2007 11:45 AMAs mentioned above, the "anomalous" result of IE is not that anomalous. In some testing we have done on JS performance, IE6 and IE7 are both perform horribly when handling strings over a non-trivial size. And, while that is avoidable in some cases, the fact is the string data type gets quite a bit of use.
Here's hoping that IE8 addresses the issue....
Peter Wagener on December 20, 2007 11:46 AMBased on the colors, the font and the style(s) used, I'm guessing the graphs were produced with Excel 2007.
Ear on December 20, 2007 12:02 PMReally, is "anomalous" really the right word to describe IE's poor string performance? It appears to be an acknowledged bug that's not fixed yet, not a mistake in the test. I'd have called it lackluster, personally.
That said, I have heard of people working around it by shoving all their strings into an array and joining them all at the same time, but that's quite a pain when you have to go to such lengths just for one browser. Of course, IE has always been like that to develop for...
Anomaly on December 20, 2007 12:07 PMAfter the digg effect, you might be hit by the slashdot effect now, Jeff ( a href="http://slashdot.org/article.pl?sid=07/12/20/1640234"http://slashdot.org/article.pl?sid=07/12/20/1640234/a ). Good job :)
could you add firefox 3 beta 2 as i think you will find it is much better
james on December 20, 2007 12:30 PMThe browser market is more competitive than it has been in years, with Opera 9.5, Firefox 3, Safari 3, and IE 8 all vying for the coveted default browser position.
All this competitiveness just means juggling with lots of incompatibilities; a href="http://www.quirksmode.org/dom/w3c_html.html"http://www.quirksmode.org/dom/w3c_html.html/a and more. While an XMLHttpRequest is rather universal (and even that took long enough), placement and rendering of elements isn't, and most mobile browsers (which support a half-assed featureset) make this even worse, because you have to scale back your design to 1998.
We'll have to deal with the spectre of IE6 for a few more years, I'm afraid.
Rob Janssen on December 20, 2007 12:36 PMonce you factor out that wildly anomalous string result.
I doubt it's anomalous. In fact, I think it's a direct consequence of MSIE's awful garbage collector and garbage collection scheme. I haven't looked at the tests but string probably generates a lot of junk objects and MSIE's behaviour degrades *extremely* badly once you reach a precise threshold (in number of objects in memory I think) although I don't remember that threshold right now.
All this competitiveness just means juggling with lots of incompatibilities
Since 95% of the incompatibilities come from MSIE and limit the JS programmer to a restricted set of features which are usually well supported cross-browser by everybody else, that usually isn't much of a problem (and I'm serious there, especially since Safari 3. Safari 2's JS support sucked balls but Safari 3's is very good)
Masklinn on December 20, 2007 12:45 PMHave they improved the font rendering in the newer safaris? I am currently running it on xp because it's font rendering is so much better than msie or firefox. (maybe it has to do with screensize/resolution? It looks really great on 15inch 1920x1200)
Andrej on December 20, 2007 12:49 PMHey there,
interesting article - you should have a look at xquery xqueryp and xscript - will rule out javascript in the near future.
Jeff: FF is usually distributed 32-bit, even on 64-bit OSes, due to Flash and other plugins only being distributed 32-bit. FF works very well when build 64-bit.
Nick on December 21, 2007 4:34 AMflavour (December 21, 2007 01:58 AM)
Perhaps you've heard the oft-quoted joke about XML and violence? I think that strongly applies to Xscript.
zoips on December 21, 2007 9:32 AMJeff, do you know an anything about IE JScript engine? JScript 5.6 or JScript 5.7? What did you test?
enternet on December 21, 2007 9:45 AMFirefox does have a faster rendering engine though, which in turn balances some of these slower results.
Ryan on December 21, 2007 10:45 AMThe comments to this entry are closed.
Subscribe in a reader
Subscribe via email
|
|
Traffic Stats |