That's why we have Wirth's Law:
"Software is getting slower more rapidly than hardware becomes faster"

Can that salary chart be right? A programmer with less than a year of experience makes more than one with 1-4 and even a little more than one with 5-9 years of experience? So if that is right and you are looking for a job just leave your resume blank until you have 10 years of experience!

I think that he is judging this as a cost via employer with benefits:

http://swz.salary.com/salarywizard/layouthtmls/swzl_compresult_national_IT10000010.html

Breakdown:

25th percentile: $46,047
50th percentile: $52,141
75th percentile: $59,139

Breakdown WITH benefits:
50th percentile: $76,448

That salary chart only has 23 samples recorded for the "1 year" experience guys vs 2000+ samples for the other bands.

It's a statistical glitch.

While I agree in principle, I'd caution against assuming that the cost of adding a server to a hosting farm is equal to the cost of buying the hardware. First there's the supporting infrastructure - more servers mean more storage, which means more backups, just to name one element.

On top of that, more servers mean more work for the systems people, whether you have them on staff or pay a hosting provider. A well designed infrastructure architecture (e.g. good automated configuration management) will mean you can scale your number of servers more quickly than your staff, but it's not insignificant.

How can the top chart be correct? After that I stopped reading, if all the conclusion were based on an incorrect chart, then why bother with the rest of the article?

Yes, the salary chart is right. That's because it's based on "newly employed programmer with x years experience" - driven by job ads.

And a newly employed person with no experience includes a small number of superstars who have just left college (and not just with a BS - some will have a masters or even a PhD) who lift the average, but no-one advertises for $100,000 jobs for PhDs with two years' commercial experience.

On high volume low cost products (a typical consumer product embedded system for example), smart coding for cheap hardware definitely pays dividends. Often the performance is limited by the need to run from batteries.

Jeff is using the wrong chart. The chart he is using is a Sr. Software Engineer/Developer/Programmer.

Here is just non-Sr Chart link

http://www.payscale.com/research/US/Job=Software_Engineer_%2f_Developer_%2f_Programmer/Salary

There less than 1 years is 57k, with 20 years or more at 81k.

Ok, it is 5:40 on the west coast. Why are you awake!?

Also. I have had a couple of employers who wouldn't spend a dime on their developers, designers or anything. I recently came to a very amazing company where each employee received a brand new computer and individual licenses for CS3, VS 2008, VS 2005, Vista Ultimate and a few others. Now that is awesome!

> Also. I have had a couple of employers who wouldn't spend a dime on their developers, designers or anything.

Obviously this is a very bad sign, and I'd avoid working for companies this dumb.

View's shared by some very smart people.

"Does Django scale?

Yes. Compared to development time, hardware is cheap, and so Django is designed to take advantage of as much hardware as you can throw at it."

Oh, you forgot: "Make it scalable". No sense in trying to spend your way out of misery if you code uses only 1 core...

We are waiting to receive our brand new workstations. My old one is no slouch and I'm still going to take a few months of transition before I finally turn it in. In the meantime I'll finally have something to run Vista. My previous machine could, technically, run it but the performance was underwhelming.

For programming I think it's easy to throw more hardware at the problem. Almost every IDE I use is memory/CPU hog. When it comes time to ship I keep an old P4 around just for testing. While my company has the resources to buy good new hardware regularly a lot of our users don't have the same capabilities.

>> Also. I have had a couple of employers who wouldn't spend a dime on their developers, designers or anything.
>
> Obviously this is a very bad sign, and I'd avoid working for companies this dumb.

That's easy to say but something I have encountered several times. Production hardware is coming out of one budget, the money for the "standardized" hardware & software out of another, so the developers get stuck with the same hardware as everybody else. Trouble is that you often only get one machine (because everybody else doesn't need more than one machine either) so you get to run a virtual machine on top of your OS and dev stuff to replace the test box you used to have 10-odd years ago.

IME this is something that mostly happens in bigger companies where the bureaucracy of acquiring additional hardware or software can be so bad that some programmers (OK, me for starters) will cough up for some tools themselves. But that's software obviously.

Sadly, sometimes, somewhere, a low-end office computer - just about the cheapest you can get - costs the same as a programmer's monthly salary.

I'm personally a little bit more expensive than that (just a little), but getting any kind of hardware (or desk, or chair, or software) is almost completely impossible.

Just like any good thing, this can be taken too far.

IME as a web developer, the single biggest performance problem stems from interacting inefficiently with a database, a poorly optimized database, or <shudder> both.

Now, however, I hear web developers tossing around maxims about the dangers of code optimization and using them as reasons to not change expensive database transactions.

> bigger companies where the bureaucracy of acquiring additional hardware or software can be so bad that some programmers (OK, me for starters) will cough up for some tools themselves

I ninja-stealth upgraded my own computers at a previous job where we had restrictions like that. I basically gutted the inside of the PC so on the outside it looked like a normal corporate box, but on the inside it had a modern mobo, PSU, CPU, hard drive etcetera. I just imaged the old drive over to the new one, and let it redetect all the new hardware.

Granted this is a little (ok maybe a lot) extreme, but that's how much I believe in giving programmers the hardware they deserve!

While I agree that companies unwilling to spend cash on developers/design is "dumb", jobs are not falling from trees anymore here in the SE US. It is a real gamble to drop your established job and pick up one with another company not knowing if it will be around in 12-18 months.

Also, can that pie chart on optimization be right? 14% sounds really high for micro-optimization. How many asm algorithms are the average developers running that 10+% performance boosts are available for optimizing them?

It seems like I just live in a backwards city where the technology companies can't do simple math like this. I've had sub-par hardware at every company I've worked for, and penny pinching managers reluctant to provide simple upgrades.

At both of my last two positions I had to fight tooth and nail to get a dual monitor set up, and at one they used smaller monitors, apparently out of spite. I had to buy my own ergonomic keyboard and optical mouse too.

Who knew they even still made ball mice?

I wish I was kidding.

> I ninja-stealth upgraded my own computers at a previous job where we had restrictions like that. I basically gutted the inside of the PC so on the outside it looked like a normal corporate box, but on the inside it had a modern mobo, PSU, CPU, hard drive etcetera. I just imaged the old drive over to the new one, and let it redetect all the new hardware.

I just bought my own Laptop, it had the hardware I wanted (for the development I do) and a second monitor output. Let work know, and told them that it will be used purely as a work machine (which it is) and until they get me a rig with the specs I want, then I'll just use the laptop.

Okay so it cost me money, and the company none. But I have pretty high spec'd laptop, that's my own, and if work supply me with the rig I want, then bonus, I've got a laptop to use at home.

You may want to read http://www.diovo.com/2008/10/the-funny-caching-problem/

Hey Now Jeff,

Great post & nice charts. It's good to see Coding Horror #3 on top 125 NOOP dev blogs for q4. Hardware is cheap.

Coding Horror fan,
Catto

I work in the games industry, in the short term at least your argument doesn't work for us. I like it that way.

Throwing hardware at a problem makes very good sense when talking about a hosted application. When you move to Desktop development, it doesn't make as much sense. Intuit can't expect all their users of Quickbooks to throw a couple hundred bucks at their machines every couple of years.

At my last job, we used quite a bit of in-house developed software. These were not (and could not be) hosted applications -- applications for drawing and marking up architectural drawings and things, software that had to run disconnected from the net, etc. It made more sense to our bosses to optimize code rather than upgrade 10,000+ machines.

But, it seems that no desktop developers exist anymore, or at least they don't read blogs and comment on them, so no one here will care. :)

Interesting distribution on the pie chart. For us it's been almost backwards, I think; recent algorithmic improvements gave us about a 20% improvement in throughput, and parallelization gave us a 300% improvement (no kidding).

This is for a specific process and I'm not sure how well it holds up with respect to general web design. For me, designing for scale has come to mean "having an architecture that will actually give you at least twice as much throughput if you provision twice as much hardware". Most of that lies in parallelism. If you don't have a well-distributed load, adding more hardware isn't going to help you.

This is true, but as always experience comes in to say when it isn't true.

In web development anyway, I've seen the case quite often that people do something really algorithmically stupid that will be needlessly expensive, big n or small n, with the 'hardware is cheap' argument when you know it'll be a problem. In one case I saw a nested loop that executed SQL that would do a query, and loop over the results of the query and do another query on top of that. Eventually the home page was doing 1500 queries, which 'worked' because of a cache, unless you were the lucky visitor that hit during expiration time. I found it because it took 15 minutes for the dev server homepage to load (since no customers were hitting it).

It took me 15 minutes to write the proper query, but more thought than what was there before.

So I think an additional consideration needs to be the complexity of the optimization. If it's easy, like pulling expensive operations out of loops if they don't need to be there, that makes sense to me and you should do it right in the first place. (I'm sure you'd agree with this). But if it requires a whole data and object remodel, buy new hardware first.

Never send a human to do a machine's job
- Agent Smith

I agree with Matt above:

"...as a web developer, the single biggest performance problem stems from interacting inefficiently with a database, a poorly optimized database, or <shudder> both."

Jeff's third step (Benchmark your code to identify specifically where the performance problems are) probably means: figure out which database queries are being run the most often and also taking the longest to execute? Then take them out of the damn for loop.

David is right above. This is the better salary chart:

http://www.payscale.com/research/US/Job=Software_Engineer_%2f_Developer_%2f_Programmer/Salary

But, regardless, Jeff's point is still valid.

All I have to say is that YMMV.

Take, for instance, a preprocessor program that removed duplicate CDRs before sending data on to billing, etc.

The program, which used to run very quickly, was now taking more time to process a CDR file than it took for a CDR file to be produced. A very non-optimal situation, of course. This was all CPU time -- no I/O or memory constrained the program.

Now, we could find a hardware twice as fast, but it wouldn't really be easy -- the hardware being used was already fast enough. Distributing wouldn't work either, unless you wrote a whole new application with a completely different processing algorithm.

But, let's stay with the hardware. Suppose we decided to take that route. That's 3 months before we can have the hardware ready to be used. I'm happy that smaller concerns could have the new machine in the afternoon, but a LOT of corporations out there just don't do things this way. Their loss? Maybe, but that's still a system we have to deal with.

Now, being unable to charge our clients for 3 months really isn't good for business. So, my manager decided to ask me -- not a member of the programming team, and a much more expensive resource -- to take a look at it.

Two hours later, I had a 5000% performance increase tested and ready to go. It just happened that the fields in each CDR were extracted through a pattern matching, and just a dozen or so fields in the first hundred or so were actually used. The total number of fields had increased from a couple hundred to more than 600 through the years, which resulted in a much slower matching. So I just changed it to ignore all fields above the last significant one before extraction.

And the moral of the story is: hardware maybe be cheaper than a month of a programmer's salary, but getting a good programmer to look at the problem for a day or so might be cheaper and faster.

I brought myself a Mac Pro with 8 cores and 10 Gig Ram to be able to virtualize Windows XP and Vista as many times as I wished. I couldn't be happier. :)

These observations might be valid for server based applications.
For desktop development, shrinkwrap software or industrial applications with a defined hardware environment, you still cannot replace senior development craftmanship with hardware.
Good that is :-)

good analysis, although I have to counter, "programmers are expensive relative to what?" Compared to server hardware costs it would seem expensive but have you considered the costs of providing a service like stackoverflow w/o software? like if stackoverflow was paper based for instance; then one would have the cost of ink, paper, printing equipment, transportation, labor, and etc. kinda makes programmers in-expensive.

i agree with this article whole heartedly.

in my experience, it is amazing how many managers and old programmers are out there that disagree on this subject.

On the other hand, sometimes premature optimization turns out to be the major selling point of a new product: http://alumnit.ca/~apenwarr/log/?m=200806#24

I must disagree with you today. You should optimize code, but not why you think you should not. There are three reasons:
1) Someone else will have old hardware, and it needs to run well for them too.
2) Scalability. You need it to be efficient, because although it might be OK to throw hardware at it and double the capacity when it is small, it becomes less reasonable when you become large.
3) Optimized code should be simpler to read and maintain (optimize by simplifying, not by clever hacks) saving future programmer time.

Your attitude is why Vista stinks as bad as it does. There is no reason that what it is doing should require the hardware specs it does. Other OS's do the the same with so much less hardware.

Throwing hardware at a problem, or benchmarking and refactoring the code, are no substitutes for having an experienced programmer design it right the first time. Demand may increase faster than hardware scalability. A good programmer can get it right early on in the development cycle, and will identity most "problem" areas before they become a problem, and will have a good idea about how to mitigate these in the future.

I see your point... and I disagree.

If Windows Vista was written as optimized as these demo programmers make their code, it would probably still run on hardware (without quality loss) that was already considered too weak to run XP.

If a process spends 5% more CPU time on doing something than it could, you will say, hey, who cares? But if you my system is running at 100% CPU and I have 10 apps that do something. As I said, all could use 5% less, that means 50% less for all apps together.

The inefficiency in your code sums up to the inefficiency of all the other code running on that system. It is a little bit like: Getting one cent from 100 people makes you just earn a dollar. Getting one cent from every human being on this plant makes you a rich man, but for every person involved, it is just one cent.

We ship a complete solution (hardware and software), so qualifying and upgrading hardware is a difficult process and causes an increase in COGS. We are in a performance-sensitive industry where you are never allowed to slow down, even when you add new features.

Mecki--good point. Generally sloppy practices don't show up as a single line item in the profiler. The code is too slow, but there's no nice pareto chart showing the one thing you need to fix.

I am expensive, therefore my salary is too low.

It seems like you cannot count on Moore's law anymore.

Until a few years ago, the CPU clock frequency increased all the time, but that development seems to have stuck at about 3.2 GHz. It's hard to find a CPU faster than about 3.2 GHz today, and that hasn't changed in over a year. And it seems like we have to wait for a while (for a technological breakthrough) before we can get faster CPU:s again.

Now we get more cores instead. But more cores doesn't automatically make your program run faster, it has to take advantage of the parallelism, and you may need to hire some expensive programmers to fix that.

Throwing more hardware doesn't quite work when you're talking about client side of things (e.g. desktop software or JavaScript)...

This is all well and good advice for server side stuff where the code is only running on a few machines. However, for client side code, where you have to multiply the inefficiency by a million, a few optimizations go a long way.

I'll also say that good programs tend to write better code to begin with, and that beginner programmers need to learn some of the basic optimizations and bottlenecks of programming. So my advice would be for beginners to spend a bit more time learning where their programs are bad so that they can become better.

@Dom - You ever hear of Vista?

Jeff,

You're arguing against yourself here. You start out by saying that the best gains are in optimization, then you say not to optimize, then you say to optimize only when your code isn't good enough. This hasn't worked for Microsoft and it sure as hell won't work for you.

Also, when I was looking at going into programming in 2002, according to the school counselor, the national average salary for a programmer was $50K. The average salary for a software developer was $55K. (The difference between a BS and a Master.)

I agree with your article Jeff, but you also have to be careful. In your case, adding hardware is an obvious choice. But what happens when you have multiple servers already running. Doubling your hardware power suddenly starts to become more costly...

For example, if you have 10-20 machines, each say at $5k. That's anywhere from $50k-$100k. It's now close to 1 year's worth of salary. And if your system is growing in traffic, well...

At some point, and it doesn't take too long, it becomes more interesting to look at software optimizations. For example a 1% optimization for Google in software will cost dramatically less than adding 1% of hardware.

Especially when you factor in the person time to build the machines, order them, maintain/update them, power them, etc.

You are in a sweet spot where it's much better to buy more hardware. Enjoy it while you can, it's a great place to be! And a fun one too :) But once you reach about a dozen or so boxes, you'll notice your metrics will flip on you.

Btw, I absolutely agree with getting the right hardware for developers. There's no nice way to put it, it's just plain stupid not to!

The world unfortunately is changing. Intel/AMD have both hit a wall on single core speed. Instead of getting about 2x performance every 18 months its about 20% in the same period and falling in growth every release. Processors are now caches with logic as they try desperately to increase performance with more transistors and many cores.

So I think the old rule of no optimisation is wrong, I think we need to make sure our applications will scale across multiple CPUs and in todays web applications that means being kind with the number of database hits we do and decent amounts of caching (that is until the database dies and is replaced with a better more scalable solution). That means giving developers multiple core boxes so they can test that.

One of the few applications capable of using the multiple CPUs efficiently is the application server. Generating web pages is easy to do in parallel (ignoring the database problem). Even this free lunch will soon come to an end. Consider what would happen if the CPUs in 10 years time had the same single thread performance as they do today. We face that very real possibility now and it means we're going to hit a new limit where adding new hardware won't help with increasingly complex and powerful applications.

Because single thread performance will remain the same and we currently in web apps marry one CPU to a user the maximum processing time per user is going to be 2 seconds. Users don't want to wait for a page much longer than that including transmission time so lets call that the maximum. Today we probably aim to deal with a user in much less than that, say 200ms purely because we run more than one user per CPU at any time. But with the apps of tomorrow we'll want to do more. Excpet we'll hit a wall at 2000ms when single thread performance won't increase further. At that point adding new hardware won't help, its all algorithms from that point on.

Scared yet? I know I am because most algorithms do not scale even remotely as well as we would like. Even an algorithm which is 99% parallel can only ever get 100x faster regardless of how many cores you throw at it. Something is going to have to change in the way we generate pages, the way we write GUIs if we are to use all these CPUs in a single application.

As the great wise man once said:

"We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil." (Knuth, Donald. Structured Programming with go to Statements, ACM Journal Computing Surveys, Vol 6, No. 4, Dec. 1974. p.268.)

Jeff, you comment is written from the point of view of a user. If you're creating an application for your own company, sure, hardware is much less expensive than programmers.

This situation is different from the point of view of a vendor. if you're creating an application which thousands of customers are going to use, *your* programmers are much less expensive than *their* hardware. You really don't want to make customers pay more for your application because it requires more hardware. (Vista is an extreme example, as an OS, but this is doubly true for vertical applications.)

For example my company Aperio makes digital pathology systems. These manage lots of really big images. If we didn't have efficient code our customers would have to buy more hardware - CPUs, disk, bandwidth, etc - and it would make our applications more expensive. We'd be less competitive and deliver less value. So for us spending programmer resources to reduce application hardware resources is the right trade-off.

BTW this discussion bears on the choice of language, too. If you're creating applications for yourself, sure, go ahead and use C#. If you're creating performance critical applications to sell to others, C++ might make more sense.

Jeff says, "If a one-time investment of $4,000 on each programmer makes them merely 5% more productive, you'll break even after the first year."

This isn't actually true for small consulting firms that bill large companies an hourly rate for programmer services. If my boss spent an extra $4,000 on me, he wouldn't see any of that money back, because every hour that I spend is simply money that he makes from our huge corporate clients. Being able to finish a project early is nice, but when you have clients who think nothing of projects going over time and over budget and will just keep paying as long as it takes, then there's little incentive to make sure things don't slip.

Your point about making developers want to keep working for you is good, but as long as the hardware I've got is reasonably decent, it's not going to make a difference as to whether I switch jobs.

Sure hardware is cheap, but sysadmins arn't.

I worked on a large project where the hardware is cheap card was played early on. We ended up with a system with 20 servers...

Oh but the system was replicated across two sites, we had a qa system, the customer had a qa system and there was a development system...so actually 100 servers got bought.

Then they needed setting up and running. We had a team of 10-15 people racking hardware, wiring and installing OS and configuring the system for well over a year.

In all honesty I think we could of put the system on 3 servers and saved a lot of time and money.

Many of the comments here bring up, in different ways, that the economics of this aren't just that clear. For my tiny one-person company, for sure I solve just about any problem by doing a few mouse clicks and buying hardware. But for my clients for whom I do embedded product development, it totally depends on the cost of goods sold (COGS) and number of products shipped. Sometimes saving pennies on hardware really pays off in high volume applications. However, I also go with what I read decades ago in the the first edition of Kernighan and Ritchie: first make it work, then make it fast.

My thoughts are to write code that performs as fast as the customer expects it to perform. A minimalist. If the code is well written and useful, then someday the application will move to faster hardware and perform better.

I would argue that the cost of insufficient hardware is actually higher, because if a developer has to wait a long time for long-running operations that should be interactive (e.g., refactoring, rebuild, run, restart the environment), he's more likely to do a quick switch to the web/mail/coffee and this will end up actually taking longer.

Also, developers may be afraid of anything that shakes the boat like refactoring. I never refactor on my old laptop, only when I'm on a fast desktop.

I think we get a little lost in the semantics sometimes.

Certainly the 'throw hardware at the problem' approach isn't a viable option in all scenarios, and it's never the ideal solution; but when it is a viable option it is often likely to be the cheapeast fastest short term solution.

It's unfortunate, sad even, that the state of development has come to this. Seems like every doubling of hardware power has been met with an even greater, pardon the expression, 'crapping up' of the software we run on it.

The developer as the most costly component of development has led to the development of RAD IDEs that do so much hand-holding that any schlep can pump junk code to market that somehow, accidentally or otherwise, manages to do most of what it's supposed to at the expense of hardware. 'Get it done' walks all over 'Do it well'. Dynamic memory allocation, automatic garbage collection, and that sort of stuff seemed kinda cool at the onset, but in retrospect I'd gladly give it all back if it would scare a few would-be developers away from the biz or make them clean up thier act.

Sorry, I'm a bitter angry man who has cleaned up a lot of sloppy source code in his time.

I understand your point Jeff, I got a laugh out of it; but I do agree with a few of your readers that indicated that often there are glaring issues in the code that a talented and experienced developer can quickly find and correct often leading to striking performance improvements.

So, by all means give 100 monkees VS08 and see what they come up with but don't publish it until you've had your more seinor prime-ape check their work to make sure it's not mucked up with their feces.

Wow I am bitter...

nice post. thanks.

Just one thing: no hardware will save you from code produced by stupid programmers.

So, it's not about hiring cheap programmers and buying expensive hardware, but giving good tools to good programmers, so they won't have to waste time doing unnecessary micro-optimizations.

> It seems like you cannot count on Moore's law anymore.
>
> Until a few years ago, the CPU clock frequency increased all the time, ...

Er...no. Moore's Law relates to the number of transistors they can cram on a chip every year, not to CPU clock frequency. Not only have they kept pace this year, but they are ahead of the pace.

Hypothesis: Harddrive space is cheap - no need to optimize our code.
Fact: MS Office for Windows 95/98 came supplied on floppy disks or and "all-in-wonder" CD that contained many EXTRAS as well... Today - it comes on DVD and doesn't contain near as many extras.

Your hypothesis is what has lead to multiple generations of code bloat. I am truly glad not everyone thinks this way.

Jeff Atwood wrote on December 19, 2008 06:12 AM
> I ninja-stealth upgraded my own computers at a previous job where we > had restrictions like that. I basically gutted the inside of the PC
> so on the outside it looked like a normal corporate box, but on the
> inside it had a modern mobo, PSU, CPU, hard drive etcetera. I just
> imaged the old drive over to the new one, and let it redetect all the > new hardware.

There was a similar policy at the place I worked for a few years ago. As I quit, corporate said everyone gets the same hardware regardless of job function, so as to save lots of money in tech support costs. There were a few developers who did the same thing, replacing the standard hardware with something better. Corporates' response was to mandate a system app to detect unapproved software and hardware and email tech support. The offender gets a similar email detailing the offense, with the addition that they will be fired if the situation is not corrected. One developer I talked to got those emails four or five times a day, soon after recompiling a company app. She quit soon after that, so I never found out if they managed to solve that problem.

Tim

1. The first rule of Optimization Club is, you do not Optimize.
2. The second rule of Optimization Club is, you do not Optimize without measuring.
3. If your app is running faster than the underlying transport protocol, the optimization is over.
4. One factor at a time.
5. No marketroids, no marketroid schedules.
6. Testing will go on as long as it has to.
7. If this is your first night at Optimization Club, you have to write a test case.

@`Josh
>View's shared by some very smart people.
>
>"Does Django scale?
>
>Yes. Compared to development time, hardware is cheap, and so Django is designed to take advantage of as much hardware as you can throw at it."

That isn't an argument against investing development time in optimisation. It's an argument for using frameworks where the development time has already been invested. To be scalable, someone must have spent time in optimisation (probably from the start at the architectural level). Django's core business is creating the framework, so it makes sense for them to spend developer hours optimising it. The user's of Django don't have to spend that time, and can scale out by hardware. Scaling out isn't the same as throwing hardware at a problem without thinking about optimisation bottlenecks.

There are other considerations apart from dollars - assuming it's linear with CPU load, a script implemented in Ruby uses 50 times the energy of an equivalent program written in (not particularly optimised) C. If economic measures come into force to reduce the world's carbon footprint, then the balance between cost and energy may be rebalanced.

Uh, yes. Programmer time *is* more important. That's why we write in high level languages now, not assembly.

Hardware is cheap, without programmers is trash.

Hmm, what and ideal picture you paint. The hardware I work with is more than adequate for my job - sure, it'd be nice to have cutting edge bits of kit and to not worry about how our services perform. WOMM (works on my machine), so I'm past caring.

No, rather I would work smarter and with a more efficient sense of mind, learning to eep out every tiny ounce of performance I can before I hit the 'Buy It Now' button in accounts. As another poster comments, telling customers to do the same would be the death of a company.

I agree with the Knuth quote and it's worth pointing out that people all too often only pay attention to half of the quote, forgetting that 3% of the time you actually should always be paying attention to the right kinds of optimization.

>> Always try to spend your way out of a performance problem
>> first by throwing faster hardware at it

Bullshit.

This should be preceded by:
1. Perf test your code
2. Profile
3. Optimize bottlenecks

only when the lowest hanging fruit has been reaped does it make sense to throw more hardware at the problem. Most of the initial perf issues are easily solved. It's only when you reach the area of diminishing returns that things become expensive.

I'll put those economics up your ass, write my web apps in C++, and run them on a Pentium IV.

I'm not being sarcastic or joking.

I agree with DMB. Throwing more hardware at the problem only masks the problem and it will rear its ugly head again, only this time with a lot more difficulty in making the true fix that should have happened in the first place.

"but no-one advertises for $100,000 jobs for PhDs with two years' commercial experience"

I would _NEVER_ hire stupid university people for IT jobs. Their knowledge and experience is nowhere compared to people who started programming as a hobby back in the 90's/80's and don't have a single certificate.

Look at the many middle eastern, Israeli monkeys with their CCNA, CISSP can administrate their 10 routers in their country lawlz.

Paying 100K for someone just to sit there every day with his 0 knowledge and phd is a waste. This is a profession what you don't learn in schools, but we all know that already.

i wish i could get a new pc at work. i'm running a pentium 4 with 2 gb of ram. it's painful to use.

Don't forget the cost of server licenses.

Microsoft's SQL Server licence policy is getting much tougher, to the stage where a busy top-range 8-way server can cost $80,000 or more. Per year. In license fees alone.

When you start to include the cost of a DBA to administer the hardware then alternatives involving more efficient code become very attractive.

My current employers are seriously considering throwing out their SQL Server installation and going with a /more/ code-intensive solution because of this alone.

How many $5000 MP3 players do you own? I mean, forget optimising, just add hardware... it weighs 10kg because there's a large battery to power the fan to cool the fast processor, but the software is simple.

Even with server systems it can be misleading, because the cost of upgrading client's hardware can be very, very high. Even ten servers at $5k starts to add up, when you have one per store in a nationwide chain it can get really ugly really quickly because roll-out costs are significant. I also have worked with people where clients would pay $100,000 for an upgrade rather than shell out an extra $5k for faster disks or more memory in their single licensed server. My conclusion was that they were running many, many more copies of the server than they were paying for but my boss convinced me they were just idiots. Sorry, "constrained by bureacrazy". Software upgrade budget A, hardware budget B... which one has money in it?

> If a one-time investment of $4,000 on each programmer makes them merely 5% more productive, you'll break even after the first year

... and their hardware will be out-of-date by then too.

Not disagreeing with post in general, though.

I think this is bullshit. I have seen developers working in such a style, and the effort was: You have a ten times slower system by letting them add features on a new hardware than the old system was on older hardware. And that for a system which is sold to customers. Customers who don't want to change a big IBM machine against a newer one, who think they can use their hardware at least until they returened the investment they made into it. Think about that. One "good" indian programmer gets ~36000 $. Isn't it cheaper to hire one for coding, one for optimization and one as spare part than one "buggy" american programmer at 100.000$ which no optimization and a need for more modern computers

So many people are getting this wrong. If you're selling to customers, then the cost of the hardware they require is multiplied by every customer, which is hopefully many times more people than the number of developers. But if you're providing the service to customers, then throwing hardware at the problem can work, and not only that, it is the best solution, short and long-term, within reason (if 99.98% of your time is spent on linear searches on sorted data, then an optimization would cost less in all metrics than a hardware upgrade).

The disk footprint of MS Office install discs is a good example. Why should they fit the installer in a few floppies? DVDs cost almost nothing on margin compared to smaller media. If we're using the installer size as a proxy for MS Office's actual installed size, the complaints makes more sense, although with that said: would you pay $10 extra for a version of Office that has 90% the HDD footprint? Or do you just on principle want optimal optimization against every metric (how soon it's released, stability, price, disc size, memory size, working set, alignment, lack of redundancy, robustness, CPU time, correctness, GPU use, I/O subsystem use, security, usability, maintainability, and computational elegance), even when they contradict?

I agree with the general thesis, however there's a place where your entire foundational logic breaks which is when you're creating libraries (like we're doing) for others to use. Since then you can multiply any performance gain with the number of users using your library which makes even the slightest performance gain grow insanely in importance...

Anyway, great writeup. BTW, I remember a friend of me kept the "number of simultaneous sprites" record in fact for more then a year I think ;)
"Ypsilon" from "IT"...
He was a couple of years older then me at the time and actually my "hero" (kind of) while we were "kids" in their views...

Though when we (ADD, later to become Power Lords - me == Polterguy) crushed them in a copy party demo contest later that year - we were all of a sudden "OK" to talk with ... ;)

Those were the times....

"Given the rapid advance of Moore's Law, when does it make sense to throw hardware at a programming problem? As a general rule, I'd say almost always."

No, not even close to "almost". Your neglecting the fact that most computer systems out there are embedded systems - where the rules change significantly. Desktop systems may be the most visible computers, yet I bet that if you take a close look at your household you'd find more microprocessors running your microwave, toaster, remote-control etc. than you have desktop/server systems. Now do the calculation again with 5 million units where the difference between a uC with 16K Flash and 32K flash may be as small as $0.02, gaining a possible cost reduction of $500,000. According to your chart that's the salary of five senior software engineers working one year around the clock tweaking the code size from 21K down to 14K to make it fit into the smaller uC with a couple of bytes to spare.

What about cases where the same software is deployed over large number of hardware? In this case it would be justified to spend more on the programmer rather than upgrading all the hardware.

Darn, I got the numbers wrong. It's just $100,000. Yet, that is still one year's salary worth of work for a 20+ year experience guy.

What about the performance cost of band-aiding a poorly chosen architecture?

Most of us don't make money by writing demo programs. I'd rank architecture choice as far more important than simply algorithm choices.

The problem is most "enterprise architects" seem to be people pulling a specific vendor's framework, etc. out of their hats. It seems that most people have forgotten what architecture really means.

You must be doing a Christmas joke,

For badly structured code no amount of hardware and power use will break even with a GOOD Developer.

You must think that only BAD Developers are in play.

In the late 70's / early 80's when run size changed from 4M to 16M, that programmers to use on mainframes, bad developers started to read whole tables into memory to "run faster". Yes that single job did run faster, but the whole system ran slower and more conflicts of sector in quue showed up.

Lastly with these "faster" computer are not faster, because most have bloat of the OS that take up that extra speed. Add to it bloat that developers leaving in their programs...

Good developers can SAVE money.

Your article assumes that the choice is between hardware and developers, and steadfastly ignores the non-linear costs of purchasing, installing, and maintaining both the servers and the data center space to house them. Running hardware is *NOT* cheap, it's merely wholly externalized to the developer.

It's often easier to believe hardware is cheap -- and create an entire team's worth of grunt work -- than it is to studiously approach the problem at hand.

"3) Optimized code should be simpler to read and maintain (optimize by simplifying, not by clever hacks) saving future programmer time."

You are essentially right, but that is "Refactoring" not optimizing. Refactor your code to make it simpler, remove redundancies, call out patterns that you see and put tests around your code.

Once you have refactored your code, it is much safer to optimize... if you still need to. Now with some proper tests around it, you are at much less of a risk when optimizing.

I avoid theoretical rules of thumb like this article. You have to look at your problem and find what works.

That said, when there is a real issue to solve in terms of optimizing, I've not found hardware to be the answer in the majority of cases.

Perhaps all you mean is when you know hardware will solve it, it's cheaper. Maybe, but the costs of that isn't just slapping a box on a rack, often your dev costs will move over to your infrastructure engineering costs. And as things scale you're going to spend more on supporting infrastructure hardware and space.

This article also must be assuming a web type app, not something like a shrink wrap, where you can't control the system requirements (marketing does that)

There is a point of no return (or at least diminishing returns) on hardware as a performance solution. Once you scale beyond "one box" architecture (or one web server to one database server), the complexity of the system spikes as locking and synchronization become fault points in the application. Ironically, the expensive programmers you skimped on are now the ones you need to design a system that doesn't deadlock a clustered web farm connecting to a clustered database farm and you have both - expensive hardware and expensive programmers. The expensive programmers, if they are really worth their salary, will also be saying that the system is overly complex, and fixing the fundamental design problems will remove the need for all the extra hardware developers are spending 90% of their time on, instead of adding features.

Bottom line; Oracle RAC is never the answer =)

fire the bad programmers!
otherwise in 5 years you will only be able to hire bad, overpaid programmers
to support the problem with faster, better hardware doesn't solve it ...

servers are perhaps cheap...but the power to run them are not. lets
hear it for "green" programmers. what is your code carbon footprint?

My first reaction when I read a post like this is that such effects are local.

My second reaction is that American programmers won't be making that kind of money much longer, hardware costs notwithstanding.

I spent a year managing a web dev team in China. I'd estimate they were 1/4 - 1/5 as "efficient" as European/American programmers in the same tech zone (I'm basing "efficiency" here on the number of revs we had to do AFTER beta.)

Most of my employees were making USD$400-600/mo. No benefits, no withholdings. That was their take home pay and the cost to the company. This was a generous middle-class salary in Xiamen China. My Sr. developer -- who spoke four human languages and easily ranked with most western programmers skillwise -- made the luxurious sum of USD$1200/mo.

So these guys were 1/4 to 1/5 as good, but made 1/10 as much as an American programmer. You do the math.

There are places in the world where it DOES pay to throw people at a problem.

BTW I definitely think you get better product from a highly paid team in a country like the U.S. But just because you spend twice as much on your product doesn't mean it's twice as good.

I believe you have been poisoned by management though. Throwing hardware at most performance problems is ridiculous. If the application isn't design from the begining to scale it will never work.

The proper solution is to design for performance, create an architecture that will be able to spread to as much hardware as possible, and then program for such architecture.

After all this is done I agree with you that programing optimizations in algorithms is mostly a non issue - compilers and frameworks usually take care of those.

I simply don't get the point of this comparison. It's like I'd say that it is cheaper to buy a fridge than a year's worth of meat for the family.

Buying hardware is one thing and really does not cost much (I'd gess you need to be a software developer to a) think that it is news and b) stop thinking there). According to most studies it covers only 10% of the costs for the lifespan of that hardware. And these studies don't take into account the personnel operate and administer the hardware.

- J

This article scares me in it's simplicity.

1) Doubling HW rarely doubles performance or throughput. However, doubling production server hw usually requires similar purchases in development, test, stress, & uat environments too. In addition excessive disk requirements can quickly lead to huge backup storage requirements. Figure a copy for everynight of the last month (30) plus a monthly for the last 7 years comes to a multiplier of 110 copies of that data for compliance purposes plus whatever copies are in the non-production environments.

2) HW is a small fraction of overall server costs.
I saw a vendor's suggested €250K hw purchase add up to €6m when software licencing was added (ftp software, clustering sw, db software, middleware, app server s/w, backups, etc). Their 'HW is cheap' attitude was working out so expensive that we were willing to send them packing.

3) I do databases. I get fed up when database developers say that the database infrastructure is performing poorly when new code is releases into the stress environment. When we investigate, we discover basic errors such as table scans.
In a recent scenario, we were asked to find out why the hw was performing slowly. We discovered a missing index that boosted performance on a key user action by 8000. I would hate to know how much hw would be required to provide the equivalent HW performance.

In summary, good HW can double / treble performance. But spending some developer time can improve performance by thousands. But it does cost time. The main question is, is it fast ENOUGH for the purpose?

Because fast hardware becomes more and more cheaper 'we' start to do slow things like using XML or programming in Ruby because it shortens the time to market.

Most developers never read database documentation.

Fast hardware makes Vista Aero possible.

"Clearly, hardware is cheap, and programmers are expensive. Whenever you're provided an opportunity to leverage that imbalance, it would be incredibly foolish not to."

That might be true for application software that just runs on a few servers.

It is generally not true for device drivers and other system software that runs on millions of desk top PCs (Vista anyone).

It is rarely true for for memory/weight/power constrained items like mobile phones, organisers etc.

And it is total nonsense to apply it to embedded software that runs on the billions of TV remote controls, network switches, keyboards, mice, mp3 players, dvd players, televisions, washing machines etc. If you're selling a hundred million units then an extra penny saved on a component can hire you a lot of programmers, as can selling an extra 10% because your device is perceived to be faster/slicker/better than the alternatives.

I was brought into a place one time to check out why a government-mandated report on mortgage fee codes was failing. The report (written in C# against SQL Server by one of my junior colleagues) was now taking so long that it didn't finish.

I walked in, looked at the table and put a table index on fee code. Instantly the report went from taking over 2 minutes to 45 seconds. Then I spent a few minutes looking at the query. I made a simple change to it (eliminated an obvious cross join) and got it down below 30 seconds.

Then I looked at the C# code. My predecessor was concatenating strings together when formatting the report. I changed it to StringBuilder Appends instead. The report now ran in 8 seconds. That was all in under an hour's worth of time and it really wasn't hard at all.

So, I agree that it's well worth it to spend an hour or two getting an expert to look at the low-hanging fruit first. Sure, faster hardware would have got it done, but why?

I think this is missing the point a fair bit. If you want to build something that is worth having a performance problem, i.e. a high volume website for programming Q&A for example, then you need good developers anyway.

Once the site is becoming popular and you need to improve performance you can either (a) fire those good developers and get better hardware or (b) let those good developers improve the performance.

Looking at this problem in isolation of having to have good people to design/build the app in the first place is kinda dumb. Sorry.

Would like to recommend "Release it" over at pragmatic progammers. It amounts to a debunk proper of this lofty talk.

Noticeable that you skip the part where one would try to figure out if capacity sinks linearly or exponentially, slowly or at a particular threshold ... Or how fast and why performance grow as you add resources.

Please mind that probably the most rapidly increasing clients at the moment is Iphone and Android. These do not have increadible amounts of memory, they use a battery that is supposed to last for quite a while, they are convection cooled, and moore's law works vs. display and battery, not nearly as much in terms of cycles. School kids to-three years from now all over the world might be entirely happy to put such a device into a screen, keyboard, battery docking device. These things will run on solar.

Social responsibility and ethics is no small problem with your thinking.

Further, it bodes for the deservedly infamous complete code rewrite down the road.

If you put your eggs in the "just add hardware" basket, basically you are making far worse a design than in terms of car analogies, the SUV.

Niall Flanagan has it correct, its a very poor assumption that doubling hardware (for example) will double performance or even raise it at all (and is that performance in terms of throughput or performance in terms of response time because whilst you can sometimes add throughput, reducing response times is often a whole lot harder just through hardware)

Many systems I see in my daily work wont go any faster however much hardware you throw at the problem, because someone (usually inadvertently) programmed in a maximum level of performance - either because of incompetence, or because "we'll never need more performance than X".

For example take a simple app that updates the exact same row in a table for every transaction, maybe a sequence number or a transaction count. At some point, you will hit a wall and after that point add all the extra disks and CPUs you want, it wont make a jot of difference.

For software which runs on a few machines, sure it doesn't pay to optimize too much. Take software which runs on thousands of machines (ala, google search), and suddenly the cost function is in favour of employing a whole team that optimizes it as much as possible.

It's all about choosing between trade-offs.

I've come to realise that there is always a cost with solving a problem, regardless of how you solve it. That means that if you have a number of different ways to solve the problem, then the best solution is the one that best solves it, and that solution is the one that best matches the requirements. Different problems might have different requirements - e.g. least time, least cost, most scalable etc.

For example, if solving the problem in the least amount of time is a high priority requirment, then something like throwing hardware at the problem is one of the better solutions. Changing the algorithm to a better one might be another option, and as long as the time invested in doing that is cheaper than other options then it might be the best way to solve the problem.

A great paper I've just read, which Programming Pearls referenced, is "Hints for Computer System Design", by Butler Lampson. It goes throught the various ways of solving performance problems e.g. use cacheing, using hints, and, of course, using brute force a.k.a throwing hardware at it. Programming Pearls and The Practice of Programming also have good sections on performance improvement techniques, that are also applicable to outside of programming.

As a performance tester of 10 years. Tuning gains the most benefit. Cheap or Expensive hardware doesn't solve the problem. I seen a maxed out Sun E10000 thrown at problems (2.5mil) (5+ years ago) the application still performed terribly few hundred users terrible times. The application couldn't scale at all. Distributed or parallel applications need lots of customization. = programmers. Typical performance problems in order are: 1. Load balancer setup 2. Network setup 3. OS/DB/Machine setup (hardware and software). 4. Layer communication (RPC). 5. Easy Application fixes. (SQL, searches, data, cache). After those in depth optimization has little effect the problem is architectural and the whole thing needs to scrapped. I seen this too.

This sounds like the typical excuse to write bad code quickly and then expect the systems guys to make it run faster with new hardware. Write good, fast code to begin with, and always having the newest, fastest hardware isn't necessary. The fact is, there are lots of crappy programmers out there.

There are situations where you cannot throw hardware at it. Take for example a portable device - throwing hardware means you have to throw batteries at it, which means the person using the device has to carry a heavier and bulkier machine. Sometimes this is unrealistic - these people may be mobile 8 hours a day, and never near a source of power.

Embedded devices is another area - throw in another DSP and CPU, and it can push up the price of a consumer device beyond what people are willing to pay, or reduce margins to make the product not worthwhile. (Also, programmer time scaled across many devices may be cheaper than the hardware - take devices like iPods which can sell millions - a beefier processor may cost more than the complete R&D effort - $10/part versus $1/device for development).

Also, throwing hardware at a problem caused by idiotic programmers is completely nonsensical. Sites like TheDailyWTF.com have plenty of listings where some programmer iterates through a database manually, making a simple update or query take 1000 times as long as it has to because the program is doing work the database can do faster, and the program is doing it 100 times in order to generate say, a web page. Or by a programmer using bubblesort instead of quicksort - proper algorithm choice can have far better ROI than throwing exponentially more hardware at the problem.

It's not premature optimization, it's short-sighted optimization that's evil. If you optimize the first piece you start writing, especially if it's done in a way that reduces flexibility and generality, you can end up with a legacy code mess during initial development.

But some early global thinking about performance, before you've frozen the organization and algorithms, can give you asymptotic speedups, and that's well worth it.

Micro-optimization can generally wait, unless you're positive that a particular inner loop is both required and performance-critical.

Problem here is that you can spend your self into a problem too.

Classic example here is 4 and 8 core processors. Wrong design and 4 core version is just as fast as the 8 and the 12 core is slower. So yes 12 core is cheaper than 8 because it basically don't work.

Be sure to throw good hardware at problem. If good hardware happens to be cheep of course then throw it at it.

Number two adding more cores on some OS's makes problems worse not better. So make sure you use a good quality OS built to scale and have developers working on fixing its scaling issues.

Be sure problem scales. Not point throwing 1000 cores at a problem if problem can only be processed single threaded.

Be sure to use a complier that optimizes correctly. Surprising how much difference spending some money buying Portland group complier and using that instead of MSVC or gcc.

I have had cases were people wasted hours trying to improve performance threw altering code. Only to find out that I was hammering them all I did was build the code they were starting with with a good complier that had done all the optimization methods they were trying automatically.

Some compliers even rewrite Algorithms removing human error. Human doing optimizations can be a complete waste of time. Some cases it would be better for long term to send your developers into the complier itself.

On the Go programming list, they recently discussed an area where throwing more hardware on a problem may not be appropriate: If you have made a change to your Go playing program, you may need to run it thousands of times against other programs and earlier versions of itself, in order to convince yourself that the change is any good. Then the code needs to be fast - the gains in productivity you gain from using a higher level language are lost again in the additional time needed to regression test it.

There might be cases where this also applies to other software: Things that need to be profiled so heavily and continuously, that it's better to use additional skill to get it right the first time than just buying more hardware (after a certain point). Maybe weather prediction systems, climate models, or other simulation/prediction systems?

This is an interesting article and one cannot in isolation argue either clause of your premise, especially the I think however, that much is missing from your list of factors with respect to the issue.

The initial cost of hardware (servers) is not the only cost, and - yes hardware is cheap, but is a drop in the proverbial bucket compared to the total cost of ownership. I believe overtime your premise is eroded by cost of:
1. Licensing software such as Oracle or Websphere on additional servers
2. Space/leasing costs
3. Power and A/C costs
4. Server admin/maintenance costs, and
5. Most importantly - if one wants to add servers to improve performance, then one will probably need to upgrade (continuously) networks and network appliances to realize the gains, and
6. Finally - not all performance issues are caused by software. Server configurations (improper configurations as it were) and under-powered network appliances can certainly contribute to performance issues.

I think your statements,

"Always try to spend your way out of a performance problem first by throwing faster hardware at it. It'll often be a quicker and cheaper way to resolve immediate performance issues than attempting to code your way out of it. Longer term, of course, you'll do both."

... amount to bad advice. Example: I go out and buy “n” servers to “cure” a performance issue per your statements above while at the same time incurring all costs listed above. A day later and while the new servers are being built out (incurring additional costs) and prior to going live – a server admin solves the problem in 15 minutes by increasing Java heap size.

Did you catch the same “virus” as this guy? :-)

http://articles.techrepublic.com.com/5100-10878_11-6044115.html?tag=search

Forgive the grammatical error of the opening sentence above. Thank you.

Jeff,

For once I disagree with you.

I think throwing hardware at the problem is like playing Monopoly and having the "get out of jail free" card. Certainly with windows servers you can only expect to play this card once every 24 months to get a significant leap forward in terms of 'raw' pace.

As for dev workstations, any software company that does not invest in tools/workstations/monitors/chairs/environment etc. for their developers (or indeed any serious PC power user) is simply being short sighted.

Mike

You've gained some interest on Slashdot:

http://developers.slashdot.org/developers/08/12/20/1441203.shtml

Wow! I'm grossly underpaid, but then I don't do much on my job and live in an inexpensive area. I really should upgrade my home computer but it would take a long time to reinstall everything on a new box. Figure at least an entire day wasted installing Visual Studio alone.

This is a pretty gross generalization.

True, spending 3 weeks to get 5% more out of an algorithm makes little financial sense.

But, for poorly designed database tables, indexes, and databases, this is very poor advice.

Heh, try getting the public service to fork out for hardware for developers. We're developing on 5-year old servers that are show as hell. I've been trying to get us new development servers for 2 years now -- with no success in sight.

In terms of effort, it's easier to hire more bums on seats than have new servers purchased at 1/10th of the cost.

Programmers are not about writing the best/fastest algorithms. The more important part is building the system/software itself. Build it reliably, to be modular and expandable. Going from general to specific. Creating and tying the pieces together. No hardware can do that.

weak argument. Throwing more hardware at problems does not make them go away. It may delay them, but as an application scales either:

1) you may get a combinatoric issue pop up outstripping you ability to add hardware

and

2) you just shift the problem to systems administration who have to pay to maintain the hardware. Someone pays either way.

All-in-all, this is the sloppy sort of thinking that leads to such screwed up software. "We don't need good software so we don't need good programmers" and the cycle perpetuates.

The worst performance I have seen has come from middleware, despite all the DB bashing that goes on. Things like constant roundtripping to the DB, bubble sorts (!), constant re-rendering of the UI/web page etc.

Oh, and most of the time, the server is *not* the bottleneck. The network is. You can never go faster than your network. So keeping queries minimal and knowing when and how to cache are important. But that takes skill, which doesn't doesn't exist very much amongst programmers.

Hello,

In India, Programmers are cheap, hardware is pretty costly...

There are still plenty of things to optimize that you can't throw hardware at. I see some people have referenced database bottlenecks and whatnot. I have spent much time optimizing the performance and load times associated with JavaScript code and other related page load times in the web world. Unfortunately, you can't throw hardware at a user's bandwidth or their system.

But, you have an interesting point, as long as your code is robust enough not to fail. You can throw hardware at a problem all day and find out that your just running massive endless loops, leaking memory and generally destroying your system. In the end, you could end up with a situation that requires more new hardware as time goes on, at an exponential rate. Although it may be helpful to double your CPU and see your memory fill up that much faster :)

milkmaid bill

This is one of those wonderful articles that ends up at Facebook and other Web2.0 applications. They utilize an unbelievable amount of hardware to make up for some phenomenally poor design decisions that they're now stuck with. Throwing hardware at the problem first is rarely the right answer.

Nice article.
I think it applies only when:
1. you have a web based project.
2. your code is THE shit and scales well(responds to addition of new hardware).
3. you have an optimized, tested and released product and it makes economic sense to add more servers than to pay for new developer.
4. you own stackoverflow.com

Jeff: I think you may have nuked the fridge with this one.

That whole article rests on the assumption that the software is only ever used once on a single product. A bussines solution that software is especially bad at.

Deploying multi server solutions is not exactly cheap. Doing it multiple times on multiple solutions is very expensive.

Furthermore it is always done in the end of a project where the deadline has long passed by, so it is stressfull too.

I'd say with both developers and hardware it comes down to the law of diminishing returns. For example if a website cannot cope with 10 users then throwing more hardware at it won't fix it. And on the other hand adding more developers to optimise code there comes a point where you get less benefit.

This would only work if you run the hardware that this runs on. If you send out the software to customer sites that is slower than the last version or the competition it just looks like a poor quality product in my eyes. If the software is made for the desktop it will often cost more in hardware upgrades than was spent on the software in the first place (e.g. Vista/Office 2007).

I think you have to pay attention to performance metrics but not necessarily do anything about them if they are not too bad. If you are trying to wring the last 10% or 20% out of something you should give up since it's not worth it. However I often find optimising bits of unloved code gets speed ups of over 10x (which is still only worth it if that code is often used or causing a problem).

Throwing more hardware at the problem most of the time is only a temporary solution used by people who don't understand the route cause of the problem (yet), and rarely works as a long term solution in my view.

@Dave

"So my advice would be for beginners to spend a bit more time learning where their programs are bad so that they can become better."

You see, problem is that they are unaware of where their programs are bad in the first place.

On this one, code on a well structured way, profile, tune area involved until before the point of dismishing return and rethink next step.

Most of the times is just sloppy code and unawareness of behind the courtains work.

I'd like to know where these first year programmers are making $85,000 a year. Where I am, the average salary for starting out is $40k-$50k tops.

I agree with Kevin...where the heck do those averages come from?

They seem exaggerated from everything I've been reading up to finding my job.

@Daved

"So my advice would be for beginners to spend a bit more time learning where their programs are bad so that they can become better."

You see, problem is that they are unaware of where their programs are bad in the first place.

On this one, code on a well structured way, profile, tune area involved until before the point of dismishing return and rethink next step.

Most of the times is just sloppy code and unawareness of behind the courtains work.

> That's why we have Wirth's Law: "Software is getting slower more rapidly than hardware becomes fast

Ooo, that is so smart. So in few years computers will not be able to run software anymore? Who knew.

It makes a lot of sense when Jeff runs his application on a single installation, but is the stupidest advice you could give in types of applications.

Take embedded software for example; even a $10 memory upgrade is expensive when you expect to sell 100,000 units.

JakeBrake is thinking along similar lines as me. Throwing hardware at a software problem has its place in smaller, locally hosted data facilities. When you're running in a hardened facility the leasing of space, power, etc. begins to hurt. One could argue the amount of time and labor necessary to design and implement a new server, along w/ the hardware costs, space, power -- and don't forget disk if you're running on a SAN (fibre channel disk isn't cheap!) -- can easily negate the time of a programmer to fix bad code. But again, that also assumes you know the problem can be fixed with code.

This doesn't hold true for portable application, where size and power consumption matter.

and sometime ago ballmer was hopping around like a monkey making fun of all schizophrenics clapping and shouting in coordinated fashion, "developers, developers, developers, developers!!"

we need new ai that can write programs for us.

In some scenarios your analysis holds true, in others it doesn't. Sometimes you can't just throw hardware at the problem - especially if the problem is IO bound rather than CPU.

Also the performance difference between well designed code and poorly designed can be extreme and more then can be made up for by simpy throwing hardware at it. Someone's already mentioned the web DB scenario and there are others.

I would say get your good guys to design the most critical/sensitive bits of the system fairly well in the first place. This also avoids optimisation as there is less to gain.

Some kinds inefficiency are inexcusable, a product of stupidity or ignorance rather than rapid development. Fast code doesn't need to be complex/bloated/unintuitive/difficult. Someone who's made a habit of writing good code can often produce better, faster code in less time than someone who happily writes piles of slow, repetitive, disorganized crap.

http://wiki.secondlife.com/wiki/LlListSort (reinvented the bubble sort wheel)

For the most part I agree with what Jeff has stated but at the enterprise level the hardware comes with a maintenance cost.

But most importantly I can't count the calls from head-hunters that have a client looking for someone to analyze their app server and set the fast = "true" setting. Instead of writing it right the first time and putting it on a server sized to handle the load they all are looking for that setting and just can't locate it. I used to take these contracts but they are usually a no win situation working for some manager who already bought the stuff and has no budget left.

No a good place to be...

Depends on the programmers you have working at your shop, but my impression is the world has replaced programmers who know about qsort being a standard feature, with those who implement their own, buggy bubble sort.

I work in a shop with 70-80 developers, so any initiative costs a "prohibitive" amount of money in absolute dollars. At the "new year kickoff" event, the most popular request from the Devs [aside from more vacation time :-) ] was dual monitors.

Management had a cow when they found it would cost about $500 per seat for a second monitor and graphics card, or $40-50k to outfit the whole group. A "pilot project with a selected group of developers" was initiated to evaluate the productivity gains. As one could guess, a year after the initial outcry we are still waiting for dualies.

I think hiring good programmers is very important.. I have a discussion going on about this over at the forums section of <a href="http://www.corkin.com">Corkin</a>

I think hiring good programmers is very important.. I have a discussion going on about this over at the forums section of http://www.corkin.com

Jeff -

One of these days (soon I hope), we'll convince customers that gathering a dozen or more people in one place for a design review is not a smart use of development funds. If ten of those people need to fly (business class for those wearing suits) to Chicago or Birmingham or ?, need rental cars, need hotel rooms, need to eat, etc., for a review, perhaps the money might be better spent. A terabyte hard drive costs less than most domestic airfares. A fully loaded server costs less than an international airfare. Doing the formal parts of design review via virtual meetings (assuming you can spend the savings on hardware), might mean higher thruputs via extra servers, or redundant mirror systems where appropriate.

On the optimization front, I've found unrolling loops by hand very effective if the N in "do 1,N" is a constant. Knuth Volume 3's index has 13 "optimization, loop" references, mostly in the exercises.

Just a guess, but I suspect there's plenty of scope for optimizing
anything written in Java.

Perhaps irrelevant, but whether I use an ATM in NYC, or Glasgow, or Auckland, I see much the same response (cash out), and crashes are almost unheard of. Large databases are being queried, correct answers are almost always returned, and in a short time. I don't think I want my bank to move to Java.

Well, at some point I agree with you. But remember that future optimization of hardware seems to be done by throwing in more cores, and (for most problems) you have to do some good old fashion programming to keep the level of serialized code to a minimum in order to scale beyond a certain number of cores.

Although, for a lot of web applications the add more hardware approach will get you far (but only so far...).

My friend (and IT exec extraordinaire) Dave Berk has a response to this article posted at his blog. http://www.lump.org/?p=135 He hits on many of the same ideas mentioned by others.

--Most software is usually not very fast to start with. Good optimization can often get you much better than the linear gains you get _at best_ by adding hardware.

--Lots of common software gets exponentially slower as you increase it's usage / throughput.

Alan Kay:

Neither Intel nor Motorola nor any other chip company understands the first thing about why that architecture was a good idea.

Just as an aside, to give you an interesting benchmark—on roughly the same system, roughly optimized the same way, a benchmark from 1979 at Xerox PARC runs only 50 times faster today. Moore’s law has given us somewhere between 40,000 and 60,000 times improvement in that time. So there’s approximately a factor of 1,000 in efficiency that has been lost by bad CPU architectures.

The myth that it doesn’t matter what your processor architecture is—that Moore’s law will take care of you—is totally false.

i really think intel went overboard, since when was 500 MHz or more required for DVD playback? I think the 3 GHz dual cores are overkill. everything's going to a browser eventually, the home operating systems like windows will be obsoleted by clouds of virtual servers. to operate a browser, does one really need more than 500 Mhz?

http://www.3v3bgh.com

an A+ programmer in Vietnam get about 5.000$ per years only,
why don't you try to outsource to Viet nam ;)

Finally someone has put this to perspective with actual numbers. Nice job! There is no excuse for giving developers inadequate tools that slow them down.

It's like giving a carpenter a wooden hammer instead of a metal hammer. Takes 2x as long to pound in the nail.

Also, you look at any Engineer (CAD, Mechanical, Electrical) and they never have slow PCs. Partly because of the CAD software and that proves that they also don't have time to wait either so why should we? For the longest time, my wife had a better PC than I as she is a Mechanical engineer. It really pissed me off that I was programming and had to wait for MS word to pop up, or VS to compile took 1 minute because I had an fing 5400 rpm hard drive at 2 of my jobs.

>>>I ninja-stealth upgraded my own computers at a previous job where we had restrictions like that. I basically gutted the inside of the PC so on the outside it looked like a normal...

When you work for any company who is giving you only 2 gigs ram, or a 5400 rpm hard drive, or inadequate processor and they refuse to give you a "developer based laptop or desktop" I say I'm all for the ninja-stealth. If they are not smart enough to provide the tools needed then you have to do what you have to do.

You don't come to work to wait for shit to load. You come to work to code and do that as efficient as possible.

One good post, then 3-4 bleeding obvious posts on this blog. Over and over again. That's how you build web properties <kicks himself>.

Optimization is for losers. Just ask Microsoft how they fared with throwing more hardware at Vista and its inefficiencies. I hear that Vista has eroded Microsoft's standing, brand and profits (somewhat) and allowed a company called Apple to gain a strong foothold in the market with their little super-efficient OS X (running on super-efficient hardware).

Jeff, with all due respect, you should take the time to read up on subjects you write on before posting such simplistic articles.

Jeff, the free lunch *IS OVER* (also for economics).

I have spent years optimizing code with profilers for scientific purposes, where hardware upgrade was not an option, and now in a business environment I don't even think of changing a line of code to do a possible optimization that most probably will simply require a new set of test to verify/certify the product.

The real optimization is improving the design of the code via refactoring such that it is easier to make changes to it.
The you save on the time spent by the developers that makes next changes. So one has to be extremely careful when deciding if it is worth. If the software will have a long history in the future and you expect a fairly large amount of changes due to new requirements, well then you might consider refactoring design.

There is an old saying: You get what you pay for. I think this saying can also be applied to the software developer.
http://www.sa-technology.com

How many times do I have to repeat it here? You compare apples with oranges. H/W has nothing to do with S/W!

With regard to the whole number of people involved in the industry only handful are really writing S/W for H/W. The majority of programmers are writing code for other domains. OK a programmer has a computer science background but that's all. What matters the most is the experience and the knowledge he/she has about a different profession. You get business programmers, accounting programmers, industrial programmers, telecommunication programmers and so on. Has someone working in the business tried to get hired in a telecoms startup? Even the most talented and gifted programmer will have hard time to do so.

It is for that programmers are much, much more expensive than H/W.

What are you looking for when you buy new H/W? Processor, memory and HD. That's all. Now let's get a look on Monster or Hotjobs or other. The less bushy add will have at least 500 words describing the whole daydream of hiring mgr. Hardly someone will fit 70-80%. 100% is impossible - this would be the God of programmers, and God doesn't need a job!

You see...

p.s. suggestion for a discussion topic: How log would take someone in love with Java to switch language?

Your cost for hardware outlay is not accurate fro a large company with managed servers, storage, and software - power - and storage management. Even in a virtualized environment the average cost without administration of the average new server is more than 10,000. A physical rack mounted server from IBM or HP is more like 25,000 with internal managed storage and memory, OS, backup and managed AV software. Add $18 per gigabyte for SAN storage and the cost of power. and hardware rapidly adds up to be on a par with the cost of labor. Especially since a developer is likely to write more than one system / year and if each system requires new/additional hardware that will require addistional administrative overhead. Take a medium size company of 200 IT employees. 400-600 servers (dev test/qa and production) is not an unreasonable size for their data center. Accounting for 1/8 of the IT staff being developers each adding 3-4 servers per year that's a growth of 120-160 servers per year.

Servers 160 * $15,000 = 2.4M
Developers 25 * $85,000 = 2.1M

HMMMMMM

My employer has many large, heavily-utilized databases. Some of our applications run for 8 to 20 hours, and scan billions of rows of data per hour. We throw prodigious amounts of hardware at this problem - expensive Netapp filers with lots of cache driving hundreds of spindles with terabytes of data. More money helps improve performance, but the biggest gains came through the application of common-sense analysis to the applications. We've seen 20% to 99% reductions of runtime as a result. (yes, at least one application dropped from hours to minutes. ) Per Wirth's law, applications tend to expand to fill available resources. When we push back, applications developers find much ways to make better use of available hardware, and our clients are happier.

I have seen it time and again as mentioned. You throw ten severs at a bad query, it's still a bad query. You re-write the query, and suddenly you only need one server. Programmers are invaluable, no matter what kind of pie chart you can manipulate into looking the other way around.

I think this is attributable to Wes Dyer, we've got it up on the wall in our office:
"Make it correct,
make it clear,
make it concise,
make it fast

in that order"

Watcom C/C++ Forever!

I'm a brazilian programmer, and the thinks here are very hard, just work, work, work, like a slave, lol, and the sallary its a shit... only a java programmer can make some money, but for other... lol, impossible....

There are limits to what throwing hardware will do for a performance problem. When those limits are hit, things get really fugly, really quick.

That is why one school of thought will say that performance/stress/load testing should be done at every stage of development.

James you are giving one of the most horrible examples, but nonetheless real out there. The problem is corporations over spending on servers by buying about 100 times more horsepower than they need. Look at Plentyoffish.com and how it runs on just a few servers for an example.

rwerwer

uh, what about the people who support the hardware? Storage engineers? Network admins?

Not to mention the fact that hardware alone doesn't solve anything.

In fact, poor and sloppy programs, not to mention developers who don't know how to manage their own tools and code, probably account for a great deal of the "need" for more power...

One must also understand that when a company pays $85,000 a year for a programmer, they are paying for the most complicated machine we, as a race, know: the human brain. There is such capacity in a human to learn, that until a computer can surpass this capability, humans will possess a more adaptive, analytical machine. Humans are set to control computers. Although computers have incredible calculating power, they do not have the brain capacity to take full advantage of it. This is where a human programmer steps in. He/she has the capacity to break down any situation into small, easy problems and then instruct the computer on which order to solve them in. That is the task you are paying for, the people who can analyze real world problems and coordinate the execution of the solution.

very true.
unless you work on IBM mainframe hardware.
here the cost of hardware is so incredibly inflated that with the money you spent for the new hardware set up you would probably manage to raise the machine memory from 512 to 1024mb and maybe get 300mb more space on the disk... (plus here in italy the average salary for a 10 years-experienced developer is around half what you quote, even taking the ongoing money exchange rate in account).

M