I work in hosting for a *large* hosting provider, so for once I have some insight to share. A lot has changed in the datacenter space over the last few years. But first the things that haven't:

- Separation of application and database duties is critical so that each can scale independently of the other - the databases scale up, and the applications scale out.
- Typically databases scale vertically (what you refer to as "scaling up") as they have a central data repository that must be kept in sync across multiple servers. So you tend to see either "big iron" or very high CPU systems act as database servers. Clustering - whether active or standby - is a must. Systems that do scale out like Oracle RAC are frightfully expensive and difficult to maintain.
- Application servers *should* scale horizontally. Just throwing processing power at something is a good sign You're Doing It Wrong. If an application properly keeps its state in a remote backend database (see Very Large Database Servers above) then you can add as many application servers as you need, using pizza boxes or whatever is most efficient.
- VERY few applications will scale linearly as processors are added, as most only have so many threads and parallelism in place. Serial operations happen, and will reduce the potential of these monster "scale up" solutions.
- For horizontal scaling you'll likely need external load balancing, which is another piece of hardware, and another thing to power, cool, and manage.
- Power (and cooling, which is tightly tied to the wattage you're pulling) is critical, so you're likely to want to find a middle ground between a 1U pizza box and a 7U monster like the HP. There are also other issues such as storage, expansion slots, etc.

Now as for what's changed. In a nutshell, performance per watt.

- It used to be you could just about count processors by rack unit. A 1U server was a single processor, 2U was dual, 4U was quad, etc. There were some minor efficiencies gained - 4U servers tended to become 3U over time - but not much. Until multi-core processors hit.
- These days, you can pack quad-core processors into most servers. A 1U server can typically now reach 8 processing cores (2 x Quad-Core). So while that 32-CPU beast may be 7U, if you just need raw processing and memory, you'll likely still come out ahead using small pizza boxes. Internal storage and expansion will be minimal, but if you're using NAS or SAN it's a non-issue, and you'll get better storage management to boot. SAN avoids burdening your network and offers much higher performance, but at a price.
- Licensing varies, but Microsoft in particular considers a single die 1 processor, regardless of the number of cores therein. This heavily favors multicore solutions and reduces the licensing penalty above.

As others have commented, your costs and numbers above only make sense if it actually took 83 of those servers to match up to the HP - something that I doubt would be the case for most well-written applications. If we're judicious and say that 16 would match it (personally I'd blace bets you'd still be ahead) then those numbers look quite different.

So in an ideal world, you'd have a nice SAN backing up your servers' data needs, a bunch of small servers for application (blade servers are actually quite nice in this niche), and a couple large clustered servers for the database, with a gigabit network for communication between them.

A lot of excellent discussion on this post - it's been an interesting read. A few more things that are particular to my background...

- As for software licensing - bbviously scale out will incur more costs in that arena than scale up (unless you're 100% free/open source); how badly that affects you is extremely application and vendor dependent.
- Scaling out is very hard to do if you didn't design for it in the beginning. Let's face it - most homegrown stuff isn't designed up front for issues like this, and so it's going to be difficult/impossible to scale many such systems horizontally.
- Horizontal scaling requires a much more dedicated approach to system management - you'll need to be able to automatically build servers, apply patches en masse, backup/restore, push content updates, etc. Content management systems, reporting and alerts - just more of everything. Moving from 20 to 80 servers isn't hard, but the move from 4 or so to 20 is. This stuff is not trivial, but depending on how much value is in your infrastructure (is this your core business, what's the size of your business, etc) it may be warranted.
- Scaling out does have other hidden costs. You'll need external load balancers, as I mentioned. You also may need a beefier network, better/more switches, NIC teaming, etc. Centralized storage like SAN or NAS - while I highly recommend it - can be complex and costly.
- And of course, you can meld the two. Not only by separating your application and database servers and scaling independently, but by splitting the database into multiple servers. This may not always be possible, but at least we're back to standard software engineering techniques of factoring and database design.
- Virtualization makes a lot of these issues even trickier, but offer the hope of major gains *if done right*. You gain a huge amount of flexibility in adjusting workloads, CPU/memory allocations, or migrating systems to better servers. You also have an additional layer to manage, and have to be very careful balancing resources between virtual machines. This is especially difficult when all systems are the same, as you can't use the differences in application design to your advantage (for instance, placing a CPU-intensive VM on the same system as an I/O-intensive VM, etc). However, don't expect it to be cheaper - far from it. Hardware cost tends to scale exponentially once you pass 4CPU (16 core) systems, so partitioning a 32CPU server into 32 x 1CPU servers won't likely be cost effective.
- Beyond virtualization there are interesting technologies like Solaris Containers and other lightweight partitioning schemes to consider. Not even going there - this is long enough. :)
- Finally, planning for failure. You definitely don't want a single huge machine, as everything is in one basket. Many folks will then move to clustering for redundancy, but then you have a new set of trade-offs: do you go with Active-Active clustering, and watch your ability to handle load drop by 50% when a node dies, or go with Active-Passive clustering and have an expensive paperweight 90% of the time? It may hurt to have a passive server sitting there, but the last thing you want to see is one server die, immediately followed by your second server cratering under the load.
- Scaling out offers built-in N+1 redundancy, in that a single node failing takes out very little of your capacity. This makes hardware management somewhat less critical, as you haven't lost much capacity (relatively speaking), and you're not sitting vulnerable on a single point of failure until it's fixed.

So... from my perspective, scaling out is the preferred method, because of the usual cost savings, flexibility, simplified growth, and redundancy. However, it's not for everyone. Your applications may not be written this way, your databases may be tied to closely to the applications, or you may not have the necessary support infrastructure for lots of smaller machines. Like so many things, it's a set of tradeoffs - but that's what keeps our field interesting.