June 23, 2009
In My Scaling Hero, I described the amazing scaling story of plentyoffish.com. It's impressive by any measure, but also particularly relevant to us because we're on the Microsoft stack, too. I was intrigued when Markus posted this recent update:
Last monday we upgraded our core database server after a power outage knocked the site offline. I haven't touched this machine since 2005 so it was a major undertaking to do it last minute. We upgraded from a machine with 64 GB of ram and 8 CPUs to a HP ProLiant DL785 with 512 GB of ram and 32 CPUs ...
The HP ProLiant DL785 G5 starts at $16,999 -- and that's barebones, with nothing inside. Fully configured, as Markus describes, it's kind of a monster:
- 7U size (a typical server is 2U, and mainstream servers are often 1U)
- 8 CPU sockets
- 64 memory sockets
- 16 drive bays
- 11 expansion slots
- 6 power supplies
It's unclear if they bought it pre-configured, or added the disks, CPUs, and memory themselves. The most expensive configuration shown on the HP website is $37,398 and that includes only 4 processors, no drives, and a paltry 32 GB memory. When topped out with ultra-expensive 8 GB memory DIMMs, 8 high end Opterons, 10,000 RPM hard drives, and everything else -- by my estimates, it probably cost closer to $100,000. That might even be a lowball number, considering that the DL785 submitted to the TPC benchmark website (pdf) had a "system cost" of $186,700. And that machine only had 256 GB of RAM. (But, to be fair, that total included another major storage array, and a bunch of software.)
At any rate, let's assume $100,000 is a reasonable ballpark for the monster server Markus purchased. It is the very definition of scaling up -- a seriously big iron single server.
But what if you scaled out, instead -- Hadoop or MapReduce style, across lots and lots of inexpensive servers? After some initial configuration bumps, I've been happy with the inexpensive Lenovo ThinkServer RS110 servers we use. They're no match for that DL785 -- but they aren't exactly chopped liver, either:
|Lenovo ThinkServer RS110 barebones
|8 GB RAM
|2 x eBay drive brackets
|2 x 500 GB SATA hard drives, mirrored
|Intel Xeon X3360 2.83 GHz quad-core CPU
Grand total of $1,150 per server. Plus another 10 percent for tax, shipping, and so forth. I replace the bundled CPU and memory that the server ships with, and then resell the salvaged parts on eBay for about $100 -- so let's call the total price per server $1,200.
Now, assuming a fixed spend of $100,000, we could build 83 of those 1U servers. Let's compare what we end up with for our money:
Now which approach makes more sense?
(These numbers are a bit skewed because that DL785 is at the absolute extreme end of the big iron spectrum. You pay a hefty premium for fully maxxing out. It is possible to build a slightly less powerful server with far better bang for the buck.)
But there's something else to consider: software licensing.
(If you're using all open source software, then of course these costs will be very close to zero. We're assuming a Microsoft shop here, with the necessary licenses for Windows Server 2008 and SQL Server 2008.)
Now which approach makes more sense?
What about the power costs? Electricity and rack space isn't free.
|Power Cost / Year
Now which approach makes more sense?
I'm not picking favorites. This is presented as food for thought. There are at least a dozen other factors you'd want to consider depending on the particulars of your situation. Scaling up and scaling out are both viable solutions, depending on what problem you're trying to solve, and what resources (financial, software, and otherwise) you have at hand.
That said, I think it's fair to conclude that scaling out is only frictionless when you use open source software. Otherwise, you're in a bit of a conundrum: scaling up means paying less for licenses and a lot more for hardware, while scaling out means paying less for the hardware, and a whole lot more for licenses.
* I have no idea if these are the right prices for Windows Server 2008 and SQL Server 2008, because reading about the licensing models makes my brain hurt. If anything, it could be substantially more.
Posted by Jeff Atwood
Seems like a 'scale up' crowd, but a couple points:
1. As pointed out, you probably could get away with a lot less than 83 servers to replace this one beefy server. This will cut down on a lot of the costs you are touching on, including the admin headache. Completely agree you have to go open source for this to work though.
2. Scalabity: Yes, it is harder to architect your system to be able to scale this way, but once it is set up it becomes trivial to scale. Want to add 25% more capacity? Just add a couple of more machines. At some point your 'beefy' setup is at capacity, and you just can't scale any more, at any price.
3. Machine failures/uptime: You are really going to have a single machine that cost you 100k? What if you really really care about failures/uptime? Hot backup? Now we are talking 200k. With the scaled out setup you don't have to worry about this as much. If one machine out of your 83 fails, it's less important. The system will still be up and running. And, it's not only failures but hardware/OS/software updates/upgrades...
EnterpriseDB also scales out very nicely, at a 1/6 of the cost of oracle.
By the way, Thank you Jeff for fixing the site where it can be accessed without running scripts. Thank you very much.
Douglas McClean: check your dictionary. Spend is both a noun and a verb.
Maintaining the 83 servers with open source isn't quite as difficult as it seems. You 'just' make an install CD, and whenever a box has a serious problem you swap parts until the install comes through again. Getting windows to reinstall that painlessly may be a bit more tricky and require pretty identical hardware.
Or swap the whole box.
Ok I like the spirit of this article, but the dollar comparison doesn't take into account which one can handle more traffic.
I'd like to see a cost per user/hit/connection/transaction, because its very possible that the dollar per unit of measure isn't linear for either of these two options.
Does MS SQL even scale up to that beefy of a box? Can you have that many instances in a cluster?
"a fixed spend of $100,000"
> Spend is not a noun. Did you mean "budget"? ;)
> Douglas McClean on June 24, 2009 9:45 AM
> Douglas McClean: check your dictionary. Spend is both a noun and a verb.
> mhuyck on June 24, 2009 1:51 PM
dictionary.com doesn't show a noun definition for "spend". This looks like a simple error.
Okay, but what if Monty Hall opens a door to reveal a netbook? Do you swap your first door, in hopes of getting the DL785?
"open source is only free if your time is free"
I'm curious about what the original commenter meant with this. If I use SQL Server instead of, say, PostgreSQL, will a Microsoft guy come home and install it for me? I agree with the phrase in some other contexts, but I'm not sure about what it means in this case. Which advantages would I get for using Microsoft instead of open source? Is the Microsoft tech support realy *that* good?
Note: I know the question sounds like trolling, but I really mean it. It sounds like the kind of scenario where open source is supposed to suck, according to non-open source guys.
Larry's point about resilience to failure is important, but the other things you need to worry about that nobody seems to have mentioned are:
1. network infrastructure - the moment your 83 servers aren't sharing a bus you have to tie them together.
2. systems management - if 6 of your 83 servers are broken somehow, will you notice? You'll also neet to monitor the network too. And your software as well as any vendors' software.
3. you'd better hope that you can buy bigger hardware before you reach the performance limit of your brute box; it has a hard ceiling somewhere. On the other hand hitting the ceiling with a many-boxes approach is a squishy affair; you start to notice that you're getting diminishing returns for adding new boxes quite a while before you get no benefit at all. (actually, larry mentioned this point)
4. If you're going to do the design work to scale to 83 boxes, you may as well fold in the relevant measures to span data centers, at least for failover. That gives you a big reliability boost, at least for many applications.
5. no question, there is more engineering work in the 83-boxes approach. How many software engineers are you going to have to pay?
I'm convinced that if Netflix ran the world we would all be happier. What a great combination of vision, UI, and infrastructure.
Does anyone know what their setup is?
>>5. no question, there is more engineering work in the 83-boxes >>approach. How many software engineers are you going to have to pay?
None, hombre. That's a hardware problem ;)
@Martin, while in most cases I do not care that much for open source software, here I agree with you. PostgreSQL is much more user friendly than SQL server, as long as you don't want to do performance tuning. Most DBA's that I have meet could not performance tune PostgreSQL. Then there is EnterpriseDB, built on top of PostgreSQL, which does not generally need to be performance tuned.
>> Nobody scales out their databases that way. That's crazy talk.
Scaling out on low cost hardware provides more computing power at lower cost. People who can conquer the issues of managing and virtualizing the large numbers of machines reap a reward. Then they can, and sometimes do, use this capability to make a big profit in businesses which otherwise could not exist.
As pointed out by the comment above, some systems can't leverage low cost deployment. This is justified by their ability to support legacy programming models.
What are some possible outcomes:
1. Low cost deployment takes over and we change our programming model is changed.
2. Low cost deployment takes over and it is enhanced to support the current SQL programming model
3. High cost deployment remains and we keep our current programming model.
I don't like the current programming model. So I hope for 1.
"Scaling up will typically involve less man-hours deploying the app. Scaling out will require more time unless you fully automate the deployment process (Azure for the enterprise anybody?)"
I don't where outsourcing hardware operation stops making sense. If it does at some point there is/was an opportunity for a vendor like Sun, to support the ability to set up an in house cluster that handled that sort of thing. Assuming of course Google does not productize their offering.
Great comments here, very insightful for the most part.
I totally agree, the upfront cost is only one part and definitely not the most significant factor.
The personal and power costs are going to exceed any software and hardware costs pretty rapidly.
Your licensing costs are *way* off. I'll ballpark our Select prices (they're likely cheaper than most, but ratios would be close).
Sql 2008 Ent with SA: ~$7500/cpu
Server 2008 Ent with SA: ~$450
Server 2008 Std with SA: ~$150
Server 2008 Web with SA: ~$45
I assume you're hosting a public website - since you're not authenticating against AD you don't require a Core CAL, but Sql Server needs CPU licenses.
So, for the HP:
$7500 * 8 CPUs + $450 = $60,450
For the 83, we'll figure 41 are DB, 41 are web, and the extra is standby (no additional Sql license needed). We can use Svr 2008 Std for the DB servers, and Svr 208 Web for the web:
($7500 + $145) * 41 = $313,455 (DB)
$45 * 41 = $1845 (Web)
$145 = Standby
Of course, if you're going to run 41 DB servers, maybe you can get by with Sql Std (~$1800). That brings the DB tier down to a more reasonable $79745. That makes the software within ballpark of each other.
Of course, it'd be quite an odd configuration to have 41 DB servers and 41 Web servers. A more realistic scenario might be 4 quad DB servers ($7780) and 40 web servers ($1800) - which makes the licensing a non-issue. Assume the quad DB servers cost 4 x your Lenovo servers, and I've got just enough left over for the SAN I'm going to need to get all those DB servers working together. ;)
* I am not a Microsoft licensing expert, but I was a reseller many years ago. However, this is only a blog comment - do not take reliance on any of this. If you have questions about software licensing, please contact your Microsoft reseller and/or Microsoft Licensing. Or, use Linux.
Most comments have already covered my concerns. Here's some separate ones:
- Are the power costs of the big iron really only 1200w? My home desktop consumes over half that. Also, with a big iron you'll need a bigger more powerful UPS to keep it running in the case of power failure. Simpler battery backup works for the scale-out solution.
- People have been quick to mention sysadmin and developer costs on the scale out solution. It's not like these are non-existent for a big iron as well. In fact sysadmin factors may be just as high on a big iron as you'll probably pay a premium to have someone sufficiently qualified to handle such an exotic beast. Meanwhile a sysdmin that can deal with a cluster of standard metal is a dime a dozen.
- How well can you share a big iron between developers? If you have a team of 12 developers, which would you rather have, one big server or a cluster of 83 small ones? Keep in mind that a bug by any single developer could take down an entire server.
NO, too many things to think about! My head hurts so bad!
No offense Jeff, but this is exactly why I'm so happy that I'm not a Microsoft guy :-)
Very interesting to see the comparison though, thanks!
I'm not sure if you also have considered the cost that scaling out brings to your software side.
It's not easy to write a reliable piece of software that will scale so well that it will run on n machines without hassle.
As to throwing a lot more processing power at one installation usually works with not too much pain on the application side.
And when that one monolithic machine dies due to any number of reasons, which one is better then?
Spend is not a noun. Did you mean "budget"? ;)
Hey Jeff, don't forget about the cooling costs too. It's a hell of a lot more difficult to cool 83 servers than one
You left out the most expensive component.
Comparing 1 DL785 against 83 RS110 is not quite fair I think, because you usually don't buy servers until your budget is spent, but until the site runs smoothly. Any idea how many RS110s you would need to replace one DL785? Might be interesting to do the calculations again with that value.
(Though I assume the result will lean toward Scaling Out, which is more similar to Googles strategy).
One thing that you didn't touch on was redundancy. It seems mighty short-sighted to have one beast of a system that runs an entire database without any sort of backup system. Especially since this db machine IS such a beast, I doubt they would have one just sitting in the corner waiting to be utilized if the primary goes down.
I could see a compromise approach working well. Have 3-4 'semi-beast' machines to run the db so that if one goes down you still have the other 2-3 running the site, you may have slower performance, but you still have performance and you don't have the large licensing/power/rack costs associated with many running machines.
This is a very simplistic comparison. Too simplistic.
What about redundancy? Even a $100k server can crash! A couple of clusters would give you so much better
Computational power? Does the applied use lend itself well to batch or clustered processing?
Datacenter fees? 7U isn't cheap but it's a lot cheaper than 83! Colocation fees per hertz is strongly weighed toward the 7U uberserver. Where does the price point cross over?
That's impressive food for thought. Thanks for another enjoyable and thought provoking read, Jeff.
I just want to 2nd what artsrc says ... just because the RDBMS doesn't scale-out, doesn't mean that databases don't. BigTable is an excellent example of this. I run HBase, the open source version of BigTable.
As my article explains, RDBMSs have abstracted away the storage of your data at the cost of many other things - scalability, performance, domain solvablitiy (SQL or nothing -- try doing PageRank in THAT), and many other things.
To all the people advocating scaling out for redundancy, isn't the promise of cloud computing that your install/deployment is as simple as with a single-server but the stability and load handling is as robust as scaling out? I haven't looked much into cloud-hosted apps so I don't know if that's the idea or not.
Another fun Microsoft license note. If you are non-profit the licenses come at a 95% (approx.) discount. So for a non-profit to scale out would be cheap.
What about the "blade" type servers that were all the rage a while ago?
It sounds like you get all the benefits of "scaling out" while ostensibly reducing power/cooling costs.
You're paying a tax for proprietary hardware for sure, but the economies of scale has to work better than a bunch of 1U racks.
I think the article would be more interesting if you took into account specing them out to similar cpu's and ram etc. That way you can see how many U's different they are along with power consumption and software cost.
Also I'm assuming M$ chargers per install of SQL. Oracle chargers per CPU core for most processors.
Building and then maintaining 83 servers is a serious time drain. You also need all sorts of new systems in place to handle updates, maintenance, log checking in any reasonable way.
Since you seem to value your time quite low (you like to do all these menial tasks yourself based on your statements on the SO podcast and this blog) but for many people, that would be a huge piece of the puzzle.
With 83 servers, you probably need to hire a full-time sysadmin, which should be factored into your figures.
"scaling out is only frictionless when you use open source software"
AND when your time is free.
captcha: "BOSTON expound"
Your comparison relies on your 83 1U servers being equal in performance to the single piece of big iron, so of course the licensing/power/rackspace costs will be far higher.
In reality, filling the same 7U space with 7 1U servers is likely provide quit a large saving in cost (especially the initial hardware) for only a small increase in complexity. This would also provide redundancy as already pointed out.
This particular example has more specific considerations than the more general "scale up/scale out" question. Here we're dealing specifically with Microsoft SQL Servers. (I don't think my SQL Server knowledge is that out-of-date, but please forgive me if it is.)
SQL Server doesn't "just cluster" in a way that you could add another server to some magical configuration and everything will work just fine. You have failover clustering, in which a second server can be primed and ready for traffic if the first server fails. You also have federated databases, which each system stores part of the data and knows how to access the other parts if necessary.
The point is, a SQL Server cluster requires a lot of custom code, either on the application side or at least the SQL DDL side. You need to take the cost of writing and maintaining all of that code into account when considering this particular question.
(Look's like SQL Server 2008 also has "Peer-to-Peer Replication". I don't think this is just magic, still, but it may help.)
This has been touched on by others, but the cost of maintaining the scaling out option, it's more complex software issues, etc, are not trivial, and you will get more hardware failures, though each one should matter less due to the redundancies.
It's probably a case by case kind of decision, but I would probably generally favour a compromise approach as well. Scale up until it starts to get nasty expensive, and then look into distributing the workload. You can do a surprising amount with one or just a few machines.
I work in hosting for a *large* hosting provider, so for once I have some insight to share. A lot has changed in the datacenter space over the last few years. But first the things that haven't:
- Separation of application and database duties is critical so that each can scale independently of the other - the databases scale up, and the applications scale out.
- Typically databases scale vertically (what you refer to as "scaling up") as they have a central data repository that must be kept in sync across multiple servers. So you tend to see either "big iron" or very high CPU systems act as database servers. Clustering - whether active or standby - is a must. Systems that do scale out like Oracle RAC are frightfully expensive and difficult to maintain.
- Application servers *should* scale horizontally. Just throwing processing power at something is a good sign You're Doing It Wrong. If an application properly keeps its state in a remote backend database (see Very Large Database Servers above) then you can add as many application servers as you need, using pizza boxes or whatever is most efficient.
- VERY few applications will scale linearly as processors are added, as most only have so many threads and parallelism in place. Serial operations happen, and will reduce the potential of these monster "scale up" solutions.
- For horizontal scaling you'll likely need external load balancing, which is another piece of hardware, and another thing to power, cool, and manage.
- Power (and cooling, which is tightly tied to the wattage you're pulling) is critical, so you're likely to want to find a middle ground between a 1U pizza box and a 7U monster like the HP. There are also other issues such as storage, expansion slots, etc.
Now as for what's changed. In a nutshell, performance per watt.
- It used to be you could just about count processors by rack unit. A 1U server was a single processor, 2U was dual, 4U was quad, etc. There were some minor efficiencies gained - 4U servers tended to become 3U over time - but not much. Until multi-core processors hit.
- These days, you can pack quad-core processors into most servers. A 1U server can typically now reach 8 processing cores (2 x Quad-Core). So while that 32-CPU beast may be 7U, if you just need raw processing and memory, you'll likely still come out ahead using small pizza boxes. Internal storage and expansion will be minimal, but if you're using NAS or SAN it's a non-issue, and you'll get better storage management to boot. SAN avoids burdening your network and offers much higher performance, but at a price.
- Licensing varies, but Microsoft in particular considers a single die 1 processor, regardless of the number of cores therein. This heavily favors multicore solutions and reduces the licensing penalty above.
As others have commented, your costs and numbers above only make sense if it actually took 83 of those servers to match up to the HP - something that I doubt would be the case for most well-written applications. If we're judicious and say that 16 would match it (personally I'd blace bets you'd still be ahead) then those numbers look quite different.
So in an ideal world, you'd have a nice SAN backing up your servers' data needs, a bunch of small servers for application (blade servers are actually quite nice in this niche), and a couple large clustered servers for the database, with a gigabit network for communication between them.
As everybody knows, open source is only free if your time is free.
What are your system admin costs for 83 servers? Not pretty. Probably cost more per year in salary and cost burdens than the big iron hardware. And don't forget to factor in some kind of serious hardware for load balancing, and admin of that.
I assume you wouldn't put sql server on every machine, right? The configuration for clustering those would be kinda crazy and probably not necessary.
Also, here's a thought...how about doing a little analysis and actually architecting your server setup to handle your user load...THEN start comparing different solutions.
Joshua: nice summary, thanks. Rounded off the original post nicely.
Joshua: You sir are spot on thank you for typing that.
This is why I don't do any ASP.NET work.
Interesting, but you can't really scale SQL "out" that way without a whole lot of application redesign.
Other types of servers, though (like web servers) would scale out (relatively) simply.
I guess the problem is that you pay the same price for a windows/sql server software that runs on XXXX CPUs than the version of windows/sql that runs on XX CPUs. I remember back then before the multi-core age that you used to pay for your software on the base of how many CPUs it would run on. That kind of scare away small developers because they can't afford their small server costing more on software than on hardware. MS should make a windows server license to run on most of 4 CPUs and be way cheaper.
Reminds me a lot of a calc 1 optimization problem, or trying to debate the merits of a 10x1 rectangle vs. a 1x10 rectangle.
Clearly, the correct answer lies somewhere in the middle, far from the two extremes you've presented.
I think the biggest tactical advantage for scaling out is that you can buy capacity a couple of grand at a time and install it without taking down the capacity that's already running.
Other than that, the big iron starts to look pretty good. I think this is a central part of the appeal of VM-based server consolidation -- to a certain extent, you can get the best of both worlds.
Great breakdown, though - thanks.
Something that others didn't seem to mention, when scaling out you're no longer running a single instance of Sql Server and Windows. With 83 separate servers you now have 83 instances of Windows and 83 instances of SQL Server gobbling up hard drive space and RAM. With 40 TB storage at your disposal drive space probably won't be an issue but I would imagine that Sql Server would act a lot differently when it has access to 8 GB RAM as opposed to having access to 512 GB.
Ubuntu 9.04 64bit Server + Apache 2.2 + MySQL 5 + PHP + OpenJDK 1.6 + Red5 0.8 = $0
There are some things money can't buy. These things are known as open source software :)
With the right planning, software licensing really can cost $0.
"scaling out is only frictionless when you use open source software"
scaling out is only frictionless when you use low/no cost software.
It's the per-server (or worse, per-cpu) licensing that frequently kills the non-opensource approach (which is not the sole difference of opensource software).
If you have a 'site' license for whatever product you are running, however, it may not be open source but may have a fixed cost, and you can still scale horizontally.
For a search engine (pre-google days) we migrated from a large 32cpu SGI machine to 70+ commodity servers allowing us to cancel the annual maintenance costs that were easily in the mid six figures (at the time). We tried to go horizontal with a J2EE engine, but found it licensed per server and found horizontal scaling limited by license costs alone.
"This is why I don't do any ASP.NET work"
Because the 13th most popular site in the US needs a big database server?
Thanks to Joshua Ochs -- great "scaling-up vs scaling-out" overview!
Thanks to Jeff too for asking the question. :-)
I did notice Plenty of Fish had got a bit faster recently ;)
Reminds me of the old "Grudge Match": a rottweiler vs. a rottweiler's weight in chihuahuas.
Scaling out implies several hidden "costs" such as synchronization overhead (replication and so on), balancing overhead and so on.
Scaling out can look nice but sometimes it will not be as nice as it looks.
The site I'm working at is shifting from a "scaled up" approach to a "scaled out" approach.
We currently have a few hundreds of thousands of users running off four or five SQL Server back-end boxes configured quite similarly to the big-end Proliant mentioned, plus further application servers and front-end boxes.
Our problem is threefold:
- huge bottlenecks into the SQL databases
- the application tier is monolithic and hard to update or scale
- the SQL Server licence costs are outrageous
So the solution chosen has been to rearchitect, from the ground up, into a set of 20-30 small services.
At the bottom, the core services each have their own database. Above this, a coordinating layer performs business logic. At the top an MVC layer does the presentation.
All services are stateless, so scaling out should be very simple: just add another box where necessary in the middle tiers.
We are replacing the single fairly monolithic database tier with a series of small databases that should share the load between them, and moving the business logic out of stored procedures to the services. This means we can move over to MySQL.
The downside? 150+ developers for about a year!
You've equated "Open Source" with "Free as in beer", which isn't necessarily true... buying RedHat Enterprise Linux is plenty expensive on its own.
Just for the fun of arguing :
are 4 virtual servers on 1 big iron (let's say 8proc/8Gb mem/8Tb hd) more interesting than 4 pizza box with the same specifications (2proc/2Gb/2tb) ?
It seems like you ought to be able to find some middle ground: somewhere between the big monster and 83 smaller systems. Perhaps 6 medium-level systems spread over 3 different sites, to improve both performance and redundancy. What would a setup like that cost? What happens when you throw cloud computing into the mix?
Even scaling out on open source software isn't frictionless. The cost of maintaining one (or two servers in an active/passive setup) is far cheaper than a hundred smaller servers... It's the difference between one big problem a year and five small changes a week. Also, the cost of developing software that gets the full benefit from distribution versus horsepower is typically more. State management and distributed transactions can consume at least one developer's full time efforts depending on the nature of the business.
Only people who have never tried scaling out a non-trivial DB app participate in these (stupid) discussions. Fact is, unless your DB can be easily partitioned, by design, you will have one heck of a time scaling it out and you will need a TON of hardware. Not everything fits into BigTable. Most applications have certain latency, throughput and concurrency requirements which often make partitioning / sharding difficult or unfeasible. If you want decent response times on a large partitioned DB and you don't want to scale up, you will have to consider things like Netezza, Teradata, Vertica, Greenplum, Aster, etc. For large amounts of data, those can cost $1M a year or more. The cost of scale up becomes easier to swallow in this case, assuming your problem can indeed be solved by scale up.
My point is, just because Google solved some of their (very specialized) problems by scaling out 10 years ago doesn't mean you should do so now. It usually makes sense to scale up within individual nodes (i.e. have 8 cores and 32/64GB of RAM instead of the typical 4 and 16GB of RAM) and have more cores and fewer nodes overall. It also usually makes sense to have a good RAID controller and a wide RAID array (600-800MB/sec read / 400-500MB/sec write). It does not make sense to use Hadoop for anything unless you can cough up more than 15-20 nodes, significantly, orders of magnitude more. See, Hadoop only begins to beat _one_ decent 8 core box when you have 10 nodes in the cluster.
I mean, Jesus tittyfucking Christ, if you haven't done anything large scale, just STFU and don't regurgitate the same old bullshit you've heard five years ago.
That is all. Thank you.
I haven't seen anyone mention using an in-memory data grid with a scale-out/scale-up approach. An IMDG scales out to handle more load while lowering the load on the database box. Buying a smaller number of 1U servers with maxed out memory and a database box a fraction of the cost in the post is more predictably scalable.
Depending on your grid capacity a database failure may not be as catastrophic as it once was. If the entire contents of your database fits in the grid then the application using it will continue to perform finds and updates just the same as if the database was up. The grid batches all updates and will update the database when it is available again.
IMDGs also offer a ton of features for graceful failover, data availability and redundancy and self-healing topologies. All this works like magic.
Two great resources for IMDGs are Billy Newport's blog (http://devwebsphere.com) and Nati Shalom's blog (http://natishalom.typepad.com).
As many have said, yes, this is slightly ridiculous. There are very few database loads which will scale from one machine to 83 without quite a lot of messing, and frankly most of them are key-value oriented and you wouldn't want to be using SQL Server anyway.
It's a very different story with application servers, of course; it's usually the more the merrier there and you can greatly reduce admin overhead by using a standard image and code pushing method.
>> And this is why Microsoft HPC will never be a serious offering. Who would want to pay that kind of licensing for 32, let alone 1000 servers?
This is why Windows HPC edition is quite a bit cheaper to license. In fact, if you buy software with support (as large clusters often do), even _plain_ Windows would be cheaper.
Good points. I think in the end you could justify either approach. There are some other factors as well:
Scaling up will typically involve less man-hours deploying the app. Scaling out will require more time unless you fully automate the deployment process (Azure for the enterprise anybody?)
Scaling up will involve less network complexity and infrastructure. Scaling out will involve more.
And this is why Microsoft HPC will never be a serious offering. Who would want to pay that kind of licensing for 32, let alone 1000 servers? I maintain a small (14 nodes, 64 cores) cluster for a couple of labs, and there would be no way they could afford that kind of iron if they had to pay to put Windows on there.
Also, your story of scaling up rather than out means that unless you also increase the number and bandwidth (like myrinet or infiniband) of your connections, they will be maxed out over a very small demand on CPU, given most web applications.
"But what if you scaled out, instead -- Hadoop or MapReduce style, across lots and lots of inexpensive servers?"
None of the calculations take into account the cost of rewriting a large portion of an existing software base so that it can leverage a distributed computing environment instead of a classic 3-Tier approach. Not to mention the reduced time to market that would be accrued in that time period. I doubt Markus was willing to wait a year while his developers rewrote everything. Even if his group did decide to leverage distributed computing he would need some machine to run everything until the overhaul was completed.
This article has some good points, but it completely ignores the most important part of scalability: figuring out what problem you're trying to solve. When you scale up, the marginal utility of each core decreases rapidly.
When you're solving a lot of "Google-Scale" problems over extremely large data sets, scale-up architectures are absurd: you're quite I/O bound. My company needs to crunch through dozens of terabytes of data on the social media space, comprised of billions of multi-KB records. Hadoop and HBase are the only way to tackle this problem.
If you absolutely need to have a 100% RDBMS like I believe the OP is talking about, then there's companies like Greenplum and Vertica, who recommend $25k boxes or so, and can process petabytes of info far more efficiently than a homegrown SQL server farm.
As for, "As everybody knows, open source is only free if your time is free.", we've spent $0 on maintaining our Linux farm besides the salary of 1 person. The marginal cost for supporting another box is 0, because of the amazing variety of cluster management tools in the Open Source world. UNIX was meant from the beginning to be a highly available, network-enabled environment.
I just wrote an article on http://www.roadtofailure.com on why Scale-Up is dead to a large portion of the problem space :)
Scaling out isn't frictionless with open source software. There's not exactly a zero-configuration scaling package that I've seen, you'd have to spend big money on your developers/admins getting it all to work together.
Two words: Cray CX1
384 GB memory
Geek factor: Priceless
Keep also in mind that not every application is originally designed to scale out. Scale out means you have to design your code and having an architecture to avoid server affinity.
I have seen projects come to a point when there a serious performance problem and they want to scale out. Then they conclude that their architecture is not well designed to work across multiple tiers, and then you have a real problem!
What happens to your comparison if you scale out to just 8 systems, giving you the same number of cores as the tricked-out DL785?
That would give a similar performance (distributed scaling issues notwithstanding) but at far lower capital cost and much more reasonable licensing and running costs.
I think Joshua Ochs hit's it best when he talks about the right hardware solution for the right application. You can argue MS SQL vs MySQL clustering and on what OS and what type of architecture, but bottom line is it depends on the application. Everyone wants a fabric network of servers like Google's or amazon's where host added and host failures are seamless to the processing demands. But until Linux or MS build us that better mouse trap to have and hold as our own, we are stuck with weighting our needs on an individual need. ...and don't point your fingers at those high end servers until you've read the specs on the redundant power supplies, nic, backbones and raid arrays. Sometimes you do get what you pay for when the application will not scale out. On the other hand much is to be said for failover one can archive with pizzaboxes as you call them. IMHO, The happy medium is scale as you grow, pay as you go and never pay for more than you need. The old saying the right tool for the right job, applies to hardware also. Good post Joshua Ochs
> And when that one monolithic machine dies due to any number of reasons, which one is better then?
captcha: lesson tinfoil :_)
For 100K$ you can get 4 SuperMicro 2U Twin², which would then have:
128 Xeon Cores
768 GB Ram
16 TB Diskspace
8U Space requirements
With a maximum power requirement of 4800W
That doesn't look to bad for me.
And this is why 2010 will be the year of linux.
Take a look at CPU vendors. Years ago they were scaling up. CPUs got more and GHz of processing power. Now they are scaling out, instead of making a single core faster, they add more and more cores.
Have system gotten faster because of this? Yes, if the application can make decent usage of all these cores they got much faster, even though we are back at around 2 GHz and were far beyond 3 GHz before. A current 2 GHz Dual Core CPU can beat the crap out of an overcloked 4 GHz CPU with a single core.
As you pointed out, the advantage of scaling up is that the software doesn't have to be optimized for anything in particular, while scaling out only works well if the software can really operate that way. However, pretty much every software can. Just take a simple LAMP project (Linux, Apache, MySQL, PHP). You can have a single Apache server in front that gets all requests and spread these across a whole server farm of Apache servers, totally transparent. Each of these run only Apache and PHP. The MySQL Databases can be scaled out to different servers again. You can go as far as having one server not for every database but actually for every single table of a database.
Managing 84 servers is not that nice, and you should expect an increased % of HW failures. And since I work in a farm with 30 "barebone" servers and some blade servers I can tell the difference
Additionally 84 servers mean clustering or redesign of the applications running on them, and more maintenance, so additional time that hasn't been counted in the calculations:
or do open source activists work for free?
I don't think this comparison is really fair.
you don't need 83 1U servers to compete with the power of the 7U server.
So probably you might end up buying just 10 1U server to reach the same amount of CPU power.
This will also reflect on the licenses, power and hosting costs, which will be just 1/8th of what you write.
And the big monster is not enough... if you are talking about DB servers, than you'd need a cluster, which might cost even more (external disk unit, 2 physical machines) and will have higher costs attached (if I'm not wrong SQL Server Datacenter edition, the only one that run on clusters, costs 30K).
This also must be evaluated based on the kind of software you are running on:
DB servers are most likely to scale up, with redundancy
Web Farms are better suited to scale out, same with application servers.
Another option you have to consider is: putting everything up in the cloud.
Nice post Jeff.
A client of mine recently got their main DB turned into a clustered DB server at a major hosting company. Clustering (for them at least) basically means you get some amount of fault tolerance at the 'mere' cost of 2x the main processing boxes (i.e. 2x Windows Clustered servers) and some _extremely_ expensive SAN storage that is used by both machines.
You, however, will quite possibly have the option of splitting your DB's into 3 (I'm assuming one for StackOverflow, one for ServerFault and one for SuperUser). Depending on your willingness to lose 'minuts' of data, you could then vary the sizes of these servers by necessity, have them at different sites, and use a circular roubd-robin approach to log-shipping. In this way, each 'DB' machine would need to be able to store it's own DB and the one it was receiving log-ships from, and computationally be able to 'muddle through' if it had to actually process two DB's at once due to one server failing.
This is not true fault-tolerance, but may offer a pragmatic intermediate approach to the big iron and clustering and so on.
Another fun possibility is to rent instances on the Amazon cloud.
Prices run as low as $428 /year for a reserved instance or can be billed by the hour if your computing demands change frequently. If you build a good AMI image, you can automatically create and destroy instances as needed for load handling.
Of course if you "brick" an instance with valuable data on it, you're *very* hosed. There's no option to drive/fly to the data center to hook up a monitor and keyboard to the thing for the purposes of stepping past a boot time kernel panic. (this has been my headache this week)
I don't mean the costs of scaling up vs scaling out in hardware, that's pretty old news. But I was never fully aware of how hard shops using proprietary software were getting screwed until I read this...
In my experience, very few problems scale out easily. Scaling out always looks good on paper and never goes nearly as well. Having a lot of small things is almost always more painful and expensive than having a couple of big things in practice. Even if the big things cost 5 or 10 times as much. Even with outsourcing, people are expensive. Even if you aren't paying them much you still have to manage them effectively. Effective management of people is the hardest thing to do in business, IMO. And the more "things" you have the more people you need to manage them.
Now I won't deny that there are some very specialized, large-scale cases where scaling out is the only option you have. But before you go that route, you better triple check your plan and consult some people that have real-world experience with more than a calculator.
Having fewer, more powerful processors actully provides you more flexibility for handling intensive, linear tasks while still being able to "scale-out" virtually. If you currently have a process like that and you could spend 50K to bring a single new server in to make it run in 1/4th the time or you could spend 50K on a project to re-architect the process to be non-linear (assuming that's even possible), going the hardware route is a no-brainer.
Markus obviously wants to be a one man shop so one big server makes sense. Clustering, replication, SANs etc. Too much work, too much to worry about with diminishing returns. You also do not want to rewrite your application ground up to support some kind of scaling out strategy. I don't see Markus touching what is working already.
Bigger operations are a sprawling jungle of complicated and confusing software with piles of servers not doing a whole lot of useful things. That would be just about every company on the planet. That's why Markus should be a scaling hero.
I've had jobs ranging from embedded development on hardware that would fit in your shirt pocket to several years spent at a national lab with a floor full of supercomputers. It's interesting to see that these scalability issues crop up in a similar fashion at all parts of this spectrum: "bigger" versus "more".
But in many ways, as the manycore folks suggest, "bigger" has become "more", as the way things have gotten bigger is to have more cores. Licensing issues aside, we are faced with transitioning our production systems from one or two core platforms to dozens of cores. This transition presents similar issues (differing perhaps only in scale or maybe not at all), whether those cores are all in one chassis or spread across a campus.
I'm guessing that we'll all have to starting thinking about the memory architectures and processor interconnects a lot more carefully, much like the supercomputer and MPP folks have always had to do. Suddenly we're faced with deskside MPP systems that aren't the exotic hardware we once may have known and loved (or hated) but are the latest commodity whitebox we bought over the internet.
I believe that embedded developers and supercomputer developers may have a leg up on everyone else because they've been dealing with these issues for decades. The manycore folks at Berkeley seem to also have the opinion that the embedded area has a lot to contribute in this domain. I think product development organizations should consider leveraging any developers they have in the embedded domain to multi-core/manycore server-side development since (I hope) the skill sets are transferable.
I think it's going to be kinda interesting.
I would suggest outsourcing the hardware infrastructure to Amazon Web Services. They have brought transparency to many of the hidden costs associated with scaling.
For instance, they have figured out the pricing of Microsoft SQL and present it to you in a way that is predictable, i.e. 12.5 cents per hour.
Obviously, a linear model doesn't work for everyone financially -- but it does take the guess work out of it.
We're a Microsoft shop (and a gold partner, too, so our cost for OS/SQLServer software for internal development use is $0). We also use virtuals for development. So for us, the key feature of a server is the amount of RAM one can jam into it, as each virtual is going to be sucking up 1G of RAM (more or less, but it is a reasonable rule of thumb). So that honking great beast of a machine will be able to host close to 500 virtuals for development, while the 83x 1U servers are probably going to be close: about 6-7 virtuals per server. On the down side, 83x 1U servers = 2 full racks of servers with a bit left over, while the big beast is less than 1/4 of a rack.
Our current AC infrastructure has a hard time handling what we've already got. It is a 20-ton unit and needs maintenance several times per year (especially when the cottonwood trees are shedding and the roof heat exchangers start looking like cubical sheep).
We're further constrained by corporate policy that states we are forbidden from obtaining "used" stuff, so the ebay items are way off limits. This was comically absurd when I was looking into an itanium server to develop against (we can check x64 builds with 64-bit vista/server 2003/2008, but for ia64, we need totally different hardware). The cheapest new itanium box comes from HP and starts around $15k, while used 1U itanium servers are in the $200 range (with RAM and HD, you're still looking at maybe $1k per).
A lot of excellent discussion on this post - it's been an interesting read. A few more things that are particular to my background...
- As for software licensing - bbviously scale out will incur more costs in that arena than scale up (unless you're 100% free/open source); how badly that affects you is extremely application and vendor dependent.
- Scaling out is very hard to do if you didn't design for it in the beginning. Let's face it - most homegrown stuff isn't designed up front for issues like this, and so it's going to be difficult/impossible to scale many such systems horizontally.
- Horizontal scaling requires a much more dedicated approach to system management - you'll need to be able to automatically build servers, apply patches en masse, backup/restore, push content updates, etc. Content management systems, reporting and alerts - just more of everything. Moving from 20 to 80 servers isn't hard, but the move from 4 or so to 20 is. This stuff is not trivial, but depending on how much value is in your infrastructure (is this your core business, what's the size of your business, etc) it may be warranted.
- Scaling out does have other hidden costs. You'll need external load balancers, as I mentioned. You also may need a beefier network, better/more switches, NIC teaming, etc. Centralized storage like SAN or NAS - while I highly recommend it - can be complex and costly.
- And of course, you can meld the two. Not only by separating your application and database servers and scaling independently, but by splitting the database into multiple servers. This may not always be possible, but at least we're back to standard software engineering techniques of factoring and database design.
- Virtualization makes a lot of these issues even trickier, but offer the hope of major gains *if done right*. You gain a huge amount of flexibility in adjusting workloads, CPU/memory allocations, or migrating systems to better servers. You also have an additional layer to manage, and have to be very careful balancing resources between virtual machines. This is especially difficult when all systems are the same, as you can't use the differences in application design to your advantage (for instance, placing a CPU-intensive VM on the same system as an I/O-intensive VM, etc). However, don't expect it to be cheaper - far from it. Hardware cost tends to scale exponentially once you pass 4CPU (16 core) systems, so partitioning a 32CPU server into 32 x 1CPU servers won't likely be cost effective.
- Beyond virtualization there are interesting technologies like Solaris Containers and other lightweight partitioning schemes to consider. Not even going there - this is long enough. :)
- Finally, planning for failure. You definitely don't want a single huge machine, as everything is in one basket. Many folks will then move to clustering for redundancy, but then you have a new set of trade-offs: do you go with Active-Active clustering, and watch your ability to handle load drop by 50% when a node dies, or go with Active-Passive clustering and have an expensive paperweight 90% of the time? It may hurt to have a passive server sitting there, but the last thing you want to see is one server die, immediately followed by your second server cratering under the load.
- Scaling out offers built-in N+1 redundancy, in that a single node failing takes out very little of your capacity. This makes hardware management somewhat less critical, as you haven't lost much capacity (relatively speaking), and you're not sitting vulnerable on a single point of failure until it's fixed.
So... from my perspective, scaling out is the preferred method, because of the usual cost savings, flexibility, simplified growth, and redundancy. However, it's not for everyone. Your applications may not be written this way, your databases may be tied to closely to the applications, or you may not have the necessary support infrastructure for lots of smaller machines. Like so many things, it's a set of tradeoffs - but that's what keeps our field interesting.
Your lisencing costs are way off.. Its a website you only need Server Plus CAL Pricing which is a tiny fraction of the cost of per cpu lisencing
Your lisencing costs are way off.. Its a website you only need Server Plus CAL Pricing which is a tiny fraction of the cost of per cpu lisencing
Your lisencing costs are way off.. Its a website you only need Server Plus CAL Pricing which is a tiny fraction of the cost of per cpu lisencing
re: [Joshua Ochs on June 24, 2009 10:10 AM]
Joshua, I've saved your comment off in a text file that I will read repeatedly. Thanks man.
Jeff, thanks for the thought provoking article, without it Joshua wouldn't have ever replied with so much good info.
Jeff, you may want to read the "Special Considerations" (http://www.microsoft.com/Sqlserver/2005/en/us/special-considerations.aspx) page about licensing on SQL Server. Basically it says that you need a CAL for each computer (or user) accessing the database, even if they access it through some other middleware application such as a web server. So if you have a public website, the only way to go for licensing is per processor. With the big 7u server with 32 processors, that would require SQL Server Enterprise edition (standard only runs on a maximum of 4 processors), which runs about $25,000 per physical processor, which would cost you $200,000 just for SQL Server licenses. On the scale out solution, you could run SQL Server standard edition, which is still $6000 per processor, which would run you $498000.
For someone who insisted that he build his own servers from scratch, you seem to know very little about the licensing of the software you installed on them. Which makes me wonder if you even have them licensed correctly.
This comparison is wrong because the Microsoft stack - Windows and SQL server - is not built (or priced) to scale out cheaply.
Systems that are built for large scale using a different set of technologies (i.e not relational database) to enable this kind of growth - map-reduce frameworks like Hadoop, column base databases, distributed data warehouses.
Scaling out is not a matter of choosing between 1 bigass server and 83 small ones. Software has to be built to scale out, and when it does costs should be cheaper...
A somewhat more interesting comparison might be how Wikipedia works: http://meta.wikimedia.org/wiki/Wikimedia_servers
It's something of a compromise (albeit with all free software)—they scale up in some places, out in others, depending on the function of the servers in question (and take care to divide up the functions a bit so that they can do this). On the whole it runs rather smoothly given the size of the databases, the number of constant I/O operations, and the number of visitors.
As others have said, the management cost of 80 servers is non-trivial. Once you have that many, you'll probably want to know if one of them has a hdd warning, an ecc memory warning, or a fan failure. For one server that's not so bad, but for 80 you're going to want to deploy one of the vendor's hardware management solutions (which are free, iirc) but non-zero to maintain. Oh, and I suspect you'll want to patch and deploy app updates to those servers, that's non-trivial and non-zero cost also.
One other difference is that the larger servers from the major vendors build redundency into the server to avoid problems like a power supply failure taking out the box. The cheaper servers don't employ those things to keep costs down and exactly because it is assumed you'll deploy them in a way that one going down doesn't take down the app. So while you have more of your eggs in one basket when it comes to the DL785, it is a more resilient system (and one big reason the chassis cost is $16k alone.) The cheaper servers are more cost effective but less reliable. It's a reasonable tradeoff but one that has to be factored in.