Idea counts, dev't is the issue of Developers, not of users. People would still go for a DOS based app if you give 'em what they need!

Don't worry. SO has started good. We are all lookin' forward to its massive success.

How is Stack Overflow doing? I see it's up to 50,000+ questions. How many users? How many hits? Are you seeing any issues with scaling?

I haven't noticed any performance related issues. My biggest problem with the site is the signal-to-noise ratio seems to be falling due to the sheer volume of questions and answers. I wish there were more ways to sort and analyze the questions to help me find what I'm looking for faster.

at first i thought it's about the image rescaling issue, which is something PoF.com clearly needs to work on as well...

Sigh. Another poor domain name choice (a la expertsexchange) -- I read that one as "plenty offish".

Yes. Back in the good old days, the programmer was much more of a tyrant. He had absolute control over everything.

Now, with the advent of - open source libraries, internet, web services, write once run everywhere languages, external data sources, dynamic loading of code even in statically typed languages - you write a program of 1000 lines which is nothing more than the tip of a 100,000 lines ice-berg. The other parts were written by other guys, who will maintain them, change them, sometime remove them, without consulting you.

Programming is now much closer to democracy (some may say, anarchy). By the people. For the people.

Hmmm, www.okcupid.com is about 10x better in every way, and is also completely free. Plenty STILL stretches uploaded user images into squares making its thumbnails useless. Plus, the people on it are generally, to be frank, trashier than those on OKCupid.

It's amazing what you can do without that much hardware. And this is even truer for a free service that doesn't need to be up 99.99...% of the time.

But even so, you have to give credit where credit is due. Markus really knows what he's doing. To be able to handle that kind of traffic by yourself, well it's legendary!

Hey Now Jeff,

StackOverflow will scale & it's a credit to ASP.NET.

Coding Horror Fan,

Catto

Alright! thanks for the link. Now to loggon and get laid! Yeeeeha!

Be careful when you say "uses ASP.NET". Part of why he was able to run so much on one server was that he didn't use web forms, server controls, etc. Raw string concat (this is according to an interview he gave on Channel9).

BTW, Markus was my answer to the Stack Overflow "who is your programming hero".

Well if Microsoft reads this they should donate him some resources in exchange for the advertising he's already doing them. Putting these numbers on a fact sheet for ASP.NET wins the competition hands down.

There is a wicked article about how he scaled at:
http://highscalability.com/plentyoffish-architecture

But it is definitely not a credit to asp.net, he used none of their controls. We are doing the same with the web app we are building, we use all the ado.net, .net xml classes and the rest of the framework except asp.net controls. We found the controls generated WAY too much html for our complex UIs. We do use http handlers and some webservices for our ajax stuff.

He's got some of the worst HTML soups ever: mixed case, tables and bad javascript... yeah i'm envious of his success! ;-)

Wow, it is great to see others, starting a web site, and making loads of income from it. Mostly, unexpectedly.

Also, this is just a testiment to the power of the word FREE, has on people.

Sure you can goto match.com or eharmony.com or millionairematch.com and PAID, but it goes to show, that a BASIC site design, with little to no CSS site design or fansy business front, is needed.

I myself run very successfull advertising sites, using the power of FREE.

I love this kind of post because sometimes is very hard to find information of this type with real numbers. I don't want to debate about other technologies here or what/who is better, but I would love to hear/read about other combination of OS + Web Framework + DB with real numbers, any link/references posted here would be awesome.

3 <I>billion</i> rows?

Do I want to know what that's a table <I>of</I>?

His table with 3 billion rows, actually flows into a much larger observation.

I've seen this in every company I've worked for. Subprime mortgages, healthcare, insurance, finance, etc. You start an app and it works great. With each passing month you have more data in your database. Eventually you have 1 million records and you start looking at indexing, then it's 5 million and you start looking at upgrading hardware. Then you have 50 million and you start fine tuning queries and such.

The longer you stay in business, the more it costs you to handle the transactions. Now presumably during this same time period, you're also automating other things and building economies of scale. This whole time the database is growing, we're talking about how can we archive off stuff, but nobody really wants to take responsibility for that answer.

It's like there is a natural progression where the older a company gets the more inefficient it becomes, and this allows for a competitive advantage to a young startup.

That is, unless you do something about this problem.

And it's not just data, it's a lot of stuff that you collect over time as a company. Could be boxes of paper. Could be a fleet of airplanes. Just interesting.

> Part of why he was able to run so much on one server was that he didn't use web forms, server controls, etc. Raw string concat

Isn't that pretty much how ASP.NET MVC works? It's certainly one of the reasons I chose it.. just get the heck out of my way!

So he's got 16 CPUs for the app servers and they are doing 500-600 hits a second. ie 37.5 requests a second. That is about right really.

Take a pretty normal Ruby on rails application and a single CPU will do about the same. Of course scaling it so linearly so easily is the key here but even so its not uber, its about right. Now if we were seeing 70 hits/sec then this would be about twice as fast as most sites can do and that would be special.

Jeff give us some simillar statistics regarding StackOverflow.com please :)

>Isn't that pretty much how ASP.NET MVC works?

MVC is a lot closer to that than Web Forms. But it still goes through the rendering pipeline, has master pages and user controls, route resolution, parameter binding, ... Markus' version is just about as close to the metal as ASP.NET can get.

I have to admit, I always look forward to Catto's comments.

Love ya Catty

More interesting than the scaling strategy (of which there are plenty of descriptions already on sites like highscalability.com) is the Plenty of Fish owner's position that "free does not scale". he makes some very interesting arguments; it would be a lot more interesting to have Jeff's take on *that* !

(edit: ... rather than a passing reference to that argument; this is a really contentious issue, and the contrarian position is poorly explored)

Here's an interview and tour with Frind from his Vancouver host Peer1:

http://www.peer1.com/aboutus/customer_stories_plentyoffish.php

@Paul Keeble, You have any high-scalability stats or links to back that rails claim up? I'm not a rails-hater, but last I heard rails wasn't 'linearly scalable' and nobody pretends it is.

It appears most of POF's high availability tricks are applicable to any language. All it's doing is reducing framework overhead. When you add generic frameworks (web forms, rails, mvc) you're making it easier and less buggy to program against, but you also make it slower overall. For most websites this is okay because you just chuck a little more hardware at it and the problem goes away.

This isn't the case for high-traffic websites, which is the whole point of the blog post.

Anyways it just goes to show you that how the program works (any program) is more important than how slick it looks. I'm also impressed that this is just 1 guy working away.

What I'd also like to know are the economies involved. The revenue from ads is high, and 2 servers aren't that expensive, but what is the bandwidth bill like?

"to be frank, trashier than those on OKCupid."

You say that like it's a bad thing ;)

@Paul Keeble, someguy:

Rails is not scalable - the database abstraction will kill a site sooner or later. Twitter is a case in point (ironically used more often by RoR advocates missing the point). It was scaled primarily with a massive amount of caching, which, of course, bypasses almost all of Rails' clunky whirring & grinding.

You shouldn't be surprised that fish scales ;)

Actually considering the hardware, I don't see this as a scaling wonder. The hardest work is probably performed by the database servers and we have no information what giant performance beasts these are.

Handling 64000 simul. connections can be achieved with much worse hardware. Each of his servers has 8 cores (2 CPUs Quad Core) and he has two of them. That 16 cores, each 2.66 GHz, can handle 64000 is not really surprising. These are 8000 connections per core and that is doable with 2.66 GHz.

If I profile our web servers, I see that Apache is actually only 20% of the work-load (and that includes the PHP scripts running in Apache) and 80% of the time is taken by MySQL performing database operations. So if we offload the database to an external server, we have 80% CPU time spare. Ok, I must admit that we pay a lot of attention to make sure that MySQL does the actual work wherever possible. PHP doesn't do much more than building the SQL statement, firing it to the database and put some HTML formating around the results.

We made sure that all heavy work-load is performed by MySQL (since if you can do something in PHP or directly in SQL, you can bet that SQL will do it faster than you ever could do it in PHP). And now that MySQL has Subselects, you can run even much more complicated requests within a single SQL statement instead of fetching data from the database in PHP and then build more SQL statements based on this data. Thus whenver our server slows down and check where the load comes from, usually always the database is the culprit.

Somehow we developers just can't convince the management that we must stop running the backend database on the same servers as the web frontend... they don't want to listen to us. We could probably once buy a set of really huge performance beasts to run the SQL backend and easily replace our web frontend servers with much weaker, more lightweight machines.

3 billion records, sounds like a script needs to be run through every so often to clear out old data, its probably message posts or something if it's that large. Just archive it off, improve speed even more.

> Part of why he was able to run so much on one server was that he didn't use web forms, server controls, etc. Raw string concat

Does anyone use the asp.net webforms/controls?
I find them extremely annoying but i use asp anyway because i love the .net framework. I only use the asp:literal or asp:placeholder to put the generated content on the page.

Not to ben an a**hole or anything, but I don't find the design of SO that much better then that of plentyoffish. It's a little better, but it better should because you guys hired a real designer for it, right?

When you compare PlentyofFish with MySpace, it makes you ask the question:

What the hell are MySpace doing will all those extra servers?

@Mecki: Ok, I must admit that we pay a lot of attention to make sure that MySQL does the actual work wherever possible. PHP doesn't do much more than building the SQL statement, firing it to the database and put some HTML formating around the results.

Refreshing to know that a youngster (you are, yes?) gets what databases are good for. Not that MySql is spiffy for a transactional application.

2x database = 6x maintenance?
do you really need a database in that instance?
(might a filesystem suffice?)

Jeff, does your wife know that you're spending so much time on PlentyOfFish?

ASP.NET controls are great for prototyping and rapid development. If you need to throw a site together and don't want to start mucking around with HTML, javascript, and input checking you can't do worse than what's available. But it's probably not a great idea to make that prototype into your main site without a lot of optimization. When I was getting started I never understood why some sites would use println after println. It made it difficult to update and maintain those pages. It became apparent why not too long after (though I still hated to maintain pages with hardcoded html).

Also, after updating so many customer databases it really becomes a nightmare to keep a database running. Sometimes it would be easier to just flush the entire thing and start fresh but they have years of information stored on those systems. You learn to tiptoe around the problems and learn the importance of transactions.

Scaling issues are always the fault of some programmer who made a tool or library that you use which is not intended to be scalable.

So, in reality it's always your fault :)

"The problem with free is that every time you double the size of your database the cost of maintaining the site grows 6 fold. I really underestimated how much resources it would take, I have one database table now that exceeds 3 billion records. The bigger you get as a free site the less money you make per visit and the more it costs to service a visit."

I'm not sure what to make of this. I've never run an ad supported website so maybe I'm wrong but I thought that advertisers paid more money if their ads were exposed to more people. So higher traffic should mean more cash from advertisers right?

@sp160n: Hmmm, www.okcupid.com is about 10x better in every way, and is also completely free.

# 1.2 billion page views per month, 500,000 average unique logins per day
# 30+ million hits per day, 500-600 per second
# 45 million visitors per month
# top 30 site in the US, top 10 in Canada, top 30 in the UK

It doesn't matter. I think that's the point of Jeff's post.

If you haven't known, check this out.

[duplicate of 2nd link in article]

he's the no.1 adsense user in Canada, making 10million a year, which is way more enough to pay off his bandwidth bill. :)

I disagree with the last quote. There were definitely hacker communities in the early days of programming (see Richard Stallman's book Free Software, Free Culture for examples) that encouraged sharing of code to a great degree. Heck, Stallman even started GNU following frustration from being unable to debug a proprietary printer driver, which was pretty much the only proprietary thing in the entire AI Media Lab, operating systems included.

Another thing to point out is that he is serving all those images through a CDN (Akamai). While very expensive, they do remove the load, space, and bandwidth from your servers

From highscalability blog: "IIS arbitrarily limits the total connections to 64,000 so a load balancer was added to handle the large number of simultaneous connections"

WHAT?!?! And people actually pay for that?!?!

"No caching functionality in ASP.NET is used. It is not used because as soon as the data is put in the cache it's already expired."

I'm thinking this is an example of "doing it wrong".

There are so many ways/technologies/techniques for building websites it's staggering ... least to me it is.

It's funny because the very first asp.net app I ever wrote simply had a server tag for returning the page generated server side. I thought it was totally hack at the time, but maybe not so much?

Then later on when I had to build some sites for a customer (and this is without, really any other experience) I used the ASP.NET server controls and whatnot for user entry and all. I really didn't have any time (as I was the sole dev) to learn some of the MVC/jquery-AJAXy style stuff ... and while the site works and is pretty solid, I read articles like this (and some of these comments) and just go "Damn, 101 ways to skin a cat indeed..."

Scalability worries me quite a bit. Very soon, me and two co-workers will be developing a web app (using C#/ASP.NET) and have very little experience in doing so as efficiently as possible.

I'm not sure our idea will directly get anyone laid, so it might be doomed to failure.

@Mecki "We made sure that all heavy work-load is performed by MySQL"

I'd have thought that you reach a point where you can only scale a database so far (i.e. better hardware), whereas if there is more logic in the web-layer, you can keep adding more servers to the web-farm to handle that load.

If you've got one shared server available (sounds like you do?) the MySQL heavy load seems fair - just wondering how you'd approach it if you had more servers available, and where you'd place the heavy lifting then?

Did you mention he makes around half a million dollars a month in adsense from the site? The limited hardware could also be done in the linux world by simply avoiding apache and using other far better (free/commercial) replacements. But it's the adsense that blows me away at $10 million in income per year.

This is very comforting as I am planning to scale my website which also runs on asp.net mvc. It is a hosted microblog platform and it is still in private beta and less than 2 weeks old and I already have 200+ sites created. My problems will be less than stack overflow because this is a paid service, so the growth will be limited. If it was free like twitter then I would run into those kinds of problems very quickly. The site I am referring to is www.yonkly.com

BTW, I am using Amazon EC2 hosting, so I can quickly provision new servers and do DNS round robin if there is a huge surge in traffic. Scaling the database would be another problem. But being on EC2 is very comforting.

I'm still doing everything myself except for customer service.

Current numbers are.

60-65 Million Pageviews per day, 4.4 million per hour peak.
72 million visitors a month
3 load balanced Windows Server 2003 x64 web servers with 2 Quad Core CPU sits at about 20% cpu at peak, 2 Quads... Makes it easy for me to take servers up and down without worrying about crashing.

Sites running on 5 DB's now, between 64 and 128 GB of ram, Got a huge 30% increase in traffic coming on Dec 26th same as every year. Every page on the site has a bar on images on it... To create that bar a radius search is done along that uses your history as well as preferences. Its not possible to cache anything as its true personalization and each query depends on the previous one. Same with all the searches.

Over 130,000 Concurrent users at peak.

500M images per day.

Asp.net for the most part is irrelevant when it comes to scaling, because it uses so little resources. I use StringBuilder, Response.Write and access the DB. C# is the last thing I think about when it comes to scaling issues, its 99% the DB.

Jeff -

In the 1970's many of my colleagues would not run a program unless they had read the source code and compiled it themselves. Those who wanted to share their work put the source on open display - it certainly was not like your love letter metaphor.

-Lepto

@Starburn: A database won't scale beyond a certain point, but where is the difference of throwing better hardware at it compared to running more web servers? In both cases you must buy hardware. The only difference is that in one case you *replace* hardware, in the other case you *add* hardware. However, no matter how many webservers you run, if they all share the same data, you'll need a central database again and this will sooner or later always become your weak link.

@BuggyFunBunny: Youngster :-) Well, I'm slowly starting to group out of this term. At least my cards don't say "Junior Software Developer" anymore, just "Software Developer". Lets see how many years to go before I'm a Senior. And all I know about LAMP I learned from the best (unfortunately one of them already left the company). When I started, my first project was a grown, old, web-based document management system. The task was: Make it run faster. This was the first time I was doing something meaningful with PHP and the first time I really had to understand SQL Selects in all their beauty. By limiting the amount of SQL statements (2 statements per page viewed instead of 31 statements per page, still with the same data in return) I was able to speed up the system dramatically. From that day on I really studied SQL statements to find out how much more work you can actually already do in the database, since it is also much nicer for a programmer to just create a statement of how the result should look like instead of getting a huge blob of information and having to manually "write code to get it into the desired state".

I think the simplicity desgin of Plentyoffish, gives users a feeling of trust, and belonging. After all, most of us (singles) are living in a messy room. When it comes to websites deals with personal matters and life, I much prefer the low key, causal design of Plentyoffish, just a sense of belonging.

Ross Lin.

i'm not sure if it's been said, and it definitely doesn't belong on this blog (except that maybe you care, since you said a few things about security).
plentyoffish has the WORST security in the history of the internet. not only does it (have to) have a completely unencrypted database of the passwords of every user, it sends em out WITH EVERY SINGLE EMAIL IN PLAIN TEXT.
why would anyone want their password sent to them every single month ? i do not know.
if one was in the habit of using the same password for everything he signed up for on the internet (not a stretch of imagination here) he could not read his email with someone standing behind him, ever.

ok the rant is over.
>> The problem with free is that every time you double the size of your database the cost of maintaining the site grows 6 fold.

that is a point to remember. and i think that's a point advertising companies should take into consideration. the problem with this is .... you hardly ever get negative growth. so eventually you will have to charge people for using your site since the ad revenues don't cover the cost anymore. which would be bad for everyone involved, the advertising company (or the sites you're advertising for) yourself, and the users.

Jeff,

I'd like to start a free budget/personal finance site and will need plenty of horsepower and storage. I'm definitely a nubee, so how does one get started in terms of hardware infrastructure? Do I need to get a dedicated IP from my ISP and obtain my own servers? Do I go with amazon or Sql Server Data Services/Azure or something like that? Is the latter services free or a discount offering available to non-profit/free services such as the budget/personal finance site I'm looking to start?

Every time I feel like I'm going numb and going into demotivational stage, I look up at Markus's work and refill my lunges with new energy.

Awesomeness!

Did you mention he makes around half a million dollars a month in adsense from the site? The limited hardware could also be done in the linux world by simply avoiding apache and using other far better (free/commercial) replacements. But it's the adsense that blows me away at $10 million in income per year.
http://sadushki.ru/

Its too bad Marcus wont share more information to help others with their web development and programming quest. I know many have helped him along the way, I am not sure why he would not want to return the favor. I would think he could at least tell someone new where to start their learning, how to optimize code, what technologies he thinks works best and what areas they should do research for the future how to figure it out for themselves how to get a high traffic automated web system up for their online ideas. Example

1.) Asp.net - Learn these functions of it, don't use this part of it and optimize your code by doing this (Study this other technology that I am thinking about changing it to for this reason possibly? Future plans?)Tips on automation and optimization. Should we think about C# or other future Microsoft technologies?
2.) Web Servers - Recommended Hardware, Windows 2003 64bit, this hardware, this setup and configuration and use it this way
3.) SQL 2005 (Or whatever) - Recommended Hardware, Windows 2003 64bit, What version, how setup, tips on utilization, use this way for reads and use this way for writes, how to optimize for high traffic, Etc.
4.) Bandwidth and other network optimizations?
5.) Good resources you used online to help you figure stuff out??
6.) Any other helpful tips on AI and other stuff you use?

You are rich now! This is one way you can share your wealth in knowledge without having to spend a cent. Just give us enough to figure out the detailed creation ourselves?

Or create a framework we can buy or something. Just help a brotha out!

Bless us oh Marcus our Friend! :-)

I think the simplicity desgin of Plentyoffish, gives users a feeling of trust, and belonging. After all, most of us (singles) are living in a messy room. When it comes to websites deals with personal matters and life, I much prefer the low key, causal design of Plentyoffish, just a sense of belonging.
http://touravus.ru

I think Mr. Frind will not share or discuss his secret..The reason it's business.