October 23, 2007
My employer, Vertigo Software, graciously hosted this blog for the last year. But as blog traffic has grown, it has put a noticeable and increasing strain on our bandwidth. Even on an average day, blog traffic consumes a solid 30 percent of our internet connection-- and much more if something happens to be popular. And that's after factoring in all the bandwidth-reducing tricks I could think of.
While I greatly appreciate my employer's generosity, I don't like causing all my coworkers' internet connections to slow to a crawl. So when my friend and co-author Phil Haack mentioned that we could share a dedicated server through a contact of his, I jumped at the chance.
I'm a big believer in virtualization, so I wanted a beefy physical server that could handle running at least four virtual servers. And I wanted it to run a 64-bit host operating system, as 64-bit offers huge performance benefits for servers. Nobody in their right mind should build up a 32-bit server today.
The contact he was referring to works at CrystalTech. And boy, did CrystalTech ever hook us up:
- Windows Server 2003 R2 x64
- Quad-core Xeon X3210 @ 2.13 Ghz
- 4 GB RAM
- 300 GB RAID-5 array
Not too shabby. It is, of course, an obscene amount of power for our relatively modest needs. Have I mentioned how much I like my new friends at CrystalTech? Or what great deals they have on hosting?
But in all seriousness, it's effectively a new sponsor for this blog, so welcome aboard.
I was already hosting this server as a VM, so here's what I did to switch over to completely new hardware:
- shut down my VM
- compacted and compressed it
- transferred it to the new server
- booted it up again
All I had to do was change the IP address in the VM and I was up and running as if nothing had changed. That's the easiest server migration I've ever experienced, all thanks to virtualization.
Phil and I are both Windows ecosystem developers, so we went with what we knew. But virtualization provides total flexibility. I could spin up a new Linux server at a moment's notice if I decided to switch this blog over to the LAMP stack. Or I could play with the latest release candidate of Windows Server 2008. And they can all run in parallel, assuming we have enough memory. That's what I love most about virtualization-- the freedom.
Although Phil and I share admin access to the host machine, we have our own private playgrounds in our virtual servers. We're completely isolated from each other's peculiarities and weirdnesses: nothing we do (well, almost nothing) can affect the other person's virtual machine. Reboot? No problem. Install some stupid software I can't stand? Go for it. Format the drive and start over? Don't care. It's your machine. Do whatever.
The only downside to virtual machine server hosting is that it can be difficult to share IPs between virtual machines. CrystalTech has provided us with a block of 6 public IP addresses, so fortunately we don't have to worry about this. One IP is occupied by the host, but that still leaves five IPs for virtual machines of our creation. That's plenty.
But let's say we only had two public IP addresses-- or we wanted to run lots and lots of virtual machines with a small pool of public IP addresses. What then? How could codinghorror.com and haacked.com share the same IP address (and port 80), when they're on two different virtual machines? They clearly can't occupy the same IP.
On a single physical server, the answer is easy-- virtual hosting, or host header routing. But that requires our websites to live side by side on the same server. Phil and I don't share our wives, so why would we share a server? No offense intended to either of our wives-- or our respective servers-- but sharing is an unacceptable solution. I like you, Phil... but not that much.
If you want two different machines (physical or virtual) to share an IP, it takes some clever trickery. In the Windows ecosystem, that clever trickery often comes in the form of Microsoft's ISA Server. (I'm not sure what the open source equivalent is, but I'm confident it's out there.)
ISA Server acts as our public interface to the world, talking through a public IP address. All DNS entries, and thus HTTP traffic, would be directed to that single public IP address. As our gatekeeper, ISA Server is in a unique position to do lots of cool stuff for us, like firewalling, caching, and so on. But we only care about one particular feature right now: the ability to share an IP address between multiple machines. This is known as a "web rule" in ISA parlance. With appropriate web rules in effect for both of our sites, ISA Server will shuttle the HTTP requests back and forth to the correct private IP addresses based on the host headers. It basically extends the host header routing concepts we saw in Apache and IIS outside the confines of a particular machine.
ISA Server 10.0.0.1:80
That's one way you can host fifty websites, all running on fifty different machines, with a single public IP address. It's a very clever trick indeed. Unfortunately, ISA Server isn't the simplest of products to configure and administer. I'm glad we have enough public IPs that we don't have to worry about sharing them between multiple machines. But it's definitely something you should be aware of, as virtual servers become increasingly commonplace.. and the pool of available IP addresses continues to dwindle.
Posted by Jeff Atwood
You might want to consider making an entry on just what your statistics look like. Or even putting up a stat/graph page. I did it with my website:
It sounds like a beefy box, but I'm wondering just how much performance is lost with virtualization? Of course that only matters if your pumping out some serious bits per second.
Ladar, all my stats are public and linked from the front page.
We use apache reverse proxying at our organisation (we're a FreeBSD house). I guess this is similar to ISA Server. This allows us to move all of the SSL processing to a one place (+backup) meaning that the application servers (custom apache builds) don't need to worry about it.
Another advantage is that all SSL certificates are held on the proxy machines, reducing the number of people with access to them.
As far as I know though, each SSL site needs it's own IP.
Hey TristanK - I agree with your assertion that wildcard certs can help somewhat with this issue, BUT I wanted to point out that due to differences in interpretation of the spec, that generally wildcard certs only work on "three-level" domains e.g. whatever.foo.bar. Four-level ones like you give in your example will cause certain very-common browsers to give warning messages and choke.
Musing, it might be one cert per socket rather than per IP. Unfortunately, that's a next-to-meaningless distinction because everyone's interested in the one port 443 (i.e. nobody wants *their* website to be the one you have to type a port number for (eg, https://example.com:8443) and it might need reconfiguration at the client-side firewall/proxy too to allow SSL tunneling on a nonstandard port...) and there's only one tcp/443 per IP.
So, let's assume one IP per certificate is a requirement based on ISA and Windows' current implementation. The problem then becomes that all those IPs need to be externally visible at some point (assuming that we're not dealing with SSL Host Headers).
If you're running, say, something.example.com, somethingelse.example.com, somethingagain.example.com and so on, these can potentially share a wildcard certificate and use only a single server IP for all those sites.
If your namespace is largely-hierarchial-but-also-quite-flat :) you may be able to significantly reduce the number of ips and certs required (think site183729.hosting.example.com - all those sitennnns could share one cert...), and it reduces both the number of IPs and certs to manage. It can be done today, so it's probably worth investigating.
See also the ISA blog here:
Bottom line: In the future, hopefully an SSL equivalent to Host Headers is implemented. Right now, not by us, afaik.
ISA will terminate SSL connections itself (if you let it, which we'd usually suggest), inspect the traffic, then forward the request to the published server internally. So internally, you can use CrazyPKI (if that's a real product name, I apologize to the vendor, but *what were you thinking!?* (and I claim first-use rights if not)) or self-signed certs, or no certs, and private IP addresses, so you're golden there. The "cost" of SSL is in terms of public IPs. Internally, you can use different certs per published site or not (or no certs at all and everything on the one IP using host headers), without preventing you from using wildcard certs externally.
It's really a question of where you want to manage your cert complexity - you can do it "out front" at the ISA Server/SSL Termination tier and organize sites however you want internally, or you can straight-through the SSL and manage the complexity at the web server.
Sorry for the rather long-winded "it's up to you"...!
What virtualization software are you using?
First, your post was serendipitous. I just signed up a client with Crystal 5 minutes ago. You've never led me wrong before...
Second, I'm curious as to the Internet connection options the PWillis mentions. I work for a bank (unrelated to the site I mentioned above) and we HAVE to use a T-1 to connect us (in Kansas City) directly to our main office in Cheyenne, WY. We actually have 5 of them, at a cost of around $1200 per month each.
The home office is paying somewhere around $1000 per month EACH for 2 T-1s that connect them to the internet.
So, even if I'm here @ 3 AM on Sunday morning, my best access will only ever be 3 Mbit/Second (I think I got that right).
Are there other options for internet access? I'm a lowly programmer, but even I have a sneaking suspicion that the guys out in Cheyenne are all old, and don't want to learn anything new, so they go with what they already know...ie, T-1s that were all the rage 10-15 years ago.
Anyone want to point me to a newsgroup where I could ask some questions and get some data to kick the guys in Cheyenne in the ass so they would get us some faster iNet access?
From my house, through Time Warner, I get 500k per second which seems literally like 10x faster than it is here at work. And that's for $50 per month!
Jeff - thanks of the info on how you got them to give you 64-bit. I've been a CT customer for years so I know they have good support. I'm pretty sure you'll be happy with them for a long time. The biggest mistake I ever made was switching one of my servers from CT to Godaddy dedicated hosting to save $80/month. Godaddy's dedicated support is abysmal.
Oh yeah? I love AQuest Hosting.
I use Nginx too (mainly for the small memory footprint) with spawn_fcgi to serve dynamic pages. I wish Nginx would support .htaccess as moving stuff around is a nightmare.
What kind of wonky 20th century Internet connection is Vertigo running? They should be able to host this blog without impact.
Let's visit some numbers:
Your homepage currently weighs 350KB.
Let's assume that tomorrow you have 60,000 page views, and that _all_ of them are for the homepage and the homepage only.
Let's also assume that all of your readers are in one time zone, so your hits are distributed across 8 hours not 24.
That means you would serve 164063 Mb (Megabits) in 8 hours, or 5.7 Mb per second.
Keep in mind that this is an absolute upper bound -- on your highest traffic day, everyone views the heaviest page and nobody looks at another page and they all do it in a compressed 8-hour day. Real-life numbers would be a fraction of this.
Bandwidth is cheap. You can get 15Mbps at home from FiOS. An office can get 100Mbps from a local fiber provider for $1000/month. So the worst impact your blog could possibly have is consuming 5% of the company's bandwidth on the nightmare day outlined above.
... unless Vertigo is partying like it's 1999 and using (gasp!) a T1 or something. So I ask you: why was this a pain point at all?
I've seen this elsewhere with a hosting company but I'm not sure how secure it can be.
If a customer wanted to host an SSL site and didn't have certificate or their own public IP associated with their account the could use the hosting company's certificate on a subdomain of the company and the traffic would be proxyed to their corresponding website.
Example: If your website would be www.mywebsite.com you could redirect your customers from https://www.mywebsite.com to https://mywebsitecom.hostingcompany.com which uses an ssl certificate from the hosting company and traffic from this site would be proxyed to your http://www.mywebsite.com
Man, Squid would have a ball serving up something as static as this blog.
You do realize that on the average weekday This site has 40,000 page views? That's frighteningly close to your absolute upper bound already. And not that I know much about business connections, or even the size of a company like Vertigo, but I'd imagine that 5.7mb a second is a good chunk of bandwidth. Even if it's 10% of the capacity of the connection, It's not going to all come at once. There will be times when this site takes 1% of the link, and others when it takes 80%, and if the combined other usage is 30% then that's a problem.
CrystalTech is rolling out the 64-bit machines sometime early/mid November. If you shoot the CT sales dept. an email they'll give out some details.
Interesting read, and I like seeing CrystalTech getting some props. I've been using them to host client websites for years now and CrystalTech has only got better and cheaper over time.
Man, no wonder you need that much hardware, if it's running Windows Server natively and several more copies of it in virtual machines. I used a Pentium 200 with 64MB of RAM as both server and firewall at school; with a full LAMP stack, MySQL, Samba, an IRC client, and other bits running, it still had no problem transferring a few megabytes/sec over the school network. I put Tomcat on there as well for one class, and that was mildly laggy, but it didn't affect anything else.
Cool - I've worked with CrystalTech quite a bit in the past. One of my clients used their dedicated servers for their 20+ customers. In my experience, their support was leaps and bounds above most web hosting/co-lo companies out there, and it was really nice to actually be able to talk to a human being when I needed to.
Thanks Fred! My understanding was that it works with an arbitrary number of levels *but* that the parent level must be the wildcarded one (at least in the Microsoft world, IIRC) - so for sitennnn.hosting.example.com, you'd need a wildcard cert for *.hosting.example.com, *.example.com wouldn't work.
(Usual caveats about often being wrong and not knowing everything apply) :)
They support PHP and ASP on the same server. Interesting...
glad I could help you guys in getting that server. But I am sad to report that I no longer work from CT. I moved on shortly before you guys got the server.
I took a look at their prices and they seem a bit high. I've been using Rackshack, which changed to Ev1Servers which was bought by The Planet (as you can see I've been using them for years) and the prices are more competitive for dedicated hosting.
I'm not sure how involved in the server maintenance these guys are though, I like having full control and maintaining it on my own schedule so The Planet is perfect for my needs.
I've been using CrystalTech for years now to host my site, and loving it. Just doing shared hosting at this point as the dedicated is a little expensive for me, but I've had great experiences. Good choice to go with them.
Sadly, there are a couple of reasons to be stuck with 32-bit servers. Some apps (Exchange 2003) still require it, 64-bit terminal servers can be really odd, and since +80% of clients are 32 bit, you get all sorts of pain with printer drivers if you're hosting a print server.
However, for a website server, 64 bit is the best way to go.
How did you get CrystalTech to give you a 64-bit machine? If I click over to their dedicated hosting services, all the machines are 32-bit with 2 gigs of ram (max). What did you do to get the big boy?
According to this five-year-old press release, Cogent has metro fiber in Kansas City:
I would start there. The *only* reason to use a T1 today is for the 24 analog voice ports. If you're doing strictly data, then a T1 makes about as much sense as using a fax machine to send messages to your co-workers.
If you were facing the multiple machines on one IP issue in the Linux ecosystem, the solution would be the Linux Virtual Server (http://www.redhat.com/docs/manuals/haserver/RHHAS-1.0-Manual/ch-lvs.html)
It essentially does the same thing, redirecting requests made to a given IP to various servers based on whatever rules are set up. Like ISA, this can give you a simple cluster or a single IP on which multiple services are hosted on multiple machines.
Of course, there are also solutions using things like BigIP, which is a boon when you are doing load-balancing and need sticky connections or other such features.
There is a lot of interesting stuff happening in the virtualization world with respect to hosting. More tier one providers are moving to a power consumption model because the scarce resource is no longer space. Multi-core processors are allowing us to do more in less space which in part drives this change. I just started some research into it as a way to spin up additional web capacity on short notice without long term hardware commitments. There are some interesting things happening in particular to promote grid-like hosting using VMs. I wish more providers would move to this model. I want control, but not necessarily dedicated hardware. In the mean time I watch Terremark, Flexiscale, and others hoping that someone goes the route of MediaTemple in pushing prices, but gives me full VM access.
If I click over to their dedicated hosting services, all the machines are 32-bit with 2 gigs of ram (max). What did you do to get the big boy?
We had to push for 64-bit. It's part of a new offering they're rolling out-- you may have to ask for it via email until they get it formally up on the website.
I also had my first support incident a few days ago. I emailed support at around 11 am, I had a response within 5 minutes, and within 45 minutes, the problem was fixed and a confirmation email sent. So I can vouch for the good CT support so far.
Bandwidth is cheap. You can get 15Mbps at home from FiOS. An office can get 100Mbps from a local fiber provider for $1000/month.
We have two T1s, and that is the deeper issue. I agree. Based on earlier feedback on bandwidth/pricing, I've urged the powers that be to look at alternatives. We have a quote coming in tomorrow.
Jeff - what virtualization sotware are you using for your site? VMware server, vmware workstation, MS Virtual Server etc. plus the version would be great to know.
We have two T1s, and that is the deeper issue. I agree.
!! What a curiosity you have there.
Hey Now Jeff,
Sound like good news I sure learned some virtual server info.
Coding Horror Fan,
The real performance hit this blog placed on Vertigo Software is not important when compared to the perceived hit. When ever the network is slow "its Jeff's fault" It is kinda like church and State- work life and personal life - keep them separate and everybody is happier.
We are using the 64-bit flavor of Virtual Server 2005 R2 SP1. Quite a mouthful.. it's Microsoft's free server-oriented virtualization environment, fronted by a web UI.
The only thing I don't like about it is the fact that guest VMs will "only" see 1 CPU, although the server will obviously use all 4 CPUs to share the work. Also of note is that the guest operating systems have to be 32-bit, which is no big deal-- as long as the host is full-bore x64, I'm cool with that. Other than those two caveats, Virtual Server 2005 has worked great for me over the last year.
whenever the network is slow "its Jeff's fault"
Exactly, and who likes apologizing for their own inherent awesomeness? It just doesn't work.
"I'm sorry I'm so awesome."
You made a great choice with Crystal Tech - I know they have great representation with the AZ .Net groups; I have yet to hear a single complaint about them.
Ok Jeff...I need your help with this one. I host 400+ unique sites across a virtual cluster. Each site *requires* an individual mapped IP address for SSL cert purposes (obviously removing the option of host headers). This IP requirement is problematic for a number of reasons not excluding load balancing and IIS configuration management. Off loading the SSL processing to the load balancers is not an option at this point. Can ISA Server help me here? Any other ideas to help reduce the number of IP addresses needed?
Woah, here's a VM that allows me to load "Live CDs" (e.g. Knoppix) into a VM instead of bothering with burning and rebooting. Sweet!
I hadn't tried VMWare since I was running Windows 98, I'm really impressed with the variety of free VMs available to run with VMPlayer. Thanks Jeff!
PWills -- thanks so much for the pointers. I'm going to try to find a newsgroup somewhere and hopefully find someone who can help me build a case to present to the company for reducing our costs and increasing our pipe to the internet all at the same time. Thanks to you I have some avenues of investigation.
Again, thank you very much.
I wish you the best of luck with CT. We used them for several years for our dedicated servers. After nothing but continual problems, server crashes, network outages, power problems, etc, we switched most of our servers to a different provider about 10 months ago and are so happy to have our servers out of CT's data centers. Not a single data center related issue since the move. At CT, it was almost a weekly occurrence. We still have a few servers at CT and not surprisingly, those are the only servers that we have issues with. YMMV.
To their credit, they do have good support and their Control Center is very handy. And for small websites, shared hosting, etc they are probably pretty decent. But for dedicated servers...not so much.
Sorry to hear about your issues with CT. I am the president of CT and have never had a customer complain of issues you stated. Can you send me your contact information or call into the CT number? If there are issues like you stated occuring, I would like to get details and find out what we can do to solve.
@Auras - you could try to redirect from https://example.com to https://example.hosting.com , but as a key part of the SSL handshake is the client verifying that the certificate matches what the user typed into the browser, you'd either:
- get a nasty "zomg the website cert name doesn't match" big red error urging you to go outside or read a book instead of using a computer, or
- you'd need *two* public IP addresses (one for example.com, one for example.hosting.com), either of which could be wildcarded, but it's still a minimum of two IPs and two certs.
You could easily (and without penalty) bounce from http://example.com to https://hosting.example.com (note the first was non-SSL); because a bunch of websites don't really need server authentication or security for most of their operation, this might actually be acceptable (what do most corporate sites use SSL for? Could it be outsourced or handled externally (for eg, PayPal, Windows Live ID, etc?))
You'll get better virtual server performance on a RAID 0, RAID 1, or RAID 1+0 array than you will with RAID 5.
This has been very interesting reading as I just recently started to play around with virtualization (and I love it). My understanding is that the ISA Server only handles http requests, what if I wanted to access all ports in my virtual machines through one IP? Is it at all possible (without Virtuozzo and the likes)?
Any comment or pointing me in the right direction welcome.
My 2 cents...
Solaris 10 offers virtualization with very little overhead. Vmware consumes much CPU and memory.. but Solaris 10 uses a new technology, called Containers (aka Zones), that shares the same kernel and you can run 100's of virtual servers on the same machine with little impact. There are relatively few hosting providers for SUN Solaris, one that I know is good and relatively inexpensive is Hosted DB.. hosteddb.com. It is worth it to check out using SUN Solaris instead of Windows with Vmware for running virtual servers.
Cool post, I have a setup like this too, runs my Windows servers and some Trac/svn JumpBoxes.
I use 64bit Gentoo as the host, it’s a headless minimal install with VMWare server Squid in reverse to fix the IP sharing problems.
All free, works a treat, and it’s fast too.
I just Joined to this this forum and I enjoy all your great posts.
I know nothing about your possibilities, but I think you all are touched by the economical crisis.
It is a great thing if you can save on your dental treatment in this situation.
I get a good tip from my friend, and I want to share this info with you.
Chek this url if you have dental issues:
Cosmetic Dentistry Hungary
I would like to know your opinion!
I'm using nginx as my reverse proxy these days. It's a very small, incredibly high performance open source web server from Russia. The thing literally screams - I have it proxying through to a couple of other web servers (Apache + mod_python, just because that's the easiest way to serve Python sites without worrying about keeping FastCGI procceses alive). It consistently uses just a few MB of RAM whether proxying or serving static files directly.
Congrats on a successful move.
You guys might want to upgrade to Windows Server 2008 with Windows Server Virtualization when it's available (WSV is supposed to be available 180 days after WS2008 ships, the CTP is available now: http://blogs.msdn.com/virtual_pc_guy/archive/2007/09/24/windows-server-virtualization-community-technology-preview-now-available.aspx ). Windows Server Virtualization puts the parent operating system in a hypervisor mode which makes more efficient use of hardware and allows for better use of the physical hardware by VM's. I'm not sure, but I'd bet it allows you to use 64bit VM's, too.
This seems like a good tradeoff if you really want to partition things so you each get your own server. You pay a performance penalty (compensated for by smokin hardware), and you have to keep 3 operating systems patched and maintained, but you don't have to worry about Phil screwing up your awesome Perl scripts with his yucky .NET code (sorry, that just slipped out).
Did anybody notice the 'great deals' link is broken?
Also I think iptables is an open-source
equivalent contender to ISA
I had one hell of a time trying to figure several virtual machines on my Core i7 950 rig. I used vmware but it took me days and days to get it working properly. The happy side? I have 4 virtual machines working right now, each with 1gb of ram memory allocated. Real nice. No pain no gain.
George of http://webhostingforacent.com/?v=g