Reducing Your Website's Bandwidth Usage

March 5, 2007

Over the last three years, this site has become far more popular than I ever could have imagined. Not that I'm complaining, mind you. Finding an audience and opening a dialog with that audience is the whole point of writing a blog in the first place.

But on the internet, popularity is a tax. Specifically, a bandwidth tax. When Why Can't Programmers.. Program? went viral last week, outgoing bandwidth usage spiked to nearly 9 gigabytes in a single day:

codinghorror bandwidth usage, 2/24/2007 - 3/4/2007

That was enough to completely saturate two T1 lines-- nearly 300 KB/sec-- for most of the day. And that includes the time we disabled access to the site entirely in order to keep it from taking out the whole network.* After that, it was clear that something had to be done. What can we do to reduce a website's bandwidth usage?

1. Switch to an external image provider.

Unless your website is an all-text affair, images will always consume the lion's share of your outgoing bandwidth. Even on this site, which is extremely minimalistic, the size of the images dwarf the size of the text. Consider my last blog post, which is fairly typical:

Size of post text~4,900 bytes
Size of post image~46,300 bytes
Size of site images~4,600 bytes

The text only makes up about ten percent of the content for that post. To make a dent in our bandwidth problem, we must deal with the other ninety percent of the content-- the images-- first.

Ideally, we shouldn't have to serve up any images at all: we can outsource the hosting of our images to an external website. There are a number of free or nearly-free image sharing sites on the net which make this a viable strategy:

  • Imageshack
    ImageShack offers free, unlimited storage, but has a 100 MB per hour bandwidth limit for each image. This sounds like a lot, but do the math: that's 1.66 MB per minute, or about 28 KB per second. And the larger your image is, the faster you'll burn through that meager allotment. But it's incredibly easy to use-- you don't even have to sign up-- and according to their common questions page, anything goes as long as it's not illegal.

  • Flickr
    Flickr offers a free basic account with limited upload bandwidth and limited storage. Download bandwidth is unlimited. Upgrading to a paid Pro account for $25/year removes all upload and storage restrictions. However, Flickr's terms of use warn that "professional or corporate uses of Flickr are prohibited", and all external images require a link back to Flickr.

  • Photobucket
    Photobucket's free account has a storage limit and a download bandwidth limit of 10 GB per month (that works out to a little over 14 MB per hour). Upgrading to a paid Pro account for $25/year removes the bandwidth limit. I couldn't find any relevant restrictions in their terms of service.

  • Amazon S3
    Amazon's S3 service allows you to direct-link files at a cost of 15 cents per GB of storage, and 20 cents per GB transfer. It's unlikely that would add up to more than the ~ $2 / month that seems to be the going rate for the other unlimited bandwidth plans. It has worked well for at least one other site.

I like ImageShack a lot, but it's unsuitable for any kind of load, due to the hard-coded bandwidth limit. Photobucket offers the most favorable terms, but Flickr has a better, more mature toolset. Unfortunately, I didn't notice the terms of use restrictions at Flickr until I had already purchased a Pro account from them. So we'll see how it goes. Update: it looks like Amazon S3 may be the best long-term choice, as many (if not all) of these photo sharing services are blocked in corporate firewalls.

Even though this ends up costing me $25/year, it's still an incredible bargain. I am offloading 90% of my site's bandwidth usage to an external host for a measly 2 dollars a month.

And as a nice ancillary benefit, I no longer need to block image bandwidth theft with URL rewriting. Images are free and open to everyone, whether it's abuse or not. This makes life much easier for legitimate users who want to view my content in the reader of their choice.

Also, don't forget that favicon.ico is an image, too. It's retrieved more and more often by today's readers and browsers. Make favicon.ico as small as possible, because it can have a surprisingly large impact on your bandwidth.

2. Turn on HTTP compression.

Now that we've dealt with the image content, we can think about ways to save space on the remaining content-- the text. This one's a no-brainer. Enable HTTP compression on your webserver for roughly two-thirds reduction in text bandwidth. Let's use my last post as an example again:

Post size63,826 bytes
Post size with compression21,746 bytes

We get a 66% reduction in file size for every bit of text served up on our web site-- including all the JavaScript, HTML, and CSS-- by simply flipping a switch on our web server. The benefits of HTTP compression are so obvious it hurts. It's reasonably straightforward to set up in IIS 6.0 , and it's extremely easy to set up in Apache.

Never serve content that isn't HTTP compressed. It's as close as you'll ever get to free bandwidth in this world. If you aren't sure that HTTP compression is enabled on your website, use this handy web-based HTTP compression tester, and be sure.

3. Outsource Your RSS feeds.

Many web sites offer RSS feeds of updated content that users can subscribe to (or "syndicate") in RSS readers. Instead of visiting a website every day to see what has changed, RSS readers automatically pull down the latest RSS feed at regular intervals. Users are free to read your articles at their convenience, even offline. Sounds great, right?

It is great. Until your ealize just how much bandwidth all that RSS feed polling is consuming. It's staggering. Scott Hanselman told me that half his bandwidth was going to RSS feeds. And Rick Klau noted that 60% of his page views were RSS feed retrievals. The entire RSS ecosystem depends on properly coded RSS readers; a single badly-coded reader could pummel your feed, pulling uncompressed copies of your RSS feed down hourly-- even when it hasn't changed since the last retrieval. Now try to imagine thousands of poorly-coded RSS readers, all over the world. That's pretty much where we are today.

Serving up endless streams of RSS feeds is something I'd just as soon outsource. That's where FeedBurner comes in. Although I'll gladly outsource image hosting for the various images I use to complement my writing, I've been hesitant to hand control for something as critical as my RSS feed to a completely external service. I emailed Scott Hanselman, who switched his site over to FeedBurner a while ago, to solicit his thoughts. He was gracious enough to call me on the phone and address my concerns, even walking me through FeedBurner using his login.

I've switched my feed over to FeedBurner as of 3pm today. The switch should be transparent to any readers, since I used some mod_rewriteISAPIRewrite rules to do a seamless, automatic permanent redirect from the old feed URL to the new feed URL:

# do not redirect feedburner, but redirect everyone else
RewriteCond User-Agent: (?!FeedBurner).*
RewriteRule .*index.xml$|.*index.rdf$|.*atom.xml$
  http://feeds.feedburner.com/codinghorror/ [I,RP,L]

And the best part is that immediately after I made this change, I noticed a huge drop in per-second and per-minute bandwidth on the server. I suppose that's not too surprising if you consider that the feedburner stats page for this feed are currently showing about one RSS feed hit per second. But even compressed, that's still about 31 KB of RSS feed per second that my server no longer has to deal with.

It's a substantial savings, and FeedBurner brings lots of other abilities to the table beyond mere bandwidth savings.

4. Optimize the size of your JavaScript and CSS

The only thing left for us to do now is reduce the size of our text content, with a special emphasis on the elements that are common to every page on our website. CSS and JavaScript resources are a good place to start, but the same techniques can apply to your HTML as well.

There's a handy online CSS compressor which offers three levels of CSS compression. I used it on the main CSS file for this page, with the following results:

original CSS size2,299 bytes
after removing whitespace1,758 bytes
after HTTP compression615 bytes

We can do something similar to the JavaScript with this online JavaScript compressor, based on Douglas Crockford's JSMin. But before I put the JavaScript through the compressor, I went through and refactored it, using shorter variables and eliminating some redundant and obsolete code.

original JS size1232 bytes
after refactoring747 bytes
after removing whitespace558 bytes
after HTTP compression320 bytes

It's possible to use similar whitespace compressors on your HTML, but I don't recommend it. I only saw reductions in size of about 10%, which wasn't worth the hit to readability.

Realistically, whitespace and linefeed removal is doing work that the compression would be doing for us. We're just adding a dab of human-assisted efficiency:

RawCompressed
Unoptimized CSS2,299 bytes671 bytes
Optimized CSS1,758 bytes615 bytes

It's only about a 10 percent savings once you factor in HTTP compression. The tradeoff is that CSS or JavaScript lacking whitespace and linefeeds has to be pasted into an editor to be effectively edited. I use Visual Studio 2005, which automatically "rehydrates" the code with proper whitespace and linefeeds when I issue the autoformat command.

Although this is definitely a micro-optimization, I think it's worthwhile since it reduces the payload of every single page on this website. But there's a reason it's the last item on the list, too. We're just cleaning up a few last opportunities to squeeze every last byte over the wire.

After implementing all these changes, I'm very happy with the results. I see a considerable improvement in bandwidth usage, and my page load times have never been snappier. But, these suggestions aren't a panacea. Even the most minimal, hyper-optimized compressed text content can saturate a 300 KB/sec link if the hits per second are coming fast enough. Still, I'm hoping these changes will let my site weather the next Digg storm with a little more dignity than it did the last one-- and avoid taking out the network in the process.

* the ironic thing about this is that the viral post in question was completely HTTP compressed text content anyway. So of all the suggestions above, only the RSS outsourcing would have helped.

Posted by Jeff Atwood
162 Comments

One thing you can do with bad RSS readers, assuming they set correct UserAgent header, is to redirect them to a bandwidth capped server.

I have a server that on IP x serves pages as fast as possible, but on IP y it servers them at speed of 4KB/s, enough to keep clients downloading the material, but slow enough to keep them hogging all the bandwidth.

Actually I have a set of IPs on that server and each IP serves data at different rate, going from full speed to half, quarter, 1/8th and then to that 4KB/s. This way I can prioritize and manage how the hosted sites use up my limited upload capacity.

Raynet on March 7, 2007 11:46 AM

I've had success with S3 -- it's very easy to setup, especially with the S3 firefox plugin (https://addons.mozilla.org/firefox/3247/).

Also, I threw together some quick numbers here in case anyone wants to compute their bandwidth costs: http://tinyurl.com/yr6rl6

One last comment on compression -- apparently this behavior is buggy in some older browsers, so be careful (http://www.thinkvitamin.com/features/webapps/serving-javascript-fast).

Great article.

Kalid on March 7, 2007 12:39 PM

If you only had a nickel for each time someone reads your blog. . . .

John A. Davis on March 8, 2007 12:29 PM

Nice article. I especially liked the ideas around hosting images and impact of that external to your site.

Also worth mentioning is http headers on certain resources on your site to tell the browsers to not hit your static content so hard.

Check my post at

http://jacdev.blogspot.com/2006/12/tips-for-aspnet-website-performance.html

Jac on March 13, 2007 11:50 AM

How can I futher reduce the size of 304 / 404 etc, as 404 response is not compressed by IIS ?

Vinod on July 25, 2007 3:28 AM

If it's that important to you to reduce the size of a 404 page why don't you replace the error page with "404 not found".

GH on July 25, 2007 11:23 AM

Thanks for this, I have been looking at reducing my bandwidth for some time and did not think of some of these ideas. Not serving images AT ALL is a totally new concept to me, brilliant!

Donetsk on September 26, 2007 5:23 AM

Great insight here! I didn't know you could TOTALLY outsource your images. I thought you could only post certain images up at places like Flickr, and have a link from your site to Flickr. Shows you how amateur I actually am! Outsourcing your images completely, fantastic idea! It makes perfect sense! And if I was getting hits like you... there's no doubt I'd be doing that as you are.
Thanks for the excellent idea, will definitely be looking more into it.

linen on October 4, 2007 8:49 AM

Great article! Thanks for the tips. I didn't know about the HTTP compression until now...

Sotek on October 24, 2007 2:49 AM

Hi guys im in need of some help just read the artical and some of the post. id like to implment some changes to my site to save bandwidth. Also tips on how to make my pages load quicker. Ive no idea how to start. My site is getting arround 500 unique hits per day and expect to rise to 1000 at least by this time next month. im roughly using about 5gb bandwidth a day and have 3600Gb a month to play with but has might sites growing with popularity i need these changes.

What is this HTTP compression how do i turn it on and what are the bad points for doing this?

Where is the best place to put my images if not on my hosting server? i want it to be cheap as possible but fast and reliable
Ive also got 2 virgin media accounts which as space 50GB i think. currently i dont use these but are these a good place to store my pictures?

pink angel on October 29, 2007 2:39 AM

why not automatically redirect only everyone came from slashdot/digg/fark/etc...

http://wiki.dennyhalim.com/htaccess

so, that would automatically 'protect' your web from any dugg even before you know it.

dennyhalim on November 22, 2007 1:13 AM

thats nothing I got 30GB in 5 HOURS! My server overloaded by those blasted Asians upload their porn

Kyle Richelhoff on January 9, 2008 3:35 AM

I'd like to plug a CSS minifier I came across recently that gets quite good compression. It seems to take a different approach than just removing whitespace,etc. Anyway, you can read about it

http://www.artofscaling.com/css-minifier/

only downside is that it isn't an online minimizer, need to run a jar

chris on January 22, 2008 3:41 AM

Outsourcing can be a very complex and complicated. Every facet of the exercise must be carefully considered and properly executed. There is very little margin of error, if the full screen for the value.

However, this does not need to trauma, yet another adventure of blind research. The potential benefits are well documented, and the outsourcing strategy is now sufficiently mature to the path to innumerable times zertreten were before.

But how do you ensure that the lessons of others (sometimes the hard Tour), for a good cause? How do you ensure that you do not have to reinvent the wheel again? How, you are exercising in any case most effectively and efficiently as possible?

Michel john on March 24, 2008 6:12 AM

I have a question - Where do All these companies that sell "bandwidth" buy theirs from? If I want to skip the middle man, what do I do?

JustWannaKnow on April 16, 2008 12:04 PM

Also worth mentioning is http headers on certain resources on your site to tell the browsers to not hit your static content so hard

Oyun on April 30, 2008 11:30 AM

Great blog, great post, my blog are dofollow too.

SHOUW on May 16, 2008 1:39 PM

Hey,

Great article, just wondering. Would it be possible to use a sort of PHP script on each page to balance the load of the images, splitting it between different servers?

For example, it could monitor how much bandwidth an image is using and switch to another service before it became to much?

I dunno how ImageShack or other image hosting services would feel about me uploading images multiple times, and linking to them after each other.

Splatzone on July 18, 2008 10:18 AM

why would you pay for an image host when www.sexyimagehost.com is a free anything goes image host

Linda on July 24, 2008 1:25 PM

http://www.sexyimagehost.com

Linda on July 24, 2008 1:25 PM

haha,

you guys are great.
I have a Photobucket Pro account and I use it to store all of my images, in my EXTREMELY image-bandwidth heavy website.

It rocks, c'ya :)

Responsible Blogging at its Finest on August 16, 2008 11:24 AM

a href=http://www.rubberduck.me/rubberduck.me/a gives you enough bandwidth (unlimited) so you do not have to worry about the above :D

Sub on September 6, 2008 8:49 AM

http://www.rubberduck.me gives you enough bandwidth (unlimited) so you do not have to worry about the above :D

Sub on September 6, 2008 8:50 AM


This company stock (ROKE) is set to take off. Worldwide client base in the mobile communications space. See the details at
www.icoft.com
any clarification contact at
icoft123gmail.com

icoft123 Roke on October 30, 2008 5:09 AM


Worldwide client base in the mobile communications space. See the details at
www.icoft.com/roke.html
any clarification at
icoft123@gmail.com

icoft123 Roke on October 30, 2008 5:09 AM

WORLDWIDE client base in the cell phone sector. Tremendous opportunity to get in the stock now. Check out how big the opportunity is at
www.icoft.com/roke.html
any clarification at
icoft123@gmail.com

icoft123 Roke on October 30, 2008 5:11 AM

WORLDWIDE client base in the cell phone sector. Tremendous opportunity to get in the stock now. Check out how big the opportunity is at
www.icoft.com/roke.html
any clarification at
icoft123@gmail.com

icoft123 Roke on October 30, 2008 5:12 AM

I wish I had read this article a month ago before I started switching to flickr and then got responses to my blog from users stating None of your images are showing up anymore.

Ugh.

David S on November 5, 2008 9:57 AM

Dear friends,
The link below is for website that I found very useful. Instead of searching for website now we can get all under one site.
www.1webmall.com
Recommend it to be bookmarked as your favorites or make this is as home page by just clicking on make this is home page tab.

Thanks and Regards
Irfan


Irfan on February 10, 2009 2:27 AM

I could definitely use some less bandwith folks. Miek
http://hubpages.com/hub/Error-Smart-Review-To-Buy-Error-Smart-Software

Mike on April 30, 2009 1:14 PM

i have been having problems with my bandwith for a while. i am so glad i read this. i hope this works for me.

best website directory on May 9, 2009 6:07 AM

Worldwide client base in the mobile communications space. See the details at
www.icoft.com/roke.html

waterless cookware on May 27, 2009 1:21 PM

For example, it could monitor how much bandwidth an image is using and switch to another service before it became to much?

naot sandals on May 27, 2009 1:23 PM

I have a Photobucket Pro account and I use it to store all of my images, in my EXTREMELY image-bandwidth heavy website.

sunbed tanning on May 27, 2009 1:24 PM

am camara from Guinea conakry west africa.I am twenty seven years old and I am 1.68 metre, 62 kilo. I am a man very handsome and honest.I am looking for friendship with the women and plus high.Why because I am bachelor but dreaming of a good,kind and loving and wife ,children as well and just happiness If you happen to know my email address sidikicamara33@yahoo.ca

okey oyna on May 31, 2009 3:18 AM

I just saw the YouTube video about web 3.0 and I have had this idea about our future economy. It’s a bit in tune with the Zeitgeist film. See, if all of our knowledge was shared, we couldn’t make money of it. I’m a musician and I kinda have the feeling that in the future people will pay for live performances, because we have computer software that is very close to sounding like a musical instrument, with a modelled human touch. So, if a person that has never played the guitar, can click a button and sound like Jimi Hendrix, than what is the coolness about all this digital music really? Infact, I think that the more advanced this gets, the less interesting it becomes. We are still in awe over a youngster that can make the violin cry and we will definately be in even more awe when a person can actually do with their fingers that what we can do in an iPhone application.

So I say. Give it all out for free, it will give more value to the ones that work for it. In the meantime I will continue selling my music to try and make a living :)

okey oyna on May 31, 2009 3:20 AM

I suppose this does not matter in some situations, and especially if the in/out stats are gathered from the router, but...

mario oyunları on May 31, 2009 4:06 AM

Good article... In fact, information about RSS feed was eye opener.

To reduce image sizes one can consider smush.it from Yahoo. [ http://developer.yahoo.com/yslow/smushit/ ]

Mahin Gupta on June 7, 2009 12:04 PM

Free image hosting companies wont help i mean who know how much they will be around? Also if you are using jquery use google api :)

okey on June 11, 2009 10:11 AM

was a very nice article thank you I'd put my archive

Okey Oyna on June 13, 2009 9:44 AM

God bless you!I really agree with your opinions.

venus on June 15, 2009 7:49 AM

thanks a lot.

Okey on June 16, 2009 9:47 AM

Very good, congratulations article

Okey Oyna on June 24, 2009 8:20 AM

even after 2 years this post is still very helpful =D

vegas on June 28, 2009 7:15 AM

thank you very very much...

okey on July 2, 2009 5:26 AM

thanks

ceysu on July 10, 2009 2:34 AM

you guys are great.
I have a Photobucket Pro account and I use it to store all of my images, in my EXTREMELY image-bandwidth heavy website.

It rocks, c'ya :)

hikaye on July 16, 2009 4:17 AM

This blog is very useful for website owners those who are uploading more number of files into their server, they can able to reduce their monthly bandwidth.

Nice and very interesting topic.

Good Luck!!

Prabhu Eswaran on July 17, 2009 9:06 AM

This website is very useful for website owners those who are uploading more number of files into their server.

Very nice and interesting topic for all the webmaster.

Thanks. Good Luck.

Prabhu Eswaran on July 17, 2009 9:09 AM

Thank you to fot make such a nice opportuny for us. Thanx a lot

yazgülü on July 18, 2009 6:07 AM

Thank you very beautiful setting and function

travesti on July 30, 2009 2:34 AM

As a precursor to outsourcing your image serving, I would have thought that you could do much to reduce the size of your image files. That 46,300 Bytes for the image seemed a lot to me. Clearly, outsourcing your images altogether is a total fix, but one day those nasty T&C may come back to bite you. Plus, smaller images will be served faster wherever they are retrieved from.

I would strongly recommend Macromedia Fireworks (now owned by Adobe, I think) and Adobe ImageReady to allow you to see the same image with different image compression levels side by side. Mind you, from past posts, I realise that you know all about image compression...

Presumably, you now have an extra step in your blogging process in terms of uploading images to your image server. I'd be interested how this effects your blogging..

medyum on August 1, 2009 1:44 PM

This is a great article. Something which would be useful when trying to explain to my clients why their gallery is costing them a fortune. Optimise your images before you upload them! Nobody wants to see a 3000px x 3000px image. Brilliant, Thanks for the info!

website design stoke on August 7, 2009 2:13 AM

çok güzel bir yazı olmuş böyle yazıları devam ettirmenizi dilerim.ayrıca yalancısın sen dizisinin fan sites açıldı buyrun girin yalancısın sen dizisi ile ilgili tüm bilgiler bu sitede.ayrıca oyunla ilgili sitemede gelebilirsiniz.oyun haberleri

serhat on August 17, 2009 9:02 AM

thanks a lot

gögüs estetigi on August 19, 2009 4:49 AM

Well said!

Jon Galloway on February 6, 2010 10:03 PM

Flickr should work for you... the price of putting a few bits of anchor taggage around the image (to comply with their TOS) is minor compared to offloading the image bandwidth.

Aaron B. Hockley on February 6, 2010 10:03 PM

For photos I really like photobucket and imageshack. But I think a good lesson is never blog on a quasi program due to peoples desire to see if they can do it.

Brandon on February 6, 2010 10:03 PM

Here's some better advice: if you're concerned about 9gb of traffic, you should probably switch web hosts. Bandwidth and storage prices drop every year--if your plan isn't automatically upgraded or priced down, that's a strong sign you're with the wrong people. My host has a limit of 70GB per day (2,100 GB/month) for less than $7/month and continuously gets better as time goes on. Features Ruby on Rails, PHP/MySQL, etc. I would strongly recommend:

http://www.servage.net/?coupon=bonusbytes

aaron on February 6, 2010 10:03 PM

This is pretty useful tutorial. Read few extra tips about How to Test Your Website Completely and Validate HTML,CSS , RSS, ATOM,Mobile compatibility
http://www.globinch.com/2010/05/01/how-to-test-your-website-completely-validate-htmlcss-rss-atommobile-compatibility/

Globinch.blogspot.com on May 1, 2010 8:07 PM

Has anyone have a dedicated server with www.gsid.net ? I am pretty sure that their bandwidth billing is not working properly, because according to my cpanel stats i used over 8TB of bandwidth, and they only billed me for 500MB.. I really hope it doesn't catch up to me.

Kelso Kennedy on July 27, 2010 2:55 PM

This, site is really a great help since I'm always posting many articles a day. I found this website quite different and unique. To check more about it just click here.

Edenhost12 on March 15, 2011 12:37 AM

«Back

The comments to this entry are closed.