Coding Horror: HTTP Compression and IIS 6.0

August 22, 2004

HTTP Compression and IIS 6.0

HTTP compression is the ultimate no-brainer. The network is really slow, and CPU time is effectively free and geting faster and, uh, "free-er" every day. Compression typically reduces plaintext size by 75 percent: that quadruples your throughput! Every website should be serving up HTTP compressed pages to clients that can accept it. The client indicates ability to accept compressed contents in the request headers:


GET /blog/index.xml HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*
Accept-Language: en-us
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)
Host: www.codinghorror.com
Connection: Keep-Alive

HTTP compression was available in IIS 5.0, but it was also horribly broken. I know that's a link from a vendor selling a competing product, but I can personally vouch for this-- it sucked. Don't enable compression in IIS 5.0. It's not worth the pain it will inevitably cause you. Fortunately, there is an alternative-- the free FlatCompression ISAPI filter. It's not very sophisticated. All outgoing content of the specified mime type(s) is blindly compressed in real time with no caching, so it's ideal for sites with mostly dynamic content. Most importantly: it actually works, unlike the built in IIS 5.0 compression, and it's free open source. If you control an IIS 5.0 server, you should have the FlatCompression ISAPI filter installed.

One of the things I was looking forward to in IIS 6.0 was a HTTP compression layer that actually worked. I thought I had HTTP compression enabled correctly in IIS 6.0 in the Properties, Service tab, but after looking at some sniffer traces.. not quite. I followed a few walkthroughs, such as the excellent Enabling HTTP Compression in IIS 6.0, and I was still getting spotty results. A few observations on my troubleshooting:

Adding the IIS compression filter .dll to the extension manager made absolutely no difference on my server. I'm not sure why people think they need to do that; It has no effect for me, and I tried it both ways a few times.
Despite what the MS documentation says, the metabase filename extension lists are not space delimited! They are cr/lf delimited.
You must restart IIS to get it to reload any changes you've made to the metabase.
There appear to be some non-obvious metabase entries that will prevent compression of script output.
Setting the file extensions to "blank" does not cause IIS to compress all content as specified in the documentation.

Getting a variety of static content extensions to compress was easy, but I had an absolute rip of a time getting dynamic script output to compress. You know, .cgi (perl), .aspx, .asmx, etcetera. I followed every suggestion out there with no joy-- the sniffer kept showing uncompressed dynamic output coming back, but all the static files I tried came back compressed just fine. I'm still not sure which metabase setings I changed to get it to work, so I will post the current working version of the relevant IIS metabase sections in their entirety:

<IIsCompressionScheme	Location ="/LM/W3SVC/Filters/Compression/deflate"
		HcCompressionDll="%windir%\system32\inetsrv\gzip.dll"
		HcCreateFlags="0"
		HcDoDynamicCompression="TRUE"
		HcDoOnDemandCompression="TRUE"
		HcDoStaticCompression="TRUE"
		HcDynamicCompressionLevel="10"
		HcFileExtensions="htm
			html
			xml
			css
			txt
			rdf
			js"
		HcOnDemandCompLevel="10"
		HcPriority="1"
		HcScriptFileExtensions="asp
			cgi
			exe
			dll
			aspx
			asmx"
	>
</IIsCompressionScheme>
<IIsCompressionScheme	Location ="/LM/W3SVC/Filters/Compression/gzip"
		HcCompressionDll="%windir%\system32\inetsrv\gzip.dll"
		HcCreateFlags="1"
		HcDoDynamicCompression="TRUE"
		HcDoOnDemandCompression="TRUE"
		HcDoStaticCompression="TRUE"
		HcDynamicCompressionLevel="10"
		HcFileExtensions="htm
			html
			xml
			css
			txt
			rdf
			js"
		HcOnDemandCompLevel="10"
		HcPriority="1"
		HcScriptFileExtensions="asp
			cgi
			exe
			dll
			aspx
			asmx"
	>
</IIsCompressionScheme>
<IIsCompressionSchemes	Location ="/LM/W3SVC/Filters/Compression/Parameters"
		HcCacheControlHeader="max-age=86400"
		HcCompressionBufferSize="8192"
		HcCompressionDirectory="%windir%\IIS Temporary Compressed Files"
		HcDoDiskSpaceLimiting="FALSE"
		HcDoDynamicCompression="TRUE"
		HcDoOnDemandCompression="TRUE"
		HcDoStaticCompression="TRUE"
		HcExpiresHeader="Wed, 01 Jan 1997 12:00:00 GMT"
		HcFilesDeletedPerDiskFree="256"
		HcIoBufferSize="8192"
		HcMaxDiskSpaceUsage="99614720"
		HcMaxQueueLength="1000"
		HcMinFileSizeForComp="1"
		HcNoCompressionForHttp10="FALSE"
		HcNoCompressionForProxies="FALSE"
		HcNoCompressionForRange="FALSE"
		HcSendCacheHeaders="FALSE"
	>
</IIsCompressionSchemes>

The last things I tried were modifying the HcNoCompression* settings, and turning HcDoStaticCompression on for gzip. It's likely one of those.

I enabled HTTP compression for the .cgi script filetype, which covers the PERL scripts that Movable Type uses, and the interface was just blasting on the screen after I did that. It's truly amazing how much faster pages appear to load, even over a 100baseT local network, with HTTP compression enabled. It is dramatic. I can only imagine how much snappier pages load over a remote network.

Posted by Jeff Atwood View blog reactions

« User Friendly ASP.NET Exception Handling Showstopper! »

Comments

Hi Jeff,

I havent looked into IIS6 ("completely new and much better than IIS5)", so I assumed that MS would have fixed the compression thing. However, this seems not to be the case.

Would it make sense to port FlatCompression to IIS6?

Yours,

Uli M.

Uli on September 20, 2004 03:02 PM

Hey Uli! Great to hear from you again.

We still use your excellent FlatCompression dll on our IIS5 servers at work with great success!

As for IIS6, the compression is fixed, it's just kind of a pain to configure. The documentation is flat-out wrong in a lot of places and misleading in others. But once you get it configured, it does work..

Jeff Atwood on September 20, 2004 07:05 PM

Not sure if you have seen this article:

http://beta.orcsweb.com/articles/iis_compression_6.0.aspx

It covers "those other parameters" which you originally missed.

Lenard on April 20, 2005 01:51 PM

this gives additional data points on actual performance before and after enabling compression. There is some coverage about SSL with compression.

<a href="http://www.intel.com/cd/ids/developer/asmo-na/eng/208335.htm">http://www.intel.com/cd/ids/developer/asmo-na/eng/208335.htm</a>;

john on July 24, 2005 11:26 PM

> this gives additional data points on actual performance before and after enabling compression.

Interesting.. so on relatively old 500mhz and 700mhz processors, IIS compression increases CPU utilization 25-35% for dynamic pages (compressed each time). And only 10% for SSL since the SSL overhead is already so high. Not bad!

Jeff Atwood on July 25, 2005 12:47 AM

Dr. Uli's code is about 3 years out of date. It uses the zlib library, and the source shows quite an old version. I wonder if it's vulnerable to the recently found and fixed zlib buffer overflow vulnerability he predicted in his readme notes?

Andrew on July 25, 2005 05:15 PM

These are old posts but I'd like to add my 2 cents. My IIS 6.0 HTTP compression was not working because I had a McAfee virus scanner running in parallel. Once I turned off the On Access Scan, it started to compress the downloaded files.

Frank Lynam on June 27, 2006 08:46 AM

I notice you're using a compression level of 10 in your example. Scott Forsythe has commented that a level of 10 can be more more cpu-intensive than a level of 9, but with little effective increase:
http://weblogs.asp.net/owscott/archive/2004/01/12/57916.aspx

Sam on November 23, 2006 06:52 PM

Can you tell me how to determine if http compression is enabled? My site is not hosted on my own server so i can't see that in the iis manager.

Any help would be appreciated.

Gerald Johnson on July 1, 2007 03:32 PM

Hi Gerald,

I use this page to test if compression is working:

http://www.port80software.com/products/httpzip/compresscheck

Jeff Atwood on July 1, 2007 04:00 PM

When I use the compressing, I receive for one of my pages a blank screen. When I open the page in notepad I get the following:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML {955246E3-6933-480d-B146-2462CE868767}="true"
{95F2CBFC-289F-4e9e-9972-50F36660DAEC}="0"
{55671E03-16F5-412c-97E7-648282E1183B}="true"
{F92F6ACC-5E50-4482-BC62-9D8DF61E5A32}="5"><HEAD>
<META http-equiv=Content-Type content="text/html; charset=utf-8"></HEAD>
<BODY></BODY></HTML>

For all other pages the compression seems to be working fine..
Any ideas on this one?

GM on August 13, 2007 07:52 AM

Just wanted to mention IISxpress here, it's a really good piece of compression software and compared to the pricing of many of the other ones it's a bargain.

http://www.ripcordsoftware.com/IISxpress/IISxpress_for_IIS.aspx

PL on August 23, 2007 12:59 AM

Hi,
I have enabled IIS compression on my server which hosts two websites. Weird thing is compression is working for one of the websites, but not the other. What steps can I take to diagnose why this is happening.

Also, on our internal testing servers, how do I check that compression is working? Since the port80 link you gave only works for sites on the internet. What is this sniffing you mentioned?

Thanks in advance :)

Bryon Chan on September 1, 2007 02:06 AM

For those posters wanting to know how to check if compression is enabled on their internal servers, here is the answer. You need to see the headers returned by your server, there are several ways to do this.

You could use a packet sniffer (Wireshark), an http proxy (proxomitron, fiddler) or a browser plugin (firebug for firefox). My personal favourite is Firebug as it allows you to see a lot more information about your site's performance, especially if you download the YSlow addon.

Alternatively, if you download the Pipeboost demo from port80, there is a header analyser tool included.

Bruce on October 28, 2007 01:36 PM

I can't get this to work w/ IIS on Windows XP Pro because the inetmgr doesn't have things like snapins and stuff..can't even find a metabase file

David on October 31, 2007 11:53 AM

works like a charm, superkewl post :>

Frikkie on November 27, 2007 01:36 AM

Here's the command line options to set it all up:

ADSUtil.vbs Set W3SVC/Filters/Compression/Parameters/HcDoStaticCompression TRUE
ADSUtil.vbs Set W3SVC/Filters/Compression/Parameters/HcDoOnDemandCompression TRUE
ADSUtil.vbs Set W3SVC/Filters/Compression/Parameters/HcDoDynamicCompression TRUE

ADSUtil.vbs Set W3SVC/Filters/Compression/deflate/HcDoStaticCompression TRUE
ADSUtil.vbs Set W3SVC/Filters/Compression/deflate/HcDoOnDemandCompression TRUE
ADSUtil.vbs Set W3SVC/Filters/Compression/deflate/HcDoDynamicCompression TRUE
ADSUtil.vbs Set W3SVC/Filters/Compression/deflate/HcFileExtensions "asx" "css" "doc" "htm" "html" "js" "txt" "xml"
ADSUtil.vbs Set W3SVC/Filters/Compression/deflate/HcScriptFileExtensions "asp" "ashx" "asmx" "aspx" "axd" "dll" "exe" "svc"
ADSUtil.vbs Set W3SVC/Filters/Compression/deflate/HcOnDemandCompLevel 10
ADSUtil.vbs Set W3SVC/Filters/Compression/deflate/HcDynamicCompressionLevel 9

ADSUtil.vbs Set W3SVC/Filters/Compression/gzip/HcDoStaticCompression TRUE
ADSUtil.vbs Set W3SVC/Filters/Compression/gzip/HcDoOnDemandCompression TRUE
ADSUtil.vbs Set W3SVC/Filters/Compression/gzip/HcDoDynamicCompression TRUE
ADSUtil.vbs Set W3SVC/Filters/Compression/gzip/HcFileExtensions "asx" "css" "doc" "htm" "html" "js" "txt" "xml"
ADSUtil.vbs Set W3SVC/Filters/Compression/gzip/HcScriptFileExtensions "asp" "ashx" "asmx" "aspx" "axd" "dll" "exe" "svc"
ADSUtil.vbs Set W3SVC/Filters/Compression/gzip/HcOnDemandCompLevel 10
ADSUtil.vbs Set W3SVC/Filters/Compression/gzip/HcDynamicCompressionLevel 9

IIS Reset, clear your cache, open Fiddler, open IE, make a request and voila, compressed content! One thing to note that on the dynamic level compression happens with each request. The blogosphere statea that the trade-off between levels 9 and 10 for dynamic compression is such that you will want to consider 9 because it takes exponentially less CPU versus the benefit.

Colin Bowern on January 8, 2008 12:22 PM

I have a question..

Is it normal for compressed aspx to load slower, like 10 seconds slower? The pages were all compressed but somehow they are now loading slower..

Wilfredo Estrada on March 4, 2008 04:29 AM

The only issue I had was my metabase would revert after restarting IIS. The trick... right click the computer in IIS, and check "Enable Direct Metabase Edit"

Jesse on June 11, 2008 04:59 PM