Another problem you don't mention above is the problem with Google's PageRank algorithms (and probably other search engines): If your page context gets indexed on different URLs, Google counts it as a duplicate website and automatically lowers the PageRank - on the site that was added last. You can use Google Sitemaps to reduce the problem, but you can't completely rule it out unless you use redirects.

I'm pretty sure there's a free version of ISAPI_Rewrite.

ISAPI_Rewrite has a free version, but the rewrite rules will get applied to every site on the server. It's perfect for the development environment.

I have had great luck with ISAPI_Rewrite and would recommend it to everyone. It's well worth the $90, and their support is pretty good.

I just wish I could figure out a way to make ISAPI_Rewrite work with the VS.NET built in web browser.

I'm a huge fan of ISAPI Rewrite. The bundled regular expressions tester is a fairly useful little tool too. I've used it to debug numerous regexps for varying languages.

I only wish I had a good way to automagically generate expressions for building applications in Visual Studio 2005. I believe in the W3C's philosophy of <a href="http://www.w3.org/Provider/Style/URI">Cool URIs Don't Change</a> and would prefer it if my .NET apps were future proofed for the next big extension change (eg from .asp to .aspx)

I agree, the ISAPI Rewrite tester is handy.

I grok regular expressions fairly well, but what trips me up with URL rewriting is I often don't know exactly what's coming into the regex-- is it the complete URL? Or just the path? There are definitely some oddities.

For example, this is directly from the ISAPI_Rewrite forum moderators:

RewriteCond Host: (.*)
RewriteRule ([^.?]+[^.?/]) http\://$1$2/ [RP]

Where the heck is $1 coming from? The Host: match?

And in this one, also lifted from the forums for the Ionic filter:

RewriteCond %{HTTP_REFERER} ^(?!HTTP_REFERER)

What I'm trying to do is say "allow blank referers", eg, "only apply this rule if the referer is not blank". I have NO idea how I would have figured this out without seeing someone do it first. The traditional way to say "allow blank referers" with mod_rewrite is this:

RewriteCond %{HTTP_REFERER} !^$

Also.. the spelling of referer has always bothered me. Shouldn't it be "referrer"? WTF.

Ah, big fat softballs (spring training is already on, so there). Let's take the thread down the other axis; may be not. The PP guys article linked references an essay on their site which is more extensive on the subject of DRY and the nature of OO. It has near the top this:

Procedural code gets information then makes decisions. Object-oriented code tells objects to do things.
--- Alec Sharp

That's from a 1997 smalltalk book. Yet, begetted by the Apache folks among others, what passes for OO code to this day, by this definition, is still procedural. Tell, don't ask.

The essay:
http://www.pragmaticprogrammer.com/ppllc/papers/1998_05.html

Are there any "correct" way an url should be formed?

I meen should it be files.ext and folders/ wise?

So this is correct:
.../blog/archives/000797/
.../blog/archives/000797.html
.../mypage.aspx

and this is wrong:
.../blog/archives/000797
.../mypage

In other words, should the url mapping represent a file system?

Why should the user be bothered about the file extension? php, aspx, ashx, etc..

referer is a known mis-spelling that was mis-spelled so long ago, they've decided to leave it alone in the name of familiarisation.

Here's an insane idea: let's just dump HTTP altogether and come up with another standard.

HTTP? I meant the URL standard. My brain no worky so good as of late.

http://msdn2.microsoft.com/en-us/library/ms228302.aspx

> Why should the user be bothered about the file extension? php, aspx, ashx, etc..

To a certain extent, that's true-- for scripting languages. Advertising your language is like painting a big "hack me" target on your site.

But for media content types, it's aggravating not to have the file extension:

http://www.megginson.com/blogs/quoderat/2007/02/15/rest-the-quick-pitch/
--
OK, a resource is a sort-of Platonic ideal of something (e.g. "a picture of Cairo"), while a representation is the resource’s physical manifestation (e.g. "an 800600 24-bit RGB picture of Cairo in JPEG format"). Yes, as you’d guess, it was people with or working on Ph.D.’s who thought of that. For a long time, the W3C pushed the idea of URLs like http://www.example.org/pics/cairo instead of http://www.example.org/pics/cairo.jpg, under the assumption that web clients and servers could use content negotiation to decide on the best format to deliver. I guess that people hated the fact that HTTP was so simple, and wanted to find ways to make it more complicated. Fortunately, there were very few nibbles, and this is not a common practice on the web. Screw Plato! Viva materialism! Go ahead and put “.xml” at the end of your URLs.
--

And with video files, it's aggravating even if you have the file extension. Good luck finding the correct magical combination of codecs you need to play that video.

DRY principles aside--why am I caring about Google page rankings? Sounds like ego-boo to me.

I like to leave all my URL extensions .html regardless of what the scripting language is. After all, that's the end product actually getting sent to the user. Then in the future, if I do decide to change to a different language, I can still keep the .html URLs.

Everything looks nice and static.

> why am I caring about Google page rankings? Sounds like ego-boo to me

You care about it, because without decent PR, nobody can find your content. Unless you're writing solely for yourself, in which case why bother putting it on the intarwebs?

But I agree with the sentiment. For me, it's simple DRY-- I want one and only one URL for any content, and I certainly don't want people bookmarking the non-www version of this URL, or the version with the annoying and unnecessary /index.htm URL.

>Why should the user be bothered about the file extension? php, aspx, ashx, etc..

In case "asp.net + URL Rewriting" you need to tell your IIS the extension of files you would like to "redirect".

You are assigning an ISAPI filter for a particular extension.

For example
http://www.featurepics.com/online/Red-Fox-Pics178160.aspx

(Redirect rules applied to
http://www.featurepics.com/image/img.aspx?fid=178160&show=image)

works, but Red-Fox-Pics178160 doesn't.

My favorite URL rewriting behavior of all time is SharePoint (WSS) v2, AKA the 2003 version. It will take a perfectly formed URL like

http://wss-site.intranet/site/subsite/

and will actively append default.aspx to this immediately. It a perfectly backwards implementation of URL rewriting: let's take a good URL, and make it less friendly! Awesome!

I guess I'm missing something. If you are using URL rewriting to map external links to different (or "cleaner") URLs, how is this helping your page rank?

If you rewrite all variations of http://(www\.)?foo\.com(/index.html?)? to http://foo.com/ then aren't you now supporting many external links - that will all get ranked sparately - to the same ultimate destination? Isn't that precisely what you were trying to avoid?

Am I misunderstanding how URL rewriting works?

> external links to different (or "cleaner") URLs

In the above case, we are mapping different URLs to a *single* URL. We are narrowing choice, not widening it.

Try it yourself. Navigate to http://google.com and see what shows up in your address bar. Or, try Scott's site. Navigate to http://www.computerzen.com and, again, see what shows up in the address bar.

> In the above case, we are mapping different URLs to a *single* URL. We are narrowing choice, not widening it.

I understand that but doesn't that mean that people could be linking to you by many different URLs and Google would keep stats on each of those "source" URLs thereby dividing your stats across many variants of the same effective URL, and lowering your rank? Or does Google resolve the URL to its destination value and see them all as the same URL?

Of course nothing stops people from making up random hyperlinks, but you should do your damndest to make sure those links NEVER appear in the address bar of the browser.

> does Google resolve the URL to its destination value and see them all as the same URL

That is the magic of 301, Permanent Redirect:

http://www.seroundtable.com/archives/007233.html

In the rules you saw above, [RP] means 301 "permanent redirect", and [R] means 302 "temporary redirect".

So, the value of URL rewriting comes from the fact that Google correctly canonicalizes links its spider finds before entering them into their index. Otherwise, it would be a net loss, as URL rewriting would allow countless distinct URLs to the same content.

Interesting. Google was successful in the beginning because its search page wasn't a crowded, cluttered mess of ads or irrelevant links, and produced good results because their index modeled the real behavior of the web reasonably well. Now implementation details of their index scoring have become an independent variable to determine how much traffic a web site gets.

Are we seeing Google transition from an "observe, and correctly evaluate" methodology to "observe, and define rules to influence mass behavior"? Something about that bothers me.

Pretty URLs originally became important because some search engines would not index files with querystring variables or with more than a few querystring variables-- probably because you could wind up indexing an infinite number of URLs on one site (trust me, having had this happen on a local copy of Inktomi back in the bad old days before I bought a clue).

Subsequently, they've taken on a life of their own and everyone has their own rules about them (file extensions always, file extensions never). They appeal to geeks because of their cleanliness. The real justification, however, is that properly crafted URLs mean you can completely re-engineer your site and have everything still live in the same place (so my preference is for no file extension in that case; it's also a free bit of security through obscurity).

You can use Google's webmaster tools to let Google know to treat several of your domains as one (i.e. codinghorror.com and www.codinghorror.com are the same page). If you have never used Google webmaster tools I highly recommend it.

Very nice topic.. Id like to add the mod_rewrite code for Apache that forces only the www.domain.com or vice-versa, and other .htaccess SEO tips at http://www.askapache.com/2006/htaccess/htaccesselite-ultimate-htaccess-article.html

Some pretty bad practice here: absolute paths!
=========
RewriteRule ^/(.*) http://www.test.com/$1 [RP]
...
RewriteRule .*\.(?:gif|jpg|jpeg|png) /images/block.jpg [I,O]
=========
Why it is bad? From the top of my head (30 seconds):
- cannot change the domain name easily (also moving to a new country: www.codinghorror.fr, www.codinghorror.co.jp)
- bad internationalization (all languages point to the same (English?) file /images/btnNew.gif)
- cannot test it properly on a local file-system, cannot make a CD with a functional image of you site, etc. (ok, works for plain html, not asp, jsp, php, etc.)

ISAPI_Rewrite is unnecessary with ASP.NET 2.0, as it's built-in. You can rewrite URLs in global.asax.

> is unnecessary with ASP.NET 2.0, as it's built-in

Really? So you can redirect http://test.com/folder/ ?

I don't think so.

I am not a fan of the "native" ASP.NET 2.0 redirection because it's fraught with limitations and holes. Unless you want all of your redirections to end in .aspx or some other file extension that the ASP.NET IIS handler understands, you have to redirect *all* file formats through the ASP.NET ISAPI filter. This is a nasty hack with lots of side-effects, and it still doesn't address the issue of redirecting folders.

If you can only redirect URLs that end in ".aspx", or if you have to remap all files to go through the ASP.NET ISAPI filter, then you don't have a true URL redirection solution IMO.

I can see you've got http://www.codinghorror.com and http://codinghorror.com and http://www.codinghorror.com/index.html and http://codinghorror.com/index.html (and few others) all resolving to http://www.codinghorror.com

Any reason why you would choose to use the www for the final URL or is it just personal preference?

Oddly... I put this in my .htaccess and it 500 errors the entire site!

The only part I used was:

# force proper www. prefix on all requests
RewriteEngine on
RewriteCond %{HTTP_HOST} ^treehugger\.com [I]
RewriteRule ^/(.*) http://www.treehugger.com/$1 [I,RP]

Nick, Make sure you're using the right syntax. Only one of the lists I posted is (mostly) Apache mod_rewrite compatible -- the top one.

I find the Author is absolut correct that one should redirect to one site only. It's a must and so simple to do.
But one should check which route is best to take when doing a redirect such as to www.mysite.com or just mysite.com.
There are tools out there, which will tell you whats best for you.
In my case I used the www route as much most of my pages were indexed
with www. And in a matter of days my non www pages all disapeared from the SE.

Jeff, there is something which is interesting.

>For a long time, the W3C pushed the idea of URLs like http://www.example.org/pics/cairo instead of http://www.example.org/pics/cairo.jpg, under the assumption that web clients and servers could use content negotiation to decide on the best format to deliver.

W3C has indeed pushed Content Negotiation (See Web Architecture) and it is working, it depends on what you want to achieve, but "content negotiation" is part of HTTP which is a spec published by the *Web Community* (yes, including W3C affiliated persons) at the *IETF*.

The idea of Content Negotiation is not owned by W3C. :)

The only issue I've had with ISAPI Rewrite is when there are too many redirects in the httpd.ini file, which can slow down the request quite a bit.

Jeff: Great post and comments!

Glad to see URL Design finally getting some respect, especially from the ASP.NET crowd who have been the 'have-nots' when it's come to clean URLs (vs. Apache web apps that have mod_rewrite built in.)

Anyway, for those interested in this sort of thing I discuss URL-related issues in depth over at The Well Designed URLs Initiative. And I am working to produce a comprehensive set of patterns and best practices but I need lots of input from web developers as I don't know it all. That's why I started the URLQuiz series where you get to test your URL knowledge and give your opinions of our URL design questions. Check it out at:

http://blog.welldesignedurls.org/urlquiz/

That's funny, I was just trying out both and I was only able to get Iconic rewrite working, isapi rewrite only worked when I put the server into IIS 5 isolation mode.

I'm not sure if the Apache extension works the same way, but with IIS ISAPI Rewrite the instructions file must be at the root of the site so won't work under a virtual directory. This only really is an issue if you develop under XP where you can only run a single web site.

Force www? Bleh. I STRIP www. An acronym that uses more syllables than the phrase it abbreviates. And who needs it? What's wrong w/ http://google.com?

"What's wrong w/ http://google.com?"

old people freak out

As a heads up for people - in the version of Ionic Rewrite that I downloaded (IonicIsapiRewriter-1.2.11b), RP is not the correct syntax for permanent redirects, it is R=301. Ionic accepted RP, but it was doing 302 (temporary) redirects. I'm no expert, but the articles I'm reading indicate that 301 redirects are the way to go. Also, I had no trouble getting Ionic to run. You do need to make sure that the IIS Guest user (ISR_<machinename>) has permission to access the folder you put the dll and ini file in, though.

I've installed isapi rewrite and iconic rewrite, iconic was actually easier to get working on one site but as soon as I started putting it on more sites on the same server it quickly killed the server.
I've recently had another problem caused by removing the index pages as well, unless your index pages are static html you won't want to add the redirects that don't show the page.

# remove index pages from URLs
RewriteRule (.*)/default.htm$ $1/ [I,RP]
#RewriteRule (.*)/default.aspx$ $1/ [I,RP]
RewriteRule (.*)/index.htm$ $1/ [I,RP]
RewriteRule (.*)/index.html$ $1/ [I,RP]

I was doing some testing with an ajax page and the request kept loosing all it's values, this was because isapi rewrite kept intercepting the request and redirecting without keeping the posted values, this also caused problems with another regular .net page all because they were named default.aspx

By commenting out the redirect for default.aspx I was able to get the pages working properly, this took the better part of a day for me to track down.

Hi There:
I am in the middle of a URL rewrite project where the requirement
is for me to have a .aspx filename in the URL for my url rewriter to get control. I have an actionless form as well.

Obviously this approach seems to not work for Directories http://www.foobar.com/categeries/Springs/ or even the main site http://www.foobar.com (when it does not have the default.aspx) when I am trying to do a postback from a control on that page. The initial rendering works ok.

(a) I think I can use Iconic to solve this problem, right?
(b) If yes, can I use the Iconic filter to push all incoming requests that are missing the default.aspx to include it so that the final url on the client browser would make have the default.aspx even if they typed in http://www.foobar.com for an web address.

Would I be able to do that with this filter?

Any pointers would be appreciated. thanks much!

KP

For those of us who use shared hosting and want url rewriting to work, I came up with a nice way to get extension-less urls working without the performance penalty for routing all content through asp.net and the solution is 100% C# code.

Take a look here for the details:
http://pragmaticprose.com/PermaLink,guid,e6d205ab-3edb-4b84-9e7c-6d6a28476e13.aspx
Any feedback on my solution is welcome.

Hi Jeff,

I have a question about using Ionic's ISAPI Rewrite Filter. In the article, you say you could not get it to work on IIS ver 5.

I am doing development work on a windows xp with IIS 5.1. So, will the ISAPI rewrite filter not work on my development machine(cos its IIS 5.1) ?

Is it possible to install IIS 6 on windows xp? (The microsoft website says IIS6 only runs on Windows Server OS)

Thanks,

I tried the isapi rewrite and I couldn't get to my website after that. I vopied the code just changed test to e-notations

Guys,

if you're serious about using a paid but serious solution that is fully compatible with apache mod_rewrite (not just similar or wannabe mod_rewrite), then you should consider IIS Mod-Rewrite.

http://www.micronovae.com/ModRewrite/ModRewrite.html

Also, the Pro version supports .htaccess. That's damn cool!

I would like to mention that that we (Helicon Tech) have released new version of ISAPI_Rewrite that is now nearly 100% Apache mod_rewrite compatible. Please see compatibility chart here: http://www.helicontech.com/isapi_rewrite/doc/compatibility.htm

As for best URL format practice my answer is: keep URLs as short and descriptive as possible. Generally it is better to use dash &#8216;-&#8216; or underline &#8216;_&#8217; separators for parameters instead of slash &#8216;/&#8217; but it mostly depends on your web site design. It is better to omit extension or use .htm/.html

Link example: http://www.somesite.com/articles/best_url_format_practice.htm

And freeawre ISAPI_Rewrite version (Lite) is still the best free URL rewriting solution available. To apply rules to specific web site you only need to precede rules with this line:

RewriteCond %{HTTP:Host} (?:www\.)?myhostname\.com [NC]

How can I avoid the following:

the original URL is http://www.pocketgadget.org/2008/01/14/serendipity-10-accidental-inventions/

but somebody has submitted it to Didd as
http://www.pocketgadget.org/2008/01/14/serendipity-10-accidental-inventions#html

so now Google has two copies of the same page. I have tried to do a 301 redirect from the second URL to the first and still nothing.

I am a bit lost with regards to this one. Anything i should be down at the server level that I am not aware of ?

Yes I agree with you.
The www in the domain name is very ugly.
You can also redirect via PHP-file to a domain without www.

The proper way of doing this is to always include a HTTP Content-Location header on non&#8208;preferred URIs. This is a lot faster than rewriting and &#8208;directing the user to other URIs.

For example, a user requests /document from example.tld; anaccompaniedt that document from that source acomanied by a HTTP header telling the user client&#8212;including search engines&#8212;that the &#8220;Content-Location: http://www.example.com/document.[eng].[utf-8].xht&#8221;. To save bandwidth, this can be a relative&#8208; instead of an absolute URI when the request indicates that the preferred host or full URI already is known.

This is a power tip for advanced users. ;-)

Ionic's ISAPI Rewrite is still actively maintained, still free, and still works! It runs on IIS5 IIS6 and IIS7. Latest new feature: Rewriting HTTP Headers the same way you can rewrite URLs.

How can I avoid the following recursive loop:

OLD URL is http://www.example.com/movies/home.aspx
for SEO we make URL as http://www.example.com/movies/

to achieve we write Rules in httpd.ini file as
RewriteRule /movies/home.aspx /redirector.aspx?URLAction=movies/ [I,L]
RewriteRule /movies /redirector.aspx?URLAction=movies/ [I,L]
RewriteRule /movies/ /movies/home.aspx[I,L]

problem is browser showing this message "Firefox has detected that the server is redirecting the request for this address in a way that will never complete"

how to resolve this recursive loop ?

IEEE article link is dead--and the IEEE site downloads a file ending in ".pdf", but containing HTML.

Try this link instead: http://snipurl.com/ieee-dry

See this with comple working sample code <a href= "http://dotnetkeeda.blogspot.com/2009/04/url-rewriting-mapping-in-aspnet.html">how to rewrite url</a>

Cheers

See Sample code here

http://dotnetkeeda.blogspot.com/2009/04/url-rewriting-mapping-in-aspnet.html