Coding Horror: Searching all Craigslist.org Cities

March 31, 2005

Searching all Craigslist.org Cities

If you've ever used Craigslist.org-- a fantastic and rather odd resource-- you may have noticed that it's heavily biased towards per-city searches. This is a pain if you want to do a national search across all cities that Craigslist.org operates sites for. A while back, I found a web page offering an all-city search, but I wasn't happy with the performance. And now it's been perma-banned by the Craigslist.org brass, so.. time to roll up my sleeves and implement my own improved Craigslist.org "all cities" jobs search.

My search is dramatically faster than Chovy's, because I use HTTP compressed queries and progressive Response.Write and Response.Flush output rendering as the queries are returned. This is one of my major beefs with ASP.NET; there's no way to do any kind of progressive page rendering using the ASP.NET architecture. (And no, iframes do not count.) The entire page renders to a buffer, then-- and only then-- it is all displayed at once. Not exactly an ideal web experience if you're building large pages.

Progressive output is particularly critical for a long-running web page process like this one. Otherwise you'd be sitting there looking at a whole lot of nothing until all ~30 cities were queried. Technically it takes the same amount of time, but there's a huge psychological difference between seeing immediate, if partial, results, and waiting the same amount of time looking at a blank screen. I always try my best to design for progressive rendering, even in Windows Forms.

Interestingly, I had to disable IIS6 dynamic compression to get Response.Flush to behave as expected. The good news is that disabling IIS6 compression can be done on a per-website per-folder basis:

I used the following commands from the \Inetpub\AdminScripts directory:
cscript adsutil.vbs set w3svc/{site#}/root/{vdir name}/DoStaticCompression False
cscript adsutil.vbs set w3svc/{site#}/root/{vdir name}/DoDynamicCompression False

To get the {site#}, click on the "Web Sites" node in the IIS manager and note the "Identifier" number in the right-hand Detail pane for the top-level website that contains the NTD application.

My favorite Craigslist posting, by the way, is this one.

Posted by Jeff Atwood View blog reactions

« What's on your keychain? Parsing: Beyond Regex »

Comments

One thing you have to admire about ads like this one, assuming it really is for real, is that some people really know what they want and are not afraid to ask for it. :-)

Is non-progressive rendering really an ASP.NET issue or a general issue with serving Web pages? AFAIK, you can't get progressive rendering of straight HTML pages. What about other dynamic rendering platforms?

Just curious, I know very little about this.

mike on March 31, 2005 10:19 AM

> AFAIK, you can't get progressive rendering of straight HTML pages

Er.. yeah you can. That's the way HTML works; it renders what it can as soon as it arrives, except for pathological cases. Here's an overview:

http://blogs.msdn.com/ie/archive/2005/02/10/370721.aspx

and here's a demo:

http://msdn.microsoft.com/workshop/samples/author/dhtml/refs/tables3.htm

The slower your connection (and/or the larger the page) the bigger the difference will be.

Jeff Atwood on March 31, 2005 10:24 AM

Welp, there ya go ... I *said* I know very little about this ... :-)

mike on March 31, 2005 12:27 PM

Also I'm 99.999% sure that posting is a cleverly constructed joke. And what a great one!

Jeff Atwood on March 31, 2005 06:19 PM

I wanted to tell you thank you very much for developing the Craigslist Search All-Cities program! I work from home as an independent contractor, and I have found your program very valuable in searching for telecommuting jobs.

Thank you!!

WAH on June 8, 2005 12:48 AM

I did the same thing you did (funny how craigslist doesnt do what so many people request) but i made it a little different, i did city and state wide searches for For Sale items on craigslist.

so like everyone elses.. my server got blocked automatically

but.. i found a way around it. and i am in the process of completing the site/program so everyone can benefit what craigslist ignores.

stay tuned. if youd like.. email me and ill give you the site/info

Eric on July 2, 2005 06:53 AM

Maybe you can help>

There are 1000s of postings on Craig's List for offers of housing to Katrina survivors. But these folks have little or no access to the web, email and craig's list etc. I have been trying to find out how I can get a file or report that includes housing offers out of their database accross all cities that can then be printed and distributed. The people who need housing need a piece of paper with phone numbers (and money to make the call). They don't have computers!

I have had ZERO luck getting anyone from craig's list to contact me. Do you have knowledge on how to get in touch with someone from Craig's list who has power to make this happen or a way to pull this data from their system?

PLEASE reply to this post or contact me directly @ glen@colemancomputers.com

Thanks...

Glen

Glen on September 5, 2005 01:49 AM

Hi Glen,

Your best bet is the craigslist forums:

http://forums.craigslist.org/?i=

Jeff Atwood on September 5, 2005 04:42 AM

can you get me the email address of that guy "Eric" who posted the below?

i would very much like to see his site and the code used to do what he claims...
contact me jionin@optonline.com
or aim: ionizer5
thanks

----------

I did the same thing you did (funny how craigslist doesnt do what so many people request) but i made it a little different, i did city and state wide searches for For Sale items on craigslist.

so like everyone elses.. my server got blocked automatically

but.. i found a way around it. and i am in the process of completing the site/program so everyone can benefit what craigslist ignores.

stay tuned. if youd like.. email me and ill give you the site/info

Posted by: Eric at July 2, 2005 06:53 AM

Jared on September 8, 2005 11:51 AM

I realize that craigslist has blocked this service, but I'm more interested in how you did it from an academic standpoint.

How did you get,parse, and then cache the data from the craigslist posts? I'm looking at the rss feeds on the site, but they generally only contain sparse information on the last 15 posts in that section.

Would you mind sharing your overarching design? or possibly source code snippets?

Andy on March 23, 2006 03:45 PM

<a href="http://www.crxsearch">http://www.crxsearch</a>; dot com

Thats my multi city, multi state, multi region craigslist search tool. i wont go into details on how I did it, but i will say that it uses RSS feeds.

See the biggest problem in this kind of search, is getting the user to be able to search craigslist via rss feeds.. without getting banned by the CL servers. The CL servers will ban an IP if it hits the CL rss feeds too often. So my task was to find a way for offer the search tool, but have it so the CL servers will not be able to block the IP. I feel like I just went in a big circle with that.. But yes.. as of right now..

As of right now... my program has
New Users per day: 1.18
We have 348 registered users
Total Craigslist searches: 4541
Total hits: 2375

Which I know is not a "lot" compared to chovys original site, but hey.. at least mine works :)

And yes, I do know that if RSS feeds gets taken off, my search will not work. BUT... there is another way, that I believe I can make this work. But hopefully I wont have to try that method because it will take even longer than this did.

Eric on April 20, 2006 12:33 AM

Eric, it works, eh?

Juan on April 22, 2006 12:51 PM

Here's another craigslist search tool. I've taken a look at what else is out there, and I'm pretty sure this is the best.

It's all web based, so no need to install anything. It won't be blocked, because it simply opens the results in iFrames. But it's set up to let you easily and QUICKLY scroll through the various cities. Anyway, just go give it a try and you'll see what I mean.

http://www.craigshelper.com

Nathan on September 17, 2006 01:06 PM

I'll through my tool onto the pile.

http://www.craig2mail.com/craigs.html

Ok, it doesn't search multiple cities. What it does do is email you when an ad that matches your search is posted. No more searching over and over just to find that someone else beat you to it. Just sit back and watch your email. I think it is pretty cool.

Craig (no not that Craig)

Craig on September 28, 2006 01:35 PM

Beyond blocking IP's is there legal action that CL has taken in the past? What are these cases?

legal authority on March 18, 2007 07:20 PM

Here's a Craigslist search you might digg:
http://cl.radiux.com

Craig McDowell on July 10, 2007 02:46 PM

Anyone tried CraigSpy? Standalone prog so potential IP bans are user-to-user, but I've been able to search about as much as 10,000 pages on CL with simple search terms. Good for getting free stuff for the house...lot of potential, really. :)

shwa on August 15, 2007 05:17 PM

http://www.craig-spy.com

shwa on August 15, 2007 05:18 PM

I discovered that if you put this in google "not ok to contact this poster" and then type in some keywords, it will bring up craigslist ads from all over the country. sample google search:

"not ok to contact this poster" katrina

look at all those results! I tried to find some wording that was in all craig's list postings, and this was as close as I could get.

Sarah on September 7, 2007 01:34 PM

He -

Thanks for the tip on http://cl.radiux.com. That's the best Craigslist browser/search I have seen. I use it every day now.

JohnM on September 7, 2007 05:52 PM

Did anyone try CLSearch from http://www.craigslistcompanion.com/

It is a standalone windows program which give you the multi-city search functionality, plus it will give you a preview of any pictures that were attached to an craigslist ad.

Best of all, it is a free download.

Max on September 30, 2007 01:04 PM

YES I AGREE - CRAIGSPY IS MUCH BETTER AND IS AUTOMATIC!!!!!

jibman on October 7, 2007 08:48 AM

Thank you for trying Jeff,
I am sure that Graigslist.org will not understand how valuable your search engine was.

Goodluck Jeff.

M on November 4, 2007 03:18 PM

This is a simple wrapper for Google-based services. Works pretty well.

http://craigfind.com

mike on November 6, 2007 10:45 AM

CLBrowser is the newest tools for craigslist.

Give it a try at http://www.craigslistcompanion.com/CLBrowser.php

max on November 29, 2007 11:12 AM

You should try Craigslist Search CLSearch at http://bouncingknee.com/?page_id=24 You can search all the cities o modify the list of cities to search only the ones you are interested in. Pretty cool program.

JohnLennon on February 4, 2008 04:24 PM

We also have a search tool "One Eye Open" for running multiple searches over whatever cities you select. Free demo available.

http://www.bigattichouse.com/oneeyeopen.html

Tomorrow (Feb 8,2008) we're running a special with Bits Du Jour, so you can buy for $9.80

http://www.bitsdujour.com/index.aspx?page=index.aspx%3freview%3d350

Mike Johnson on February 7, 2008 07:49 AM

Google will search them all also, type in search characters (i.e. 1975 Pontiac Grand am) followed by "site: craigslist.org". Unlike most the tips above, this will work on any computer, Mac or PC.

Rick on February 13, 2008 07:10 AM

I use perfect automated tool to search craigslist. Check their videos, and how it works. It takes Craigslist on totally new format, efficient, precise, organized, really nifty.

www.CraigsPal.com

Dan on March 4, 2008 03:44 PM

well F#$K the man... this would have been nice

bill lambeer on March 14, 2008 01:23 PM

I just go to google to search craiglist...

In the searchbox, I enter "Craigslist:"

After the colon, include your search items.

Easy as pie (and free).

dale on April 20, 2008 08:44 AM

To search craigslist cities by state, you can use http://www.craigslist-search.com

Hope this info helps.

CraigslistSearchByState on July 6, 2008 08:12 PM