Searching all Craigslist.org Cities

If you've ever used Craigslist.org-- a fantastic and rather odd resource-- you may have noticed that it's heavily biased towards per-city searches. This is a pain if you want to do a national search across all cities that Craigslist.org operates sites for. A while back, I found a web page offering an all-city search, but I wasn't happy with the performance. And now it's been perma-banned by the Craigslist.org brass, so.. time to roll up my sleeves and implement my own improved Craigslist.org "all cities" jobs search.

My search is dramatically faster than Chovy's, because I use HTTP compressed queries and progressive Response.Write and Response.Flush output rendering as the queries are returned. This is one of my major beefs with ASP.NET; there's no way to do any kind of progressive page rendering using the ASP.NET architecture. (And no, iframes do not count.) The entire page renders to a buffer, then-- and only then-- it is all displayed at once. Not exactly an ideal web experience if you're building large pages.

Progressive output is particularly critical for a long-running web page process like this one. Otherwise you'd be sitting there looking at a whole lot of nothing until all ~30 cities were queried. Technically it takes the same amount of time, but there's a huge psychological difference between seeing immediate, if partial, results, and waiting the same amount of time looking at a blank screen. I always try my best to design for progressive rendering, even in Windows Forms.

Interestingly, I had to disable IIS6 dynamic compression to get Response.Flush to behave as expected. The good news is that disabling IIS6 compression can be done on a per-website per-folder basis:

I used the following commands from the \Inetpub\AdminScripts directory:

cscript adsutil.vbs set w3svc/{site#}/root/{vdir name}/DoStaticCompression False
cscript adsutil.vbs set w3svc/{site#}/root/{vdir name}/DoDynamicCompression False

To get the {site#}, click on the "Web Sites" node in the IIS manager and note the "Identifier" number in the right-hand Detail pane for the top-level website that contains the NTD application.

My favorite Craigslist posting, by the way, is this one.

What's on your keychain?

It's a geek rite of passage: what's on your keychain? Here's mine:

• 512mb Sandisk Cruzer USB 2.0 thumbdrive
• Leatherman Squirt S4
• Arc AAA LED flashlight

I carried a Leatherman Micra for years, but I forgot to ditch it prior to a business trip and it got confiscated at the airport. Very frustrating, but it forced me to look for alternatives, and the Squirt is superior. It's available in colors, it's slightly smaller, and most importantly-- unlike the Micra, the ancillary tools can be used without opening the scissors/pliers. It's incredibly handy.

And you can have it engraved. Mine is, of course, a homage to Pulp Fiction. I think it's much funnier on a tiny multi-tool, but you can actually get the real wallet if you're so inclined. There's an alternate, non-embossed version available here, with some related trivia and media.

USB thumbdrives, on the other hand, are all basically the same. There is some variance in read/write speeds based on the type of flash media used, but even the fastest ones are too slow to substitute for a real hard drive unless you're extremely patient. With any luck, the USB thumbdrive will become the new floppy drive. I've talked before about how to make your USB thumbdrive bootable, which is always convenient. I picked the Sandisk Cruzer simply because it happened to be on sale at Costco; I had to drill a small hole in it to make it suitable for keychain mounting. It does have one feature I like-- the giant LED on the rear of the device, which makes it quite obvious when it's connected or transferring data. It's also fairly small as these devices go.

USB thumbdrives are way more boring than LED flashlights or multitools-- it's what you put on them that makes them interesting:

• Jeremy Wagstaff's A directory of programs designed for USB drives
• Engadget's what do you keep on your USB keychain drive?
• Nauman Leghari's what's on my USB?
• Steve Makofsky's what's on your keychain?
• what's on Mike Gunderloy's keychain

There are some great ideas in those threads. A lot of it is what you'd expect: anti-virus, anti-spam, remote access, encrypted passwords, and various utilities. I only recently added my USB drive to my keychain*, so I'm not sure how much of this stuff I'll actually need or use, but I'm game to find out. I did discover one particularly intriguing item in these lists: uniform server, which is a tiny WAMP (Windows, Apache, MySQL, PhP/Perl) distribution. Pretty cool.

update: Here's my 2006 edition of the keychain.

* after going through a Palm trial in 1996 and trying again with a refurb Tungsten C last year, I've concluded that PDAs just don't work for me. I guess I'll be needing that mythical perfect smartphone, but there's one small problem: I don't typically carry a cellphone with me, either.

John Carmack on Java, Phones, and Gaming

John Carmack, the primary developer of Doom and Quake at id Software, posted some great comments on his recent experiments with cellphone game development in Java. My favorite?

there is something deeply wrong when text editing on a 3.6 ghz processor is anything but instantaneous.

That's quote of the month material.

Building Mht Files from URLs revisited

I finally finished updating my Convert any URL to a MHTML archive using native .NET code CodeProject article. It's based on RFC standard 2557, aka Multipart MIME Message (MHTML web archive). You may also know it as that crazy File, Save As, "Web Archive, Single File" menu option in Internet Explorer. It's basically a way to package an entire web page as a (mostly) functonal single file that can be emailed, stored in a database, or what have you. Lots of interesting possibilities, including quick and dirty offline functionality for ASP.NET websites using loopback HTTP requests.

This was a truly painful total rewrite, but it offers tons of new functionality:

• Completely rewritten!
• Autodetection of content encoding (eg, international web pages), tested against multi-language websites
• Now correctly decompresses both types of HTTP compression
• Supports completely in-memory operation for server-side use, or on-disk storage for client use
• Now works on web pages with frames and iframes, using recursive retrieval
• HTTP authentication and HTTP Proxy support
• Allows configuration of browser ID string to retrieve browser-specific content
• Basic cookie support (needs enhancement and testing)
• Much improved regular expressions used for parsing HTTP
• Extensive use of VB.NET 2005 style XML comments throughout

If you're interested, you can download the VS.NET 2003 solution from my blog until the CodeProject site gets updated. Here's a screenshot of the demo app packaged with the Mht.Builder class:

On Necessity

When working with users, I am frequently reminded of this conversation in David O. Russell's movie Three Kings:

GATES
What is the most important thing in life?

TROY
What are you talking about?

GATES
What's the most important thing?

TROY
Respect?

GATES
Too dependent on other people.

VIG
What, love?

GATES
That's a little Disneyland, isn't it?

DOC
God's will?

GATES
Close.

TROY
What is it then?

GATES
Necessity.

TROY
As in..?

GATES
As in people do what is most necessary to them at any given moment. Right now what is most necessary to Saddam's troops is to put down the uprising. We can do what we want, they won't touch us.

TROY
All right. I'll be wearing fashionable Kevlar.

This is a wartime version of what Steve Krug, in his book Don't Make Me Think, calls satisficing:

When we're designing pages, we tend to assume that users will scan the page, consider all of the available options, and choose the best one.

In reality, though, most of the time we don't choose the best option -- we choose the first reasonable option, a strategy known as satisficing. As soon as we find a link that seems like it might lead to what we're looking for, there's a very good chance that we'll click it.

I'd observed this behavior for years, but its significance wasn't really clear to me until I read Gary Klein's book Sources of Power: How People Make Decisions. Klein spent 15 years studying naturalistic decision making: how people like firefighters, pilots, chessmasters, and nuclear power plant operators make high-stakes decisions in real settings with time pressure, vague goals, limited information, and changing conditions.

Klein's team of observers went into their first study (of field commanders at fire scenes) with the generally accepted model of rational decision making: Faced with a problem, a person gathers information, identifies the possible solutions, and chooses the best one. They started with the hypothesis that because of the high stakes and extreme time pressure, fire captains would be able to compare only two options, an assumption they thought was conservative. As it turned out, the fire commanders didn't compare any options. They took the first reasonable plan that came to mind and did a quick mental test for problems. If they didn't find any, they had their plan of action.

Make sure you're designing for what is necessary rather than what is possible. Designs that assume more investment on the part of the user are doomed, because they are designing for idealized behavior rather than actual behavior. Users really don't care about your application-- they have specific goals and will do only the absolute minimum necessary to achieve those goals. For web sites, Steve lists a few reasons users may behave this way:

• We're usually in a hurry
• There's not much of a penalty for guessing wrong
• Weighing options may not improve our chances
• Guessing is more fun

For a more in-depth discussion, I strongly recommend picking up a copy of Don't Make Me Think.

Are You There, God? It's Me, Microsoft.

Although you eventually outgrow them, any developer worth his or her salt bears the scars of a thousand tiny religious wars. It's an occupational hazard, as Steve McConnell notes in Thou Shalt Rend Software and Religon Asunder:

Religion appears in software development in numerous incarnations-- as dogmatic adherence to a single design method, as unswerving belief in a specific formatting or commenting style, or as a zealous avoidance of global data. Whatever the case, it's always inappropriate.

Blind faith in one method precludes the selectivity you need if you're to find the most effective solutions to programming problems. If software development were a deterministic, algorithmic process, you could follow a rigid methodology to your solution. But software development isn't a deterministic process; it's heuristic, which means that rigid processes are inappropriate and have little hope of success. In design, for example, sometimes top-down decomposition works well. Sometimes an object-oriented approach, a bottom-up composition, or a data-structure approach works better. You have to be willing to try several approaches, knowing that some will fail and some will succeed but not knowing which ones will work until after you try them. You have to be eclectic.

I think it's great that we are passionate enough about what we do to have these kinds of discussions. As long as everyone retains their sense of humor. However, I can't imagine the fire and brimstone that results when you mix software religion with.. that old time religion, as FellowshipChurch.com has:

Microsoft continues to improve in this area, and there is always a new version just around the corner that will make everything better, but this is a fact of life. Microsoft products, from the server to the development environment, will inexplicably stop working with no outside interference. Each of our development machines begs to be rebuilt after six months of use. Servers that haven't been rebooted for a couple of weeks begin to have issues. Code that has worked for months stops working for no apparent reason.

I built a number of Linux servers a couple of years ago and nine months later I had to be reminded that they existed as they hadn't been touched or rebooted since they went live.

Could it be.. satan?

Trees, TreeViews, and UI

I somehow doubt this is what Joyce Kilmer was thinking of when he wrote the poem Trees:

I think that I shall never see
A poem lovely as a tree.

It's unfortunate that the treeview is one of the standard widgets in a usability designer's toolkit, because trees aren't usable. They're a pain in the ass. They may be necessary for developers who are forced to work in the strict, rigid world of software development, but they are unnatural, restrictive, and just plain unnecessary for average users. Where do I begin?

1. Trees force a rigid hierarchy
   There's an episode of the old television show Gomer Pyle, USMC where the quartermaster gets sick and Gomer is put in charge of the Army PX. Gomer proceeded to reorganize every item in the PX into three categories: animal, vegetable, or mineral. Hilarity ensued. Lesson: rigid categorization may seem like a good idea, but it doesn't work very well in practice.
2. Trees are difficult to browse
   Good luck finding anything in a tree; it's a navigational nightmare. Expanding and collapsing folders constantly causes items of interest to fall out of view, and loss of context in the hierarchy. Expand enough, and you'll end up scrolling not only up and down but also left to right. Interactively searching trees is awkward, if even supported.
3. Categorization is an expert activity
   If left to their own devices, your users aren't likely to do any better than Gomer Pyle-- unless they happen to be experts in library science and information mapping. Categorization is extraordinarily difficult to do correctly unless you're an expert in the field. And even then, there is disagreement.
4. Trees imply a parent/child relationship
   On top of all the rigid hierarchy baggage, there's an additional connotation of ownership-- both physical and logical-- that goes along with putting items a tree. Are you sure that item has one clearly defined owner and one clearly defined parent?

Any time you're tempted to add a TreeView to your application, consider carefully. Whenever I've encountered TreeViews, I've found that a flatter, less rigid representation of the data is almost always possible-- and much easier for users to understand and manipulate. Don't blindly fall back on a full-blown tree without weighing the alternatives.

It's true that treeviews are appropriate for a few specialized situations. A HR diagram of managers and employees, for example. In my experience, however, trees get horribly abused. The canonical example of unnecessary tree use is in email clients. Google has an excellent solution in Google Mail's labels:

The old way You create an elaborate filing system of folders and subfolders, then decide where to file a single message. The Gmail way Instead of folders, Gmail uses labels to give you the functionality of folders, but with more flexibility. In Gmail, a single conversation can have several labels, so you're not forced to choose one particular folder for each message you receive. That way, if a conversation covers more than one topic, you can retrieve it with any of the labels that you've applied to it. And, of course, you can always search for it.

I've aggressively adopted the label approach, because it's so much more reflective of the fluid way things are organized in the real world. Programmers may love rigidity-- to each item its appropriate folder and meticulously named class hierarchy-- but users prefer simple, flat lists.

If You Like Regular Expressions So Much, Why Don't You Marry Them?

All right... I will!

I'm continually amazed how useful regular expressions are in my daily coding. I'm still working on the MhtBuilder refactoring, and I needed a function to convert all URLs in a page of HTML from relative to absolute:

''' <summary>
''' converts all relative url references
'''    href="myfolder/mypage.htm"
''' into absolute url references
'''    href="http://mywebsite/myfolder/mypage.htm"
''' </summary>
Private Function ConvertRelativeToAbsoluteRefs(ByVal html As String) As String
  Dim r As Regex

  Dim urlPattern As String = _
    "(?<attrib>\shref|\ssrc|\sbackground)\s*?=\s*?" & _
    "(?<delim1>[""'\\]{0,2})(?!#|http|ftp|mailto|javascript)" & _
    "/(?<url>[^""'>\\]+)(?<delim2>[""'\\]{0,2})"

  Dim cssPattern As String = _
    "@import\s+?(url)*['""(]{1,2}" & _
    "(?!http)\s*/(?<url>[^""')]+)['"")]{1,2}"

  '-- href="/anything" to href="http://www.web.com/anything"
  r = New Regex(urlPattern, _
    RegexOptions.IgnoreCase Or RegexOptions.Multiline)
    html = r.Replace(html, "${attrib}=${delim1}" & _HtmlFile.UrlRoot & "/${url}${delim2}")

  '-- href="anything" to href="http://www.web.com/folder/anything"
  r = New Regex(urlPattern.Replace("/", ""), _
    RegexOptions.IgnoreCase Or RegexOptions.Multiline)
    html = r.Replace(html, "${attrib}=${delim1}" & _HtmlFile.UrlFolder & "/${url}${delim2}")

  '-- @import(/anything) to @import url(http://www.web.com/anything)
  r = New Regex(cssPattern, _
    RegexOptions.IgnoreCase Or RegexOptions.Multiline)
    html = r.Replace(html, "@import url(" & _HtmlFile.UrlRoot & "/${url})")

  '-- @import(anything) to @import url(http://www.web.com/folder/anything)
  r = New Regex(cssPattern.Replace("/", ""), _            
    RegexOptions.IgnoreCase Or RegexOptions.Multiline)
    html = r.Replace(html, "@import url(" & _HtmlFile.UrlFolder & "/${url})")

  Return html
End Function

Each regex is repeated because I have to resolve relative URLs starting with forward slashes to the webroot first--and then all remaining relative URLs to the current web folder.

One of the BCL team recently recommended pretty-printing regular expressions, eg, using whitespace to make regexes more readable with RegexOptions.IgnorePatternWhitespace. I agree completely. We do this all the time with SQL. I can think of a half-dozen tools that will block of SQL and pretty format it-- but I am not aware of any regex tools that offer this functionality. I guess I'll email the author of Regexbuddy and see what he has to say.

And here's an interesting bit of trivia: did you know that the ASP.NET page parser uses regular expressions?

BetaBrite LED Sign API completed

As I mentioned in Automated Continuous Integration and the BetaBrite LED Sign:

I'm currently working on some .NET classes that wrap a BetaBrite-specific subset of the Alpha Sign Communications Protocol. This requires serial communication via a 25 or 50 foot RS-232 serial to RJ-12 cable, so you'll need a physical PC with either a serial port or a USB-to-Serial adapter to get this working.

This is now ready for public consumption. I posted it as a new article on CodeProject, BetaBrite LED Sign API - A simple API for controlling a BetaBrite LED sign via RS-232 serial commands. Here's a sample:

Dim bb As New BetaBrite.Sign(1)
With bb
  .Open()
  .UseMemoryText("D"c, 128)
  .UseMemoryText("E"c, 128)
  .UseMemoryText("F"c, 128)
  .AllocateMemory()

  .SetText("D"c, _
    "<font=five><color=green>This is <font=seven>file D", _
    Transition.Rotate)
  .SetText("E"c, _
    "<font=five><color=yellow>This is <font=seven>file E", _
    Transition.WipeLeft)
  SetText("F"c, _
    "<font=five><color=red>time is <calltime>", _
    Transition.RollDown)

  .SetRunSequence("EDF")
  .Close()
End With

It really came out nicely. Now go get you some of that sweet, hot LED sign action!

Make Mine XCOPY

Steve "what the heck does furrygoat mean" Makofsky crystallized a lot of my thoughts in his recent rant on software installers. One of the biggest advantages of using the .NET framework is the way it enables XCopy deployments for the first time*. Installing a program by copying it to a folder was an utter fantasy in the VB6 world. In addition to the VB6 runtime, It took multiple first and/or third party OCX controls to do anything useful in a real app. And each of those OCX controls had their own dependencies.

XCopy is the gold standard for correctly architected .NET software projects. We should always be working toward that goal-- making our software deployment as simple as dropping a few files in a folder-- not putting in processes that make it easy for us to backslide into the bad old days. That's the whole point of Microsoft abandoning the registry, COM+, and all the other associated meta-dependencies that made our life hell for so many years. If the design of your .NET project precludes XCopy deployment, take a long, hard look at what you're doing. Don't just treat the symptoms of the disease.

Now, there are other reasons you may want an installer anyway. For one thing, I'm not sure users are ready for XCopy deployment. Do we really expect users to be able to unzip an archive to a folder? No, I'm not being sarcastic. There are other ameneties users expect in a software install, such as creating start menu icons, desktop icons, and integration with the add/remove programs section of Control Panel. Honestly, do you really think your program would end up in the Program Files folder if you left it up to the user? I'll tell you where it would go-- on the desktop. Along with every other piece of software and every other document.

The installer is a distraction, but a necessary one for users' sanity. If someone can propose another workable alternative, I'm all ears. I think the real lesson here is to keep your project dependencies to a minimum, and to absolutely master the dependencies you must have. Can your app run from a USB keychain? I realize that it's not practical for all real world projects, but it should always be one of the goals.

* Delphi has also offered this kind of dependency-free "install" for years. Which may be one reason why it's so incredibly popular amongst win32 utility authors.