Bayesian Kryptonite - spoofed email

I use POPFile bayesian filtering to keep email spam at bay. With a little training, this works amazingly well-- I'm at 99.8% accuracy, and that's with a little over a month of "training" precipitated by a recent server migration. But bayesian filtering has one big weakness that I'm seeing more and more: spoofed emails.

You know what I mean-- emails titled Your Account Has Been Violated from, ostensibly from service@paypal.com. The body is a direct cut and paste from a real PayPal email:

Security Center Advisory!

We recently noticed one or more attempts to log in to your PayPal account from a foreign IP address and we have reasons to belive that your account was hacked by a third party without your authorization. If you recently accessed your account while traveling, the unusual log in attempts may have been initiated by you.

If you are the rightful holder of the account you must click the link below and then complete all steps from the following page as we try to verify your identity.

Of course, the spoofer is desperately hoping you won't notice that the crazy URLs in their email ..

http://paypaldemo.com.previewyoursite.com/source/service/ema/helpextsourcepage/PaypalISAPIruhttp3A2F2Fmyebamcom3A802Fws2FeBayISAPIdll3FMyeBay26ssPageName3DH253AH253A/
http://ebay.doubleclick.net/clk;13012399;10693575;h?http://cardsavetransfer.com/cmdr_login/index.htm
http://ebay.doubleclick.net/clk;13012399;10693575;h?http://paypalcardstraznact.com/cmdr_login/index.htm

.. aren't actually pointing to paypal.com (or ebay.com), and you'll key in your account and password on their servers.

These spoof emails contain so-called "kryptonite" because they so closely mimic actual emails from PayPal with valid words and phrases. Bayesian filtering is useless against this type of spam; if the spammer knows what any email in your actual inbox looks like, he can construct one that will beat any Bayesian filter. This is a a strict requirement at the very heart of bayesian filtering itself; any knowledge of valid contents (eg, things that "get through") has to be strictly eliminated.

I usually just delete these emails from my inbox; what else can I do? One thing is for sure: popular web-based services can no longer communicate via email with their customers. That's like giving spoofers a free pass; once they have the "template" email they can copy and paste it into a spoof email that is almost guaranteed to get past bayesian filtering for users of that service.

eBay, for example, has almost given up altogether on email communication. You have to visit eBay.com and check your web-based "message center" to communicate with them. I can't say I blame them; what other choice do they have?

For Best Results, Forget the Bonus

The anonymous mini-microsoft blog has a fascinating entry on the pitfalls of Microsoft's curve rating system:

I totally accept that we need to have a rating system, especially to reward our kick-butt super-contributors who end up doing most the hard work around here. I have not, however, come to accept the bucketing rating system we employ, especially around the 3.0 review score. Now then, I have met dev managers who have reached acceptance with our system and with dolling out the 3.0s. In their presence, I experience a cult-like allure to these folks as they tell me how they are totally behind the peer relative 3.0 review rating and that's how we do business in managing our performers. They make it sound so calm and simple that I feel like if I could just drink up that Kool-Aid I, too, could not rip myself up inside over this.

I touched on this briefly in managing with trust. Performance metrics are useful, but only privately to the person who is being measured. The minute you make these performance metrics available to the manager as anything other than a sanitized average, you've turned them into a self-defeating cudgel to beat people into submission. I know, it seems like a logical idea-- measure and reward performance-- but there's tons of data supporting the harm this does. Peopleware goes into great detail on this; without belaboring the point, for best results, forget the bonus.

"Do this and you'll get that." These six words sum up the most popular way in which American business strives to improve performance in the workplace.

And it is very popular. At least three of four American corporations rely on some sort of incentive program. Piecework pay for factory workers, stock options for top executives, banquets and plaques for Employees of the Month, commissions for salespeople -- the variations go on and on. The average company now resembles a television game show: "Tell our employees about the fabulous prizes we have for them if productivity improves!"

Most of us, accustomed to similar tactics at home and school, take for granted that incentives in the workplace are successful. After all, such incentives are basically rewards, and rewards work, don't they?

The answer, surprisingly, is mostly no. While rewards are effective at producing temporary compliance, they are strikingly ineffective at producing lasting changes in attitudes or behavior. The news gets worse. About two dozen studies from the field of social psychology conclusively show that people who expect to receive a reward do not perform as well as those who expect nothing. This result, which holds for all sorts of rewards, people and tasks, is most dramatic when creativity is involved.

I suppose this only works for software development if you consider it a creative activity, but the arguments against the use of rewards are quite compelling:

• for every reward, there is the lack of a reward, an implicit punishment.
• rewards discourage cooperation and encourage competition.
• rewards are one-size-fits all approach that ignores causality.
• rewards deter risk.
• rewards undermine interest by focusing on the reward rather than the process itself.

Uncrippling Windows XP's IIS 5.1

Scott Mitchell says the best new ASP.NET feature in VS.NET 2005 is the integrated webserver. I agree. No more ditzing around with annoying IIS dependencies and install issues: aspnet_regiis, anyone? Tight coupling of VS.NET to IIS is also number three in K Scott Allen's worst of the .NET 1.x years. Good riddance IIS, hello Son of Cassini!

Unfortunately, we're still stuck with VS.NET 2003 in the meantime, and Windows XP's IIS 5.1 is thoroughly crippled out of the box. It allows only one root website, and a maximum of ten concurrent connections to that website. Microsoft really, really doesn't want us hosting slashdot.org on our XP Pro box. These limits are so aggressive that they can get in the way of legitimate localhost development. But there are workarounds.

1. Increase the 10 concurrent connection limit
   
   Are you getting 403.9 "Access Forbidden: Too many users are connected" errors on an XP Pro website? You're limited by default to 10 concurrent connections by design, but this can be increased. First, make sure your default windows script host is set to the console (cscript.exe) one.

   cscript //h:cscript
   

   Next, let's increase the connection limit to 40.

   C:\Inetpub\AdminScripts\adsutil set w3svc/MaxConnections 40
   

   Note that this is a hard-coded limit; it can't be increased any further unless you like patching windows system files. You can, however, make the IIS connection timeout more aggressive so connections don't last as long.

2. Run more than one root website
   
   IIS 5.1 only allows one root website. This is fine if your websites run under subfolders:

   http://localhost/MyWebsite1
   http://localhost/MyWebsite2
   

   But it's kind of a pain if your websites must run as root, or need to be tested when running as root:

   http://MyWebsite1/
   http://MyWebsite2/
   

   In that case, you'd have to edit your hosts file, and switch the default home directory for the default website. But there's a better way. You can hack up multiple web sites in IIS 5.1 via command line tricks, or you can use this nifty little GUI utility which automates that for you. It works great-- you'll even see multiple websites show up in the IIS manager. But bear in mind that, unlike the server versions of IIS, only one website can be active at any given time.

Comic Sans, the Font Of The Gods

You may be familiar with the font Comic Sans MS:

Over the last 5 years, my wife and I noticed that this annoying font is inordinately popular "in the wild" -- we've seen it in the strangest places. Enough so that it has become a running joke whenever we see it.

We may be weird, but we're not alone: there's a Ban Comic Sans website.

Like the tone of a spoken voice, the characteristics of a typeface convey meaning. The design of the typeface is, in itself, its voice. Often this voice speaks louder than the text itself. Thus when designing a "Do Not Enter" sign the use of a heavy-stroked, attention-commanding font such as Impact or Arial Black is appropriate. Typesetting such a message in Comic Sans would be ludicrous. Though this is sort of misuse is frequent, it is unjustified. Clearly, Comic Sans as a voice conveys silliness, childish naivete, irreverence, and is far too casual for such a purpose. It is analogous to showing up for a black tie event in a clown costume.

There's also a rather amusing history of comic sans presentation which I recommend-- now we know to blame Vincent Connare for this font. Vincent does his best to defend the font on his website:

Comic Sans was designed because when I was working at Microsoft I received a beta version of Microsoft Bob. It was a comic software package that had a dog called Rover at the beginning and he had a balloon with messages using Times New Roman.

Comic Sans was NOT designed as a typeface but as a solution to a problem with the often overlooked part of a computer program's interface, the typeface used to communicate the message [in a comic character's message balloon].

There was no intention to include the font in other applications other than those designed for children when I designed Comic Sans. The inspiration came at the shock of seeing Times New Roman used in an inappropriate way.

Proving, yet again, that anything which came from Microsoft Bob cannot be good. The misappropriation of Comic Sans is truly epidemic. At this point, we're bracing for the inevitable Comic Serif.

Keyboarding: Dell Minimalist

My quest for the perfect keyboard continues. Scott Stanfield turned me on to an interesting keyboard from, of all places, Dell. Scott recently tested out one of the Happy Hacker keyboards, but abandoned it in favor of this compact Dell model. Details are sketchy, but the model number is RT7D50, or SK-8115.

It's not ergo, but it has a standard layout, it's USB, and it has a pleasing, zen-like black minimalism. I believe this is Dell's standard keyboard for servers, where the keyboard is rarely necessary. There are a slew of these available on eBay for roughly $17 including shipping, or you can buy one direct from Dell for $18 plus shipping.

The Broken Window Theory

In a previous entry, I touched on the broken window theory. You might be familiar with the Pragmatic Progammers' take on this:

Don't leave "broken windows" (bad designs, wrong decisions, or poor code) unrepaired. Fix each one as soon as it is discovered. If there is insufficient time to fix it properly, then board it up. Perhaps you can comment out the offending code, or display a "Not Implemented" message, or substitute dummy data instead. Take some action to prevent further damage and to show that you're on top of the situation.

We've seen clean, functional systems deteriorate pretty quickly once windows start breaking. There are other factors that can contribute to software rot, and we'll touch on some of them elsewhere, but neglect accelerates the rot faster than any other factor.

That's excellent advice for programmers, but it's not the complete story.

The broken window theory is based on an Atlantic Monthly article published in 1982. It's worth reading the article to get a deeper understanding of the human factors driving the theory:

Second, at the community level, disorder and crime are usually inextricably linked, in a kind of developmental sequence. Social psychologists and police officers tend to agree that if a window in a building is broken and is left unrepaired, all the rest of the windows will soon be broken. This is as true in nice neighborhoods as in rundown ones. Window-breaking does not necessarily occur on a large scale because some areas are inhabited by determined window-breakers whereas others are populated by window-lovers; rather, one unrepaired broken window is a signal that no one cares, and so breaking more windows costs nothing. (It has always been fun.)

Philip Zimbardo, a Stanford psychologist, reported in 1969 on some experiments testing the broken-window theory. He arranged to have an automobile without license plates parked with its hood up on a street in the Bronx and a comparable automobile on a street in Palo Alto, California. The car in the Bronx was attacked by "vandals" within ten minutes of its "abandonment." The first to arrive were a family--father, mother, and young son--who removed the radiator and battery. Within twenty-four hours, virtually everything of value had been removed. Then random destruction began--windows were smashed, parts torn off, upholstery ripped. Children began to use the car as a playground. Most of the adult "vandals" were well-dressed, apparently clean-cut whites. The car in Palo Alto sat untouched for more than a week. Then Zimbardo smashed part of it with a sledgehammer. Soon, passersby were joining in. Within a few hours, the car had been turned upside down and utterly destroyed. Again, the "vandals" appeared to be primarily respectable whites.

Untended property becomes fair game for people out for fun or plunder and even for people who ordinarily would not dream of doing such things and who probably consider themselves law-abiding. Because of the nature of community life in the Bronx--its anonymity, the frequency with which cars are abandoned and things are stolen or broken, the past experience of "no one caring"--vandalism begins much more quickly than it does in staid Palo Alto, where people have come to believe that private possessions are cared for, and that mischievous behavior is costly. But vandalism can occur anywhere once communal barriers--the sense of mutual regard and the obligations of civility--are lowered by actions that seem to signal that "no one cares."

There's even an entire book on this subject. What's fascinating to me is that the mere perception of disorder-- even with seemingly irrelevant petty crimes like graffiti or minor vandalism -- precipitates a negative feedback loop that can result in total disorder:

We suggest that "untended" behavior also leads to the breakdown of community controls. A stable neighborhood of families who care for their homes, mind each other's children, and confidently frown on unwanted intruders can change, in a few years or even a few months, to an inhospitable and frightening jungle. A piece of property is abandoned, weeds grow up, a window is smashed. Adults stop scolding rowdy children; the children, emboldened, become more rowdy. Families move out, unattached adults move in. Teenagers gather in front of the corner store. The merchant asks them to move; they refuse. Fights occur. Litter accumulates. People start drinking in front of the grocery; in time, an inebriate slumps to the sidewalk and is allowed to sleep it off. Pedestrians are approached by panhandlers.

At this point it is not inevitable that serious crime will flourish or violent attacks on strangers will occur. But many residents will think that crime, especially violent crime, is on the rise, and they will modify their behavior accordingly. They will use the streets less often, and when on the streets will stay apart from their fellows, moving with averted eyes, silent lips, and hurried steps. "Don't get involved." For some residents, this growing atomization will matter little, because the neighborhood is not their "home" but "the place where they live." Their interests are elsewhere; they are cosmopolitans. But it will matter greatly to other people, whose lives derive meaning and satisfaction from local attachments rather than worldly involvement; for them, the neighborhood will cease to exist except for a few reliable friends whom they arrange to meet.

Programming is insanely detail oriented, and perhaps this is why: if you're not on top of the details, the perception is that things are out of control, and it's only a matter of time before your project spins out of control. Maybe we should be sweating the small stuff.

UI is Hard

Some users commenting on the poor pre-game user interface in EA's Battlefield 2:

Poster #1: They need to stop hiring angry little men and romantically spurned women to design user interfaces.

Poster #2: But doesn't that describe most programmers?

Poster #3: No, that describes all programmers.

It's funny because it's true. Not the romantically spurned part, mind you, but the accusation that most programmers are bad at designing user interfaces. That's partly because UI is hard:

GUI builders make GUI programming look easy. Nearly anybody can whip up a decent-looking GUI in no time at all using a GUI builder. Done.

It is much harder to whip up a quick and dirty EJB system, giving the impression that server-side coding is harder to do. A bad programmer will continue to struggle with EJB, but a good programmer will find ways to automate nearly every aspect of EJB. That's the secret of server-side programming: it is very well-defined and repetitive. Thus, it can be automated.

Take your favorite Model-Driven-Architecture (MDA) tool. They work best when generating server-side code, things like EJBs, database access code, and web services. They might be able to generate a rudimentary GUI, but a really GREAT GUI cannot be automated.

But programmers are partly to blame, too. Most programmers begin by thinking about the code instead of the user interface:

John almost hit on the most important point in all of this. No one else did. When you're working on end-user software, and it doesn't matter if you're working on a web app, adding a feature to an existing application, or working on a plug-in for some other application, you need to design the UI first.

This is hard for a couple of reasons. The first is that most programmers, particularly those who've been trained through University-level computer science courses, learned how to program by first writing code that was intended to be run via the command line (Terminal window for you Mac OS X users). As a consequence, we learned how to implement efficient algorithms for common computer science problems, but we never learned how to design a good UI.

The second problem is that the tools we use to create UI are often good tools for more simple usability issues, but tend to fall well short when it comes to designing UI for a more complex set of user scenarios. Forms designers are great when you're working within the problem domain that forms are intended to solve, but once you step outside those problem domains, the work gets much harder. Use a more flexible tool, like Xcode's nib tool and the Mac OS X HIView object, and you're going to have to write considerably more code just to manage the UI objects.

This is also known as UI First Development, but I can't find many other references.

High Dynamic Range Lighting

At the nVidia 7800 launch event today, one of the video rendering technology highlights was high dynamic range lighting. Almost all video cards in use today are limited to 32 bit color values-- that's 8 bits for red, green, and blue, with the "rounded" 8 bits typically thrown away*. 24 bits is enough to represent most of the colors the human eye can see. But those 8 bits per color also represent intensity. That means the brightest white is 255, 255, 255-- only 256 times brighter than the blackest black. This vastly underrepresents both the dynamic range of light in the real world (10¹² to 1) and the dynamic range of the human eyeball (1000 to 1).

That's why HDR lighting uses 64-bit values to represent color, which offers both 16 bits of color precision and 16 bits of brightness. As the chief scientist at nVidia points out in a recent Q&A session, don't underestimate the importance of good old black and white:

Reader Question: What is your opinion about some of the new graphical features that are being implemented in games? Some are quite beneficial to GPU performance, such as normal-map compression and virtual displacement mapping. But others are very costly to performance, specifically high-dynamic-range lighting. After seeing the extreme over-saturation of light with HDR in Far Cry (even on the lower levels of HDR) and the performance hit it took, I personally am not convinced that HDR is a method that should be pursued any longer. What are your opinions on this subject? -- cfee2000

David Kirk: I think that High Dynamic Range Lighting is going to be the single most significant change in the visual quality over the next couple of years. It's almost as big as shading.

The reason for this is that games without HDR look flat. They should, since they are only using a range of 256:1 in brightness -- a small fraction of what our eyes can see. Consequently, low-dynamic-range imagery looks flat and featureless, no highs, and no detail in the shadows, the lows. If you game using a DFP (LCD display), you probably can't tell the difference anyway, since most LCD displays only have 5 or 6 bits of brightness resolution -- an even narrower 32:1 or 64:1 range of brightness. On a CRT, you can see a lot more detail, and on the newer high-resolution displays, you can see not only the full 8 bits, but even more. There are new HDR displays that can display a full 16-bit dynamic range, and I can tell you that the difference is stunning. When these displays become more affordable in the next year or two, I don't know how we'll ever go back to the old way.

Here are some comparative screenshots from Far Cry, which supports HDR lighting as of the 1.3 patch:

High dynamic range lighting (64-bit color)

Standard lighting (32-bit color)

The effect is not always obvious, but it's just as significant as the move from 16-bit color to 32-bit color in my opinion. It's like going from dial-up to broadband; you don't really know what you've been missing until you've seen it in action.

Of course, TANSTAAFL: increasing the data payload from 32-bit to 64-bit hurts performance. Although the GeForce 6 supported HDR lighting, the new GeForce 7 is supposedly the first generation of video card that can deliver HDR without significantly impairing performance.

* The data has to be rounded to the nearest power of two for performance. I'm not entirely sure what is done with the "extra" bits in 32-bit and 64-bit color framebuffers.

World Zone Pricing

Cory Doctorow is releasing his new novel under a creative commons license.

As with my first and second novels, I've posted the entire text of this book online under a Creative Commons license that allows the unlimited, noncommercial redistribution of the text. You can send it around, paste it into a chat, beam it to a friend's PDA, or print out a chapter to hand out in the university common room. Like Woody Guthrie said, "Publish it. Write it. Sing it. Swing to it. Yodel it. We wrote it, that's all we wanted to do."

I immediately assumed he was releasing it under the standard Creative Commons license, like that recent documentary about BBS culture, but not so. He's releasing it under the Developing Nations License. If you live in a medium/low income country, the book is in the public domain. Otherwise, you pay. That reminded me of last year's blgo entry about software pricing. Microsoft charges US customers $299 for a copy of Microsoft Office, and Thai customers $38.

Isn't this the same pricing strategy that drug companies get clobbered for? Drugs cost more in the US, so the US ends up subsidizing a big part of the R&D bill. That's why seniors take buses to Mexico or Canada to buy their drugs; they don't want to be charged US rates. It's also known as "zone pricing" in the oil industry:

Such price variations may seem odd, but they are not unique to Anaheim. On any given day, in any major U.S. city, a single brand of gasoline will sell for a wide range of prices even when the cost to make and deliver the fuel is the same.

The primary culprit is zone pricing, a secret and pervasive oil company strategy to boost profits by charging dealers different amounts for fuel based on traffic volume, station amenities, nearby household incomes, the strength of competitors and other factors.

It's a controversial strategy, but the courts have thus far deemed it legal, and the Federal Trade Commission recently said the effect on consumers was ambiguous because some customers got hurt by higher prices while others benefited from lower ones.

Granted, the life preserving qualities of drugs makes zone pricing a little more questionable in that case. Nobody's going to die if they can't fill up their Hummer H2 with premium. Or afford a copy of Microsoft Train Simulator 2006.

Giving developing countries a break is clearly a good thing. But I wonder if these zone pricing strategies will continue to work in an era of increasing global communication. Why not buy drugs online from Canada? Or check a website to see where the cheapest gas is in your area? Any system that expressly prohibits goods from moving from zone to zone-- like DVD region coding-- doesn't work so well.

Why Anyone Can Succeed

In who needs talent when you have intensity, I proposed that success has very little to do with talent. This blog entry by Brad Wardell offers even more proof:

In 1992, OS/2 came out and I felt I could get a competitive advantage by pre-loading OS/2 onto the computers I sold. I became quite familiar with OS/2. In early 1993, I thought there would be demand for an OS/2 video game. The only problem was that I didn't know how to program. So I bought a book called "Teach yourself C in 21 days" and a book called "OS/2 Presentation Manager Programming". With those two books I programmed Galactic Civilizations. Anyone who ever played the game and also knows programming can verify that only the techniques in OS/2 PM programming are in that game. What that meant is that all the "graphics" were merely iconic windows, not real images because OS/2 PM programming didn't include chapters on how to do graphics programming and I couldn't afford any more books.

Even still, my plan once I graduated was to go work at some cool company. I sent my resume to Texas Instruments and tons of other big companies. No response. I later learned from a friend who worked at Ratheon at the time that these big companies won't look at resumes from minor colleges like WMU. He got in because he went to (wait for it) University of Michigan, the school I was accepted at but couldn't afford (my friend's parents paid his way through and drove a firebird).

So while failing to get a job, the little game I made did really well. Except that I didn't get paid any royalties. Our publisher ripped us off and because I was poor and couldn't afford a lawyer, I was screwed. If you bought a copy of Galactic Civilizations, that's cool but I never got a penny of it. But the game was such a success that I was able to use the good publicity to build Stardock up with other products while I waited to see if I could get a job anywhere.

Because I worked 3 jobs during college my gradepoint was only 2.6 so I was at a competitive disadavantage. But that time allowed me to keep working on Stardock stuff which led to Star Emperor (a GalCiv derivative I wrote for IBM) and Object Desktop for OS/2 which I teamed up with a fellow OS/2 fanatic Kurt Westerfeld to bring to market.

And the rest is history. By 24 I was a millionaire. And during the stock bubble, my company's valuation was ridiculously high (wish I could just stick with that number ) which fortunately coincided with my 10 year class reunion! So what's the excuse of able bodied people who are poor? I'm not particularly intelligent. I don't enjoy working any more than anyone else (just ask my mom). But I did what I had to do. And now I pay 6 figures in income taxes of which a large percentage of it goes to other people. If I can make it, why can't others? If you are able bodied, what's stopping you?

Even if you've never heard of Galactic Civilizations, you may know one of Brad's other products. The most notable are WindowBlinds and Political Machine.