Have You Met Your Dog, Patches?

The Gamasutra article Dirty Coding Tricks is a fantastic read. One part of it in particular rang true for me.

Consider the load of pain I found myself in when working on a conversion of a 3D third person shooter from the PC to the original PlayStation.

Now, the PS1 has no support for floating point numbers, so we were doing the conversion by basically recompiling the PC code and overloading all floats with fixed point. That actually worked fairly well, but were it fell apart was in collision detection.

The level geometry that was supplied to us worked reasonably well in the PC version of the game, but when converted to fixed point, all kinds of seams, T-Junctions and other problems were nudged into existence by the microscopic differences in values between fixed and floats. This problem would manifest itself by the main character (called "Damp") simply falling through those tiny holes, into the abyss below the level.

We patched the holes we found, tweaking the geometry until Damp no longer fell through. But then the game went into test at the publisher, and suddenly a flood of "falling off the world" bugs were delivered to us. Every day a fresh batch of locations were found with these little holes that Damp could slip through. We would fix that bit of geometry, then the next day they would send us ten more. This went on for several days. The publisher's test department employed one guy whose only job was to jump around the world for ten hours a day, looking for places he could fall through.

The problem here was that the geometry was bad. It was not tight and seamless geometry. It worked on the PC, but not on the PS1, where the fixed point math vastly magnified the problems. The ideal solution was to fix the geometry to make it seamless.

However, this was a vast task, impossible to do in the time available with our limited resources, so we were relying on the test department to find the problem areas for us.

There's never time to do it right, but there's always time to do it over. If this sounds familiar, you're not alone. I have at times found myself slipping into this pattern, continually patching the same code over and over. It's the kind of code that, when submitted for code review, you're tempted to self-deprecatingly introduce as have you met my dog, patches?

While "patchiness" isn't always a bad thing -- the venerable Apache HTTP Server is testament to that -- it's probably an exception.

the original FAQ on the Apache Server project's website, from 1996 to 2001, claimed that "The result after combining [the NCSA httpd patches] was a patchy server."

Reading the Gamasutra article shamed me into to attacking a section of extra-patchy Stack Overflow code. Code which I constantly had to tweak in various ongoing ways to get it to work right. But first, I had to get over my fear. Fear. That's what led to all the patching in the first place. It was obvious this was working code which had become crufty over time, but it was working. And it's one of the core bits of functionality in the site. In those circumstances, it's easy to mentally justify "just this small change, just this once" rather than the fundamental rewrite you really need.

When you finally realize you've spent the last six months telling yourself the same "just this small change, just this once" lie, then you've met your dog patches, too.

Now what are you going to do about that rascally scamp?

That Means It's Working

We may kid ourselves into thinking we're writing out of some sense of public good, or to create connections, or contribute some small bit of knowledge to the world. But let's face it. Most of us blog because we're raving egomaniacs. We not only love to hear ourselves talk, we're incredibly eager to hear other people talk about us, and the more the better. I think Dale Carnegie put it best.

Nothing is sweeter to someone's ears than their own name.

So it should come as no surprise that I have an automatic Google ego search set up for my name. Nothing special about that. It is considered neighborly to have your ear to the ground (within reason), and to politely comment on relevant articles mentioning you and your "stuff". All very standard, banal, ego-fluffing stuff.

But of all the mentions I've gotten, nobody has utterly nailed it in the way that Brian Gianforcaro has.

Right on. That's one thing you and I have in common, burny.

As a software developer, you are your own worst enemy. The sooner you realize that, the better off you'll be. In fact, that's the tipping point between amateurs and professionals in our industry: the professionals realize everything they write sucks.

So, to the extent that I can become a conduit for other programmers to have that same epiphany in their own programming careers, that means it's working.

The Only Truly Failed Project

Do you remember Microsoft Bob? If you do, you probably remember it as an intensely marketed but laughable failure -- what some call the "number one flop" at Microsoft.

There's no question that Microsoft Bob was nothing short of an unmitigated disaster. But that's the funny thing about failures -- they often lead to later successes. Take it from someone who lived and breathed the Bob project:

I was the one who sent Bill Gates email at the height of the positive Bob-mania that said we were likely to face a horrible backlash. Tech influentials had started telling me that they were going to bury Bob. They not only didn't like it, they were somehow angry that it had even been developed. It was personal.

And that's exactly what happened. Bob got killed. But first, it was ridiculed and stomped.

For Microsoft, it was a costly mistake. For the people who worked on it, Bob taught many lessons. Lessons that came into play for subsequent products that made a big impact, both at Microsoft and beyond.

How many people know that the lead developer for Bob 2.0 was also the co-founder of Valve and the development lead for Half-Life, which became an industry phenomenon, winning more than 50 Game of the Year awards and selling more than 10 million copies?

Or that Darrin Massena - development lead for Bob 1.0, most recently named Technical Innovator of the Year here in Washington State - and Valve co-founder Mike Harrington are the co-founders and partners behind Picnik - which is now the world's leading online photo editor, attracting almost 40 million visits a month and a million unique users a day.

And then, of course, I'd be remiss if I didn't mention that Melinda French -- Bill Gates' future wife -- managed the Microsoft Bob project at one point. Bob was the first Microsoft consumer project that Bill Gates personally had a hand in launching. Well, at least he got a wife out of it.

Yes, Bob was an obvious, undisputed and epic failure. We can point and laugh at Bob. But to me, Bob is less of a comic figure than a tragic one.

Unless you're an exceptionally lucky software developer, you've probably worked on more projects that failed than projects that succeeded. Failure is de rigeur in our industry. Odds are, you're working on a project that will fail right now. Oh sure, it may not seem like a failure yet. Maybe it'll fail in some completely unanticipated way. Heck, maybe your project will buck the odds and even succeed.

But I doubt it.

I own a boxed copy of Microsoft Bob. I keep it on my shelf to remind me that these kinds of relentless, inevitable failures aren't the crushing setbacks they often appear from the outside. On the contrary; I believe it's impossible to succeed without failing.

Charles Bosk, a sociologist at the University of Pennsylvania, once conducted a set of interviews with young doctors who had either resigned or been fired from neurosurgery-training programs, in an effort to figure out what separated the unsuccessful surgeons from their successful counterparts.

He concluded that, far more than technical skills or intelligence, what was necessary for success was the sort of attitude that Quest has -- a practical-minded obsession with the possibility and the consequences of failure. "When I interviewed the surgeons who were fired, I used to leave the interview shaking," Bosk said. "I would hear these horrible stories about what they did wrong, but the thing was that they didn't know that what they did was wrong. In my interviewing, I began to develop what I thought was an indicator of whether someone was going to be a good surgeon or not. It was a couple of simple questions: Have you ever made a mistake? And, if so, what was your worst mistake? The people who said, 'Gee, I haven't really had one,' or, 'I've had a couple of bad outcomes but they were due to things outside my control' -- invariably those were the worst candidates. And the residents who said, 'I make mistakes all the time. There was this horrible thing that happened just yesterday and here's what it was.' They were the best. They had the ability to rethink everything that they'd done and imagine how they might have done it differently."

I recently watched the documentary Tilt: The Battle to Save Pinball.

It's a gripping story of a pinball industry in crisis. In order to save it, the engineers at Williams -- the only remaining manufacturer of pinball machines in the United States -- were given a herculean task: invent a new form of pinball so compelling that it makes all previous pinball machines seem obsolete. I don't want to spoil the whole documentary, so I'll gloss over exactly how that happened, but astoundingly enough -- they succeeded.

And then were promptly laid off en masse, as Williams shut down its pinball operations.

Unlike Microsoft Bob, the Williams engineers built an almost revolutionary product that was both critically acclaimed and sold well -- but none of that mattered. It's sobering to watch the end reel of Tilt, as the engineers involved mournfully discuss the termination of their bold and seemingly successful project.

Everyone was in awe. They couldn't understand why it happened. Here we'd just done this thing that from all we could tell was a total success. Why would they do that?

We succeeded. Management gave us an impossible goal, and we sat there and we actually did what they thought we couldn't do.

You know, we didn't really win... we lost. I gave it everything I had. I think that those fifty guys that worked on it, they also passionately did everything that they could.

Sometimes, even when your project succeeds, you've failed. Due to forces entirely beyond your control. It's depressing, but it's reality.

The trailout isn't all doom and gloom. It also documents the ways in which these talented pinball engineers went on to practice their craft after being laid off. Most of them still work in the video game or pinball industry. Some freelance. Others formed their own companies. A few went on to work at Stern Pinball, which figured out how to make a small number of pinball machines and still turn a profit.

These two stories, these two projects -- the abject failure of Microsoft Bob, and the aborted success of Pinball 2000 -- have something in common beyond mere failure. All the engineers involved not only survived these failures, but often went on to greater success afterwards. Possibly as a direct result of their work on these "failures".

Failure is a wonderful teacher. But there's no need to seek out failure. It will find you. Whatever project you're working on, consider it an opportunity to learn and practice your craft. It's worth doing because, well, it's worth doing. The journey of the project should be its own reward, regardless of whatever happens to lie at the end of that journey.

The only truly failed project is the one where you didn't learn anything along the way.

All Programming is Web Programming

Michael Braude decries the popularity of web programming:

The reason most people want to program for the web is that they're not smart enough to do anything else. They don't understand compilers, concurrency, 3D or class inheritance. They haven't got a clue why I'd use an interface or an abstract class. They don't understand: virtual methods, pointers, references, garbage collection, finalizers, pass-by-reference vs. pass-by-value, virtual C++ destructors, or the differences between C# structs and classes. They also know nothing about process. Waterfall? Spiral? Agile? Forget it. They've never seen a requirements document, they've never written a design document, they've never drawn a UML diagram, and they haven't even heard of a sequence diagram.

But they do know a few things: they know how to throw an ASP.NET webpage together, send some (poorly done) SQL down into a database, fill a dataset, and render a grid control. This much they've figured out. And the chances are good it didn't take them long to figure it out.

So forgive me for being smarmy and offensive, but I have no interest in being a 'web guy'. And there are two reasons for this. First, it's not a challenging medium for me. And second, because the vast majority of Internet companies are filled with bad engineers - precisely because you don't need to know complicated things to be a web developer. As far as I'm concerned, the Internet is responsible for a collective dumbing down of our intelligence. You just don't have to be that smart to throw up a webpage.

I really hope everybody's wrong and everything doesn't "move to the web." Because if it does, one day I will either have to reluctantly join this boring movement, or I'll have to find another profession.

Let's put aside, for the moment, the absurd argument that web development is not challenging, and that it attracts sub-par software developers. Even if that was true, it's irrelevant.

I hate to have to be the one to break the bad news to Michael, but for an increasingly large percentage of users, the desktop application is already dead. Most desktop applications typical users need have been replaced by web applications for years now. And more are replaced every day, as web browsers evolve to become more robust, more capable, more powerful.

You hope everything doesn't "move to the web"? Wake the hell up! It's already happened!

Any student of computing history will tell you that the dominance of web applications is exactly what the principle of least power predicts:

Computer Science spent the last forty years making languages which were as powerful as possible. Nowadays we have to appreciate the reasons for picking not the most powerful solution but the least powerful. The less powerful the language, the more you can do with the data stored in that language. If you write it in a simple declarative from, anyone can write a program to analyze it. If, for example, a web page with weather data has RDF describing that data, a user can retrieve it as a table, perhaps average it, plot it, deduce things from it in combination with other information. At the other end of the scale is the weather information portrayed by the cunning Java applet. While this might allow a very cool user interface, it cannot be analyzed at all. The search engine finding the page will have no idea of what the data is or what it is about. The only way to find out what a Java applet means is to set it running in front of a person.

The web is the very embodiment of doing the stupidestsimplest thing that could possibly work. If that scares you -- if that's disturbing to you -- then I humbly submit that you have no business being a programmer.

Should all applications be web applications? Of course not. There will continue to be important exceptions and classes of software that have nothing to do with the web. But these are minority and specialty applications. Important niches, to be sure, but niches nonetheless.

If you want your software to be experienced by as many users as possible, there is absolutely no better route than a web app. The web is the most efficient, most pervasive, most immediate distribution network for software ever created. Any user with an internet connection and a browser, anywhere in the world, is two clicks away from interacting with the software you wrote. The audience and reach of even the crappiest web application is astonishing, and getting larger every day. That's why I coined Atwood's Law.

Atwood's Law: any application that can be written in JavaScript, will eventually be written in JavaScript.

Writing Photoshop, Word, or Excel in JavaScript makes zero engineering sense, but it's inevitable. It will happen. In fact, it's already happening. Just look around you.

As a software developer, I am happiest writing software that gets used. What's the point of all this craftsmanship if your software ends up locked away in a binary executable, which has to be purchased and licensed and shipped and downloaded and installed and maintained and upgraded? With all those old, traditional barriers between programmers and users, it's a wonder the software industry managed to exist at all. But in the brave new world of web applications, those limitations fall away. There are no boundaries. Software can be everywhere.

Web programming is far from perfect. It's downright kludgy. It's true that any J. Random Coder can plop out a terrible web application, and 99% of web applications are absolute crap. But this also means the truly brilliant programmers are now getting their code in front of hundreds, thousands, maybe even millions of users that they would have had absolutely no hope of reaching pre-web. There's nothing sadder, for my money, than code that dies unknown and unloved. Recasting software into web applications empowers programmers to get their software in front of someone, somewhere. Even if it sucks.

If the audience and craftsmanship argument isn't enough to convince you, consider the business angle.

You're doing a web app, right? This isn't the 1980s. Your crummy, half-assed web app will still be more successful than your competitor's most polished software application.

Pretty soon, all programming will be web programming. If you don't think that's a cause for celebration for the average working programmer, then maybe you should find another profession.

Are You a Digital Sharecropper?

Will Work for Praise: The Web's Free-Labor Economy describes how many of today's websites are built by the users themselves:

It's dawn at a Los Angeles apartment overlooking the Hollywood Hills. Laura Sweet, an advertising creative director in her early 40s, sits at a computer and begins to surf the Net. She searches intently, unearthing such bizarre treasures for sale as necklaces for trees and tattoo-covered pigs. As usual, she posts them on a shopping site called ThisNext.com. Asked why in the world she spends so many hours each week working for free, she answers: "It's a labor of love."

This raises some disturbing parallels. Are users being turned into digital sharecroppers?

MySpace, Facebook, and many other businesses have realized that they can give away the tools of production but maintain ownership over the resulting products. One of the fundamental economic characteristics of Web 2.0 is the distribution of production into the hands of the many and the concentration of the economic rewards into the hands of the few. It's a sharecropping system, but the sharecroppers are generally happy because their interest lies in self-expression or socializing, not in making money, and, besides, the economic value of each of their individual contributions is trivial.

It's only by aggregating those contributions on a massive scale - on a web scale - that the business becomes lucrative.

In essence, any website where user generated content is the website, that is also a for-profit business (not a non-profit organization, ala Wikipedia) -- is effectively turning their users into digital sharecroppers. Digital sharecroppers typically get nothing in return for the content they've provided, and often give up all rights to what they've created. At least a real world sharecropper would get to keep a percentage of the crops produced on the land.

The tone of the relationship between virtual land owner and so-called digital sharecroppers is critical. When crowdsourcing goes sour, there can be mass revolts.

Wikia is a for-profit corporation launched by several high level people involved with Wikipedia, such as co-founder Jimmy Wales. Wikipedia has no significant financial connection to Wikia. But an enormous publicity benefit accrues to Wikia due to Wikipedia's fame: $14m of venture capital has been invested in Wikia. Its business model is to have a "community" (writers who work for free) to build a wiki website about a topic, and then to sell advertising on those pages. In short, Wikia hosts sites in return for all the ad revenues.

At the start of June, Wikia's CEO announced that many changes would be made to the appearance of sites, mainly to have more advertising and for the ads to be more prominent. As Wikia's community development manager put it: "We have to change things in order to make Wikia financially stable. Unfortunately, Google ads in the footer pay pennies a click, and nobody clicks". He went on to explain that ads paying based on view count were needed. And that type of advertiser wants their ad to be displayed where viewers are sure to see it, such as within an article, near the top.

In reaction, various content creators made it clear they understood the needs of the company and had no objection to advertising per se. But putting ads inside content risked changing their material from articles into decorated billboards. The conflict between management and (unpaid) labour became acrimonious. There were declarations such as: "If Wikia does not resolve this situation to our satisfaction, then we will leave, taking our content, our communities' inward links, our established service marks and our fellow editors with us."

This is a topic that I've spent a great deal of time thinking about, because we happen to run a site chock full of user generated questions and answers. The last thing I want to do is exploit Stack Overflow users for corporate gain, even accidentally. That's horrible.

So if you spend a lot of time creating content on someone else's website -- whether it's Stack Overflow, or anywhere else on the internet -- I think you should be asking yourself some tough questions:

• What do you get out of the time and effort you've invested in this website? Personally? Professionally? Tangibly? Intangibly?
• Is your content attributed to you, or is it part of a communal pool?
• What rights do you have for the content you've contributed?
• Can your contributions be revoked, deleted, or permanently taken offline without your consent?
• Can you download or archive your contributions?
• Are you comfortable with the business model and goals of the website you're contributing to, and thus directly furthering?

There should always be a healthy, reciprocal relationship between you and any websites you're contributing to. I like to think that Stack Overflow gives back to the community as much as it absorbs -- both in the form of Creative Commons shared ownership of the underlying data, and the strong emphasis on showing off the individual skills and knowledge of your fellow programmers.

Ultimately, you have to decide which is more important -- building your own brand, or building the brand of the website you're contributing to? While these two concepts are not necessarily opposed, I strongly urge everyone reading this to err on the side of building your own brand whenever possible. Websites tend to come and go; the only sensible long term strategy is to invest in something that's guaranteed to be around for the rest of your life: you.

But I guess I'm biased.

COBOL: Everywhere and Nowhere

I'd like to talk to you about ducts. Wait a minute. Strike that. I meant COBOL. The Common Business Oriented Language is celebrating its fiftieth anniversary as the language that is everywhere and nowhere at once:

As a result, today COBOL is everywhere, yet is largely unheard of among the millions of people who interact with it on a daily basis. Its reach is so pervasive that it is almost unthinkable that the average person could go a day without it. Whether using an ATM, stopping at traffic lights or purchasing a product online, the vast majority of us will use COBOL in one form or another as part of our daily existence.

The statistics that surround COBOL attest to its huge influence upon the business world. There are over 220 billion lines of COBOL in existence, a figure which equates to around 80% of the world's actively used code. There are estimated to be over a million COBOL programmers in the world today. Most impressive perhaps, is that 200 times as many COBOL transactions take place each day than Google searches - a figure which puts the influence of Web 2.0 into stark perspective.

Every year, COBOL systems are responsible for transporting up to 72,000 shipping containers, caring for 60 million patients, processing 80% of point-of-sales transactions and connecting 500 million mobile phone users. COBOL manages our train timetables, air traffic control systems, holiday bookings and supermarket stock controls. And the list could go on.

I have a hard time reconciling this data point with the fact that I have never, in my entire so-called "professional" programming career, met anyone who was actively writing COBOL code. That probably says more about my isolation as a programmer than anything else, but still. I find the whole situation a bit perplexing. If these 220 billion lines of COBOL code are truly running out there somewhere, where are all the COBOL programmers? Did they write software systems so perfect, so bug-free, that all these billions of lines of code are somehow maintaining themselves without the need for legions and armies of COBOL programmers, decades later?

If so, that's a mighty impressive feat.

And if COBOL is so pervasive, why is it number one in this list of dead (or dying) computer skills compiled in 2007?

Y2k was like a second gold rush for Cobol programmers who were seeing dwindling need for their skills. But six-and-a-half years later, there's no savior in sight for this fading language. At the same time, while there's little curriculum coverage anymore at universities teaching computer science, "when you talk to practitioners, they'll say there are applications in thousands of organizations that have to be maintained," says Heikki Topi, chair of computer information services at Bentley College in Waltham, Mass., and a member of the education board for the Association for Computing Machinery.

When you dig in and read about some of these real world COBOL systems, you can get a glimpse of the sort of difficulties they're facing.

Read says Columbia Insurance's policy management and claims processing software is 20 years old and has 1 million lines of COBOL code with some 3,000 modifications layered on over the years. "Despite everyone pronouncing Cobol dead for a couple of decades, it's still around," he says. "We continue to enhance the base system. It's still green-screen, if you can believe that."

Read says getting younger workers to take on Cobol chores is a "real challenge, because that's not where technology is today." He simply tells them they must do some Cobol work, promising a switch to other things at the earliest opportunity.

Remember how the common language runtime of .NET promised rich support for a plethora of different languages -- but none of that ever mattered because everyone pretty much just uses C# anyway? Well, that CLR didn't go to waste, because it is possible to write code in COBOL.NET.

See for yourself.

private void button1_Click(object sender, System.EventArgs e)
{
    button1.Text = "Call COBOL";
}

That was C#, of course. Here's the COBOL.NET version of same:

METHOD-ID. button1_Click PRIVATE.
DATA DIVISION.
LINKAGE SECTION.
01 sender OBJECT REFERENCE CLASS-OBJECT.
01 e OBJECT REFERENCE CLASS-EVENTARGS.
PROCEDURE DIVISION USING BY VALUE sender e.
    SET PROP-TEXT OF button1 TO "Call COBOL".
END METHOD button1_Click.

Is there anything I could write here that the above code doesn't make abundantly clear?

COBOL: so vibrantly alive, yet … so very, very dead.

Software Pricing: Are We Doing It Wrong?

One of the side effects of using the iPhone App store so much is that it's started to fundamentally alter my perception of software pricing. So many excellent iPhone applications are either free, or no more than a few bucks at most. That's below the threshold of impulse purchase and squarely in no-brainer territory for anything decent that I happen to be interested in.

But applications that cost $5 or more? Outrageous! Highway robbery!

This is all very strange, as a guy who is used to spending at least $30 for software of any consequence whatsoever. I love supporting my fellow software developers with my wallet, and the iPhone App Store has never made that easier.

While there's an odd aspect of race to the bottom that I'm not sure is entirely healthy for the iPhone app ecosystem, the idea that software should be priced low enough to pass the average user's "why not" threshold is a powerful one.

What I think isn't well understood here is that low prices can be a force multiplier all out of proportion to the absolute reduction in price. Valve software has been aggressively experimenting in this area; consider the example of the game Left 4 Dead:

Valve co-founder Gabe Newell announced during a DICE keynote today that last weekend's half-price sale of Left 4 Dead resulted in a 3000% increase in sales of the game, posting overall sales (in dollar amount) that beat the title's original launch performance.

It's sobering to think that cutting the price in half, months later, made more money for Valve in total than launching the game at its original $49.95 price point. (And, incidentally, that's the price I paid for it. No worries, I got my fifty bucks worth of gameplay out of this excellent game months ago.)

The experiments didn't end there. Observe the utterly non-linear scale at work as the price of software is experimentally reduced even further on their Steam network:

The massive Steam holiday sale was also a big win for Valve and its partners. The following holiday sales data was released, showing the sales breakdown organized by price reduction:

• 10% sale = 35% increase in sales (real dollars, not units shipped)
• 25% sale = 245% increase in sales
• 50% sale = 320% increase in sales
• 75% sale = 1470% increase in sales

Note that these are total dollar sale amounts! Let's use some fake numbers to illustrate how dramatic the difference really is. Let's say our hypothetical game costs $40, and we sold 100 copies of it at that price.

If this pattern Valve documented holds true, and if my experience on the iPhone App store is any indication, we've been doing software pricing completely wrong. At least for digitally distributed software, anyway.

In particular, I've always felt that Microsoft has priced their operating system upgrades far, far too high -- and would have sold a ton more licenses if they had sold them at the "heck, why not?" level. For example, take a look at these upgrade options:

Putting aside schoolyard OS rivalries for a moment, which one of these would you be more likely to buy? I realize this isn't entirely a fair comparison, so if $29 seems as bonkers to you as an application for 99 cents -- which I'd argue is much less crazy than it sounds -- then fine. Say the Windows 7 upgrade price was a more rational $49, or $69. I'm sure the thought of that drives the Redmond consumer surplus capturing marketing weasels apoplectic. But the Valve data -- and my own gut intuition -- leads me to believe that they'd actually make more money if they priced their software at the "why not?" level.

I'm not saying these pricing rules should apply to every market and every type of software in the world. But for software sold in high volumes to a large audience, I believe they might. At the very least, if you sell software, you might consider experimenting with pricing, as Valve has. You could be pleasantly surprised.

I love buying software, and I know I buy a heck of a lot more of it when it's priced right. So why not?