Dealing With Bad Apples

Robert Miesen sent in this story of a project pathology:

I was part of a team writing an web-based job application and screening system (a job kiosk the customer called it) and my team and our customer signed on to implementing this job kiosk using Windows, Apache, PHP5, and the ZendFramework -- everyone except one of our team members, who I will refer to as "Joe". Joe kept advocating the use of JavaScript throughout the technology deliberation phase, even though the customer made it quite clear that he expected the vast majority of the job kiosk to be implemented using a server-side technology and all the validation should be done using server-side technology.

The fact that the customer signed off on this, however, did nothing to deter Joe from advocating JavaScript -- abrasively. Every time our project hit a bump in the road, Joe would go off on some tirade on how much easier our lives would be if we were only writing this job kiosk in JavaScript. Joe would constantly bicker about how we were all doing this all wrong because we weren't doing it in JavaScript, not even bother to learn the technologies we were actually using, and, whenever fellow teammates would try and gently bring him back into the fold (usually via email), Joe would just flame the poor guy. At the height of Joe's pro-JavaScript bigotry, he would regularly belt off comments like, "Well, if we had only done it in JavaScript," to such an extent that the team would have been better off if he had just quit (or was reassigned or fired.)

After reading this story, I had to resist the urge to lean forward, hand placed thoughtfully under my chin, brow furrowed, and ask -- have you tried JavaScript?

Robert thought this story was a cautionary tale about technology dependence, but I see something else: a problem team member, a classic bad apple.

I'm sure "Joe" had the best of intentions, but at the point where you're actively campaigning against the project, and working against your teammates -- you're a liability to the project.

The cost of problem personnel on a project is severe, as noted in Chapter 12 of McConnell's Rapid Development: Taming Wild Software Schedules.

If you tolerate even one developer whom the other developers think is a problem, you'll hurt the morale of the good developers. You are implying that not only do you expect your team members to give their all; you expect them to do it when their co-workers are working against them.

In a review of 32 management teams, Larson and LaFasto found that the most consistent and intense complaint from team members was that their team leaders were unwilling to confront and resolve problems associated with poor performance by individual team members. (Larson and LaFasto 1989). They report that, "more than any other single aspect of team leadership, members are disturbed by leaders who are unwilling to deal directly and effectively with self-serving or noncontributing team members." They go on to to say that this is a significant management blind spot because managers nearly always think their teams are running more smoothly than their team members do.

How do we identify problem personnel? It's not difficult as you might think. I had a friend of mine once describe someone on his team as -- and this is a direct quote -- "a cancer". At the point which you, or anyone else on your team, are using words like cancer to describe a teammate, you have a serious project pathology. You don't have to be friends with everyone on your team, although it certainly helps, but a level of basic personal and professional respect is mandatory for any team to function normally.

Steve outlines a few warning signs that you're dealing with a bad apple on your team:

1. They cover up their ignorance rather than trying to learn from their teammates. "I don't know how to explain my design; I just know that it works." or "My code is too complicated to test." (These are both actual quotes.)

2. They have an excessive desire for privacy. "I don't need anyone to review my code."

3. They are territorial. "No one else can fix the bugs in my code. I'm too busy to fix them right now, but I'll get to them next week."

4. They grumble about team decisions and continue to revisit old discussions long after the team has moved on. "I still think we ought to go back and change the design we were talking about last month. The one we picked isn't going to work."

5. Other team members all make wisecracks or complain about the same person regularly. Software developers often won't complain directly, so you have to ask if there's a problem when you hear many wisecracks.

6. They don't pitch in on team activities. On one project I worked on, two days before our first major deadline, a developer asked for the day off. The reason? He wanted to spend the day at a men's clothing sale in a nearby city -- a clear sign he hadn't integrated with the team.

Let me be quite clear on this point: if your team leader or manager isn't dealing with the bad apples on your project, she isn't doing her job.

You should never be afraid to remove -- or even fire -- people who do not have the best interests of the team at heart. You can develop skill, but you can't develop a positive attitude. The longer these disruptive personalities stick around on a project, the worse their effects get. They'll slowly spread poison throughout your project, in the form of code, relationships, and contacts.

Removing someone from a team is painful; it's not fun for anyone. But realizing you should have removed someone six months ago is far more painful.

The Ultimate Software Gold Plating

Some developers love to gold plate their software. There are various shades of .. er, gold, I guess, but it's usually considered wasteful to fritter away time gold plating old code in the face of new features that need to be implemented, or old bugs that could be squashed.

Developers are fascinated by new technology and are sometimes anxious to try out new features of their language or environment or to create their own implementation of a slick feature they saw in another product -- whether or not it's required in their product. The effort required to design, implement, test, document, and support features that are not required lengthens the schedule.

But gold plating your code isn't all bad. Perhaps the most remarkable tale of successful developer gold plating I've ever read is the one Blake Patterson outlines:

Not long ago I purchased a new-in-box Atari Jaguar, complete with Jeff Minter's psychedelic sequel to Tempest, Tempest 2000. It's an amazing game that's been ported to many other platforms, but the consensus is that none are as solid as the Jaguar original. Having played several of the ports, I'd have to agree.

An interesting thing about "the world's first 64-bit console" -- its controller was, as the Brits would say, fairly pants. It was large, sported a calculator-button array for game overlays (like the Intellivision controller), had no shoulder buttons, and featured only a D-pad for directional control. (ed: certainly one of the weirdest members of the game console controller family tree, to be sure)

As the arcade original is controlled with a rotary spinner knob, the D-pad falls rather short of providing ideal game control.

But, of course, being such a savvy chap, Jeff Minter realized this.

Jeff wrote in support for an analog rotary controller ... that did not exist. Neither Atari nor third party manufacturers produced such a controller in the Jaguar's heyday. Jeff, as I understand it, hacked his own together by wiring an Atari paddle controller into a Jaguar controller. In the years since the Jaguar's passing, a few small operations have offered modified Jaguar controllers with spinners wired into them for purchase.

Jeff Minter's an interesting historical figure in the computer gaming community, as the author of several 8-bit computer era game classics. I've talked about his long-standing interest in audio visualization here once before. He's still creating games today; his latest is the Xbox Live downloadable title Space Giraffe. Jeff has a blog that he updates fairly regularly.

Still, I'm amazed that Jeff added code to a commercially shipped console game to support a completely optional homebrew spinner controller of his own creation. That's the very definition of "not required". This code lied dormant in the game until a handful of enthusiasts, fourteen years later, cobbled together custom controllers to play the game as it was originally intended by the author.

If that isn't the ultimate case of gold plating your software, I don't know what is. My hat is off to you, Mr. Minter.

Maybe Normalizing Isn't Normal

One of the items we're struggling with now on Stack Overflow is how to maintain near-instantaneous performance levels in a relational database as the amount of data increases. More specifically, how to scale our tagging system. Traditional database design principles tell you that well-designed databases are always normalized, but I'm not so sure.

Dare Obasanjo had an excellent post When Not to Normalize your SQL Database wherein he helpfully provides a sample database schema for a generic social networking site. Here's what it would look like if we designed it in the accepted normalized fashion:

Normalization certainly delivers in terms of limiting duplication. Every entity is represented once, and only once -- so there's almost no risk of inconsistencies in the data. But this design also requires a whopping six joins to retrieve a single user's information.

select * from Users u
inner join UserPhoneNumbers upn
on u.user_id = upn.user_id
inner join UserScreenNames usn
on u.user_id = usn.user_id
inner join UserAffiliations ua
on u.user_id = ua.user_id
inner join Affiliations a
on a.affiliation_id = ua.affiliation_id
inner join UserWorkHistory uwh
on u.user_id = uwh.user_id
inner join Affiliations wa
on uwh.affiliation_id = wa.affiliation_id

(Update: this isn't intended as a real query; it's only here to visually illustrate the fact that you need six joins -- or six individual queries, if that's your cup of tea -- to get all the information back about the user.)

Those six joins aren't doing anything to help your system's performance, either. Full-blown normalization isn't merely difficult to understand and hard to work with -- it can also be quite slow.

As Dare points out, the obvious solution is to denormalize -- to collapse a lot of the data into a single Users table.

This works -- queries are now blindingly simple (select * from users), and probably blindingly fast, as well. But you'll have a bunch of gaping blank holes in your data, along with a slew of awkwardly named field arrays. And all those pesky data integrity problems the database used to enforce for you? Those are all your job now. Congratulations on your demotion!

Both solutions have their pros and cons. So let me put the question to you: which is better -- a normalized database, or a denormalized database?

Trick question! The answer is that it doesn't matter! Until you have millions and millions of rows of data, that is. Everything is fast for small n. Even a modest PC by today's standards -- let's say a dual-core box with 4 gigabytes of memory -- will give you near-identical performance in either case for anything but the very largest of databases. Assuming your team can write reasonably well-tuned queries, of course.

There's no shortage of fascinating database war stories from companies that made it big. I do worry that these war stories carry an implied tone of "I lost 200 pounds and so could you!"; please assume the tiny-asterisk disclaimer results may not be typical is in full effect while reading them. Here's a series that Tim O'Reilly compiled:

• Second Life
• Blogline and Memeorandum
• Flickr
• NASA World Wind
• Craigslist
• O'Reilly Research
• Google File System and BigTable
• Findory and Amazon
• MySQL

There's also the High Scalability blog, which has its own set of database war stories:

• YouTube
• PlentyOfFish
• Google
• MySpace
• Amazon
• Twitter

First, a reality check. It's partially an act of hubris to imagine your app as the next Flickr, YouTube, or Twitter. As Ted Dziuba so aptly said, scalability is not your problem, getting people to give a shit is. So when it comes to database design, do measure performance, but try to err heavily on the side of sane, simple design. Pick whatever database schema you feel is easiest to understand and work with on a daily basis. It doesn't have to be all or nothing as I've pictured above; you can partially denormalize where it makes sense to do so, and stay fully normalized in other areas where it doesn't.

Despite copious evidence that normalization rarely scales, I find that many software engineers will zealously hold on to total database normalization on principle alone, long after it has ceased to make sense.

When growing Cofax at Knight Ridder, we hit a nasty bump in the road after adding our 17th newspaper to the system. Performance wasn't what it used to be and there were times when services were unresponsive.

A project was started to resolve the issue, to look for 'the smoking gun'. The thought being that the database, being as well designed as it was, could not be of issue, even with our classic symptom being rapidly growing numbers of db connections right before a crash. So we concentrated on optimizing the application stack.

I disagreed and waged a number of arguments that it was our database that needed attention. We first needed to tune queries and indexes, and be willing to, if required, pre-calculate data upon writes and avoid joins by developing a set of denormalized tables. It was a hard pill for me to swallow since I was the original database designer. Turned out it was harder for everyone else! Consultants were called in. They declared the db design to be just right - that the problem must have been the application.

After two months of the team pushing numerous releases thought to resolve the issue, to no avail, we came back to my original arguments.

Pat Helland notes that people normalize because their professors told them to. I'm a bit more pragmatic; I think you should normalize when the data tells you to:

1. Normalization makes sense to your team.
2. Normalization provides better performance. (You're automatically measuring all the queries that flow through your software, right?)
3. Normalization prevents an onerous amount of duplication or avoids risk of synchronization problems that your problem domain or users are particularly sensitive to.
4. Normalization allows you to write simpler queries and code.

Never, never should you normalize a database out of some vague sense of duty to the ghosts of Boyce-Codd. Normalization is not magical fairy dust you sprinkle over your database to cure all ills; it often creates as many problems as it solves. Fear not the specter of denormalization. Duplicated data and synchronization problems are often overstated and relatively easy to work around with cron jobs. Disks and memory are cheap and getting cheaper every nanosecond. Measure performance on your system and decide for yourself what works, free of predispositions and bias.

As the old adage goes, normalize until it hurts, denormalize until it works.

Monkeypatching For Humans

Although I love strings, sometimes the String class can break your heart. For example, in C#, there is no String.Left() function. Fair enough; we can roll up our sleeves and write our own function lickety-split:

public static string Left(string s, int len)
{
    if (len == 0 || s.Length == 0)
        return "";
    else if (s.Length <= len)
        return s;
    else
        return s.Substring(0, len);
}

And call it like so:

var s = "Supercalifragilisticexpialidocious";
s = Left(s, 5);

Fairly painless, right?

But with the advent of C# 3.0, there's an even better way -- extension methods. With an extension method, we "extend" the String to add the missing function. The code is fairly similar; I'll highlight the changed parts in red.

public static string Left(this string s, int len)
{
    if (len == 0 || s.Length == 0)
        return "";
    else if (s.Length <= len)
        return s;
    else
        return s.Substring(0, len);
}

And now we can call it as if this very method existed on the String class as shipped:

var s = "Supercalifragilisticexpialidocious";
s = s.Left(5);

Pretty slick. It's difficult not to fall in love with extension methods, as they allow you to mold classes into exactly what you think they should be. This is fairly innocuous in C#, as extension methods only allow you to add new functionality to classes, not override, remove, or replace anything.

But imagine if you could.

Well, that's exactly how it is in other, more dynamic languages such as Javascript, Python, Perl, and Ruby. Something as prosaic as C# extensions is old hat to these folks. In those languages, you could redefine everything in the String class if you wanted to. This is commonly known in dynamic language circles as monkeypatching.

If the idea of monkeypatching scares you a little, it probably should. Can you imagine debugging code where the String class had subtly different behaviors from the String you've learned to use? Monkeypatching can be incredibly dangerous in the wrong hands, as Avdi Grimm notes:

Monkey patching is the new black [in the Ruby community]. It's what all the hip kids are doing. To the point that smart, experienced hackers reach for a monkey patch as their tool of first resort, even when a simpler, more traditional solution is possible.

I don't believe this situation to be sustainable. Where I work, we are already seeing subtle, difficult-to-debug problems crop up as the result of monkey patching in plugins. Patches interact in unpredictable, combinatoric ways. And by their nature, bugs caused by monkey patches are more difficult to track down than those introduced by more traditional classes and methods. As just one example: on one project, it was a known caveat that we could not rely on class inheritable attributes as provided by ActiveSupport. No one knew why. Every Model we wrote had to use awkward workarounds. Eventually we tracked it down in a plugin that generated admin consoles. It was overwriting Class.inherited(). It took us months to find this out.

This is just going to get worse if we don't do something about it. And the "something" is going to have to be a cultural shift, not a technical fix. I believe it is time for experienced Ruby programmers to wean ourselves off of monkey patching, and start demonstrating more robust techniques.

Try to imagine a world where every programmer you know is a wannabe language designer, bent on molding the language to their whims. When I close my eyes and imagine it, I have a vision of the apocalypse, a perfect, pitch-black storm of utterly incomprhensible, pathologically difficult to debug code.

I was just looking at random PHP plugin code the other day, and it was, frankly, crap. But that's because most code is crap. Including my own. It is, sadly, the statistical norm. That's why sites like The Daily WTF are guaranteed to have more material than they can possibly ever publish for the next millennia. (Note to self: invest in this website). I can only imagine what that PHP plugin code would have looked like, had its developer been granted the ability to redefine fundamental PHP keywords and classes at will. These are the sort of thoughts that drive me to drink Bawls. And that stuff is disgusting.

You might say that PHP, sans the fundamental dynamic language ability to monkeypatch, is just another crappy Blub language. But there's also a ton of incredibly useful PHP code out there. So it seems to me that the ability to monkeypatch doesn't stop people from producing a huge volume of useful code, even in a kind of.. horrible language. Some of it is even good!

While I acknowledge the power and utility of dynamic language monkeypatching, I know enough about programmers -- myself absolutely included -- to know the vast majority of us have absolutely no business whatsoever re-designing a programming language. There's a reason some of the most deeply respected computer scientists in the world end up as language designers.

Perhaps then, given the risks, monkeypatching should mean reaching for the meta-hammer as infrequently as humanly possible. This is a position that Avdi himself espouses in a followup comment:

I'm afraid a lot of people have missed the actual meat of my argument -- that dynamic extension of classes is currently overused in Ruby, in ways that are:

• Needless - another technique (such as a mixin, or locally extending individual objects) would have worked as well or better.
• Overcomplicated - the use of a monkey patch actually created more work for the author.
• Fragile - the solution is tightly bound to third-party internals, reducing the usefulness of the plugin or gem because it is prone to breakage.
• Excessively wide in scope - by hardcoding extensions to core classes, the author takes the choice to scope the change out of the plugin/gem user's hands, further limiting utility.

My point is that there are alternatives - often alternatives which are actually easier to implement and will make your plugin or gem more useful to the user.

While I enjoy the additive nature of C# extensions, even those are enough to make me a little nervous, as mild as they are. Full-blown dynamic language monkeypatching goes even further; it might even be the ultimate expression of programming power. Is there anything more pure and godlike than programming your own programming language?

But if wielding that power doesn't scare and humble you a little, too, then maybe you should leave the monkeypatching to the really smart monkeys.

iTunes is Anti-Web

Ever find yourself clicking on links to music or videos and getting blasted in the face with this delightful little number?

That's right -- links to any sort of music, TV shows, movies, podcasts, audiobooks or anything else available through Apple's iTunes store requires custom software to be installed on your computer before they will display thing one to you.

Is it so unreasonable to expect links in your browser to resolve to, oh, I don't know, web pages containing information about the thing you just clicked on? Is there anything more anti-web than demanding users install custom software to display information that could have just as easily been delivered through the browser?

So here's what we know:

• Apple wants us to install iTunes.
• iTunes is necessary to sync media with Apple's iPods and iPhones.
• iTunes is only available for Windows and OS X.
• iTunes is required to browse or buy anything from the iTunes Store.

That's all well and good for people who own iPods and iPhones -- and happen to be running Windows or OS X, I suppose.

But what about the rest of the world? Why lock them out with the ultimate login barrier? We might like to browse the iTunes Store, too. At the very least, I might want some basic information about the media I just clicked on. Right here in my browser where I already am. Information like what the heck it is, some artwork, maybe some audio clips, how much it costs -- sweet talk me. Make me want to buy it through the Apple Store. Dazzle me with your simplicity and ease of use. Beguile me with your wares!

Or, you could bludgeon me with the digital equivalent of a giant stop sign.

Hey, "it just works". Except when it doesn't.

I'm certainly able to click through to eminently purchaseable media on dozens of other places on the web using nothing more than my web browser. Let's imagine, for a moment, how utterly ridiculous it would be if I had to install the Amazon application to browse and purchase media from Amazon.

And yet this is exactly how the iTunes Store works. Or doesn't work, depending on your perspective.

I can understand requiring iTunes once you want to sync your media with Apple hardware devices -- although I would argue syncing should really be a fundamental, built in function of the operating system. But I'm not trying to sync anything! All I did was click on a link. It's downright user hostile to demand installation of a special application merely to browse the store, and it is most certainly against everything the web stands for and was built on.

The last "application" I can recall needing to install to get to things online was AOL.

And we all know how great that turned out.

For all the buzz about the Apple "it just works" mystique, the current iTunes Store design surely doesn't -- at least not the same way the rest of the web does. And I, for one, can't get behind that.