Nice article, Jeff.

Interesting you should refer to "Postmortems". In the military we use what is known as an After Action Review (AAR) after any mission, training, whatever.

The format is pretty simple:

* What was supposed to happen?
* What happened?
* 3 Sustains (Things that went according to plan or even better)
* 3 Improves (Things that went wrong or hurt the mission)

You do this long enough, you begin to catch possible problems before they happen. I propose that, instead of doing a postmortem after the game fails, teams conduct one along every milestone.

For more on the Army AAR, see the following:
http://www.commonknowledge.org/userimages/aar_guidelines.pdf
http://www.armystudyguide.com/content/bm~doc/after-action-review-aar.ppt
http://www.myfirecommunity.net/documents/AAR_Guide_CALL.doc
http://www.signetconsulting.com/methods_stories/proven_methods/after_action_reviews.php

The last article is very relevant as it deals with how the AAR format may be used in corporate settings.

FYI, Gamasutra is essentially the online division of Game Developer. Gamasutra postmortems often are taken straight out of the magazine.

> Gamasutra is essentially the online division of Game Developer

Interesting-- there's a digital subscription offer for Game Developer where you can download 12 issues / year, or just one issue on a pay-as-you-go basis. I assume as PDFs. I didn't think they were related.

Thanks for the clarification.

I also like the Behind the Games series at Gamespot by Geoff Keighley (<a href="http://www.gamespot.com/features/btg/">link</a>). Reading the <a href="http://www.gamespot.com/features/btg-daikatana/">Daikatana</a> story was like reading a book entitled "How Not To Start a Start Up". The words "fail fast/small" seem to be conspicuously absent and it really highlights some of the management issues people should think about.

I too like to use the "X wrongs, X rights" format for postmortems (although, I'd guess any video game one is more interesting than every single one of mine), but I usually go with X=3. Each developer is to bring 3 of each to the postmortem meeting. We start the meeting with a short game of some sort, a favorite being Google Words (the moderator gives everybody a noun, they add an adjective to it and whoever comes up with the combination that renders the most search results on a Google search wins).

Based on the order in which the different members of the team finished in the game, that establishes the round robin order for people individually sharing one wrong. Go round the room one wrong at a time until everybody gets to share their 3. Get all the wrongs compiled (there's usually a grouping of them) and then go in the opposite order for the rights. From the compilation, talk about how to fix the top 3 wrongs and keep doing the top 3 rights.

That format lets people blow off a little steam with a purposefully lame game (yet it still sparks competitiveness) and gives everybody a chance to contribute equally to the postmortem. There's obviously a lot of ways to do this, but that's my favorite.

Pete Johnson
HP.com Chief Architect
Personal blog: http://nerdguru.net

> because game development is such a pressure cooker. It's incredibly
> challenging software engineering, with unclear goals (eg, "fun") under
> intense deadlines.

Total agree with this. Beside unclear goals, there is also another frustrating things about game development: there is no perfect goals, no
final goals.

Hey count0. :)

Indeed. Alot of goals in game development are very subjective.

Some factors are entirely quantifiable. Frame rate, memory, load time, network lag, when it will be finished, etc.

Other factors are more subjective. How fun is the game? Do you like how it looks? Do the controls make sense? Does it really feel like you are doing X? Those rely more on the personal tastes and experiences of each individual, and typically these are different from person to person.

One interesting thing I find is how different disciplines fit together in the team. Some domains are very subjective: designers and artists. While programming is typically more quantitative. Although there are always exceptions: Technical Designers, Technical Artists, and Gameplay & AI Programmers. Often times the subjective areas push the quantitative boundaries which can sometimes begin to affect framerate, memory, etc. But somehow it comes together.

"Gamasutra is essentially the online division of Game Developer"

I thought that was the case but I haven't logged into Gamasutra in so long I wasn't sure.

"there's a digital subscription offer for Game Developer where you can download 12 issues"

I got a subscription to the dead tree edition some years ago ... somehow. I've never had to pay for it since. As long as I re-up before it expires, I keep getting it for free. I think I said I was interested in game development. They occasionally send me emails inviting me to sign up a friend.

I've been reading the postmortems in Game Developer for years. Subscriptions are free to anyone in the game industry who is willing to answer "Yes" to the obvious "Answer yes to this question and you'll get a free subscription" question on the application. Apparently you have to know a subscriber to get that form though.

The trouble with these postmortems is that they are pretty much always the same. They go something like this:

What went right:
* Lots of iteration
* Experienced team
* Early focus on tools
* Lots of control over scope
* Took advantage of feedback from actual players
* Partnership with <company we like>

What went wrong:
* Not enough iteration
* Absolutely horrible scheduling
* Out of control scope
* Poor tools
* Inexperience
* Trouble hiring good people
* Partnership with <company we don't like>

Sometimes the specific headings are a little different, but usually the content comes around to one of these. The trouble is that there are so many disciplines at work on a game that the top 5 bullet points are never specific enough to actually benefit anyone. It's all well and good to say that you should have a well-scoped schedule with plenty of time for iteration and tools, but actually pulling that off is much more difficult. The postmortems never go into specifics on HOW because they are only 5 pages long.

A better way to get useful information is to attend one of the postmortem talks at the Game Developer's Conference. The best way is to hang out at a bar with one of the developers long enough that the drink loosens their tongues. Unfortunately none of those approaches scale, so I still find myself reading those Game Developer postmortems and wishing there were more details. :)

I have been enjoying reading the Gamasutra postmortems for a long time. There are few others that I found elsewhere.

"Devastro" - http://www.catnapgames.com/blog/159
"Democracy" - http://www.positech.co.uk/democracy/postmortem.html
"Unga Bunga" - http://gpwiki.org/index.php/Unga_Bunga_Postmortem

And some more here: http://www.gamedev.net/reference/list.asp?categoryid=64

Joe:
I agree, most gamedev postmortems read-alike. It's as if there's only 10 items on each of the "what went right/wrong" list for the writers to choose from. I still enjoy reading them, because if a postmortem is presented well you can really read between the lines and feel what it must have been like to work on that project, with that team, even though they did the same mistakes like so many other software developers.

Which makes me wonder: do other software projects fail or succeed for the same or comparable reasons? I don't remember reading any "run of the mill" software development postmortem but i suppose they do. I get that notion from Peopleware and other such books.

Simply put, people keep making the same mistakes. That might just be the learning curve of new people coming in to new positions. Other times, people just underestimate the challenges associated with a production, design or technical decision.

One important thing to realize is that what affected that project could also affect your own project. So people should either try to make decisions knowing peoples' past experiences or be prepared to deal with issues that may come up.

On the positive side, postmortems are a great way to find out what really worked well for another team and try to adopt those best practices.

It turns out the form to get Game Developer for free is online after all: (Thanks to somebody who responded when I reposted the comment above on my blog)
http://www.gdmag.com/freeyear/

I assume that scheduling, staffing, and controlling scope are problems that every software project faces. The non-game projects I've worked on haven't had the same need to spend many man-years on tools that you're never going to ship, so that one's unique to games. In general I bet the lessons transfer pretty well.

The SWAT3 postmortem link points to an Unreal portmortem. Whichever was fine, but I thought I'd mention it.

Yeah, I've made a few small-scale games, and I have to agree that the best learning is the one that happens after you made mistakes.

Most people don't think to research other's mistakes.

"Experience is what you get just after you need it."

I would love to do postmortems, but in my environment I am the only developer, and the projects never really end. I call the projects "tar-babies" because they stick to me forever.

Strongly agree, I teach game design and I'm asking students to do a post-mortem after their first iteration. It works well because they have a game do before spring break and separate class for refining their ideas right after the first game is due. Of course, we start with reviewing a whole bunch of industry postmortems for various games. I'm still trying to fight the standard patterns of mistakes (aka traps) we all have seen on projects.