programming and human factors
by Jeff Atwood
Scott Stanfield forwarded me a link to Roger Sessions' A Better Path to Enterprise Architecture yesterday. Even though it's got the snake-oil word "Enterprise" in the title, the article is surprisingly good.
I particularly liked the unusual analogy Roger chose to illustrate the difference between iterative and recursive approaches to software development. It starts with Air Force Colonel John Boyd researching a peculiar anomaly in the performance of 1950's era jet fighters:
Colonel John Boyd was interested not just in any dogfights, but specifically in dogfights between MiG-15s and F-86s. As an ex-pilot and accomplished aircraft designer, Boyd knew both planes very well. He knew the MiG-15 was a better aircraft than the F-86. The MiG-15 could climb faster than the F-86. The MiG-15 could turn faster than the F-86. The MiG-15 had better distance visibility.The F-86 had two points in its favor. First, it had better side visibility. While the MiG-15 pilot could see further in front, the F-86 pilot could see slightly more on the sides. Second, the F-86 had a hydraulic flight control. The MiG-15 had a manual flight control.
The standing assumption on the part of airline designers was that maneuverability was the key component of winning dogfights. Clearly, the MiG-15, with its faster turning and climbing ability, could outmaneuver the F-86.
There was just one problem with all this. Even though the MiG-15 was considered a superior aircraft by aircraft designers, the F-86 was favored by pilots. The reason it was favored was simple: in one-on-one dogfights with MiG-15s, the F-86 won nine times out of ten.
How can an inferior aircraft consistently win over a superior aircraft? Boyd, who was himself one of the best dogfighters in history, had a theory:
Boyd decided that the primary determinant to winning dogfights was not observing, orienting, planning, or acting better. The primary determinant to winning dogfights was observing, orienting, planning, and acting faster. In other words, how quickly one could iterate. Speed of iteration, Boyd suggested, beats quality of iteration.The next question Boyd asked is this: why would the F-86 iterate faster? The reason, he concluded, was something that nobody had thought was particularly important. It was the fact that the F-86 had a hydraulic flight stick whereas the MiG-15 had a manual flight stick.
Without hydraulics, it took slightly more physical energy to move the MiG-15 flight stick than it did the F-85 flight stick. Even though the MiG-15 would turn faster (or climb higher) once the stick was moved, the amount of energy it took to move the stick was greater for the MiG-15 pilot.
With each iteration, the MiG-15 pilot grew a little more fatigued than the F-86 pilot. And as he gets more fatigued, it took just a little bit longer to complete his OOPA loop. The MiG-15 pilot didn't lose because he got outfought. He lost because he got out-OOPAed.
This leads to Boyd's Law of Iteration: speed of iteration beats quality of iteration.
You'll find this same theme echoed throughout every discipline of modern software engineering:
When in doubt, iterate faster.
What a perfectly apt analogy. Once again, you make reading about engineering as interesting as actually engineering.
Tyler on February 8, 2007 5:04 AMSoftware testing is about failing early and often.
Very true.
Matthew on February 8, 2007 6:27 AMI've always heard it as an OODA loop (D for decide as opposed to P for planning):
http://en.wikipedia.org/wiki/OODA_Loop
PS - great post. Gave me a pause as I think about what my team needs to be doing better (quicker iteration).
Mike Duffy on February 8, 2007 7:11 AMNit pick. Looking a little too high tech there. F-86 cockpits were all analog dials.
http://www.militaryfactory.com/cockpits/f-86_cockpit.asp
However, great post. My experience has been that iteration speed often correlates with successful execution.
Blaine on February 9, 2007 3:12 AMI experience the same when writing programs. A quick compile-link-execute cycle helps a lot. 1 second is perfect, 5 is O.K., 10 is annoying, 1 minute and I forget what I wanted to run.
That's true. Once we had a problem with project. We switched to making deliveries and having meetings with customer two times a week instead of one time a week. It did help.
Iggi on February 9, 2007 4:51 AMHeh...
For a minute there, I thought you were going to tell me never to use a recursive algorithm, since an iterative version would be faster. :)
Randolpho on February 9, 2007 5:56 AMOne might question whether to write unit tests at all, so that iteration could be even faster. However on second thought, unit tests might be quickly justified by avoiding breaking things constantly at each iteration.
Might a moderated approach be useful: only write unit tests for things that are likely to break, or as "sanity checks," and leave the rest to chance, at least during the rapid iterative phases?
John on February 9, 2007 6:37 AMJohn: I'm wondering the same thing. I've got no unit tests on my current large-ish project.
I'm definitely breaking previously-working code with new changes, but it's usually been in an obvious, trivial-to-fix way. Right now, I'm thinking I'm spending a lot less effort on these quick fixes than I would have to on a full set of unit tests.
I'm considering writing a small number of unit tests for the most sensitive parts, but I really can't justify the cost of writing a full-blown test suite right now.
Eam on February 9, 2007 7:22 AMWrite them unit tests!!! And keep them up to date, you'll thank yourself in the future. If for no other reason then you finding your mistakes instead of the system testers. Face saving, it's not just for managers anymore ;-)
John Minadeo on February 9, 2007 7:43 AM...another important point this post brings up is the importance of ease of use. If it's hard to do (i.e. using a manual flight stick) then it detracts from the user experience (and in the jet fighter example, might get you killed!) It's easy to make things work, but oh so hard to make things easy to use.
Ken on February 9, 2007 8:00 AMHere though, the ease we're primarily talking about is ease of the development process.
Bringing back the flight metaphor again, it should be easy to maneuver fast repeatedly, *and* not get tired out from the repetitions.
I dare suggest again, that one way to help do this super-fast iteration could be to write fewer tests.
One question is, at what point does this come back to bite us? Some might answer, "right away," but I wonder if this is always necessarily true.
Also, to what extent do tests help to go even faster? A guess is that some tests do help go faster, and some don't. It's certainly impossible to accurately predict risk of failure for each and every piece of the software, but I wonder if teams have experimented with only writing tests for the riskier items.
John on February 9, 2007 8:48 AMI only worked at one organization (that wrote code for its own use) that embraced rapid iterative development. All the others failed.
They didn't fail on policy, but when time came to make the effort to support it. My one employer that succeeded did it on three pillars:
* Robust, realistic test environments. I can't move fast if I can't test fast.
* Automated code build and release framework, complete with regular, frequent production releases. If you're moving fast, everyone must know all the rules. Humans can only get involved in negotiating the release process in severe circumstances. At one time, my employer re-installed their entire worldwide production codebase every WEEK.
* Airtight version control and recordkeeping, and a standard, quick rollback process. If you can move forward quickly, you MUST be able to move backward quickly.
In a decent-sized organization, setting these facilities up takes resources and company commitment, and precious few have the vision to make that commitment. But the payoff for rapid iteration on a large scale is astounding to see.
John Pirie on February 9, 2007 9:14 AMAgain, something I already do to an extent. I focus on each feature and run the code almost obsessively, tweaking minor code fixes again and again and again. I always thought I was being a little ridiculous... but it always seemed a waste of time to try to make massive updates that would require massive bug-hunts...
Jae on February 9, 2007 10:31 AMI think you could sum up 80% of what Microsoft is doing wrong (eg, Vista release cycle) with the same advice: "Iterate Faster". How many versions of OSX has Apple released in the same time frame? five?
Of course, one wonders whether the market is capable of *absorbing* a new version of Windows every year or year and a half.
Jeff Atwood on February 9, 2007 11:04 AMThere was one point not mentioned here that is really important:
The Mig-15's had a high-T tail, while the Saber had a mid-rudder tail.
Why was this important you ask?
In a dive (approaching transonic flight), Mig-pilots would completely lose lateral control, while the 86's were fine.
Speed only help if it can be applied properly and control can be maintained. For instance, my first card was a '78 Oldsmobile 98 regency deluxed. It had a 350-V8 with a nice 4-barrel carbureter (sp?). It was very fast... as long as you were going straight...lol. At the first turn, that beauty would turn like the blimp that it was.
Nyquist says you need 2 sample to make a sine-wave... hahahaha! In practical applications, my experience is that you probably need at least 10. This backs this iterative theory.
-Doug
Doug on February 9, 2007 11:36 AMOOPA, loop-a, doop-it-y deet,
This comment's pointless, feel free to delete.
This is fun to read, but more than a little goofy. The whole precept is this: 1950's-era dogfighting is directly analogous to 21st-century software development.
Um, no. Oh, and did I mention no? Pilots in dogfights are under extreme amounts of physical and mental stress. Sure pilots are trained to handle this stress and they're professionals performing a job, but under the worse stress I've ever experienced as a developer, I've never seriously felt that failing to code a function correctly would cost me my life.
I get the point the author is trying to make, and in certain situations in business I think it's valid. But overall, it's too simplistic, too pat, and too subject to meaningless parody:
[removed]
Totally.
Master P. on February 10, 2007 1:50 AMJust like anything else in agile...
TOO MUCH OVERTHOUGHT.
That's right. I don't oppose what iterations try to accomplish, but having seen the same process fail as often as it succeeds, the overhead and pressures that this type of methodology brings it just too much.
Remember when 'killer apps' came out almost monthly? Yeah, me neither, cuz it was a long time ago. What did they do different? They designed, coded, played, tested, relaxed, coded, played, tested, relaxed, designed some more... all they had to worry about was delivering something thta worked in the end. They didn't have some manager pushing for a deadline that didn't mean anything except to a salesman who was trying to make a quota so an executive could get his goals met so his bonus was larger.
Don't put so much thought and anxiety into developing... junits are great for covering all the bases. Iterations just get in the way, because for EVERY creative process, time is always going to be the enemy. Go too fast, quality suffers, and THAT really does matter when it comes to software. And gees, don't start flaming MS for their quality... I don't see a single OS that is any better, and I've used them all. They're not worse, but not better either.
In regards to the article, the one big missing hole I see is that TRAINING was never mentioned. You can have the worst pilots in the world flying the best jets, if they have the worst training, they'll lose a lot more than win. Maybe TRAINING approaches should be just as heavily scrutinized in IT like it is in the Military... here endeth the rant.
Dan on February 10, 2007 10:55 AMNow two plus years of being on multiple scrum teams and projects I think the key of the original post:
observing, orienting, planning, and acting faster bringing a higher percentage of successes
is the key of agile. Sometimes we do 2 week iterations and sometimes a month, never like the old year long waterfall cycles. Task completion and feedback (builds and testing) come much more often and frequently which helps lead to successful outcomes. All of this goodness can also become a burden on the developers and project when supporting tools and processes end up being one of the main tasks for everyone.
Unit Tests have been a major topic in my current scrum with views across the spectrum. We have measured 30% to 50% longer coding task times for full as opposed to minimal unit tests being included. We have taken the view that this tool should be used when it looks useful in an ongoing sense and not require 100% coverage. This has allowed delivery of more features in the last 5 months and it remains to be seen if it bites us in the end (assuming the project is a success and has a long lifetime).
They should have asked some Russian pilots to fly that Mig. You know, people who actually know the airplane like their own ten fingers and have thousands of hours of flight experience with it. I bet the results would not be in F-86's favor.
DMB on February 11, 2007 11:44 AMI'm working on an agile process now. Iterations are 4-6 weeks. But we're on a multi-site team employing something akin to 60 developers and artists, many of them pretty senior, and we're building a fairly massive client-server-SDK-content development environment. I work on the client app and right now we're doing a lot of financial management stuff (the kind that has to be right).
The company I work for has done agile or similar sorts of things for a long time now. They're pretty good at it. We get some pretty decent ratings and seem to mostly operate successfully.
From the trenches, I can say agile is good in some ways (sloppiness gets called out sooner, there is more focus on frequent end-to-end testing which is great in a client-server environment, and you're always *trying* to take manageable bites). We couldn't do a two week iteration with our size of project distributed over multiple sites. Sometimes even the 4 week iterations, if we count the QC cycle at the end, are pretty darn short.
You succeed, as one sharp reader pointed out, if you can get everyone working in parallel, if the overhead created by agile frequent releases isn't too onerous, and if your utilization goes up. This is really what happens. You probably do work more, at least some of us senior folks always seem to be on a critical path and pushing to get stuff tight and solid for an iterative release. But the end result is we fairly frequently know where we're at - risk is identified much more aggressively and mitigation actions are taken early on. That alone makes agile worth the price of admission.
And the customers get confidence when you deliver regular releases. Our project schedule was fifteen to eighteen months. We do deliveries about every 5-6 weeks. The customer sees progress, doesn't get antsy that things are off the rails, and gets a chance to feedback regularly along the way. If you don't think any of that has value, then you haven't worked on many customer driven and funded projects. Also, as we almost always have some sort of release 'canned and ready', sales demos become much easier. And our customer can show it off to their customer and integration partners. That's a big plus for them too.
Sometimes you bite off more than you can chew for an iteration - some parts of a task just have a heavily serialized nature or are innately tricky. That's when agile hurts a bit because you *are* still pushing for your iteration endpoint and that means a harder push.
Our rule, and it is part of what keeps the customers coming back, is 'always hit your dates' and 'always deliver core promised content'. Sure, we sometimes slip on the edges, on 'good to have' or 'would have liked to' aspects, but we always deliver core content on time. This is in part due to the dedicated team, but in part due to management being on top of things - agile keeps everyone more focused and problems get brought up, mitigations decided on. You don't have time to screw around. You do, the iteration gets blown.
Agile works, but it does require a certain approach to project and risk management as well as a focus on regular builds for testing. Automation of testing and regular user driven smoke testing identify problems early and lead to better release quality. You don't try to get everything right, just the big things... and that helps a lot.
I've done the waterfall model and seen six month or year long projects go off the rails. There are fewer checkpoints if you wait nine months for the first production build. And if someone in the food chain is not being honest about where the development effort is at, you don't find out anywhere near soon enough. Agile forces a certain integrity on behalf of developers and middle management, a not inconsiderable management advantage.
kaladorn on February 12, 2007 7:29 AMCheck your history...the Russians were in some enemy sorties...Shot a few times, then bugged out. If they didn't have an easy kill they didn't hang around. Chickens.
Q'diddy on February 12, 2007 8:14 AMGreat post. I emailed you about it. Robert Greene just posted a similar piece you might enjoy. http://www.powerseductionandwar.com/archives/ooda_and_you.phtml
Ryan Holiday on February 25, 2007 11:08 AMIt's funny - before I saw this post, I had posted an entry at by blog on how Agile = OODA loop. I mailed the link off to Jeff, and he pointed me here.
Anyway - my post:
http://kg2v.blogspot.com/2008/04/agilexp-programming-and-ooda-loop.html
Interesting read. I find the analogy extremely fun to read but purely anecdotal. I think most would agree after reading this analogy that iterating faster accomplishes more work over time, if it works for F86s why not software right? If find it no different than saying working faster (acting faster) accomplishes more in a shorter period of time. I'm sorry but DUH!
The real question is why if a team does approximately the same amount of total work, how does the duration of an iteration effect the ultimate timeline when the same amount of total work needs to be accomplished? While most agree that an iteration can be too long, could it also be argued that an iteration is too short thus negatively impacting the duration of a project?
It goes without mention that iterative development incurs overhead. The overhead consists of planning each time, delivery/deployment each time, etc. The more interations you have the more overhead is incurred. So why is iterative development so hailed?
Enter parallel processing. Given a fixed amount of work to be completed the fastest (from a time perspective) way to complete that work is to have work performed in parallel as much as is reasonable. Reasonable meaning that doing things in parallel makes sense up to the point where the cost of overhead starts to outweight the benefit of parallel processing. Iterative development like parallel processing sacrifices efficiency for better utilization.
Ever worked on a puzzle? Yes those antiquated cardboard pieces that are put together to form a picture. If you have one person working on the puzzle they will do and X amount of work. If you have two people work on the same puzzle they will do X+O(verhead) amount of work, why? Simple, I've worked on a puzzle before with someone and have had to hunt for a piece that I had seen before. After a few moments of not finding it I ask the other person, have you seen that piece, it was right there a minute ago. They respond yes it's over here, I'm trying to see if it fits in this area. Though we're now doing more work, we also have twice as many people doing the work in parallel which shrinks the duration. Now extrapolate this to three, four or more people. Eventually depending on the size of the puzzle adding more people would actually increase the duration as we'd be bumping into one another.
The main difference with a software project/puzzle is that not everyone is performing the same function. There are architects, developers, testers, etc. No matter how hard you try you can't always work in parallel as some things are sequential/serial in nature. This is true even for the puzzle example but to a lesser degree. Regardless, the name of the game is to increase the parallelism (minimize the serialization) as much as possible even though you incur overhead in doing so. Given a fixed team size (processors) and fixed amount work to be completed how can a team increase the parallelism of their work? Simple, interations... Why have a tester sitting there doing nothing until all development is complete why not chunk the work up so the tester can test what is ready. Same type of examples can be drawn up for business anlaysts, business stakeholders, customers, etc. They key is add some overhead in the form of more frequent comunnications, hand-offs, deliverables, deployments, etc. also known as iterative development so more people/processors have greater utilization.
The end result is NOT
1. Working faster
2. Working less
3. Working more efficiently
Rather the end result is:
1. Working more, with less inactive time
2. Reduced duration (THE ULTIMATE GOAL)
3. Working less efficiently as you have introduced overhead.
So rather than Boyd's law of iteration (which is merely an anecdote) the real reason for this cause and effect has to do with Amdahl's Law: http://en.wikipedia.org/wiki/Amdahl's_Law of parallel processing. This can explain both having to few processors (long interations) and to many processors (extremely short interations) for a given amount of work. No doubt Amdahl's law and my pathetic puzzle analogy is more boring than the fighter plane analogy. Personally I'd rather associate with fighter aces but the analogy unfortunately doesn't explain what's really going on. It's very inspirational however and the masses prefer a good story. I tip my hat for the creative story. Well done!
Daron Cox on February 6, 2010 10:02 PM"It goes without mention that iterative development incurs overhead. The overhead consists of planning each time, delivery/deployment each time, etc. The more interations you have the more overhead is incurred. So why is iterative development so hailed?"
I think part of it is that at least some of that "overhead" are really things that should be done anyway: proper documentation, automation, testing, etc. The fact that you have to go faster exacerbates this problem and forces you to fix the leaky plumbing and peeling paint.
Aaron H on February 6, 2010 10:02 PMDo them iterations shorter and faster!! Right with you on that one. I learn that from my own experience - speed of iteration is one significant contributor to winning the game of making killer products.
But wouldn't you agree that its futile to expect a different result by doing the same things again and again? Isn't "inspect and adapt" another significant contributor too? I know its's implied and good developers/professionals will have this discipline in their blood. But may be its worth spelling out.
Is it fair to say that its important to "shorten the feedback loop" and "do better the next time"?
Nirmal Merchant on March 5, 2010 7:38 AMThe comments to this entry are closed.
Subscribe in a reader
Subscribe via email
|
|
Traffic Stats |