programming and human factors
by Jeff Atwood
In Lean Software Development: An Agile Toolkit, Mary and Tom Poppendieck describe a counter-intuitive technique for making better decisions:
Concurrent software development means starting development when only partial requirements are known and developing in short iterations that provide the feedback that causes the system to emerge. Concurrent development makes it possible to delay commitment until the last responsible moment, that is, the moment at which failing to make a decision eliminates an important alternative. If commitments are delayed beyond the last responsible moment, then decisions are made by default, which is generally not a good approach to making decisions.
Paradoxically, it's possible to make better decisions by not deciding. I'm a world class procrastinator, so what's to stop me from reading this as carte blanche? Why do today what I can put off until tomorrow?
Making decisions at the Last Responsible Moment isn't procrastination; it's inspired laziness. It's a solid, fundamental risk avoidance strategy. Decisions made too early in a project are hugely risky. Early decisions often result in work that has to be thrown away. Even worse, those early decisions can have crippling and unavoidable consequences for the entire future of the project.
Early in a project, you should make as few binding decisions as you can get away with. This doesn't mean you stop working, of course – you adapt to the highly variable nature of software development. Often, having the guts to say "I don't know" is your best decision. Immediately followed by "… but we're working on it."
Jeremy Miller participated in a TDD panel with Mary Poppendieck last year, and he logically connects the dots between the Last Responsible Moment and YAGNI:
The key is to make decisions as late as you can responsibly wait because that is the point at which you have the most information on which to base the decision. In software design it means you forgo creating generalized solutions or class structures until you know that they're justified or necessary.
I think there's a natural human tendency to build or buy things in anticipation of future needs, however unlikely. Isn't that the Boy Scout motto – Be Prepared?
As a former Scout myself, I think we should resist our natural tendency to prepare too far in advance. My workshop is chock full of unused tools I thought I might need. Why do I have this air compressor? When was the last time I used my wet/dry vac? Have I ever used that metric socket set? It's a complete waste of money and garage space. Plus all the time I spent agonizing over the selection of these tools I don't use. I've adopted the Last Responsible Moment approach for my workshop. I force myself to only buy tools that I've used before, or tools that I have a very specific need for on a project I'm about to start.
Be prepared. But for tomorrow, not next year. Deciding too late is dangerous, but deciding too early in the rapidly changing world of software development is arguably even more dangerous. Let the principle of Last Responsible Moment be your guide.
This is madness. Today's minds have been overly affected by short attention span music videos, video games, film edits that skip around every .4 seconds, etc.
People are no longer able to focus and hold a thought, hence their "requirements" never settle down, hence "agile development", extreme coding, etc.
I wonder what methodology the space shuttle folks use.
You shouldn't humor this stuff, it's a serious disease.
Steve, good comment. I never even had a project where the project specs were clear beforehand. at mostly have to actually make specs based on vague premonitions, impressions, meetings, etcetera,.. making specs is putting chaotic ideaes into a somewhat structured form so you can present them back and get an "aha-erlebnis" : "now that's what i wanted", or not. and then you can start over. and over until they finally agree. I usually end up deciding myself what they want.
Andy
Steve, I hope you're trolling.
I love TDD, not because I can't focus, but because I recognize the limits of my planning abilities and those of my clients, so I'm very interested in being able to change my software while retaining confidence in its correctness.
I really feel like most opponents of agile and XP are tilting at strawmen, or at very least considering the views of some impractical XP extremists as the accepted definitions.
Kig on October 19, 2006 2:30 AMIn general I'd agree that for some things, making the decision at the right time is key. However, that right time might not necessarily be "as late as possible".
A few years ago I met a guy who used to be president of one of the largest insurance company in my country (it had just been sold to a german company), and he said something that seemed to make a lot of sense for a lot scenarios: "It's better to make a decision early. That way, if you're wrong, you still have time to turn around and change your mind. If you leave it to the last possible moment, and you choose wrong, you're screwed".
Tomas Restrepo on October 19, 2006 2:36 AMSteve you can continue to rail against the perceived evils of the day and continue to long for the mythical lore of yesteryear, or you you can take the red pill and realize that no matter what you wish to be true, the reality is that people have trouble visualizing a system until it's complete.
Sure, you'd like to convince all your clients to sit down and really *think* about it, but you might as well try and convince everyone to stop fighting wars.
A better choice is perhaps to adapt to the reality. People have trouble making decisions on software systems early on. A software system today has more moving parts than any architectural structure build by humanity pre-computers.
Sure this doesn't work for the Space Shuttle, but we're not all writing space shuttle code.
Haacked on October 19, 2006 2:40 AMProcrastination always pays off now... :)
Ben Scheirman on October 19, 2006 2:47 AMI agree with the principle of Last Responsible Moment, but it's sure hard to get consensus on when that is.
In my last job, I'd try to get upper management to commit to a simple design decision and they'd often give me a lecture on how making a decision too early can paint you into a corner. So we wrote a lot of needless (and complex) if-else code knowing half of it would never be used, but that way we could deliver things that would run even if underlying decisions weren't made until the 11th hour or were reversed after going live.
Bob Dole was apparently famous for not making decisions until necessary -- those guys quoted him fairly often on this kind of stuff.
that right time might not necessarily be "as late as possible".
As late as *responsibly* possible. As Doug points out, nobody can really tell you when that is. There are a bunch of guidelines in the Poppendieck book. A large excerpt from that section of the book is available here:
http://www.ddj.com/dept/architect/184415014
--
In his essay "Delaying Commitment" (IEEE Software, May/June 1988), leading British computer scientist and professor Harold Thimbleby observes that the difference between amateurs and experts is that experts know how to delay commitments and conceal their errors for as long as possible, repairing flaws before they cause problems. Amateurs try to get everything right the first time, so overloading their problem-solving capacity that they end up committing early to wrong decisions.
--
--
I like your points about home tools. I'm getting to the point where I realize that my home time is different from work time. I like working on some things so I'll actually not by a time saving tool because what else would I have done with that time anyway? Watched TV?
Can this type of logic be applied to software development in rare cases? Or is faster always better? Kind of like the chef that used a knife instead of a food processor. He likes the feel of the knife and gets to really think about what he is doing. Maybe some thinking while coding isn't such a bad idea.
Well I pretty much disagree with 100% of this. I am known as a decisive and the feedback I get from my team is that this is a good thing. Better an incorrect but definite decision now than a delayed decision. Sometimes it makes sense to delay a decision to get more data points, but on the whole it is better to make amends than to make excuses.
Ole Eichhorn on October 19, 2006 3:43 AMLet's face it requirements don't stop changing. Business processes will continue to evolve in any organization, even ones that are leaders in the industry.
If you wait until all requirements are gathered and then develop then your processes have become stale or the system will not match the processes when it is finished being developed. Before I started my current job, they had spent 1 year collecting requirements, 6 months coding and when deployed the business stopped using the system because it didn't match their current processes.
The whole point about agile development is being agile and not rigid. You have to understand that the business is going to change, you can simply hope that it doesn't change so much that you scrap a lot of work.
Spend 3 months collecting large requirements and identifying any process that could derail your efforts. Come up with a core design that is not so rigid that you have to scrap it if something changes (you should have identified risks). Start the development effort while final requirements are being gathered (limited requirements or identifying work arounds due to technology limitations can help reduce scope and limit customer expectations).
A lot of people like to be proactively lazy, have all the requirements collected up front. Pick and chose what is possible, and what can be done in what timeframe. Split things out in to phases, start development, not deliver on time. Have to deal with scope creep and change management. Deliver a system that is out of date and that the customers don't use.
I would rather build a system based on what I know about the requirements. Change it as (if needed) the business changes. Deliver what I can, when I can. Keep my customers happy and keep the system up with the pace of the business.
There hasn't been too many times that I have scraped an entire design. Perhaps a small section that I simply comment out, because they typically will want it (they just don't know it yet).
Tim on October 19, 2006 4:11 AMback in the 1970s (before most of this blog neighborhood was born, alas) there was a "methodology" which preached not having finished specs and building from incomplete ones. as the "real" specs came to be, it was prophesied, the product would look like the spec. forget the name, of course, but I have these back of the brain stem memories that the worst fiascos came out of this.
as earlier threads have discussed, BDUF focuses the mind. this is a good thing.
One of my favorite quotes ever - I think it sums up the argument nicely:
"This is not to say that design is unnecessary. But after a certain point, design is just speculation."
Philip Chu
Yeah, well, maybe.
1) what do you want it to do?
2) understand the business as much as you can
3) draw a line in the sand for that which you can safely deliver pretty soon
4) build a system that is extensible, something that can be added on too fairly easily, because changes are coming (that is agile-ness)
5) charge for changes
"Sure, you'd like to convince all your clients to sit down and really *think* about it, but you might as well try and convince everyone to stop fighting wars."
Actually, most of the world has stopped fighting wars. If we could get Richard Cheney, Haliburton and their Texan puppet to stop, we might have nearly 99% of the world's population adhering to that. Note: This blog software won't let me put in Richard Cheney's common name - Dic.
Now, on a more software related note, getting people to think about what they want is sure a heck of a lot easier with analysis/design tools like activity diagrams, use cases and domain models than it is with code. So, if that makes me a BDUF proponent then I guess I am. Most customers do know what they want - where the failing is will usually be in the translation of their business model/processes into requirements. They are not used to thinking in terms of requirements and therefore cannot verify that the requirements say what they want. They are not used to critiquing requirements, therefore they read into them what they want to see. The requirements change when they do realize, based on what they have seen/heard since the initial writing, that what is being built has little to nothing to do with what they want. Sure, they'll make mistakes too but not as many as development teams will claim. Customers need something closer to their way of thinking, often a business-oriented way of thinking, in order to understand if their requirements are being properly recorded and met. Activity diagrams approximate business flows. Use cases (as per Alistair Cockburn) provide a more storylike approach to telling them what they will get. Domain models tell them what "things" in their business world are going to be available in the eventual software system. Try accomplishing that with code.
As for delaying decisions to the last responsible moment ... ... I went to a talk given by Mary Poppendieck on this topic and I chatted with her somewhat afterward. My interpretation of what she was saying is that delaying a decision to the last possible moment is NOT procrastination or laziness. Actually, I might argue it is hard work. You must not simply wash your hands of the decision until it is forced upon you, driving down the road to deadline oblivious to that decision. Rather you need to be constantly tracking and assessing the impact of the various decision options. How else would you know when the responsible moment has been reached or breached? You must consider in your design (and if you simply cannot think unless the debugger is doing it for you, then in your code too) so that whichever of the candidate decisions (or even new candidates that arise) eventually gets chosen due to cost/benefit, feature completeness or whatever metric, your system can rapidly (dare I say agilely) advance towards effecting that decision.
Alas, I suspect that as with XP, Agile, Test Driven Development and all the others, developers will bastardize what Mary is saying into "we don't have to think about that because we are delaying that decision to the last responsible moment". This will, of course, like all the other valid methodologies destroyed by the extreme interpretation by hackers, leave Mary's idea as just more roadkill along the path to more decades of inferior, expensive and failed software projects.
Tim Dudra on October 19, 2006 8:01 AM
Like all things, there is the finding of balance.
How do you come up with a good design when you are forced to wing it with barely defined goals?
How do you keep yourself from getting locked in to a dead end early on?
Etc....
In general, I believe there is an obsession with rapid change for changes' sake. It's the Red Bull Syndrome of our current culture. And it makes software that is buggy and sucks.
The best programs I've written had good design with well-defined specs, and a fair amount of time to work it out and test. Decisions were made early on, but again, with *good design* the code was not a rigid tower of uncooked angel hair pasta. Non-trivial change was discussed and dealt with, while larger changes had to be backed up with a good reason, not arbitrary whim.
The worst programs I've written were the ones where people could not make up there mind, changed with the daylight passing by the window, and suffered from endless last-minute decision making that chewed into rigorous testing. Yes, unit-testing, TDD, etc... Something will slip through as you are pressed for time. And the most fun of all... those long hours this half-ass approach produces.
I guess I sit in the middle of the argument for/against Agile and the like.
Yahootin on October 19, 2006 8:03 AMMy apologies - I have a semantico - kind of like a typo but where the glitch was in my brain ...
Where I typed "delaying a decision to the last possible moment is NOT procrastination" I should have typed "delaying a decision to the last responsible moment is NOT procrastination"
My god! Even I make mistakes! And those of you reading this blog are fortunate enough to witness the very first one!
I'm sorry, this is retarded. Decisions made early are risky, but so are the decisions made late in the project when a shitload of it is already designed and built. I'd rather see decisions made early, and only minor changes, tweaks, feature cuts, etc - later. That in my opinion is the only way to build something high quality, maintainable and coherent.
DMB on October 20, 2006 2:45 AMI agree with haacked, Steve.
And the space shuttle analogy is actualy a point against monolithic, "commit early" approach. When compared to the original requirement/promise, the shuttle program is wildly over budget, under performs by orders of magnitude, and suffers from a ~2% catastrophic failure rate. And you want to lable that as the way things should be done?
Let's bring some agile ideals to the shuttle program!
Stu
stuinzuri on October 20, 2006 3:09 AMIn order to decide when is the right time, you have to know all your options, and what the cost of each one will be (no need for great precision, but to have an idea).
This is the only way that allows you to really choose. Otherwise, it is not you making the decision.
Back to the tools thing: if I know I will need to cut some wood for this project, I know I will need a tool for that. I consider all the wood-cutting tools, put in balance the price, how easy is to find the tool (do you have to order it?), if I know how to use it, how long will I use it, what is the productivity with the various tools.
And then I can wait with the real decision until I find out more.
I can even do the decision in steps. In two days I will have to decide if I order that chain-saw from Japan. If I decide not to buy that, I can wait two more weeks for the other sub-decisions, because the other tools are available at Home Depot.
There are a couple of things that get solved pretty well if you just wait long enough. Of course you don't know in advance which of the things of you have in hand will qualify for this approach, so delaying to the last responsible moment increases the chance that things just get solved by themselves :)
Mrio Santos on October 20, 2006 6:31 AMsome good comments but let's face it. I hear complaints about analysts getting requirements and developers developing systems from these requirements that are obsolete before they are complete. That is almost always because no designer, or competent designer has identified from the requirements where potential variations to business process may occur so developers develop to the requirements. It's the designer that bridges the gap between the requirements and the developers. The designer knows that the developers favorite line is "I built what they asked for" and ensures that all ambiguity is removed before the code is written. This process doesn't mean teams can't be nimble (not agile).
Gene on October 20, 2006 6:46 AMSo this is basically "You Ain't Gonna Need It" methodology. Google for "YAGNI" to read more.
catfood on October 20, 2006 7:48 AMAn efficient decision is made with 60-70% of the information required for the decision. Any more or less is likely to cost time and/or money.
The better qualified you are for the task, the more likely you'll consistently make a decision in the lower 60% range.
Sam on October 20, 2006 8:30 AMIn the words of e: "Luck favors the prepared, dahhling".
But I see the point. And I have to ask (I just HAVE to ask) where's the balance? It's all very well and good to say "don't make a decision until, you know, you HAVE to make the decision." I worked a $21mil deathmartch where that was the standard mantra. Unfortunately, in the end no one made decisions (and they got made by default) and everything went to hell.
My point is that there will ALWAYS be rework. Always, always, always, always. [stamps foot for effect] It's a given. As Brooks said in No Silver Bullet, "build one to throw away". Agile, BDUF, some unholy bedding of the two, there's aways gonna be "waste".
Aaaaand...(someone with more dev experience out there correct me if I'm wrong..) what about the times when lack of upfront planning was the direct cause of major rework. (ie: "geez...if we'd thought about this just a little more..")
Seems to me that there's just gonna be rework. Peopleware (Lister/DeMarco) talks about managers taking the attitude with their folks of asking how many blind alleys they'd been down recently. And making sure that "none" is not the right answer.
Hindsight is always 20/20.
Some decisions need to be made up front. Can't continue without them. Some decisions can wait.
And it seems there's a damn few folks who can always tell the difference.
(yes, I use my compressor probably every other week, assembled my kids' bikes with the metric socket set, and the wet-dry vac...we'll, it's just a happy thing. The $500 dewalt planer, on the other hand, has gotten minimal wear...)
Hard work pays off later, laziness pays off now, right? :)
David House on October 20, 2006 9:24 AMThis is why I liked storyboarding so much, especially when I'd been tasked to develop GUIs. A blank wall and lots of postings, regularly updated, that people can move around and look at where we are, where we've been, and where we're going.
When we didn't know where we were going, someone unrelated to the project often wandered by and identified something we all missed. The meeting that followed the "gee whiz" moment usually came up with the responsible decision.
This is too uncool for the company where I work now. We have to have a wiki. Back to ad hoc development..
Margherite on October 20, 2006 9:35 AMI'm guessing you are pulling everyone's chain a bit, no? :)
But you should have called it "The Walking on the Edge" software development model, that sounds much better :)
I have a fair number of unused tools in the garage. Every now and then I toss ones that are clearly not getting used.
They are there because I do not want to have to add the overhead of buying a tool on top of a rush home project. If I have to go out and buy a level, a drill, or a screwdriver to hang a new TV, I have just pushed that project out from an easy evening to staying up late. I would rather get it done when I have the need.
Put simply, putting off a decision may be a good idea if you lack sufficient information, but may really bite you if you suddenly need to use the decision, without the time to make it.
Scott
Scott Ellsworth on October 20, 2006 10:12 AMI alway referred to this as the Bachelor Rule of Design - avoid commitment until the last possible moment. And lest you think that derogatory, I consider to be an optimal approach for most systems I design.
John on October 20, 2006 1:48 PMI'll add another consideration, that bears on design pardadigm. There are two types of software developer (those that divide...): those that build VAR systems, and those that don't. The fundamental difference is that those that do are implementing a superior (at least, that's what claim to sell) business method; the others are merely trying to placate some users, acquired somehow.
If you're selling a software manifestation of a superior business method, then *you* know what the software is going to do, BDUF is the sane approach. Who ya gonna ask, your evil twin? Spolsky operates from this position. The job is to convince folks to buy your superior business method; the software is along for the ride. SAP is the canonical example. They've managed to turn mission critical software for MegaCorps (what had been a differentiating vector for such organizations) into shrinkwrap. It is mind boggling.
I've worked with and for VARs aiming at the mid-size market, and it's a tough thing to do. SAP pulled a rabbit out of the hat. They've told Ford, et al, how to run their businesses. Hmm. Maybe it was all a EuroConspiracy???
If you're working for a MegaCorps satisfying captive users (and you're a captive developer, too); then you don't know the superior business methods. Your users will make damn sure you know that. It is a poisonous situation, generally. Been in that one, too.
buggyfunbunny on October 21, 2006 6:26 AMI agree with Steve, this is madness. As a software designer, I run through many decision trees as early as I can in the process so that I have many contingency plans as the requirements change throughout the SDLC. It’s called risk mitigation as developing software is risky business. Software development is like playing Beat the Clock - we don’t even have enough time to decide anything let alone waiting for the “last responsible moment.”
Margherite made a great comment on storyboarding. Heavily used in the creative process for making multi-million dollar movies to mitigate the risk before the shoot. We in the software world can learn a thing or two from the film industry.
Tools? What tools. The tools we have in the software world today are equivalent to the hammer and chisel from the 19th century. That’s one of the reasons why we have all of this process and methodology hoo haa, along with the embarrassing names (I hope history won’t laugh at us too much). If our tools enabled us to build software intensive systems in a month or two, you would never hear about “agile”.
Be prepared, I think is a great motto for the software world. A fellow software developer, who happens to be mountain climber, had a modified motto, be prepared or die. While no-one is going to die developing software (I hope not!), we could do a lot better job by being prepared to deal with the only constant in our world – change.
The difference between "as late as possible" and "as late as responsibly possible", from personal experience:
As late as responsibly possible:
Looking at the Haynes manual to find that I need to buy a set of small metric sockets, _before_ taking the driver's side-view mirror off of my car.
As late as possible:
Driving to the hardware store to pick up a set of small metric sockets, with my driver's side-view mirror dangling by a bundle of wires, because I read the manual _as I went along_ instead of looking it over before beginning.
Now that's freaking hilarious.
Jeff Atwood on October 23, 2006 11:19 AM"If you're working for a MegaCorps satisfying captive users (and you're a captive developer, too); then you don't know the superior business methods. Your users will make damn sure you know that. It is a poisonous situation, generally. Been in that one, too."
I'd been thinking this all along. "All" you have to do as a VAR is build, and sell, your "better mousetrap". If you're building for an in-house group, you have to build the exact mousetrap they want and at any given time they feel they have the right to change the mousetrap in any way they like because they "pay your wages".
Mike G on October 25, 2006 6:42 AMI both agree and disagree with ALL of this. I tend to be a decision maker, and to run the decision tree early and often. But decision in any project can also be like code you are working on in a svn tree: why sould you COMMIT before anyone else needs to use what you are working on, or it is mature?
That would be: decide early, but do not tell your decision before someone needs to know :)
Varkhan on October 27, 2006 7:53 AMthere are studious approaches to decision making in organisations
one is http://en.wikipedia.org/wiki/Six_Thinking_Hats
tronic on August 23, 2008 7:50 AMI'd note that the key word here is "responsible". While deferred decision-making is, in general, a good thing, I've seen a few too many Agile Addicts wait until the last *possible* moment (or later), as opposed to the last *responsible* moment.
This is a good philosophy as long as it's kept in check with a solid understanding of which decisions can't be put off, which requires an ability to actually analyze or at least estimate the potential impact of those decisions rather than simply assuming that they don't matter now. There's a fine line between deferred decision and indecision!
Aaron G on February 6, 2010 9:51 PMSam, where did you get the 60-70% number? That sounds about right.
My experience has been that chasing down that one last risk and getting someone to commit to that last vital decision is a complete waste of time. Let's face it, your customer can absolutely sign a contract in blood that says that the tire swing will not actually contain a tire, has pictures of it without a tire, shows use cases without a tire and then when it's built, their boss asks where the tire is and you get nailed to the wall. That's life in the biz, and you better find a way to stick that tire in somewhere.
Steve Jackson on February 6, 2010 9:51 PMThe comments to this entry are closed.
Subscribe in a reader
Subscribe via email
|
|
Traffic Stats |