programming and human factors
by Jeff Atwood
I've been unhappy with every single piece of software I've ever released. Partly because, like many software developers, I'm a perfectionist. And then, there are inevitably … problems:
The list goes on and on. Reasons for failure on a software project are legion.
At the end of the development cycle, you end up with software that is a pale shadow of the shining, glorious monument to software engineering that you envisioned when you started.
It's tempting, at this point, to throw in the towel -- to add more time to the schedule so you can get it right before shipping your software. Because, after all, real developers ship.
I'm here to tell you that this is a mistake.
Yes, you did a ton of things wrong on this project. But you also did a ton of things wrong that you don't know about yet. And there's no other way to find out what those things are until you ship this version and get it in front of users and customers. I think Donald Rumsfeld put it best:
As we know,
There are known knowns.
There are things we know we know.
We also know
There are known unknowns.
That is to say
We know there are some things
We do not know.
But there are also unknown unknowns,
The ones we don't know
We don't know.
In the face of the inevitable end-of-project blues -- rife with compromises and totally unsatisfying quick fixes and partial soutions -- you could hunker down and lick your wounds. You could regroup and spend a few extra months fixing up this version before releasing it. You might even feel good about yourself for making the hard call to get the engineering right before unleashing yet another buggy, incomplete chunk of software on the world.
Unfortunately, this is an even bigger mistake than shipping a flawed version.
Instead of spending three months fixing up this version in a sterile, isolated lab, you could be spending that same three month period listening to feedback from real live, honest-to-god, annoyingdedicated users of your software. Not the software as you imagined it, and the users as you imagined them, but as they exist in the real world. You can turn around and use that directed, real world feedback to not only fix all the sucky parts of version 1, but spend your whole development budget more efficiently, predicated on hard usage data from your users.
Now, I'm not saying you should release crap. Believe me, we're all perfectionists here. But the real world can be a cruel, unforgiving place for us perfectionists. It's saner to let go and realize that when your software crashes on the rocky shore of the real world, disappointment is inevitable … but fixable! What's important isn't so much the initial state of the software -- in fact, some say if you aren't embarrassed by v1.0 you didn't release it early enough -- but what you do after releasing the software.
The velocity and responsiveness of your team to user feedback will set the tone for your software, far more than any single release ever could. That's what you need to get good at. Not the platonic ideal of shipping mythical, perfect software, but being responsive to your users, to your customers, and demonstrating that through the act of continually improving and refining your software based on their feedback. So to the extent that you're optimizing for near-perfect software releases, you're optimizing for the wrong thing.
There's no question that, for whatever time budget you have, you will end up with better software by releasing as early as practically possible, and then spending the rest of your time iterating rapidly based on real world feedback.
So trust me on this one: even if version 1 sucks, ship it anyway.
I second what Bado said.
There is a difference between the functionality of software and its internal architecture. I think what Jeff is talking about here is the internal architecture of the software you release may not be exactly to your liking but the functionality should still work correctly without crashing or serious bugs.
For example maybe there was no time to wire in a IoC container and poor mans constructor injection was used instead. Or certain
components were not factored exactly to our liking but after we ship V1, we get feedback that the feature supported by those components is not needed. If we had spent time refactoring those components it would have been a waste of time and would have delayed the release.
It's not just version 1 but recently it appears may major releases of software are riddled with bugs... some like Adobe's Premier Elements 8, which I had the misfortune of buying, are almost unusable.
The problem is that most software companies feel compelled to release a new version every year or two without even bothering to issue patches for the bugs in the last release.
cyclo on December 4, 2009 2:38 AMMartinC: "Yet again Jeff is thinking soley in terms of web software, desktop PC software and devices that can update simply, and forgets about the world of non-updateable embedded devices."
The notion that a short blog entry addresses all possible situations is a bit silly. One would hope that an intelligent reader would be able to determine reasons the advice might not be applicable to their situation.
Thanks for that great post Jeff. As always on coding horror, you see things spot on.
It's worth noting that web software makes it soooo much easier to release a crappy "beta" version without (too much) guilt, because you can deploy the changes and bug fixes to all your users very fast. It's not so easy with desktop-based computer, even with automatic update systems.
It's also worth mentioning that the most annoying users who care enough to take the time to complain about your product are worth A LOT, and we should cherish them. In fact, if nobody's complaining about your product, chances are you're falling into mediocrity.
"Version 1 should not suck, and you shouldn't foist sucky software on your users."
I think people are obsessing too much on the word "sucks".
Atwood's point is that to a perfectionist any version of any quality is going to "suck" unless it's "perfect" (which is impossible).
Clearly, the version can't suck from the customer's perspective.
njkayaker on December 4, 2009 3:17 AMBoy, I really hope that my company's competitors follow this advice to the letter.
erchuck on December 4, 2009 3:31 AMThat mostly what Agile is all about right?
tathagata on December 4, 2009 5:38 AMJust don't release software so poor your customers lose faith. That's the hard choice - you can't polish the software forever or you never get the first sales, but you can't release a product so riddled with quality issues that you never get an additional sale (or worse, returns.)
Scott on December 4, 2009 5:56 AMHi. So, I'm 100% cool with the idea of releasing v1 into the wild to ferret out the known and unknown knowns as a means of testing and refining your work.
But in the world of commercial software, who should pay for that v1? Should it be the developer - the person/company who will eventually release a fully-realized product and make lots of money on it? Or should it be the end users - the people who are paying to do the testing in the first place... HOPING they already HAVE a fully-realized product?
All in all, I'm ok with the end users paying something... but there should be some give and take with the developer of such a product to provide future versions to those early customers at a reduced (or no) fee at all. The problem with enterprise-level software today, however, is that the first customers don't get freebies... they get screwed.
How do we solve THIS problem?
Jeff Gordon on December 4, 2009 5:56 AMYup; ship crap, wait for marketing and sales to start pretending they're surprised that it's crap, then improve it.
If your PR is good enough to placate the clients you just pissed off, it actually works. If not, you're SOL.
Inaimathi on December 4, 2009 5:59 AMIsn't that how microsoft works?
Anonymous on December 4, 2009 6:06 AMHave public betas replaced embarrassing v1.0's?
Benjamin D. Bloom on December 4, 2009 6:06 AMYet again Jeff is thinking soley in terms of web software, desktop PC software and devices that can update simply, and forgets about the world of non-updateable embedded devices.
MartinC on December 4, 2009 6:08 AMAmen!
(After 5 years of alpha and just releasing the beta)
Louis Kessler on December 4, 2009 6:12 AMAnd to add to that, it's better to release something that has only some of the features you wanted to include, but is solid, rather than something that has all the bells and whistles, but crashes occasionally (all right, a lot).
It's way better to be able to say, "We can add that in 2.0" than to worry about how many 1.x releases you're going to need to make all the fixes that are required to 1.0.
Alex Beamish on December 4, 2009 6:15 AMI bet the guys at VersionOne (http://versionone.com/) hate the title of this post!
Rick Kierner on December 4, 2009 6:15 AMI find it amusing that the Captcha for posting this comment is "American pos Itzhak" That reads as American POS, It's a Hack... that about sums up most of the software I've worked on!
WA
Wayne on December 4, 2009 6:19 AM> MartinC
You have a point, but even in that case early *internal* releases are a good thing.
I think the real point is to get software out there (for varying definitions of "there") and get it hammered as soon as possible. The later any mistakes get found the more they cost.
Jim Cooper on December 4, 2009 6:21 AMThank You! That post just made my day. Cause we have just released v1.0 and the clients were disappointed by the lack of a fancy UI.
a.b. on December 4, 2009 6:31 AMI'm with Scott: Don't let perfect be the enemy of good, but don't confuse unusable with good.
I understand Jeff's point, but I think there is a quality threshold below which the software should be canceled, not foisted on users. On the other side are the quality levels you might be able to achieve only after getting thorough feedback. Finding a reasonable point in the middle is probably one of the artsy parts of development.
Bernard Dy on December 4, 2009 6:31 AMSo Jeff, you're basically saying it's okay to use your customers as beta-testers? Great! I can fire my entire QA department and save $$$!
No, I totally disagree with your "ship it no matter what" policy. Perhaps that policy has a tiny glimmer of hope when what you "ship" is free, but for paying customers, it's a policy that will destroy your company's reputation.
John W on December 4, 2009 6:35 AM"There's a big difference between the apps that Jeff is talking about (webby type apps and consumer software, kiddie software, social media software, ect) and real world software that has to work right. (e.g. financial trading platforms, IV syringe pumps, flight control software)
Jeff clearly has written software only for the former. That's nice. But in the grown-up world where software has to work and your company does not get a second chance"
I pretty much agree with this poster. Your free web forum software or your 99 cent iPhone flashlight can go out the door before they are fully baked and it won't be fatal.
If, on the other hand, your customers are business who are paying money for your app with the intention of using it to help them reduce their costs or increase their profits then that's another kettle of fish entirely.
Business will never really be 'dedicated users of your software' in the sense that Stack Overflow users are dedicated.
You might get two strikes with a business customer, but you might not. You certainly won't get three.
@anonymous no that's not the microsoft policy.
Microsoft uses a lot of statistics how many bugs could eventually be in a software at what level and how much bugs a customer wouldn't take care before switching to something else. If your customer thinks you v1 is totally crap he won't buy a second version.
If microsoft have reached the level of quality, alphas and betas are released. first to msdn /technet subscribers who know what prolems an alpha/beta can make. At os public betas follow. And if they are succesful (note, you are at this point always far over the first level where a customer would buy the next version), it's released. And at this point still the next or next 2 versions are in development.
sunfire on December 4, 2009 6:56 AMThere's a big difference between the apps that Jeff is talking about (webby type apps and consumer software, kiddie software, social media software, ect) and real world software that has to work right. (e.g. financial trading platforms, IV syringe pumps, flight control software)
Jeff clearly has written software only for the former. That's nice. But in the grown-up world where software has to work and your company does not get a second chance, Jeff's advice is tantamount to negligence and incompetence.
I can cite quite a number of examples of shipped software that failed -and failed big. The results were death or millions of dollars lost. That's a far cry from poor software causing an outage of an app that wastes your time 140 characters at a time.
Be careful of the advice you follow.
tim on December 4, 2009 7:00 AM@tim
Your point is correct. However, I think such mission-critical systems are the exception rather than the rule. I would says something near 90% of programs can fail only causing minor annoyance rather than fiscal disaster.
I think anybody (programming or not) would modify their strategy as soon as they realized that this is the sort of program that cannot fail.
Practicality on December 4, 2009 7:06 AMVoltaire quote (sort of) - Perfect is the enemy of the good enough.
Sam Beskur on December 4, 2009 7:06 AMJeff, let me modify your statement:
Yes, you did a ton of things wrong on this project. But you did a ton of things exactly right, too.
And you should ship them, so everyone can benefit. Or more commercially: you should ship them, before another one gets the credits!
hinek on December 4, 2009 7:07 AMYou should always be willing to put your reputation behind version 1.0. If you say software is good enough for widespread use, it should be. This means that you don't package up your first efforts and call them 1.0.
Release beta-test software instead (either to the public or a limited number of people). Don't ask people to pay for stuff that's not actually useful as is.
And make sure the beta version does something good, even if it's buggy about it. You want your beta testers to look at it and think "This will be cool when the bugs are out" rather than "Why would anybody use this even if bug-free and polished"?
David on December 4, 2009 7:14 AMShort iteration cycles are good but not if the result is that you foist upon your customer a heaping pile of crap early and often. This has to be coupled with some other things - knowing the rules (what we trying to solve here anyway? what are the domain requirements? ), having good testing practices in place, having good team communications (doing daily stand-ups?), realistic expectations, honesty, and customer interaction and input.
If the customer *knows* he's going to be beta-testing the product and that his feedback is appreciated and will help drive the product toward a better solution and is OK with that, great. As long as the customer's expectations are known and everyone is on the same page.
At the end of the day, you want to know early when you are having issues and resolve them early. You want to keep the customer in the loop as much as possible and if you are going to provide incremental releases and the customer is OK with that, leverage the power of the feedback loop they'll provide. In some circumstances you can't drop a 'beta' on them because they can't afford to be running software that is buggy. There may be SERIOUS implications due to a bug. In that case, you don't release early, you release when it is well tested and trusted to work well.
It'll depend upon your end user and the industry as to whether you can ship a sucky version 1. You deliver a sucky version one of a medical app or a military guidance system and people could die due to errors in the code. You deliver a sucky version 1 of a social website and all that is probably in jeopardy from buggy code is your rep.
Just my opinion.
Matthew Cuba on December 4, 2009 7:15 AMOne of the projects I worked on slipped because the project lead decided that in the state it was in, it was very unlikely to actually be adopted at all. (Of course, technically it was a V2, where V1 had suffered from the product not having any customers.)
Honestly, I think Jeff's advice here is poor. It's assuming that you can't differentiate between "this is not ideal" and "this does not solve the problem it says it does", and if at the end of a project *someone* can't tell the difference between the two, you have problems than cannot be solved by slipping the schedule.
Merus on December 4, 2009 7:19 AM@Practicality The problem is that Jeff sees the world only though his own experiences and then spouts off this nonsense. As a "thought leader" (I had to really choke down a laugh there) he should be more careful about the nonsense he publishes - since he apparently has a popular blog about "software development". Kids in high school or college might take his advice as gospel. Since Jeff helped create a very successful site then we should emulate him and do like he says...
I'd also like to point out that this sentiment is nothing new - this is a repeat of the same stuff people have been saying for a few years now - "ship it and iterate."
Well, that's not exactly the whole picture and Jeff should take more responsibility for the crap he puts out there in the internets.
Of the many jobs I've had 4 had to do with things like real-time financial data, trading platforms and real-time data involving naval ship positioning, threat detection and messaging.
For me, the ratio is more like 50/50, not 90/10.
My working habits would make me a bad fit for companies like twitter and their ilk, likewise people like Jeff would not do very well at companies like Wombat Financial or Raytheon.
tim on December 4, 2009 7:20 AMWhat if v2.0 sucks and v3.0 sucks. I usually know/think v1.0 sucks, and sucks hard...
Patrick on December 4, 2009 7:22 AMEssentially release or not depends on SW usefulness in it's current state. If it solves problem cheaper, in shorter time or more comfortably than number of pulled hairs, customers scared away, houses burn, and planes crashed then yes - release. Problem for safety critical applications is that usefulness threshold is exactly zero crashed planes :o)
MaR on December 4, 2009 7:36 AMMy vision : release first version as version 0.9beta
Then.. once you have that feedback.. you can release the "official: verision 1 ;-)
OLi on December 4, 2009 7:42 AMTesting
Having a devoted testing team, while they haven't saved us completely (nobody can perfectly mimic a user), has helped reduce the amount of crap we ship.
Robb on December 4, 2009 7:45 AMThe other thing to remember is to make sure you have an audience.
About a year ago, I released an early build of one of my programs as a tech preview and a way of gaining feedback about any bugs I didn't know about. I released a second, less buggy version near the beginning of this year and am preparing to release a third.
To date, the only piece of feedback I've gotten was some guy on Wakoopa saying "Sucks!" I think one of the problems is the program is way too niche.
(The other may be because I'm really bad at figuring out what "good enough" is, and cut the rope at the wrong length. ^^ Oh well!)
MarkKB on December 4, 2009 7:51 AMAgreed. Ship it already. I am currently in the middle of project that is overdue and needs to ship ASAP. It's not nearly ready, but the moment to hold my nose and ship is rapidly approaching.
Joseph Crotty on December 4, 2009 7:55 AMWhat a perfect article, I just got finished with a huge project with a tiny timeline, and we lost a developer to a pregnancy right in the middle. And now I am bug fixing all month long with compromise after compromise.
Thanks for the perspective.
Joe on December 4, 2009 7:55 AMJust slap a Beta tag on it... that's what Google does :)
Kris on December 4, 2009 7:57 AM@Simon
Like many other posters you are being a little obtuse and missing the point that "releasing" your software to a large team of testers and a simulated office environment fits exactly with what Jeff is saying. Here your "customers" for the first few (many?) releases are the testers, whose feedback you then use to iterate on the product. You then release "Version 2" to the real customers.
Agreed, for sure.
The only exception is when you have something that would generate tremendous forward-compatibility issues or that can't really be fixed in the future unless you fix it now.
-Max
Max Kanat-Alexander on December 4, 2009 8:04 AMWhat happened to the mentality of "Ship half a product, not a half-assed product?"
Jeffrey on December 4, 2009 8:05 AMI think that the sentiment should be "Version 0.5 Sucks, But Beta Test It Anyway". By the time you get to V1, you should *already* have gotten it out in front of real users in the real environment.
Version 1 should not suck, and you shouldn't foist sucky software on your users.
Eric on December 4, 2009 8:09 AMHow does this apply to the worlds of
* "you have to ship this on Whatember the Xth, with all the features we wrote in the first doc and all the features we asked later for free" (also known as "Service" world)
* "sure, version 1.0 will have this feature, and it'll be available on Wathember the Xth" (also known as "Product sold to other companies by desperate salespersons" ?)
I wish choosing when to release was really an option, and I could listen to customers feedback. However no matter how perfectionnist we get, the only feedback that I tend to hear is "this does not work", "this does not work as I expected" or "this does not work as I need it to".
pht on December 4, 2009 8:10 AM
The counter-point to this is never to release an interim product. That is, a version that implements things differently than the final version does.
I had this problem with a community* website I developer in 1998 as a quick and dirty interim site while the final version was still in production.
Due to a lot of different issues, the "interim" site is still the production site 11 years later, with nearly 30,000 records that will have to be imported and verified whenever someone (not me by preference) actually creates a new version of that site.
*As in I didn't get paid to develop it.
R. Bemrose on December 4, 2009 8:11 AMtim on December 4, 2009 7:20 AM - had I a big old rant prepared in response to this article. But you nailed it on the head.
And to add....
"Stating the obvious" articles like this really infuriate me. Seriously, if you're as unhappy about every software release you've put out there (as you state at the beginning of this article) then why stay in the biz? If I was that unhappy with everything I'd produced in my day job then I'd be seriously thinking about a major career change.
Don't get me wrong, I love writing code, I love writing good code, I love writing bug-free testable code and all those other good things, but seriously Jeff, portraying yourself(/us?) as some kind of martyr(s) to the cause of releasing perfect code is tiresome stuff. It's our job.
How about some more articles on some of the challenges overcome when building the SO trilogy, that's far more interesting and practical than this fluff. I know you can do better.
I believe what Jeff means (and certainly what I said in the 'if you aren't embarrassed by v1.0 you didn't release it early enough' article linked to above) is that it is OK to ship a v1.0 with minimal features, thin documentation and/or a less than beautiful GUI. But it isn't acceptable to charge customers for a product that crashes all over the place, gives incorrect results or corrupts data. Obviously acceptable quality levels differ for a web log analyzer and an air traffic control system.
Andy Brice on December 4, 2009 8:27 AMGreat post, Jeff! I just wish they had touched on this "real world" stuff in college.
Having released an Android game back in March, I can honestly say I am embarrassed by my 1.0 version, but as you said, there is nothing, NOTHING, like getting real world feedback from your customers. They'll actually tell you what they want, and really yell at you for the things they don't want, but it's all good!
I just wish my day job had this much exposure to actual customers. Ah - the corporate life...
Mike on December 4, 2009 8:36 AMI resigned yday just because I got rushed (long hours, stress, lunch at desk, missus left me) through our biggest project this year.It was paramount to release it beginning of November. and after that it came out we did not have an external client so it is not used. so if you can afford it, then take your liberty to leave :-)
Michael on December 4, 2009 8:49 AM>> Now, I'm not saying you should release crap.
>> So trust me on this one: even if version 1 sucks, ship it anyway.
So basically you're saying to release something if it sucks but is not crap, but don't release it if it's crap that sucks. And you provide absolutely no metric or even thought as to what is "crap" vs. what "sucks". So basically your whole post is a meaningless rant about releasing stuff somehow earlier than people normally would, but not so early that there would be consequences of releasing earlier than normal ... ie. never.
Gotta say, this is not one of your better posts.
Jeremy on December 4, 2009 9:13 AMJust re-read my post, and it came across a bit harsher than I meant; "meaningless rant" isn't exactly true. But hopefully you catch my drift; saying "release early ... except when you shouldn't release early" and not actually discussing the issue of "well when exactly is earlier than normal but not too early to ship?" kinda avoids dealing with anything real, and leaves this all in the high-level thought (aka fantasy) realm of discussion.
Jeremy on December 4, 2009 9:18 AMAnyone who found themselves nodding along with this article would REALLY enjoy the writing of Eric Ries.
Eric maintains a blog, "Lessons Learned", and is (AFAIK) one of the earliest and most vocal proponents of the "Minimum-Visable-Product" philosophy.
http://www.startuplessonslearned.com/
Portman on December 4, 2009 9:22 AMCrap. That should have read Minimum-Viable-Product.
Portman on December 4, 2009 9:25 AMI think there is a difference between polishing the product (visible to the end user) and polishing the product (visible to the developer). I think cleaning-up code at the expense of a later release is a bad idea. Cleaning-up code to fix known bugs that the user will see is a good idea.
In the real-world (not the imaginary agile world of just-in-time bug fixes for web pages, but medical devices for example), you can't ship buggy code. Everything must work on V1.0; you don't let the patients and the doctors beta test your buggy software. If you think firmware programmers are expensive, try lawyers.
Mark on December 4, 2009 9:30 AMAm I the only one who interpreted "things wrong" as things that don't work the way the customer wants or expects (interface/training issues), as opposed to things that don't work at all (bugs)?
Larry Coleman on December 4, 2009 9:36 AMOur version 1.0 was awesome, that's because we were 4 months late (4 months according to management, 2 months according to me and tech team!)
It's our version 2.0 that sucked - that's because it was released on time :( Thankfully, all the sucky bits are/were behind the scenes and so far the nastiest problems were handled before the users spotted them (touch wood).
Anonymous Monkey on December 4, 2009 9:53 AM@Simon, and a number of others: You sound like the ones being obtuse. I work in heavy manufacturing, where a minor but important mistake could mean a user's lost hour, a thousand-dollar lost sale, or a million-dollar factory crash. So we have a whole testbed, and will mock up larger testing facilities as necessary to ensure that it works as we need it to. We certainly don't throw software into production no matter how much testing the company claims to have done.
Yes, the scale is much higher - we expect software to be at 99% instead of 90%, even if it costs ten times as much and takes ten times as long - but we also expect to be able to bang on it before it's done and send our feedback, because no application's ever been perfectly polished or perfectly fit our needs, no matter how well we tried to spec it out - in-house, outsourced, or off-the-shelf. Sometimes we even run on half-baked systems because it has to be done now. The risk of losing profits from falling too far behind the competition is too great to demand perfection.
The only place attitudes like yours are valid is when there is no competition and lives are at stake. But the software you write is no more "grown-up", it's just polished to more 9s because you might be liable.
foxyshadis on December 4, 2009 10:01 AMSurely there will be parts of the software that sucks, but the important question is not whether we should still ship the software.
No, the important question is, can we sell those parts as additional feature.
zefi on December 4, 2009 10:31 AMSo true. Even major releases should be considered "v1" releases to some degree. Last month I released Paint.NET v3.5 after over a year of working on it. It had a lot of significant and tricky under-the-hood changes. After watching my inbox for a week I found some really bad/strange/rare bugs that people were running into. But I would've never found them in "the lab"! Two weeks later I shipped a ".1" update that fixed most of the issues. If I'd waited another 2 weeks, guess when those bugs would have been fixed? Yup, another 2 weeks.
In .NET you can get really nice call stacks when things crash; in fact, if you plop the PDB next to the EXE and DLLs, you get line numbers. Make sure these crash logs get to you! On a web app it's easy, just toss it somewhere appropriate on the server. On the client it's a little more difficult but doable (mostly to find a location that won't "fill up", or get annoying, etc.) Along with this, I watch to see what crashes happen frequently and then I put a try/catch block around those specific lines. But, I don't handle the exception. I rethrow a new exception that includes extra detail that I hope will help me fix the problem -- for example, "the size of the image is ____" or "i tried to allocate ____ bytes and failed", or "this object [was/was not] disposed". If it's an argument out of range, I make sure to include the argument's value as well as its acceptable range, e.g. "y=-1 but the allowed range is [0, 20]".
So if there's a bug in version N and I can't reproduce it, then by the time version N+1 gets released I'll start getting more info back on that crash. Hopefully for version N+2 I can then fix it. This all helps to relieve a lot of the anxiety around, "omg we need to ship" vs "omg we need to wait". If you can't solve a bug that people are reporting ... add instrumentation, release anyway, then fix it for the next "dot" update.
(Oh and v1.0 of Paint.NET was pretty awful. But it worked!)
Rick Brewster on December 4, 2009 10:34 AMThere definitely has to be a balance here - you obviouslly can't release, say, flight control software for an airliner and wait for feedback.* And if you release something that is so buggy it is unusable you will not get any user feedback either.
But if you have a usable, stable product with some issues, like crashes in obscure cases, odd or inconsistent behavior, or unpolished behavior in some scenarios, you should try to release as soon as possible. The thing is, during development, you are guessing what bugs are important in your triages. You are guessing how customers use the software. You are guessing at the impact of a bug - if it is something that will effect all customers, some customers, or no customers. You could expend effect on getting these issues fixed. Or you can release, get feedback, and focus on things that actually impact customers, including things you never would have considered. No guessing at all - bug X is a crashing bug that you get 500 reports a month for. Bug Y prevents customers from deploying in a configuration you would never have tested it.
* I had to exercise some self-control here to not make a tasteless pun on the word crash.
Michael on December 4, 2009 10:57 AMI think the point that got lost is not "release buggy crap", but rather:
1) The engineering and internals of your 1.0 may not be where you want it
2) The visual design may not be where you want it
3) The feature set may not be as robust as you want it
etc.
Get a stable 1.0 and out to the customers. It should be high quality (as bug free as possible, like any software) and with at least the minimum viable feature set.
While that's out in the world, the engineering can be firmed up, the internals can be changed, etc, for the next release. You'll be receiving feedback from customers that will help shape the engineering, and help shape the UI, not to mention what features need improvement, what doesn't work the way it is envisioned, what is missing etc.
Bado on December 4, 2009 10:59 AM@JeffGordon Assuming you have a refund policy, shouldn't it be up to the user if the product is good enough for their needs? I think users will often buy a product not because of what it is, but because of what it may become. If the users can see your vision, or have their own and want to participate, it is worth it for them -- it's their choice, so let *them* choose... but be kind if it doesn't work out.
@Portman Eric's writings are wonderful. I too highly recommend it.
I think the main jest of what @codinghorror is talking about is new products. Products whose producers are trying to discover the exact need being met. For this target audience, it's absolutely critical to get feedback early, and get it often. Steve Blank has a book /The Four Steps to the Epiphany/ that discusses this process in detail. If you're writing software involving many unknowns, like many of us, it's a must read.
Kaleb Pederson on December 4, 2009 11:00 AM...or perhaps we could do XP, be ready to release tested code at the end of every iteration, never slip the date, and adjust scope with the customer during the planning game.
We could release early and often, quickly respond to feedback, and not reduce quality.
Or we could make our paying customers into involuntary beta testers.
Bill Sorensen on December 4, 2009 11:09 AMThe best motivation for me to ship a better product is not to be embarrassed about the product I made. That's motivation enough for me!!! An internal application in our company acutally will be used by tens of people daily and even all day long every day, and then it's crucial that's bugfree and free of most small annoyances! Reach for the stars and get to the treetops!! Good also!!
Code monkey on December 4, 2009 11:43 AMIt's often strange that this is how it works in the software world. We're software _engineers_. In what other engineering world can we they release a "sucky" version 1?
If version 1 of my office building was buggy, it might collapse and kill me. Version 1 of the bridge I cross on the way to work better not suck either. And Version 1 of a space shuttle leads to Challenger-like instances.
Sean on December 4, 2009 12:20 PMMaybe my experience is tainted by the job I do (I design software for medical devices, so high reliability is the name of the game), but as a user, this kind of buggy early releases are enough to make me reject a product.
It doesn't take a lot of crashes for me to lose faith in a product if it's meant to serve any kind of useful role in my job (even if that role is, say, storing my knitting projects wish list. My time is precious.)
And I've seen my mom - a complete newbie - just give up on potentially useful applications because she just couldn't get beyond the bugs. What I'd call sloppy UI and harmless but annoying bugs, she translates into "I guess I'm too dumb to use this application".
And the less critical your application, the less it takes for users to abandon your product...
Bossy Kena on December 4, 2009 12:49 PMI disagree.
I'd rather ship a version with limited functionality that works well.
Then there are those amazing people who release software that just works, and works, and works...
Lutz' Reflector, DVD Shrink, etc.
Steve on December 4, 2009 12:51 PMIt is ironic that Jeff talks about shipping what your customers/users want, but when it comes to careers.stackoverflow he and Joel seem to refuse to listen to what one side of the user base wants.
Jeff's not being consistent or honest with himself here.
tim on December 5, 2009 2:45 AMExcuse me!
Isn't this why we have Beta, Alpha, and UAT releases of our products?
When a product is in development, it should be shipped to the customers BEFORE IT IS OFFICIALLY RELEASED and BEFORE ALL FUNCTIONALITY IS COMPLETED.
Of course, I'm not talking about web server enabled software like Gmail which can be updated without ever a customer knowing about it. But, standard here's-the-CD-now-install-it-on-your-PC software is suppose to be released to a dedicated set of customers who test the stuffings out of it by using it every day.
If I buy a piece of software, even Release 1.0, and it contains some zingers of bugs, I don't update to Release 1.0.1 or Release 2.0. In fact, I usually never buy a piece of software from that company ever again.
You should release to customers a beta copy that is usable. You should be working on one feature at a time, and making sure that feature works before you start on another one. You should be writing all of your unit tests before you write a lick of code so you know what each function has to be able to do before it works. You should be using CruiseControl or Hudson or whatever build automation tool you choose that builds and runs all of your unit testing on each check in.
If you do all of that, you'll ship software that may not be 100% bug free, but is something users won't regret buying.
David W. on December 5, 2009 3:33 AMJeff's advise is to be interpreted with care. First, as mentioned before, you cannot and should not apply this advise to commercial software development, software products and embedded software. It will ruin you business if v1 sucks.
As for the consumer web, clearly Jeff's area of expertise, yes, you could consider the early release cycle methodology, but only if v1 does not "suck" as bad as to chase users away forever. Call me old fashioned, but what is wrong with doing a normal beta with limited users? What is so outdated about acceptance testing your software before releasing it?
I guess it depends on what you mean when you say v1 sucks. If your back-end code is not optimized and no user will notice, yes, you can release early. If the user experience or actual features suck, you're in for a big risk, even on the consumer web.
All of this release early hype comes from the desire to have a short time to market. To be there before the competition. The result is that we now have TVs with bugs in them. Appearantly it is now normal practice to deliver crap on time and then fix it later.
You know who has the best software planning approach in the world? Blizzard. They do the exact opposite of what you do. They don't care when they release, they release when they are ready. Everytime they do, they hit the jackpot first time. The whole rush of software releasing becomes irrelevant when you take your time and do it right from the start. Blizzard does it, and so does Google.
Ferdy on December 5, 2009 4:07 AMIf users haven't been using your software before you ship it... You have already failed.
John Sonmez on December 5, 2009 4:39 AMAmen,
I am all too familiar with this.
There is no perfect software @ version 1.0, the best tries I know have been in beta for ages and are lingering somewhere around 1.2, and they are all relatively small programs...
StarLite on December 5, 2009 5:14 AMI started reading the article..felt was Interesting..kept reading until i reached the quote of "Donald Rumsfeld"!
I stopped reading. I don't want to read someone who is quoting a guy like Donald Rumsfeld : this man is a war criminal (the butcher of Iraq).
If you want people to read you don't quote problematic persons like this war criminal called Donald Rumsfeld!!
Anonymous on December 5, 2009 5:36 AMJeff,
I have just experienced this issue from the customer side and have discovered that pushing it out works great.
There were bugs coming in all directions, so it has made the launch very stressful. But our customers are finding things that we never even dreamed of and I am pushing them back to the developers.
I am still in awe of how quickly these issues can be addressed by the programmers, once the system's bones are in place.
I was very critical, and really pretty impatient with the development team to get everything perfect before going out. But now I see how well the customers partner with us in producing a better product.
Happy Holidays to you and your family, Jeff.
Bruce Christensen on December 5, 2009 6:26 AMHummm... my comment the day this article came out is missing... It was brilliant, but I'm reduced to paraphrasing it, because it was but a brief moment of literary genius.
This is all well and good for the programming department, and we all know that nobody expects the sales department to actually tell you the truth, but this 'policy' REALLY screws over your support staff. BIG TIME. We're the ones who deal with the angry customers. We're the ones who have to make excuses and lie about why things don't work. You pull this one too many times, and you start losing updates and support packages.
It's always best to ship what works, and the new features that are still being polished get released in an update. This will keep people in a renewal plan, release only features that work properly, reduce pressure to do bad fixes, and gives a much better impression about your product. Ship 90% today, and the last 10% are updates/new features.
Do this, and you might actually be able to keep your support staff, long enough that they are actually good, or move on in the company. They are the only ones talking to users every day. They are the ones that keep the good feelings going when there's problems. Here's a quick test.. if your support staff is happy, your customers are happy.
Of course, if you outsource it, nobody's happy.
Rocketboy on December 5, 2009 8:10 AMThe problem with this approach as I see it, is that software is no longer as simple as it used to be. There are multiple layers full of contractual obligations to each-other. Rushing a first version and allowing crap will inevitable have negative consequences, i.e. the broken window syndrome.
We may think we can always come back to fix it, but we all know this rarely happens. Project managers don't understand technical debt, and marketing people scream "ship it" as soon as they see more than a mock-up.
So I favor "Version 1 is delayed, but ship a feature-deprived preview" over "Version 1 Sucks, But Ship It Anyway".
Casper Bang on December 5, 2009 8:34 AMalot of people here are saying "oh he meant release as a beta." he did not say beta, he said ship. ship means (to me) slap it on CD and put it in the stores.
THIS is the vile infliction upon software development today. Force the customers to betatest, CUSTOMERS are NOT beta testers, they are consumers. You do not give them crap and expect proper feedback. Give a proper alpha test and beta test, once you are sure you have fixed the major flaws and have polished up the shiny interface, THEN ship as v1.0.
Jeff you missed the mark something fierce. I agree with many here, your experience with web apps is clouding your judgement. In game design, or even basic software, there are people who buy it who never have access to the internet. You cant fix their problems, so they are stuck with them unless you release a fixed CD which costs even more money. Or if you force the customer to pay for that fixed CD as "version 2" your even more disgusting.
Nesetalis on December 5, 2009 9:09 AMFor any product there is a set of criteria which is "good enough". Anything less will be suboptimal and anything above will be somewhat "wasted" in the sense that there could have been lost opportunity (to say code in another highly value-added feature or made a shipped feature more usable). Of course, defining "good enough" is an art and a science unto itself.
Andy on December 5, 2009 9:12 AMI think this is sort of naive navel gazing. At the heart of this is nothing more than a cliche. Sure, you shouldn't keep polishing version 1.0 for years and years and never release. Ok, fine. But, you could can get that information from any and every source on building software. This is just kind of cliche.
Charles on December 5, 2009 10:00 AMThere's a couple of issues I'd like to address with this article
1) First impressions might be the last
Many who try out software might only use it once or twice before discarding it judging it to be of "no use" due to annoyances and "small" issues and never coming back.
That makes providing a polished experience with everything from installation, getting started etc to the actual interface very very important.
2) User-driven developement
Iterative developement is clearly the best way to develop software, no one is going to design and build the perfect software in one fell swoop. Getting feedback and bugs reported can be very useful. *However* letting the users drive your development is usually a big mistake.
First off users lack what you might call the "cohesion of the design" that the designer has, you can't build software to please everybody but need to inflect the user interface to avoid builing a frankenstein hybrid between a car, truck and landmower that no one wants. What you chose *not* to put into your application is more important than what you put in.
Secondly, to paraphrase Steve Jobs "The user doesnt know what the hell he wants". You should never ask a user what he wants, only what problems he has, and what goals he is trying to achieve. Then it's up to the designer to come up with a proposal and then perhaps get feedback on it. Start out with a smaller focus group for feedback before releasing it to everybody.
3) The cost of doing major redesign/refactoring once you ship is exponentially higher. If you ship your product and then realize you left out some important piece of architecture or concept it will be very hard to implement, first you have to convince the powers to be that you should be allowed to do it, and then handle the users who might have gotten used to the interface and then provide version compability.
Software in development is agile (with the right designers/programmers), shipped software is rapidly solidyfying concrete.
4) Aren't we tired of software that over promises and under delivers? Nowadays it's almost the default to start hyping your software long before it's released, release a crappy and/or or buggy alpha and then spend two years before the application is finally "decent". Meanwhile only the most dedicated users have stayed around.
One day someone is going to release a unhyped piece of well-tested and innovative software and I hope I'm alive to see it :)
5) Programmers aren't designers. This is a delicate subject sure to piss of alot of people but sadly it's through. Most programmers can't design, but most think they do. It's unfortunate that alot of software is built without some kind of interaction designer ever been involved.
You only have to look at alot of software to see that most maps more clearly to the programmers vision of doing tasks than the users needs and goals. There are a few exceptions, programmers who also can design, of course every programmers thinks he is one of them (even me) but must aren't.
Get a good iteraction designer, it will make a world of difference
On the other side, designers who lack understanding of technical issues and possibilities won't be able to do the most optimal design either, and unfortunately technical adept designers are about as uncommon as programmers who can design.
So to summarize my take on it. No software will ever be perfect, but be sure it's polished and "conceptually" finished when you release it. It's always better to release a little later and better than "on schedule" and flawed. That's how you build user loyalty and not get content user but passionate ones.
Never ever ever release sucky software because that will be it and no one will care enough to give it a second chance.
(Sorry for the rant)
MattiasK on December 5, 2009 10:14 AM@Jeff Gordon: VMware Fusion comes to mind. When Fusion 1.0 came out, I bought it, knowing VMware was new to the Mac market. Parallels had already released their VM software for Mac long ago. I bought Fusion, and "suffered" through 1.0 (it wasn't really that bad). As a reward, VMware gave 1.0 customers free upgrades to 2.0 when it was released. I was so happy with that, I bought 3.0 when it came out, even though it didn't necessarily scratch any itches. That's one model to solve the "who pays for 1.0?" problem.
@Joe: Don't you *gain* a developer with a pregnancy, rather than lose one? :)
@Jeff Atwood: I agree. Immeasurable are the number of things I've learned by releasing 1.0 and monitoring feedback that I never would have learned in a lab. After you see your program in the wild, you can really get to tightening nuts and bolts. Yes, each nut and bolt should be there in 1.0 and be more than finger tight, but it's okay to save the torque wrench until 1.1 sometimes. (I'm talking about software, not cars or airplanes.)
Patrick (a different one) on December 5, 2009 11:41 AMI can't agree more, and I can't agree less.
When you ship depends on you, your team, your market, your risk, your cash flow, your sanity, your investors or lack thereof, your happiness, your experience, your upbringing and probably even your mood on the day you pick the day to ship. For every should and shouldn't that someone takes the time to blog about, there are hundreds if not thousands of counter-arguments that are proven by successful companies and products.
I want to see us working in whatever way makes us and our customers happy. If we pull that off there don't have to be any rules.
Alan on December 5, 2009 11:47 AMWhat about the unknown knowns? That is to say, the things we don't know we know. I'm surprised by those quite often. :)
GR1000 on December 5, 2009 12:46 PMWell, I never feel bad, and I never find feedback too useful.
There's a difference between input and feedback, and a difference between problem domain and design feedback.
If you don't know the thing you are trying to make, you can't make it. That's where input and learning your domain matter. When you undertsand the problem, you solve the problem instead of endlessly complicating it. When you don't, it's completely impossible to solve no matter how good you are at programming or interface design. If you have good input, you don't really need the feedback except to point out bugs.
If you know what the problem is then feedback on the design you made is going to be largely useless. If someone can figure out a better interface that's just as easy to program as the one you made and they are a secretary or engineer or whatever then you've got some serious problems. Not to say there's not always simpler seeming ways to do something, but you should have already thought of them all already and chosen the best one that fits your problem (and your budget).
If you are getting big surprises it's usually having no idea when it comes to the problem, but you should not only not be releaesing then but possibly not be starting til then. If you know the problem, and the users can outdesign you for a complicated system you are in trouble but that's usually not the case.
The problem people get sucked into here is blindly adding features without understanding why the users want them. That doesn't solve anything, just makes more mess. If you understand why they want them then you can address that. Or else determine it's not worth the time an effort, offer them a bill to do more, etc.
Of course you don't have control all the time or maybe most of the time depending who you work for, but if you require a lot of feedback it's usually a symptom you made serious mistakes and you should not be releasing because trying to appease angry customers by hacking fixes on your bad design is only going to make it into an unusable garbage product over time.
Antiguru on December 5, 2009 12:48 PMTry Agile/Scrum/Kanban/Lean.... I prefer Scrum and you will yield far better results as this is more focussed on What customer/stake holder wants...
Nik on December 6, 2009 1:07 AMI agree with GD's comment about your cheapening our profession. Have some pride man! You are in a great position to improve our craft, yet you write articles about condoning shipping software that sucks.
Grow up man and take some responsibility about what you say to your +100K readers. Your words mean something you know. As someone that is trying to turn our craft into a profession, you are not helping!
--If users haven't been using your software before you ship it... You --have already failed.
--John Sonmez on December 5, 2009 4:39 PM
I totally agree with mr Sonmez, and all the oters who said "Agile", "Iterative development" and scrum or other software building ideas.
I do understand part of the blog, but I don't see why someone like you (an expert on coding horrors and a perfectionist) would be advocating this type of releases with as primary reason: user feedback. I'm sure you are using agile development theories, so I must be missing something.
So what I try to ask is, what are we missing?
(Also about iterative development, I can really really recommend Craig Larman's Applying UML And Patterns: an introduction to Object-Oriented Analysis and Design and Iterative Development, long title I know, but really a great book about I.D, I'll try to put up a review about it on my blog. (http://roy-t.nl, or click my website)
Roy Triesscheijn on December 6, 2009 3:31 AM@Mitch don't take it so hard. It's just smack-down at it again.
See you can't just make a rational, well argued point, you have to take extremes and shove it down someone's throat, which doesn't really work for this particular post (or any if you ask me).
I have to say, the writing was very cloudy. There were a few times of the "but this is wrong" rhetoric that I completely lost what the point was. Then I realized I don't care.
flipper on December 6, 2009 7:01 AMThe approach is totaly correct,
Alpha testing should be more on the security part , other aspects of the software must be handled by the beta phase where the annoying and frankly users will reach the system.
Best Regards.
I blogged about this same issue from a personal application standpoint!
Here is the URL: http://oldschooldotnet.blogspot.com/2009/02/so-finish-something-already.html
It feels good to actually agree with you once in a while :-)
L. Lee Saunders on December 6, 2009 9:36 AMA couple years back, fresh out of campus, we were thrown onto a project that required us to create a nationally integrated system with several other players. Of course, we were given the project three months after everyone else. But we stuck to our guns.
Our code was buggy to say the least. Functionally, the B word is Bad. But we did release. And it basically broke all over the place from day 1. But then, as Jeff pointed out, we could easily fix problems and in the real world, the fix was something we would never have foreseen (in all our code monkey wisdom). Also it helped that with the spec changing every week, it was amazing the amount of things we WOULD HAVE misconstrued but didn't, because it was pointed out by real end users. Not testers (in all their wisdom).
Those first two months were rocky to say the least. However now three years later, v3 has been running for 21 months with only minor bugs. And (un)surprisingly our system was has faired pretty well against the other systems. And has been hailed as an overwhelming success. All because during development, we got it mostly wrong :)
Ipkiss on December 6, 2009 11:04 AMJeff, I don't know why you always want to cheapen the profession. Bugs are tantamount to mistakes, and mistakes in any profession are best avoided or fixed before your customer sees them. Especially if they are glaring, in-your-face mistakes.
You should also either go easy on yourself, or try harder. I've been 90 - 100% HAPPY with every bit of software I've released, and absolutely smitten when the resulting bug fixes are relatively few and mostly trivial, and occassionally devistated if anything major comes back. I'm certainly not getting ready to take notes on the flood of issues/bugs/abuse coming back on what I've shipped, like you suggest I should be doing. And what other profession operates this way? None that I can think of. None that would hope to stay in business for long.
If it's not a beta release or a release that is agreed by all to be for user testing, then it should be your best effort at a sound, final product. I guess it depends on your definition of "shipping a flawed version", but taken at face value, this certainly doesn't sound like a product I'd want to be selling or buying.
GD on December 6, 2009 12:43 PMIncredibly refreshing words. Just released a drastically scaled-down 1.0 a week ago to the iTunes App Store, and had some initial pangs of guilt about it.
This post definitely soothes me a little bit more.
Can we downvote this post?
This is exactly the sort of talk you hear from software dinosaurs refusing to adhere to best practices.
If you release a beta, you should be addressing all the issues you mention here, without frustrating your user base.
I hanker after the old days of console game development (and it's also true now on certain platforms).
You couldn't have any bugs left in the game at release. None. No excuses. Just no bugs.
I think we were better programmers back then - I'm much more lazy nowadays, and it's a shame.
v1.0? Fools. Real software is distributed as v0.1, before it even does anything useful, as source. Then everyone that finds it, can fix a bug, add a feature, etc, and re-contribute it to the project. Then by the time you got to v0.9, the project is basically in full working order, and *doesnt* suck.
I think you are right.
I believe that looking at commercial part of a software business we cannot afford to streamline and perfect a software for ever and ever.
We have to draw a line.
I personally believe that we should always have some defects in software otherwise how are we going to develop V 2 of that software!
I adopt this technique in my software so that I can always give a new version to my customers. But I also take care to give my customers more than they have asked for so basically I am not cheating them in any ways.
Just my views though.
Yogi Yang on December 7, 2009 3:27 AMI feel the software should be released to the customer only when it has passed all the stringent quality tests that have to be setup by the QA team. Surely after the customers use the product, there will be significant real world issues that crop up. But a developer shouldn't expect to be fixing issues such as the software crashed after using the application 10 times after it's release.
If the software can't reach the acceptable minimum threshold set by the QA, then we perfectionists should wait [:D]
Kevin Rodrigues on December 7, 2009 5:25 AMChrist... of all people you have to quote Donald Rumsfeld. The guy credited with letting Bin Laden get away by 'saving costs' and not sending in more troops to the Pakistani border.
Other than that we all agree in principle that shipping schedules have to be realistic, and failing that, the software being shipped should reflect as close to that shipping schedule as possible.
Leo on December 7, 2009 6:09 AM@sunfire on December 4, 2009 6:56 AM:
"Microsoft uses a lot of statistics how many bugs could eventually be in a software at what level and how much bugs a customer wouldn't take care before switching to something else."
Perhaps that could be called the peanut butter approach to release management?
Andy Lee on December 7, 2009 6:41 AMThe comments to this entry are closed.
Subscribe in a reader
Subscribe via email
|
|
Traffic Stats |