While narrow estimates are unlikely to be right, wide estimates are unlikely to be useful. For example, I have no real feel for how many books have been published in the US since 1776, so to get a 90% confidence interval, I used 200,000 - 2,000,000,000. McConnell would say that that's a good range, since it actually includes the correct answer, but -- geez -- the upper bound is 10,000 times the lower bound.

Imagine a customer (or boss -- whoever you work for) asking how long a change is going to take. "Well, between 3 hours and 3 years." "WHAT!?!?!" "Well, I'm 90% sure it will fall in that range."

In some cases, in order for an estimate to be accurate, it has to have such a massive margin of error that it's no longer useful.

Estimation should be done by: SME (Subject matter experts) and the actual resource that will execute the task. Anything different than that, just confirm you are a newbie. I can't estimate tasks on a space ship construction project... And that's ok...

So, this excercise in the book sounds more like "You are bad and you didn't waste your money in my book" crap.

While I feel that this is a very interesting quiz, and I agree that the results are interesting, I do feel that the fact that no one has ever got ten right is in part due to the nature of the questions. Personally I knew the surface temperature of the sun and the year of Alexander's birth (to within ten years), so it was superfluous to give wide ranges of answers. The latitude of Shanghai is likewise an answer that anyone with any knowledge of the scale of latitude is liable to be able to make a fairly safe stab at. The impossible questions are the volume of the great lakes and the length of the coastline. The first because the units of measurement are befuddling - cubed measurements always are. The second is geuinely impossible. How did you measure the coastline? At what definition? A straight line between every cove or round every face of every grain of sand? If it were the latter then the true answer would be billions of miles. I'm all for encouraging people to give wider ranges, but if I gave a range of 0 to one trillion for every question I would get all ten right. I doubt that approach would help me in any practical setting, however.

I think this excercise misses one of the greatest problems that exists in software estimation. Estimates don't exist in a vacuum. How often have you tried to provide what you felt was a "safe" estimate only to have your boss or your customer come back and tell you that your estimate was unreasonable. They immediately begin trying to pressure you into committing to a lower estimate which they will subsequently use against you when the project starts slipping.

Government contractors are notorious for "low-balling" projects, knowing that the real costs will be much higher than the winning bid. But if you provide a realistic estimate you will not win the contract. This is mentality that permeates our industry.

Quite honestly, I am not sure you can give any reasonable estimate that would satisfy the people who pay the bills. There are so many variables in a given software project, as the size of the project grows the range of possible estimates will also grow. As Boofus said, the estimate will ultimately be so vague as to be useless.

Just looking at the number of overbudget projects that continue to occur tells me that you have more chance of creating an accurate estimate using a dart board as you do using any of the estimation techniques used over the last 40+ years of software development.

Hmm. I'd be interested to see how well the test worked the other way around. i.e. Give people a randomish range and then ask them how confident they are that the result is in that range.

Not really on-topic I suppose, but you might want to correct the volume of the Great Lakes: 23,000 km^3 = 2.3 x 10^13 m^3 = 2.3 x 10^16 litres. The quoted 6.8 x 10^20 m^3 is about 2/3 of the volume of Earth. (I didn't check your imperial numbers.)

I refused to answer 8 of the 10 questions, because I had no way to make an educated guess at 90% confident. I got 2/2, or 100%.

Not giving an estimate is much more useful than giving one too wide - at least then the question can be posed to someone who has a better chance of getting it right.

While I am a bit of a fan of McConnel's books, I think I'll skip this one. As a developer, I don't really care to improve my ability to make "accurate" estimates, nor to widen my ranges of estimates so as to have the actual figures lie within my range 90% of the time.

In most _real_ businesses, the idea of giving estimate ranges multiples of orders of magnitude wide to management are simply a fantasy.

If McConnel wants to help me with this problem, he'd teach me how to convince management that I shouldn't waste my time _making_ estimates that have no real hope of being accurate - at least to any degree to be useful.

I'm about 90% confident such a book would be more useful.

I think it's a valuable exercise, and I'm dismayed that at 4 out of 10 I'm doing fairly well, based on McConnell's histogram.

Of course people don't like uncertainty, but it's good to be as realistic about it as possible - I've given people (managers & customers) estimates that reflected my uncertainty, and they are well served by that (they'd be even better served if I could do better than 4 out of 10 when I'm shooting for 9, of course). Tasks I'm confident of have a tight range and tasks with more unknowns have a broader - sometimes much broader - range - and that allows the customer to make better choices among them (including, "what can you spend some time on to reduce the uncertainty in this task?")

> In most _real_ businesses, the idea of giving estimate ranges multiples of orders of magnitude wide to management are simply a fantasy.

That's right. So the next logical question is, given the high level of uncertainty at the beginning of *any* software project, how do you eliminate or reduce that uncertainty?

It's all about gathering data-- both as you go, and before you start, through research. Ideally, you'd have some historical project data from your own organization.

> Just looking at the number of overbudget projects that continue to occur tells me that you have more chance of creating an accurate estimate using a dart board as you do using any of the estimation techniques used over the last 40+ years of software development.

You're saying that software estimation is impossible. I don't think that's true. It's a very hard problem, but it's not unsolvable. I think the main problem is most organizations don't gather enough data from their past and existing projects (bugs, bug fix rate, function points, etcetera), so they're starting from a blank estimation slate every time they launch a new project.

6 of 10 of my estimates were a range that spanned a single order of magnitude, the rest smaller. And only half were correct. But that is just not a technique I could use in estimating my coding projects. A response of "two to twenty weeks" would get me laughed at.

It is my experience that more accurate estimations require significant time to complete. Five minutes spent coming up with an estimate of two months is probably going be well off. But a week or two spent on creating a two month estimate is likely to be much more solid.

It would be nice if estimates were not deliverable until already into a project. :) Updating estimates during a project has always been critical for managing expectations with me.

All this talk reminds me of something I read a few years ago about the most consistently reliable estimate being made by measuring *how thick* the requirements documentation. (This was in the context of large projects.) Can't remember the source.

This mostly demonstrates that most people can't give blue-sky estimates with any accuracy. Which is tautilogical. Being surprised at these results is what surprises me. If you'd required estimates to be within an uncertainty range *then* you might get useful results.

I limited my estimates to two orders of magnitude (and got 5/8, excluding a couple of exact answers), because I glanced through and decided that that was more useful than the estimate that answered all of them (+/- infinity was my first answer, since you/Steve asked a stupid question).

I don't remember where did I read this, but I did somewhere:

An estimate is just that - an ESTIMATE. You don't have to be 100% right or something, it's just an estimate of what it will be. Look at weather forecast is it exact? Always? Ever? :) An estimation is just a forecast so there is (should be) no shame to be wrong... I guess.

As people have pointed out, giving an estimate with a range of several orders of magnitude is generally a non-starter in most organizations.

But I think that the point of the exercise is that if you are requested to honestly and to the best of your ability produce an estimate with a 90% confidence range and the result covers several orders of magnitude; then, for a well functioning organization, this should be a crystal clear signal that significantly more research needs to be done before the project is undertaken.

Of course, most of us work in disfunctional organizations where inconvenient truths are not welcomed. And I would hope that other chapters of McConnell's new book will cover what to do when estimation reality collides with management fantasy. (Does it Jeff?)

Unless you are an expert on all of the subject matters above, you couldn't hope to give an accurate estimate on them. The point was to show that if you were forced to, would you be able to account for the inherent uncertainty.

If I was asked to give an estimate on how many lines of code a transaction system would have, without any more information, my estimate would have to be a broad range. The results of this experiment, however, tell me that most people's wouldn't be broad enough.

Bill nailed this one: if people simply give a fixed point or too narrow band because it is the corporate culture, that culture is avoiding the problem of the underlying uncertainty.

If the developers are giving wide estimates, it means the problem is not well understood. If better estimates are wanted, more research will be required and a more thorough understanding of the problem needs to be developed.

The idea that most organizations don't collect enough data to estimate correctly assumes that those organization that do collect and use data are successful at estimation. While I don't have the statistics, I do have empirical evidence. I can see the results from Microsoft, Oracle, Sun, Developer Express and many others, many of whom maintain statistics about code estimation and yet continuously miss estimates. Microsoft is the poster child for missed delivery dates, even when slashing features.

Do I think the task is impossible? No. But I think that many of the development methodologies make the cost of achieving good estimates prohibitively expensive. I much more prefer an agile/xp approach which is only concerned with much smaller estimates, and therefore can be much more accurate since there are fewer unknowns.

These kind of analytical estimation problems are often referred to as Fermi Questions (named after Enrico Fermi who used to ask such questions of his students).

http://www.physics.uwo.ca/science_olympics/events/puzzles/fermi_questions.html

>Do I think the task is impossible? No. But I think that many of the development methodologies make the cost of achieving good estimates prohibitively expensive. I much more prefer an agile/xp approach which is only concerned with much smaller estimates, and therefore can be much more accurate since there are fewer unknowns.

I work for a defense contractor that typically works on 1 million lines of code or more projects. The agile/xp approach just doesn't scale up for projects that size but you have to come up with an estimate to win the contract. In this case being good at estimating is a competitive advantage. I have even heard that when the country of Italy puts out a work request, they take all of the companies estimates and average them. The company closest to average gets the contract.

I don't think becoming good at estimating is a huge cost, it is just hard to stay disciplined and actually collect the metrics needed for the next project while you are working on the current one. Everyone says that having previous project metrics is important for making accurate estimates, but do you think Microsoft is keeping track of the number of overtime hours their employees are working to get XP out the door? I would guess not.

> The agile/xp approach just doesn't scale up for projects that size

To be clear, agile/xp proponents state that it doesn't scale up to projects that big.

> but do you think Microsoft is keeping track of the number of overtime hours their employees are working to get XP out the door? I would guess not.

Umm, if they want to have a balance sheet that makes sense, you bet they're keeping track. That doesn't mean they use the data to help, say, Vista completion time :)

Very early on in my programming career (1997), I was told, "the main difference - no - the only difference between an amateur and professional is the ability to estimate the time it will take to complete a project". By that definition (and several others), I'm still an amateur. Of course, I try not to tell that to my customer.

And that's the problem, in my opinion, with wide estimations for consultants. When you say, "I'll deliver what you want, and it will take anywhere between two weeks and two months", it sends a strong message that I don't know what I'm doing. That I'm an amateur. I'm better off saying "three weeks, no more!", sounding like a professional, and then extending the deadline with excuses, if need be. This sounds terrible, and it hurts the customer, but at least I got the job in the first place.

I think the best, most honest answer to the question, "How long will this project take you?" is the answer, "I don't know, but within a week of research, I'll be able to give you an educated guess". Of course that week might turn into two.
:)

The length of the pacific coastline is not well defined. Coast lines have a fractal dimension greater than one, and thus the measured length increases without limit as the measuring stick used grows smaller. Any value above 50000 miles is therefore correct for some stick-length.

The one other commenter who made the same point made the mistake of coupling the observation with the ridiculous and wrong statement that the volume of the great lakes is also not well defined.

Whatever reference produced that precise value for the Pacific coast length is either: a) teasing, or b) another instance of that special mixture of self confidence and ignorance that plagues so much of our world.

What is interesting about the study results are that we have a marked tendency to underestimate (if our estimation is 30% accurate is debatable). How many times have you estimated that a project will last _less_ than what it actually did? My interpretation of this 90% vs. 30% anomaly, is that our estimations (excuses for generalization) are about a third of the real value, when we haven't researched enough on the problem at hand, as it turns out to happen in the design of this quizz. That doesn't mean, of course, that with a decent research we cannot approach more to the real value, and possibly that's the core subject of the following chapters in the book.

Most of the comments here are interesting to read but a bit off the mark. I wonder how many of the people making the comments actually read the book. The “estimation” exercise was part of a bigger discussion of “estimation, targets and commitments”. Estimation, according to the author, is used to help the planners gauge the likelihood of having a successful project. Of course, estimates will be done by qualified individuals. Of course, the narrowness or breadth of the ranges matters. Of course, some people will know some of the answers. None of that is the point! The point is that a huge percentage of people who took the test were asked to get 9 out of 10 questions right yet the vast majority could only achieve 2-4 correct answers. Those of you who blame “management” need to look in the mirror and READ the test instructions.

The quiz is stupid. Most of the questions involve estimating things that are incomprehensibly huge; they're just a number. There's no sense in trying to ascertain the value of something when you have absolutely no conception of the scale at which you're dealing with. The guy who got 2/2 right, 100%, has the right idea.

Scheduling programming, on the other hand, involves a directly tangible sense of days and months--at least, it should to an expert programmer. So it's possible to make meaningful estimates, even if they are wrong.

Your quiz is unfair because it is designed to make people wrong. Scheduling programming is just disposed to hurting people who underestimate.

This quiz might be better if it asked for 50% confident estimates instead. Those who guess +/- infinity are actually estimating to 100% confidence, which is not the point of the quiz.

If I know Alexander's birth-date to within a century, my idea of 90% confident is much narrower than if I only know it to within 5 centuries or have no clue at all. Because of this, the amount of knowledge you have on the subject doesn't matter much.

90% confidence is difficult to distinguish against 100% confidence, so you get the impression that you *should* get them all right. But at 50% confidence, the best result is half right and half wrong, and 100% right is nearly as bad as 0 right.

That's why I think 50% would be more meaningful, assuming I have interpreted the situation correctly.

I estimated 9 of 10 correct, I had no basis for the blue whale question and foolishly picked too low [40-140tons], and I knew some of the questions directly [the temp of the sun, alexander's birth, lat of shanghai].

I think what Mr. McConnell is trying to say with the higher accuracy crowd being too narrow is that we feel we missed the question we missed because the answer was slightly outside of our range. Given the methodology on the previous questions, I felt my range was large [almost 400% my low], however on most other questions, my range high was always single or double digit percentages of my range low.

A narrow estimation range does not make my estimation skills wrong, it just means I have a lot more basis of expertese [or none at all] to answer the questions given with a high degree of accuracy.

Estimations without care for accuracy are useless. [ie: the time until the sun goes supernova, % chance of an asteroid hitting the earth, chances of dying in a fatal car accident]

I answer right 5. But I think Jeff is right, since my correct answers was due to my previous knowledge, and I fail almost all estimations (except for U.S. currency, with a factor of 8 range).

After know my results, I realize that I tend to narrow the range, without need it. With previous knowledge you can narrow it, but if you're totally in uncertainy, the best practice would be to open the range and make clear that you don't have any real data to narrow it.

As Ryan says, estimations without acuracy are useless, but so are estimations based on few data. You can estimate chances of dying in a car accident, but if you don't know anything of the driver (frecuency of drive, drive times, country...) an acurate estimation is almost impossible.