Yay First post!

It's amazing how many people read and post back to this blog.

I enjoy it very much :-)

I love the fact that this makes my missing the last project deadline not such a big deal :)

Another entry that I'll have to post on my cubicle in the vain hope it can enlighten my colleagues and, especially, managers. At least it covers that ugly gray fabric.

Re: Being a competent software engineer means choosing appropriate strategies for the size of the project you're working on.

Step one – how to become a competent software engineer:

http://softwareindustrialization.com/PreparingTheWorldForSoftwareEngineeringPart1.aspx

Hey Now Jeff,
Your right on with this post, the size of the project is very important.
Coding Horror Fan,
Catto

This validates my method of making estimates: come up with the best number you can, then double it.

> Along the way Steve applies his considerable estimation
> and project planning skills to the project

As you read my post, please keep in mind that one of the main points is that I *didn't* apply my estimation skills to the project. Because it was a small project, I never created a "real" estimate. One of the main points I tried to make in the post is that IF I had applied any real estimation skills at all I could have eliminated many of the estimation errors I made. In other words, my estimation errors on the fort/doghouse project were nearly all errors of *omission* rather than errors of *commission*.

That's a classic one where I work - "It's just a small piece of work, it's not worth doing the analysis and estimation".

WRONG! Invariably there are things that we might have noticed if we'd taken the time to do the analysis.

I belive it's the number of "domains" that are important, where compromises are needed to be decided. Physics? Laws?...and when the number of domains increases the connections (dependencis) between them increases exponential like when the number of communication nodes increases.

This reminds me of the identity management project I was working on back at my old job. The initial estimates of getting the OID server up and populated was about 6 months, factoring in the training for OID itself and creating the proper jobs for merging and managing the identity data (coming from a seperate database).

...and THEN the politics of what the actual "ID" should be. Since this was a campus wide implementation, slated to be "THE" ID for everybody, everybody had their 2 cents. It was decided they wanted to use the ID provided by a seperate entity that wasn't even controlled by us. Adding this layer of complexity then exposed TONS of data integrity issues that turned everything into a nightmare. There were two seperate data sources that had to be merged, processed, and THEN fed into a final repository, which was feeding OID itself.

I believe a year to year a year and a half I quit my job there (not because of this project 8^D) and they are still working on it a year later to my knowledge.

As my olde Pappy used to say "measure twice, cut once". Actually he still says it when he gets the chance. It's true no matter what the size of the project.
I think that half of the problem is that the training we receive in school is more geared towards large and very large projects. No ones going to trust one of those to a graduate fresh out of school. So we get given the small projects; attempt to apply our large project training to them; find that the overheads compared to the benefits are small and just wing it. By the time you get given a large project you've become cynical and resistant to large project techniques leading ultimately to failure.

Steve should have went down to his local home improvement store and hired some "dayworkers" for the brush clear, hole digging and lugging the stuff up the hill. Outsourcing could have kept him on schedule.

Seems the the "expert" of software project estimation gets a F for estimating his DIY project. Nice looking fort, although my kids could sack it in about 10 minutes. Where's the defenses?

what went unsaid: he got so far behind just because he didn't know what he was doing. nothing more. he was incompetent. nothing more. now, in context, that's not a problem. he was building he kids' fort.

on the other hand, many people who purport to be software engineers exhibit the same behaviour. but...... rather than 'fess up that they're incompetent, they wave their hands and say that it's not predictable. nonsense. Boeing knows, and knew, how long it would take to make a 787.... until they hit their area of incompetency: how to manufacture fibre fuselages. but nobody had done that before. they didn't have records of time/materials for tasks which, taken together, added up to a fibre fuselage. they had no idea.

the lesson: you can only estimate with accuracy that which you have done before and recorded. industrial organizations, as Boeing, have been doing this since at least Taylor (see the Wikipedia article for Time and Motion Study). software folk prefer to think that each project/task is very unique, and thus not truly estimable. we're kidding ourselves.

> he was incompetent.

I don't know if this is a fair statement. I couldn't have done anything he described. I don't even own a chainsaw!

I do agree with Jon R. that hiring day laborers could have helped with the early grunt work, though-- and moving those giant sacks of concrete, and the wood from the driveway.

Funny that when I read about a dad building a "fort" for his kids the first thing that enters my head is something crazy simple, like a fort made out of refridgerator boxes or something like that. Then upon closer look he means "fort" as in an out right club house with everything shy of running water!

Not only should you pick the proper strategy for the project, but make sure everyone knows the difference between a doghouse and penthouse.

You can get this philosophy by watching any of the "build shows" on discovery, A&E, or TLC. No firm schedule ever works. On every episode of American Chopper, there's concern for hitting the deadline, and while you may just think that they're making it up for TV drama, it's really true on any scheduled project. Disciples of Murhpy's Law know that you had better plan some serious buffer time in that schedule, because something somewhere will break. That's the interesting part about those kinds of shows. If everything were as easy as "hey, let's go build a chopper" then I wouldn't watch it, but I love seeing what new problems will pop up in the process, like a bent rim or a minute measuring error that screws the whole frame up.

I like Scott's original comment. I quote here in full glory:

Someone comes to me with dried paint roughly in the shape of a doghouse, they want me to fill in the doghouse underneath, but they want me to change the color of the paint, and, to be perfectly honest, what they really want is a 5 bedroom house for their family and they figure that since the doghouse is roughly the same shape as a person-house it should be easy to just make the doghouse-paint bigger right?

>> he was incompetent.

> I don't know if this is a fair statement. I couldn't have done anything he described. I don't even own a chainsaw!

well, if you had emoticons, the tongue-in-cheek one would have been shown. i guess that wasn't obvious. the gist was to segue to what developers do. and that the reason our estimates are lousy is the same reason, alas; we're supposed to be competent at software. certainly meant no serious observation of Steve.

As far as his incompetence is concerned, he clearly did a poor job of estimating. As I was reading his post it was very clear to me that he was WAY off in his estimates. I've done enough similar projects to know that digging holes and pooring concrete will easily take six hours or more. And that's without clearing brush and hauling tons of wood up a hill. I don't know how he thought that he was going to frame the whole thing in a day along with putting in windows and doors.

But isn't that the point? The more you do this stuff the better you get at estimating. Not a month goes by that some management type or newbie programmer claims that they can whip something up in a couple of weeks. I just sigh and shake my head. Let them try. I'll be here two months from now when they give up and hand it to the pros to "do it right".

I bet Steve wouldn't make the same mistake again...

> Often, you can't even begin to accurately estimate how long something will take until you start doing it.

This is the case in my experience. One corollary is it is much easier to get estimates right when developing additions to a previous release that embarking on a completely new development.

I'm sure my comment'll be rapidly covered by the conversation, but I wanted to congratulate you about this great blog. I've followed it for just a month but it has positively surprised me post after post.

Thank you for sharing your creativity and knowledge, and good luck with it, the feedback and the advertisers, you've won it.

Greetings from Spain.

Some observations:

- doing things never done before: yes, you'll miss the estimate but you will be much better the 2nd time.

- hiring day laborers: still think one should have personal experience first before doing this.

- post hole digging: should have rented a portable motorized one; this would have save MUCH time. Point --> knowing what tools are available and choosing the best one.

To all those who write code and have never built a tree house, a fence, etc., get off your asses!

Um...

Isn't the point of building your kids a fort that you do it and don't spend just as long planning as doing? Sure, some planning is necessary, but um........ Being disappointed because you are 'off schedule' seems to ruin the point of taking a week off to play with a chainsaw.

That said, I can see there are some parallels between building a fort and building a large piece of software, but there are some things that are disparate, too... The biggest difference in this case, to me, is that one is a hobby and one is a profession.

A full time GC who was as good at his job as Mr. McConnell seems to be would be beside himself about over-estimating the capabilities of his crew, and scrambling to fix his time schedule so the next queued up job could be approached on-time. Ever seen that happen in a programming context? Add to that that the quality issue is completely in the builder's hands and you have a problem that seems to be defined in a completely different space.

A hobbyist's ethic is vastly different from a professional's.... The entire mindset is different... This changes how the problem is approached as well as what 'solved' is defined as.

The funny thing is, I work with the folks who build the space shuttle (or write the software, anyway). You'd be surprised what it's like here. Because of efficiency concerns, stuff like readability and expandability has to be sacrificed quite often (the computers onboard have 1MB memory and run at 40MHz)

Not really relevant, but I found your reference amusing. :-)

Innovatics

What is Innovatics? You cannot find it neither in google or any other leading search engines, nor in wikipedia or alike. Innovatics, we invent it from Innovation, just like Informatics from Information.

What is Informatics? Informatics is the study of natural and artificial systems that sense, store process and communicate information. It includes the disciplines of Artificial Intelligence, Cognitive Science, and Computer Science.

Now what is Innovatics? Innovatics is the study of how to innovate; it is the science of creation. It includes the disciplines of engineering science, management science, psychology and philosophy.

Frontier Blog - Innovatics Pionner
http://www.hwswworld.com/wp

A doghouse eh? I think a RABID development process would be suitable for that... (sorry, couldn't help myself...)

I think there's two aspects to this. One is the ever changing client requirements and the other is the programming part. As for the programming part, it's all about the framework. If you have a solid framework (custom built), then you are WAY ahead. Everything about a house is built around its frame. Same thing for planes. You can make the same claim about anything large scale like transportation networks for example. This way, you can attach yet more large frameworks all the way down to a doghouse.

So I take objection to the size argument. No matter the size, you should have a framework. Most confusion seems to stem from what is a framework and what is the data used by that framework. "Should I pass this object along or is the object itself supposed to handle the interaction?" style of questions. If you've never asked this kind of question, then OUCH!

The reason small projects work without frameworks is because there's very few things that interact in a N to N fashion.

he also builds a fort here:
http://blogs.construx.com/blogs/stevemcc/archive/2007/09/23/building-a-fort-lessons-in-software-estimation.aspx

i talk about that here:
http://tirandolineas.wordpress.com/2007/09/24/estimando-tiempos/

but is in Spanish, sorry

Excellent summary! I love Steve's doghouse summary and in today's world of RAD, small problems can put a big project behind

Wow, that article was bizarre. It sounds like his kids didn't even help at all. If my Dad had been building it then spending time with the kids and, depending on their age, teaching them some stuff about construction would have been the primary goal. Of course we kids preferred the dangerous platforms we nailed up to trees ourselves anyway. What kind of spiritless kid would want an adult approved one?

Maybe programmers SHOULD build the space shuttle? The actual software used for the shuttle is only 150,000 lines long. It's checked, Tested, integrated, checked and tested again. It's one of the most scrutinzed pieces of code in the history of programming and in 25 years there have only been 15 versions. As far as code goes it's almost perfect.
http://www.fastcompany.com/magazine/06/writestuff.html

His estimates were off because he had absolutely zero experience with the subject at hand. 10-15 minutes to clear brush?!?

And he was surprised that wood and concrete is heavy?!?

Without such basic knowledge, he didn't stand a chance.

And yes, before anyone asks, I garden and do DIY construction projects. Working on finishing the basement currently.

The examples of building a House is old news.

The really interesting part, is to consider that most houses built in world history never had any kind of architect, engineer etc...

Just the local builder.

The local builder knew how to build houses because he had served a long and painful apprenticeship. If we apply that knowledge to software engineering we come up with an interesting answer.

Most engineers employed today can not possibly be expected to build a project because they are to young. This is not as outrageous as it sounds, look around. Find out how long it takes to become a real lawyer, or surgeon.

Think about it.

If you walk into a new project, is anyone over 40 or under ?

As some one a wee bit older than many of your readers, I would like to point out that trying to get employment over 45 as a software engineer is very, very painful.

MODULAR TO THE RESCUE!... To build a entire house is alike to build a doghouse but if the project allow to, you can build many doghouse rather a big house.

The problems is not related with the SIZE of the project but the complexity.
A complex projects can take months only to determine a single task, in fact a complex projects will required to spend time (and resources) in research, sadly this research (and R&D) can be indefinite, even a seasoned programmers cannot determine a correct time for research.

For David Ginger: almost any > 40 year old guy must work currently in a programmer-less job just administrative, organization and such (for this age contacts and the net are more precious rather the number of certification) OR to be a good programmers (chief in projects) with a focused specialty (to be the best in a specific matter).

Also there always the choice to form a own business. A guy that worked 10 or more years programming without leveling up can be cataloged most likely a lousy programmer or simply a digitizer/transcriber, this own business cannot be exclusively a IT matters but any you like to, from to invest money, to own a farm.

As always, so very true. Even genuinely small projects (like that little helper application you wrote to make that task "quicker") can grow hopelessly out of control if you're not careful. The art of software development estimation just that. I rarely have much of an idea how long a project will actually take until I am some way through it, and even then, I'm sometimes wrong.

Excellent blog mate, really glad I found it.

I very much agree with what your article says. The size of the project is really important!

I will be saving this page to my favorites for sure...