I think the real problem lies in trying to convince people that the project already *has* failed. That it's hard to maintain, doesn't meet the functional requirements, and doesn't make the end users life any easier.

But are those the only signs that a software project has failed? Is shipping something, ANYTHING ,the only real criteria that means anything?

Even shipping may not be enough:
http://www.codinghorror.com/blog/archives/000773.html

I agree with the big three for sure, but I do think that an IDE (or lack thereof) can make a considerable impact on the day-to-day development. Consider writing C# code in Notepad versus Visual Studio. Yes, that might be an extreme example, but I've worked in environments where crappy editors become bumps in the road that slow down the daily work and increase the number of "stupid" error that are otherwise caught by the smart IDE editor. Color syntax highlighting, auto-formatting, things like refactoring method names, etc. Those things might be blips on the radar to some but having good tools makes the real "work" more efficient and less error-prone. I used vi, vim, gvim, nedit and SlickEdit when I did development for Unix and hands down SlickEdit won simply because I was able to let the smart editor clue me in on the stupid typos and such that I made. With Visual Studio, I save TONS of time being able to find references quickly, do some refactoring with a couple of clicks, build and see build errors within the editor, etc. Certainly having a great IDE isn't going to save a project that lacks one or more of the other items mentioned, but having one certainly can save a company money, particularly if the programmers are making use of the features it offers.

@Scott

Seems like keeping the customer (assuming a product is being built for a particular customer) in the loop during the development cycle would help to keep things on track better (provides an accountability 'partner' for the developers) and manages the customer's expectations. Far better in my estimation that the customer understands the problems early and what is being done to address them. Otherwise, if they aren't in the loop and I ship a buggy mess to them so that I can say I shipped it on time, they're definitely going to be upset that I was dishonest with them by letting them *believe* that everything was good.omer wanted to here - that everything was good.

Scott, I think there's a definite difference between shipping *something* and shipping something of quality...

Jeff,

One of the reasons I like Scrum and XP and other agile methods is that they turn a large project into many small ones (one month or less) and that allows stakeholders and team members to have a good chance of doing the important stuff successfully early on. That said, I also strongly agree that a good set of tools is critical. I've worked on projects where the tools got in the way (e.g. VSS, WebLogic) and on projects where the tools really helped (subversion, cruisecontrol). The difference even just in morale can be substantial! No wonder tools make such a big difference!

These three pillars play a large part in why i am curently leaving a company. With all the time I have spent making and updating spreadsheets on my progress... i coulda writen a stupid web UI with a sql express database that MADE spreadsheets for my boss. </rant>

I sometimes think that because failed projects are the norm people say, 'Why bother?'. I don't want to make crappy software that people hate to use and say 'Thank god we don't use THAT anymore.' after they throw it in the trashcan.

Thanks for your blog yo.

I think that today's software tools and IDE environments are at least part of the problem. There are a myriad of ways to do something wrong, too many new features in reasonable amount of time. I mean, just look at the tutorials that exist for simple CRUD apps in .NET.

Too many other things must occupy the mind of a developer and crowd out the necessity for solid algorithms.

Another huge factor is that most companies do not have managers that manage the right things.

Jeff, great suggestions for building a foundation for success.

An important next step toward success is risk management. Identify and face the toughest technical challenges first. Progress will feel slower because most of the easy tasks won't show progress, but completing the rough components is a better reality check for your anticipated schedule.

It's important to remember that sometimes we can do everything right and still fail (to produce the deliverable on time). In those cases, the best we can do is quickly identify the we won't be able to deliver in time. Manage expectations as soon as possible rather than dropping a bomb on the project sponsor in the 11th hour. Give the customer time to plan ahead and you might get the extra time you need to make the project a success.

Great post.

I think what this industry sorely needs is a return to the fundamentals, both at the coding level and the project management level. It's as if the pressure a customer puts on a project manager makes them forget all about the simple things that lead to success. Little things, you know, like making trade-offs between time, features, resources and quality, instead of promising all the above in a highly compressed schedule.

Deadline Driven Development (DDD) seems to be in fashion these days. Work-life balance is the first casualty. Project quality goes, too. Then, perhaps your best people after that.

Estimates and plans go out the window. Overtime is required and don't even think about not working on the weekends. lol.

There is hope. Folks like McConnell, who keep beating the drum over and over about how critical the fundamentals are, may have a ripple effect over a long period of time. It would also help if our high schools and colleges improved their courses in programming, comp sci and software engineering.

To answer the question posed - in all probability, yes your project will fail. People want everything, they want it cheap, they want it with high quality, and they want it all yesterday.

"So if the three most important factors determining the outcome of a software project are...

1. Project size
2. Kind of software being developed
3. Personnel factors "

I disagree with 2 out of 3 of your items. Here's my top 3:
1. Personnel factors
2. Money available
3. time available

Your resources - people, time, and money. If you have enough resources, you'll succeed. If you don't, you won't.

Determining ahead of time if you have enough resources is hard though.

The three pillars is a great set of tools to promote. I think it is the kind of thing a lot of us know, but which hasn't been formalized enough, yet, as something that makes the difference between good software and bad or non-existent software. It is nice to see the good parts of reality validated through a good article.

"Kind of software being developed" and "Personnel factors" are often out of the control of the developers and you have to work with what you have leaving only "Project size".

The three pillars you have mentioned are tools. Sort of like carpenters arguing which hammer is the best, there may be slight differences between them but there are larger gains to be found elsewhere.

I can't think of any instance of a past project that I have been involded in failing because we didn't use one of the tools mentioned.

What has worked for us:

1. Client Buy-in
2. Programming Development Standards
3. Iterative Development

Client Buy-in:
Have the Use Cases been developed with the client?
Are actual end users of the software working with the Devs and not some management type?
Has the client prioritized them?
Has the client walked through them?
Has the client signed them off?
Has the client worked with Demos/Prototypes?
Is the client using vague/abstract terms or defining the project more in terms of what they don't want instead of what they want?

Programming Development Standards
Have you created a glossary of terms? Does the word "Customer" mean the same thing to everyone on the team for example?
Is there a common naming strategy/convention used for the whole team?
Are there templates illustrating common operations such as function definitions/database access/error handling
Do the devs have access to common source code and utility functions used by each of the devs? Are they stored in a common area so your devs aren't re-inventing the wheel (repeatedly)?
Do you have a standardized testing standard/strategy?

Iterative Development
Are you delivering iterations of the software regularly?
Walking through them with the client?
Here is where you manage the project size by developing the parts of the application needed the most first, and add functionality with each iteration.

If the client isn't properly involved, the project is dead. If the customer isn't reviewing the use cases and baselines the project is dead.

If the team isn't speaking a common vocabulary the project is going to have a hard time integrating its work and probably end up dead. If there are no templates/standards developers can't be easily moved in and out and the project will have a hard hard time being completed.

Test cases allow you to test the interfaces of the designed classes before actually coding them. If you can't write a test case it’s a pretty good sign it’s a bad object. The ability to unit test all of the classes makes the software more resilient and easier to make large changes to the code base (I don't consider unit testing a tool here, you can test in many ways, just having a tester follow numbered steps on paper for example)

Monolithic development allows the team to spend years not delivering anything and the project will end up dieing. Iterations force the developers to deliver regularly and allow you to gauge forward progress.

Over reliance on tools would be part of the recipe for failure from my past experience. They can help to create a Single Point Of Failure in the system and things can grind to a halt if something goes wrong. They may be overly complex and result in the devs working around them or just ignoring them.

Sorry for being so long ....

"I suspect designing for the Web instead of around it is at least as important as language choice."

I think both are important. The best framework would be the one that made both things well supported, by allowing for direct Web programming and indirect Web programming while sharing libraries/tools/practices at the same time. Unobtrusive JavaScript and obtrusive JavaScript are two sides of the same coin, but where's such coin?

To answer the question posed by the title: Yes.

Hang on -- I see four pillars in that picture...

unless it's one of those three-D pictures where you cross your eyes... and then there's only three of them... and they appear to be floating in space and suddenly... WOW! A Sailboat!

Thanks Jeff!

lb

p.s. Also, I can't really reconcile "build system" as being up there in importance with the other items... maybe on some projects...

LB - I can reconcile it. Especially on projects with more than one developer. Consider the fact that with every developer you add to the team, you effectively add one new "build environment". The "It Works on my machine" effect starts to take hold. The build environment handles that.

I've written about this before: http://haacked.com/archive/2004/08/26/creating-a-sane-build-process.aspx

I'm currently in the situation where the build and dev environment is a pain to use. I can't get the unit tests to work on my machine because of the huge # of dependencies on things that only work on someone else's machine. This has put a hamper on my productivity as I'm still learning a new and very large codebase and would like the confidence that the unit tests would provide me that I'm not breaking anything obvious.

When you have a smooth working build system, it really does speed up development.

I will say that build engineering is probably the most challenging part of that trifecta. Work item tracking and even source control are fairly painless in comparison.

Build engineering is hard work. It's a stone's throw from real programming and requires substantial effort from at least one developer. And like all programming, it can be done badly-- so you have to be careful. That's been my experience, anyway.

Isn't it better if the sales department tried to convince the costumer that doing a quick feasibility study before the real project starts?
That way, you and the costumer, can actually see if the project is possible too carry out before too much money and time is spent?

Recommended book for managers is: Peopleware by Tom DeMarco and Timothy Lister.

If you read that book, I believe that you have much better chances to succeed with your software projects.

> I will say that build engineering is probably the most challenging
> part of that trifecta

Not all projects require a sophisticated solution, but on those that do, I agree that it is definitely another programming task.

I'm not sure I agree with Bill de h'Ora's 3 pillar theory though. There are more than 3 pillars, and in different projects different ones become more important. I'm sure many of us have seen insane methodologies bring down projects, for example, regardless of what tools were used.

There are indeed a lot of reasons why software fail.
Based on my experience you have to have the right balance between:

-Management
-Employee
-Client
It seems that a very important role in software success is the communication on all level's between these 3 factors.

Andreas
http://www.nueronic.com/

They didn't use an Agile programming methodology!!!!!

So, if we all use VisualStudio and TeamFoundation, we'll have all of our software projects succeed? I've got a bridge in Brooklyn I'd love to sell you.

There is only one and truely one thing you need to know in order to make sure your software project succeeds:

* Do you know what the hell you're doing?

Most of the time, the answer is "no". Most projects have wide general concepts like "Integrating our quote tracking system with the news engine", but not have no real clue on how they plan to do that. If they do succeed, it's because somewhere along the way, they've stumbled into knowing what they are doing.

Take a look at most of the open source projects. Most use CVS and no built in IDE. Many didn't even start out with any version control system before they came out with their first public release. However, the successful ones new what they wanted to accomplish, and had very specific ideas on how their project would work.

I found successful projects always start out fairly simple trying to get some very basic and standard features to work -- just enough to start being useful. As time goes on, more features are then added to the framework, but in an organized manner. At times, the whole project will be examined, and decide if it is still heading in the right direction. Sometimes, basic bits and pieces of the project are rewritten from scratch, but rarely in a whole general rewrite.

One of the best projects I've ever worked on was written in a proprietary language with no version control or defect tracking system. We had a logbook where problems were written down and tracked. The president of our company knew exactly what he wanted, we had clear specs of what was expected out of the code, and we had a team of dedicated testers banging the hell out of it.

One of the worst projects I've worked on spent over $20,000,000 on the whole Rational suite of tools (and this was for five developers and two QA testers). They had a dozen designers and database engineers, but no idea what they wanted. This company burned money as every two or three weeks, the project lurched from one idea to another.

Having top flight developers, extensive development environments, the most integrated tool suite, or massive amounts of money cannot do anything for you if you cannot explain exactly what you are planing to accomplish.

Steve:
>> Another huge factor is that most companies do not have managers that manage the right things.

The vast majority of companies (even some that do nothing but make software) don't understand two things:
1. Software development management is a highly skilled specialty, with a whole field of study devoted to it.
2. Most managers don't even know the issues involved, nor have they any training on dealing with them.

Most programmers are trained to produce code, not develop a product. So if you don't have a skilled manager to make sure other activities get done (documentation, testing, etc.) you are almost doomed to failure.

By the way, Jeff, another great article.

Steve:
>> Another huge factor is that most companies do not have managers that manage the right things.

The vast majority of companies (even some that do nothing but make software) don't understand two things:
1. Software development management is a highly skilled specialty, with a whole field of study devoted to it.
2. Most managers don't even know the issues involved, nor have they any training on dealing with them.

Most programmers are trained to produce code, not develop a product. So if you don't have a skilled manager to make sure other activities get done (documentation, testing, etc.) you are almost doomed to failure.

By the way, Jeff, another great article.

Although this post's topic matter has been covered well in the past, this is a good & concise reminder... Once again glad I watch Coding Horror.

Scott: "I think the real problem lies in trying to convince people that the project already *has* failed. That it's hard to maintain, doesn't meet the functional requirements, and doesn't make the end users life any easier."

Man, Scott that hits quite a cord w /me & my current situation. There are probably a lot more projects that your comment relates to than people want to admit. So, people should make sure they don't miss it:)

Jeff,

Most of your readers are probably aware that the mistakes that cripple a project are made at the beginning of the project.

The most deadly are the invisible decisions . . . The things that do not get debated such as the version control, build system, and IDE.

Every project should be unique, in which case standard solutions should be questioned. The last project I completed I choose to write a set of very simple tiny scripts that built the project.

One very simple benefit was reclaiming screen real estate by not using an IDE.

110% in agreement:

I've blogged about this topic a while ago:

http://rubako.com/2007/05/23/estimation-art-or-science

Tnx for nice blog ;)

Hmm... I think methodology is far more important than tool choice. Of course brain-dead choices in version control and work/bug tracking will cripple you - but there are plenty of brain dead chocies that will sink you.

In fact any good methodology will incorporate handling bugs and work items and will require version control - but they are really incidental factors in my experience.

In fact, a focus on the tools points to a lack of a methodoloy IMHO...

"the version control system is a first order effect on software, along with two others - the build system and the bugtracker."

Yup. That's exactly why I put the chapter on SCM first in "Practical Development Environments", and why that's where I start when I'm talking to my consulting clients. In fact, the back of my business card says "Version Tracking, Build Systems, Bug Trackers".

Sometimes I feel like the software equivalent of a good plumber :-)

>>>
I agree with the big three for sure, but I do think that an IDE (or lack thereof) can make a considerable impact on the day-to-day development. Consider writing C# code in Notepad versus Visual Studio.
<<<

That's less of a problem than most people might think. An average software engineer tends to spend only about 10% of his time writing code. So you better shoot at optimizing the other 90% (like testing, debugging, getting the requirements right, all that stuff).

For me, pencil and paper are the tools of choice. Once you get the paperwork done, writing the code is, hmm, well, just boring. You know it'll work. ;) Of course, I'd miss syntax highlighting, it catches a lot of stupid typos. But apart from that - it doesn't matter at all.

Or to put it bluntly: If you spend your whole working day at the computer staring at the code and putting a statement here and another there until you it somehow seems to work, you're doing it wrong, anyway.

"Consider writing C# code in Notepad versus Visual Studio. Yes, that might be an extreme example, but I've worked in environments where crappy editors become bumps in the road that slow down the daily work and increase the number of "stupid" error that are otherwise caught by the smart IDE editor."

Matthew, I agree - you should use the best environment you can. Aside from my bias that simplye says vcs/build/ are more important, there are two reasons I don't elevate IDEs, First because editing environments are emotionally important and so personal, trying to standardise them will inevitably put people out. Second, not standardising on an IDE prevents developers from allowing the IDE to trump the build system, something I believe is quite common.

"That's less of a problem than most people might think. An average software engineer tends to spend only about 10% of his time writing code. So you better shoot at optimizing the other 90% (like testing, debugging, getting the requirements right, all that stuff)."

Vinzent - so here's an antipattern I think is real - powerful editing environments prop up poor solutions. I think this is true especially in the Java space, where tools like IDEA are ridiculously good, not just at refactoring but at *browsing code* - code that would otherwise just be scrapped. Case in point - I believe Struts1 remains heavily deployed, despite its many flaws, in large part because it has widespread IDE support. I've even seen IDE support justify its use in a design. IMO, I need to be eternally vigilant that I'm not allowing a complex, toxic design to live on just because I have a rad suit (pun intended) that lets me live with it.

"I think is far more important than tool choice."

Michael, I doubt methodologies matter as much as we think - for example I'd say payment models trump methodology comfortably.

"In fact, a focus on the tools points to a lack of a methodoloy [sic] IMHO..."

In truth, I did think about methods (and IDEs when I wrote that) But my point was that if you don't have those 3 things nailed, you're hosed no matter what the method is. Methods assume the tooling fundamentals are in place.

I don't see why Build system is so important. Can you even have a bad one? (MSBuild, Ant are fine) To me, it sounds like if you said having mouse with scroll wheel is important. Of course it is, but is not worth mentioning.

"I believe Struts1/ASP.NET/WPF/WWF/BizTalk/J2EE remains heavily deployed, despite its many flaws, in large part because it has widespread IDE support. I've even seen IDE support justify its use in a design"

I adjusted that statement a little.

Inv: Think automated continuous integration build system, like CruiseControl and the like. And it's also very possible to have a decent build system (NAnt) but have it configured less than great.

Why waste all this time with analysis that's hardly ever necessary?

The question was "Will My Software Project Fail?"

Just answer "Yes", and you'll be right 90% of the time. This is a gold mine. Where else can you be right 90% of the time?

How did the thread get this long without someone using the word "scope"? David W. all but says it -- if the software development team and the consumer don't agree very early on what the darn thing is supposed to do, and what it's not supposed to do, project bloat and overruns will kill it. "Consumer", by the way, doesn't necessarily mean end-user -- it could be the internal customer, the sales team, the company president, but there must be some identifiable entity with whom the development team can make its agreement.

A lot of people have been commenting that the customer is important, and without customer input the project will fail.

This is true, but if you're developing a general product to be consumed by _many_ customers, it becomes difficult to know what is the right feature-set to develop, and to develop a product that will be acceptable to the target market.

This is where Product Management and friends step-in to research product requirements based on Marketing requirements. Now if the PM f*ck up because they don't know what features the customer really wants, the software product (not project) is dead. And it's easy to f*ck up primarily because it's hard to get a feel whether feature X is useful to 10 customers or 1000 (you want the latter because == $$$ revenue). And this is regardless of the fact of how the software product was developed.

What's worse than a dead-product is that Marketing decides that the company really needs the dead-product since they shipped it to 10 customers (and it looks good on a power point slide to partners). So now developers not only have to keep the dead-product alive, but have to maintain it forever on newer code-bases and keep it backwards compatible.

Large software projects fail for many reasons, but I will throw my $.02 cents in here. Part of the problem is that the bar for quality is binary: Works (1) doesn't work (0). Works really well (3?) doesn't exist. a 100 "else if else if else if" statements might work, but it clearly is not flexible and is difficult to maintain. Will you ever be given time to fix it? Nope. Why? "If it ain't broke, don't fix it?" (insert similar logic of choice).

The problems with software are institutional. Imagine if Intel released HW that had as many critical bugs as Windows? Intel would be sued broke. But, for some reason, there is no such storm for years of buggy software -- and not just from MS.

Another problem: Price. Releasing bugged HW is nearly impossible to remedy once it's out the door -- and requires a fortune when it has to happen. Software: "We can always issue a 'hot' fix"

Boils down to this: There is no incentive to produce good code other than your own pride.

For several years now, I am using a different fundamental approach to estimating software projects by targeting specific risks that plague most projects: choice of development tools, QA inflation, infrastructure inflation and red tape, policies applied to the project before project is even done ("speed bumps"), VM inflation (VM cuts the developer's productivity in half because it is horribly slow for server-side development), "big bang" releases (often, iterative releases look like "big bang" releases because of unrealistic expectations midway through the project... final thing to avoid is tasks inflation. Why does every development task have to be broken down to quarks (or even protons and electrons)? If you have two things to do that don't do much on their own, why not put them together in the first place and cut down QA inflation?

fxfibbin lives at gmail.

I'll simplify it for you: 78% of IT projects with a staff budget of less than $500k succeed, while only 3% of projects with a budget of greater than $10M succeed. Keep your project small, dividing into distinct sub-projects if needed.

so, what bug tracker you use anyways?

I'm looking for live whole project flow... Kindly help me on this....

I'm looking for live whole project flow... Kindly help me on this....
I want to deploy on this to my system....

I'm looking for live whole project flow... Kindly help me on this....
I want to deploy on this to my system....