July 18, 2009
I was utterly floored when I read this new IEEE article by Tom DeMarco (pdf). See if you can tell why.
My early metrics book, Controlling Software Projects: Management, Measurement, and Estimates , played a role in the way many budding software engineers quantified work and planned their projects. In my reflective mood, I'm wondering, was its advice correct at the time, is it still relevant, and do I still believe that metrics are a must for any successful software development effort? My answers are no, no, and no.
I'm gradually coming to the conclusion that software engineering is an idea whose time has come and gone.
Software development is and always will be somewhat experimental. The actual software construction isn't necessarily experimental, but its conception is. And this is where our focus ought to be. It's where our focus always ought to have been.
If your head just exploded, don't be alarmed. Mine did too. To somewhat reduce the migraine headache you might now be experiencing from reading the above summary, I highly recommend scanning the entire two page article pdf.
Tom DeMarco is one of the most deeply respected authority figures in the software industry, having coauthored the brilliant and seminal Peopleware as well as many other near-classic software project management books like Waltzing With Bears. For a guy of Tom's caliber, experience, and influence to come out and just flat out say that Software Engineering is Dead …
… well, as Keanu Reeves once said, whoa.
That's kind of a big deal. It's scary.
And yet, it's also a release. It's as if a crushing weight has been lifted from my chest. I can publicly acknowledge what I've slowly, gradually realized over the last 5 to 10 years of my career as a software developer: what we do is craftsmanship, not engineering. And I can say this proudly, unashamedly, with nary a shred of self-doubt.
I think Joel Spolsky, my business partner, recently had a similar epiphany. He wrote about it in How Hard Could It Be?: The Unproven Path:
I have pretty deeply held ideas about how to develop software, but I mostly kept them to myself. That turned out to be a good thing, because as the organization took shape, nearly all these principles were abandoned.
As for what this all means, I'm still trying to figure that out. I abandoned seven long-held principles about business and software engineering, and nothing terrible happened. Have I been too cautious in the past? Perhaps I was willing to be a little reckless because this was just a side project for me and not my main business. The experience is certainly a useful reminder that it's OK to throw caution to the wind when you're building something completely new and have no idea where it's going to take you.
Yes, I could add a lot of defensive software engineering caveats here about the particulars of the software project you're working on: its type (mission critical, of course), its size (Google scale, naturally), the audience (millions of daily users, obviously), and so forth.
But I'm not going to do that.
What DeMarco seems to be saying – and, at least, what I am definitely saying – is that control is ultimately illusory on software development projects. If you want to move your project forward, the only reliable way to do that is to cultivate a deep sense of software craftsmanship and professionalism around it.
The guys and gals who show up every day eager to hone their craft, who are passionate about building stuff that matters to them, and perhaps in some small way, to the rest of the world – those are the people and projects that will ultimately succeed.
Everything else is just noise.
Posted by Jeff Atwood
Software is almost an organic process, how many times have we seen huge monolithic software which over the years has become completely unmanageable, yet companies still persist in the "just add another feature" policy! :D and who'se responsible for that.... I point you in the direction of the Marketing department!....
( and why do some software "craftsmen" treat software like the computing equivilent of scrap-heap challenge! )...
Quoting David W:
'If I am a structural engineer, and I design a bridge, and its being built, no one is going to come to me in the middle and say We changed our mind, we decided it should go from point "A" to point "C" instead of point "A" to point "B". Or, we really need a bike path on it. Or, it appears that our traffic projections are wrong, we need it to be six lanes and not four lanes.'
You think this doesn't happen? Of course it does! And then the customer complains that YOU didn't think of adding a bike path, or going from 'A' to 'C'. A good civil engineering craftsman has learned how to deal with it. And so has a good software engineering craftsman.
So we are not Engineers.
Architects? Come on..
Being a Developer is quite lame too.
Unless you have concrete evidence that most of the people buy from farmers and not from supermarkets, I would still believe that software engineering is almost extinct, giving way to mass-manufacturing process.
The economies of scale that drive real-world mass production simply don't exist for software. Mass production historically required orders of hundreds of thousands of units (muskets during the civil war) to get started. Even with hardware engineering techniques refined from hundreds of years of experience, and it being the default way engineers are taught, businesses producing below a few hundred units annually are generally based on skilled craftsmen:
There is no software company in the world that produces for sale ten thousand units of software a year. A few organisations (USAF, NASA, NHS) approach that scale for internal use, and might just hit the break-even point for mass-manufacturing techniques.
To summarize a bit of the better-thought-out above comment mess...
1)Engineering is about using a messy situation and producing a working design, to tolerance. This involves both creativity and correct practice.
2)Software is more abstract and has almost no correct practice(only better/worse). This is probably caused by the newness of the discipline.
3)Engineering is partly about the process as well as the work itself.
4)DeMarco was flaming the project bean-counter mindset, not the idea of rigorously constructing a project.
That's much less flame-baity than Jeff....
Federico is completely correct that programming is a design activity, not a building one. We have totally outsourced the manufacturing of our programs to our toolchain: compiler, linker, loader. We put in a detailed blueprint of what we want them to build and our tools assemble a flat-pack of components and a set of instructions that, when they get to our customers' systems, are assembled by the loader and the OS into one or more running processes. We have performed this automation so successfully that we do not even recognize it as the manufacturing activity. Instead we look at the last level of human involvement and decide it must be manufacturing, then wonder why it is not predictable.
Once we have successfully solved a problem once, it tends to go into a library of well-tested code, usually with parameters to control its behaviour. Would that a civil engineer had a self-assembing suspension bridge that could place its own footings! If we have solved a problem once, we often don't need to re-solve it again unless our original solution doesn't quite meet the criteria that we had before. This is where software engineering tells us to make the minimal changes to fit the new problem while remaining compatible with the old one; craftsmanship might instead tell us to design a new component that's incompatible.
In tools like the .NET Framework and our operating systems, we have vast libraries of solutions to our problems, but as people who want to believe that we are making a real contribution, we will tend to decide that such components somehow aren't quite right for our problem and waste time producing something custom. Software engineering is the discipline to resist this temptation.
If a software engineer is sufficiently experienced to know the wide range of tools at his disposal and to understand the problem sufficiently well to know exactly what to build, it is possible to predict and control the activity of designing it. If it's that well known, though, it has probably already been designed and built.
Engineers we can be, craftsmen (or craftswomen) we might but we are definitely not artists.
Anyone who says otherwise is a retard.
Let's not over emphasis our creative abilities to the extent that we believe the programs we create are works of art. Clever and sophisticated they may be but more than that they ain't Mr Knuckle head.
No one is going to hang a print out of your code from the Sistine Chapel, a polymorphic composition of 2 overloaded methods that converge in the middle touching at the angled brackets. Art transcends nature revealing hidden mysteries, software programs do not.
Those who work on projects with other programmers are more like engineers following standards and applying acceptable practices that others can emulate and understand. Those tinkering with code on the side or with full autonomy are free to ply their trade as a craft and enjoy the thrill of the chase.
But artists we are not. Did you get that yet Mr Knuckle head.
Easily one of the more dreadful articles about navel gazing.. sorry I mean software engineering I've ever read
iterative : yeah. (or recursive it depends)
artistic :someday (algorithms and maths are arts, rights?.. I believe so).
+ Art transcends nature revealing hidden mysteries ; s/Art/Computer Science/g
so maybe I am retarded...
but I agree on some points to. maybe there is art out there, but first we think, project, milestones, specifications and productivity..
Computer science is mathematics.
Programming is the act of writing programs.
Computer science != programming. It's maths.
Software development != programming != software engineering.
Software engineering is the engineering field of creating software. For example, a software engineer creates control systems for nuclear reactors. A software developer is just someone who develops arbitrary software, like some payroll app. Both program.
I'd also say that programming is not art because software can be demonstrably wrong. Art can be good or bad, but cannot be wrong - a novel doesn't stop working just because of a misplaced semicolon.
I think many commenters are missing the fact that software development is an incredibly broad discipline. I would never dream of calling the process I follow as 'engineering,' but I am not writing sorting libraries or mathematical simulations - indeed if I were to require that functionality, I'd look for a library of pre-written pre-tested code instead.
What others are confused about is that they don't realise that other types of engineering are not perfect. Developing a new component for an aeroplane requires a significant amount of iteration, both in the design and development stages. Classical engineering is fortunate to have decades more 'patterns and practices' to build upon, but many of the fundamental issues are shared with software development. I also recognise that of the hundreds or thousands of engineers involved in building a large-scale project, such as a new bridge, they will have many different roles (at many levels of abstraction) - just like software devs specialise in a particular area.
What is also apparent is that there is a massive difference between the level of abstraction in a project and how it is perceived. Sure, working on a driver for a realtime embedded system may be regarded as engineering, but me contributing to my team's web-based application is not the same (and I wouldn't regard either piece of work as less worthy than the other). There seems to be a tendency for some people to think in black and white terms here, which is a shame. I feel a little aggrieved when some people above regard us as arrogant or lazy when we say we're not strictly engineers - the fact is that there is definitely a creative aspect to many parts of programming, and just because we're not writing fourier transforms doesn't mean that we're not worthy of being in the industry.
As for the original article, Jeff's guilty once again of over-egging the pudding. And what happened to a bit of research? This argument has been raging for years and there's much prior art.
I have to agree with Dave Markle and others. Maybe not everything all software developers, including some who call themselves software engineers, is engineering. But what I do certainly is, and I've seen others do it in many situations.
And I am actually a "real" engineer originally. Not a software engineer, not a computer engineer or an electrical engineer. I dealt with physics and materials and stresses and so forth. And then I developed software, and of course it's different, but it absolutely is still engineering that I'm doing.
Sadly, much of the audience here assumes that large projects contain only highly skilled and motivated software developers. Eager to produce, despite the efforts of idiot managers to curtail them. Sadly, that just isn't the case. The majority are ordinary, unmotivated and uninterested in really improving their skills. The audience reading this is simply unrepresentative of the real world
I don't like his "one-week" horizon - it means you can never do anything that might take more than a week to complete!
A major issue is that web development has been around for "only" 10 years or so. And yet there are millions of developers.
That is a formula for a lot of issues.
In the document, Tom DeMarco defines software engineering:
I still believe it makes excellent sense to engineer software. But that isn’t exactly what software engineering has come to mean. The term encompasses a specific set of disciplines ... All these strive for consistency of practice and predictability. Consistency and predictability are still desirable, but they haven’t ever been the most important things.
I'm not sure when "engineer" got co-opted by MBA types to make their organizations sound smarter.
To be an "engineer", of any kind- in college you took; chemistry with chemists, physics with physicists, differential equations, linear algebra and buckets of calculus with mathematicians. You took your actual engineering classes towards the end, and you did most of it by hand so you can effectively eyeball an answer, and can tell when your model is lying to you.
Being able to think logically and solve problems is something some second graders can do, it does not qualify you to be an "engineer."
Engineering in other fields is tied to the scientific limits and equations that are developed for the materials being used. There are certain Physics involved that are STANDARD across the board. There are only a few ways you can do things successful in those engineering fields.
With Intellectual property rights and abundant creativity, there is no STANDARD set of libraries that everyone uses, there is no STANDARD OS that everyone uses, software engineering is a misnomer. It is still the same craft of ideas you see on Hack-a-day or any other site that will show you new innovative ways to re-use or remake something from things you never thought about.
Software development is still the same CRAFT as when I was 6 years old with a TI-994a banging out BASIC code to cassette tapes. You have an idea and passion, you make it happen. Just like society in general, once you start adding rules and government it takes the FUN out of it and strips the people of their passion unless they are WISE enough to know how those rules will benefit them in the long run.
Essentially the commercialization of knowledge has yes expanded it to an unreal extent, but has debased it as well to the lowest common denominator.
Recursive division has that effect.
Most architectural engineers never build the same house twice. That doesn't mean what they do isn't engineering; the underlying physical rules remain the same, regardless of what house they build. The fact that we haven't yet had thousands of years of knowledge to figure out what exactly these rules are doesn't mean what we do has to be a craft.
Saying "well, it's not engineering, it's a craft" is the cheap way out. Sure it's a "release", like saying "I don't have to worry about my life anymore, I'll just believe that god will guide me" is a "release."
Agreeing with others that say this debate is pointless. As an electrical engineer by training, we just have, oh, about a hundred years of experience over this brand new software development thing. So we have very specific design specs that need to be met, very specific testing requirements based on the laws of physics (is this lumped capacitance going to affect things? is this electric field too high in practice and going to interfere with this antenna? is this control system sensitive enough?) I think that we are much more able to automated quantitative tests than software guys, to be able to give very accurate assessments of the quality of the device in question.
Does that demean or make software development any less intellectually challenging? Hell no. Does that make it "engineering"? Who cares, semantics.
Also you'd be absolutely fooling yourself if you think that other engineers don't have fickle clients that change specs at the last minute. Imagine how much worse it'd be when designing an engine or an integrated circuit.
That's the business, engineers have been dealing with that for decades, it's nothing new. You don't see too many of them inanely blogging about every little design mod they have to make...
I'm justing adding to the long line of "hear, hear"s on this post.
As somebody who took Software Engineering in university, I realized this a long time ago. Firstly, that nobody wanted to pay the money, and take the time to do a properly "engineered" piece of software, save for NASA and a couple other organizations. And that even if you did all the engineering process stuff, it doesn't help when you have a bunch of meatheads doing the coding. People make the mistake all to often of equating coding with assembly level or construction work. They think, if you put enough controls in place, that somebody with minimal amount of training can create good software. This doesn't happen. What you have to realize instead, is that each person doing the coding, is doing their own design, their own "engineering". You have to have really smart people doing the coding. I think this is pretty much the road that Joel has taken with Fog Creek. Hire really smart people, and you don't really even need so much process, because really smart people who get things done will figure it out. Same reason that Jeff, Jared and Geoff were able to create Stack Overflow in a few months, and up with a good product despite not having that much unit testing, and not really even having a project plan. Saying that programming is a craft is very true. Throw 100 woodworkers together and get them to design a nice hutch and you will never get a good product. But give a good craftsman 1 month, and you will probably have a really nice end product.
I gotta say I disagree that Software Engineering is dead. The classic processes are dead. But, I've found a difference between a programmer, doing development, and engineering a product.
Clean consistent APIs that make sense and use good patterns are engineering.
Do people actually agree on what the definition of "software engineering" is?
Just because one programs, one is not an artist.
Programming many times is a "McGyver" experience: you have some tools, a goal, a time frame -- see what you can toss together.
Before making grand pronouncements you guys should read a bit about other engineering disciplines. I recommend you start with Henry Petroski's "To Engineer is Human".
What we call "hacking" in software, happens in all other engineering disciplines. Sometimes because the science is wrong, sometimes because the requirements are missing, sometimes because there is no science (eg. flood control and civil engineering).
Engineering is the craft and the art of designing useful artifacts that do not fall down....
I agree with Aaron G. DeMarco seems not to be against engineering but the larger issue of how software projects are managed.
In an interesting article from Computerworld a year or two back, there was a write-up on think tanks like DARPA. One of the principals they interviewed who'd been there when they did so many great things in the 60's said that today DARPA has changed and would be less likely to achieve what it did then. Some of his opinion might be egotistical self-promotion for the past achievements of his team, but I found it interesting that he called out the management, not the newer generation of workers. He specifically referred to the leadership there being not engineers, but project management people who have a focus on the short-term delivery; he felt this would not allow the environment to encourage the same developments the teams in the past delivered. A lot of great things happened when engineers were allowed to spend a lot of time just "thinking" and he's got a point; most project management types would not be happy to see a room full of people not delivering the latest status report.
I think this is what DeMarco is referring to when comparing the two projects in his article: one is heavily project managed and only delivers the small, containable, measurable, justifiable thing, while the other was less project managed and happens when people think big and projects aren't just canceled because they didn't deliver anything in the first 30 days.
That said, I also got a laugh out of Rod's responses and thought he had some good questions in the thread above.
Great writeup :)
I've been advocating for years that SW development is *art* and not "engineering", but I think I'll from now on actually start adopting "craftsmanship" instead in fact. I think that word more accurately sums it up in fact ... :)
Saying that software engineering is dead for the reasons stated here, is akin to saying that literature is dead because everyone and their dog has a blog that is, largely, a piece of chit.
Believe it or not... but there's still people out there who know what they're doing. (No, I'm not necessarily one of them)
I think you hit on the key, Jeff...
"The guys and gals who show up every day eager to hone their craft, who are passionate about building stuff that matters to them, and perhaps in some small way, to the rest of the world -- those are the people and projects that will ultimately succeed."
If you have a solid team in place with a culture of support for team goals and teammates, the debate over art versus engineering becomes moot.
If developing software is craftsmanship, then the boat is sinking. It seems like this is an excuse for an industry that has been slipping for years. I have been in technology for almost 20 years. I remember working next to Digital developers and being in awe at the level of ability, intelligence, and care. Today I see software as being slapped together. The tools that do so much for developers seem to limit their ability to solve problems.
Maybe this is because I work in Health care, the software that is created for Health care never mentions up to some of the stuff I saw come out in my early days. Lack of error checking, dependence on pre-compiled wasteful code, and a total waste of resources.
How about we cut through the crud and give ourselves new names that have no historic categorisation and therefore we can define what these new titles by our actions and profession.
Software Developer: Ni
Software Engineer: Peng
Software Architect: Nerwong
Database Administrator: Ekiekiekiekifertangzerbongmfrm
or something like that...
Mostly I agree with @Federico that software development is a design, not *manufacturing* activity.
When I hear people say "why can't programming be like building a bridge" they aren't talking about whether building software is like civil engineering. They mean that programmers should be able to use business specification as a blueprint and turn that into working software a manufacturing process. I guess in civil engineering this would be like they mayor saying that he wants a new bridge and then going out and hiring a construction company to build it, and then expect it to come in under budget, without first having an engineer do all the technical drawings. Then again I wouldn't put that past the current mayor of my city.
I have always believed that the closest craft to programming is poetry(not the free verse crap). there are definite rules to follow but the writer must figure out a way to piece the language together so that it both meets the rules for the type of poem, makes logical sense and is pleasing to to the reader.
In the last few years we have been seeing right brained designers and left brained developers and a manager trying to get the two types to work together. Good programmers have the ability to use both the left and right hemispheres of their brain.
Maybe it is, and certainly now that companies try to conserve money, it's basically only used as guidelines because the actual programmer has to link the ends. And sometimes it's used for software documentation but that's it basically
There are more moving parts in the average SQL Query than in the Big Dig. Computer programming relies heavily on automation to assemble these moving parts into a working whole, usually with 1% of the time and .5% of the budget of a comparable (in complexity) mechanical engineering project.
This doesn't leave a lot of room for getting things perfectly right, if that's even possible. The body of running code out there could be humanity's greatest achievement, an 8th wonder of the world. Don't let it go to your heads, though :)
The first version that does what you want is done through craftmanship. The second version built to fix all the maintenance problems is done thorugh software engineering.
I'm pretty sure that for the first few hundred (thousand?) years, building bridges, large buildings, cities, etc... was all just as much an art as programming is now.
Engineering has come to mean formulas and reliability because it's been done so much that it's predictable--just about every variation has either been tried or can be understood based on mountains of existing data.
Programming won't always be like it is now. I'll leave it as an individual exercise to imagine what it will be like, that's outside the range of my crystal ball.
Giving up on being able to reliably build things is a huge step backwards. Sure, it's popular and all the rage, but we're never going to exploit the potential of our hardware unless we admit that consistency requires discipline and monitoring, which is pretty much the opposite of making it a free-for-all. We know, and have known for a long time, that a massive chunk of unstructured code is nearly worthless. Even if it runs, it's impossible to extend; it's at the end of it's life. The only way to add structure is to actually build it in initially or add it later. Either way, we have to know what good structure is first, which requires a design. Accidential structure is unlikely.
One of the differences between software engineering and other forms of engineering is that software engineering rules get codified into tools. For example, Apple's development tools force you into using a Model-View-Control design. Ruby on Rails does the same, as well as enforcing DRY (Don't Repeat Yourself), and encouraging you to use a database rather than arbitrary files for back-end storage. Java developers using Eclipse get bad code style flagged as you type.
In contrast, a mechanical engineer needs to know the rules up front, and then test each design against those rules.
I believe their should be a distinction between what software development should be and what software development is in reality.
I still believe that software development should be a combination of craftsmanship, art and engineering.
In reality it is unfortunately often none of the above.
And people wonder why Microsoft said it will launch Windows 7 when "it is done", and their primary goal this time is to do it right...
Don't forget: Tom DeMarco is in the business of selling books.
Cultivating that profession since 20 years, I have intuitively come to the same conclusion, but the meaning of that isn't as liberating as you may think.
What pisses me off is: anyone can start as a programmer, and be hired as such, without ever having served as a journeyman under one or several master craftsman, without being member of a guild, without any third party authenticated proof of his experience.
It is the typical wild west situation on a new frontier, and it is my conviction that only a proper regulation akin to the one associated with traditional craftsmanship professions can bring an unprecedented improvement in quality on all levels of software development.
Don't panic. Software engineering and software project management are two different things. DeMarco complains about the second one.
My father is a retired structural engineer. He worked for a few large companies and lectured, but spent his last few working years at a small consultancy.
When I was a kid we built meccano together, and I can remember him working out how to get something I was trying to do to work, and saying that engineers are the people who work out how to make people's ideas real.
Most of the work he did as a consultant was talking to the clients ( "requirements capture" ) and making sure that the builders built the right thing ( "validation" ). I can't think of a single use of metrics in his engineering practice, which is what the 'no, no, and no' of the article relates to.
My career went through electronics to software ( particularly modelling of hybrid systems ). I've worked a bit in safety critical circles; the commenters who say that software doesn't have design codes simply like physical engineering doesn't work on anything that will kill people - off hand I can think of def.stan., JAA, AEC and MISRA standards for critical systems.
Fundementally, the difference between engineering and craft is that engineering artifacts are products of societies and crafts are products of individuals. You have teams of hundreds of engineers on typical systems engineering projects, some of who are software engineers. Even for small projects, you need to apply the engineering principles of breaking down a system into subsystems by purpose, function, structure and behaviour in order to reduce it to a tractable problem.
Sorry, but translating an article saying metrics only matters on unimportant projects to rejecting all engineering is bollocks. Much of what you've done on SO is engineering - making an idea real and habitable.
I think it all depends on the maturity of the craft itself. Sure, if you do someting for the first time, it's craftsmanship as you haven't done it before therefore you need to be inventive, etc.
Just as in the case of the first cars - see, there was this amazing variety of solutions for the same problem. Cars had any number of wheels from three to like ten, more types of engines you can imagine, etc. After a while, patterns started to emerge (the McPherson suspension, the Diesel engine, whatnot). Patterns are easier to use and more beneficial to refine than starting from scratch.
Fast forward to today: all the cars look the same. (still it takes 4-6 years for a new model to get out of the gates)
I see similar things in software: you have databases so you don't need to write one. You have compilers, toolchains, standard libs, whatever. You don't need to do everything from the ground up. Sure, there is still plenty of work to do, but a good percentage of code in probably any given software is not written by your software team (or yourself).
Sure, software is inherently easier to develop and distribute, so there is a greater diversity. However, there ARE some standards (*ML, SQL, just to name a few) and best practices you can follow. Testing is not as thorough (this is why all sw comes with a long EULA - you can't get away with a serious problem in case of a car this easily...)
However, engineers do break out of the rules and formulas sometimes - one calls it innovation. The rest is engineering.
What I seem to see in a lot of code of clients is if it is not embeded machine code, safety critical or on a huge project like government etc then the code is bloated, full of quick hacks and generally a nightmare to read never mind work with.
So I think in smaller projects there is less care for the engineering involved.
If you had a magical replicator that could make copies of any design for free how would that change your process to make a car?
That's software engineering in a nutshell.
It's not that software engineering doesn't exist it's just that we haven't even come close to figuring it out. The lack of physicality of software is both liberating and baffling.
Programming is intrinsically meta, because programs are a form of data. This means whenever you have some non-creative programming process, you can write a bit of software to automate it. Thus you tend to be left with just the creative parts. That's why it feels especially "artistic".
Normal engineering has the creative parts, but they also tend to have more non-creative parts. You can automate a factory with robots, but if you are building a bridge out in the field, you pretty much need a bunch of men with welding kits. That's hard to automate. From a distance that kind of labour can mask the creative side.
Izkata: Right now, I'm 21, and have been programming since I was 13 or 14. I'll be finishing a BS in CS within a year, and I've been saying for almost that entire time [...]
That's because, until now, 100% of your programming has been done by yourself, in your bedroom.
I think this turned out to a pointless discussion of what it means being an engineer or a craftsman.
If this discussion doesn't bring any improvement to the field, so we're all losing time reading and answering all of this.
To me, building a software is constructing something that does what it meant to do. But I can build a wooden brigde, a stone one, or even a steel, or I dont know. The point is, it's going to work ? Yep ? So let my creativity run free. (as long as the project doensn't suffer.)
If you are reading this article, you are part of the minority that care genuinely about their craft. But that isn't the majority of programmers.
On a real project, you won't get all burning, committed craftsmen. You won't even get that as a majority. And it is easy to poison a group.
You won't get an open ticket to innovate/create... at least not forever. You won't get turned on to, "go build the Taj Mahal"... You'll get turned on to build a new community center in Podunk.
So how are you going to energize your programming Narcissists when it's just a community center?
"Why is software any different from hardware engineering?"
As an actual engineer (Aerospace) I'll answer your question:
Physical solutions are built upon immutable knowledge - the tensile strength of steel, the conductivity of copper. There is nothing immutable about software.
Dave Markle: "Big-O notation? Who cares, we're artists! Rules engines? I will craft the most beautifully indented set of nested IFs! Knowing how a B-tree works so I can understand WTF my database indexes actually do an then intelligently apply them? Sounds like nonsense to me. Why not just add some more indexes until it gets fast!?"
Craftsmanship doesn't just mean artistry, and even if it did, artistry doesn't mean "arranging things in a pleasant way". Being able to work out what to implement, which parts should be involved and how these parts should function internally and externally is exactly what programming as software craftsmanship is. It's about using your knowledge of programming and of the situation at hand to create a good solution. There are tons of solutions for most problems, and those that work better than others are both finer crafted and better engineered; as far as I'm concerned, it's the same thing.
The reason it sounds like nonsense to you is because no one actually said that. It was your straw man. Don't make the mistake of tying craftsmanship or artistry to "beauty" or "usability" or "fanciness", or engineering to "performance characteristics". They are tools to be applied across all facets of the design, not handles by which you wage ridicule at poor solutions.
If you think of the "craftsman" thing in terms of a specific craft - clockmaking - it becomes even more interesting.
Imagine the difference between designing a lot of "clock parts" that are engineered to be beautiful, precise and interchangeable, and buying these parts to build a unique clock with a beautiful hand-carved body, hand-calligraphed face, etc.
Most of us are the clock-making artisans. Many of us still need the true engineers to make the precise, interchangeable parts. We understand what an escapement is and how it works, but we couldn't build a respectable one to save our own lives. We can use it well, though.
Sad to see an obviously highly regarded person come up with this revised view. I think he has left the real world.
The real world (where I am a project manager that needs IT now and then) has only so many chances to build Google Earths or Wikipedias - both of which have started as free/not-for-profit apps.
It also consists of many businesses that try to make money by improving/evolving through "relatively useless [marginal] projects", as in mature businesses there are not many "tranformation" break-throughs.
The real world also changes not that often.
The whole world is talking about continuous improvement and kaizen, but DeMarco thinks it's useless?
Agile is a great approach, but the inertia and ignorance in the IT world is also great - try and get a system built where customer focus and involvement are paramount and the system works incrementally from early on. Many IT companies still love the Big Bang.
DeMarco is coming up with a similarity of the lamest excuse in the book: you can't standardise MY work, I am an artist.
See what crappy, ever-changing interfaces these artists build! Try and train users to work with several different behaving apps. You artists are consistently reducing efficiency.
I think DeMarco's lost his marbles. Maybe we need to write a wonderful product called MarbleSearch!
Smells like beardy sandal spirit.
Thanks Jeff for your great site and work over the years!
OK, SE is supposedly dead. But the notion that we should just throw up our hands and despair of ever managing programmers, while perhaps attractive to certain workers, is ridiculous. Craftsmen are easily managed.
If coders are craftsmen, then we have ample precedent for how to "make" them. It's the same as any other building trade. Send them to school for a few months to learn a bit of theory, then apprentice them to a master who teaches them how to do things in the real world. Eventually they become a journeyman capable of working on routine stuff without immediate supervision; after many years, they become masters themselves capable of supervising the work of others. This system has been around for centuries and it works quite well at producing skilled craftsmen.
Lately, even in the building trades the process has become short-circuited. In the interest of cost-cutting, contractors have started using labor that, to put it charitably, is less than properly trained. Sadly, this works because most Americans care more about getting the lowest possible bid than getting quality work. I see the effects all the time in the residential and landscaping fields. I shudder to think what civil structures might be built with such labor. *Most* software is built with similarly poorly-trained programmers. No wonder most software is so flaky.
Programmers are fundamentally no different than carpenters or other skilled tradesman. The fundamental problem with "software engineering" is that we try to have the carpenters also be the architects and the civil engineers and the plumbers and everything else. This may work for simple projects, just as a skilled carpenter can probably adequately design and build a typical house that involves no unusual features. But putting an ironworker in charge of architecting a skyscraper would be madness. Unfortunately, this is exactly what happens in most software projects.
The central problem with software is not that it involves craftsmanship. Nor does this fact disqualify it from being an engineering discipline Nearly every engineered system involves craftsmanship at some point in its construction. Tin must be bent. The problem is our routine conflation of craft roles with engineering roles.
Finally someone said out loud. I've been always wondered why the cs department belongs to the engineering faculties if this shares few things in common with another engineering, and so it does with sciences. Well, thanks jeff
It's fine to just wing it sometimes when building apps, especially when it's a new idea and you aren't even sure exactly what it will become.
But I also think software engineering principles, designs, documentation, requirements, use case libraries, etc. are very important when they are used to build the foundations that all of us rely on (the .NET framework, or an OS for example).
I always considered a large part of SE way out of my picture for reasons as:
1- I feel that most, if not all, concepts after The mythical man month is academic. It is sometimes useful, but not even closely as they want me to believe.
2- The project schedules and budgets I have learnt to deal with does not support even 1/10 of the effort required to write the docs, test the app, review it and such as they tell me I should.
3- I know for sure my competitors don't use them at all, yet they win over us.
I know (3) for sure because a customer choosen a different studio to go for a SW he needed. After a few weeks he came back here to ask for a technical assessment.
In retrospect, we should have charged him way more.
As seen before nobody had a benefit from reading blog posts. In most cases it costs the reader time which we all know is money. The only people getting money for this are bloggers and merketing weasels.
I don't want to participate in the argument over semantics here because it seems trivial and could go on ad infinitum.
However, as a side note from a global perspective, in many countries, you are legally restricted from using the word "engineer" or "architect" in your title unless you have a very specific degree or license.
I had to cringe when someone of the baby-boomer generation recently referred to me as an engineer. It just sounded so ... wrong. As a Web applications developer, I consider myself more of an artist or something more poetic. "Engineer" just sounds so stuffy.
Read Object Thinking by David West (Microsoft Press) if you haven't already.
Seriously opened my mind on this matter.
> Normal engineering has the creative parts, but they also tend to have more non-creative parts. You can automate a factory with robots, but if you are building a bridge out in the field, you pretty much need a bunch of men with welding kits. That's hard to automate. From a distance that kind of labour can mask the creative side.
In the UK, welders, bricklayers and other skilled technicians are called craftsmen. It's a different role to an engineer, and does not denote lack of science. There's a different argument between 'you need some craftsmen to build it' - a hell of a lot of software projects are made by welding existing components together - and 'software engineering is dead'.
Damn straight. I'm an artist. My medium is code. Have been for 32 years.
The word engineer comes from the Latin 'ingenium' meaning 'Cleverness'. As is the word 'ingenious' (marked by originality, resourcefulness, and cleverness in conception or execution - Merriam Webster)
I would argue that software engineer is the most appropriate title for what we do, and it is other types of engineer that are mis-labelled
I don't agree - programming in the higher level very often is engineering, in the sense that one applies science to solve problems.
If the issue is that programming projects often are messy, I don't at all see how that makes it differ from other engineering areas.
Sadly I have to agree with "Sales and Marketing". The big problem with programmers is that we think businesspeople actually CARE about "engineering" or "design principles" or all of that good stuff that we repeatedly say is the "right" way to develop software, when in fact ALL that matters is to get working software that meets the business' needs out and into the hands of the people who need it.
This is why a hack who writes spaghetti code is infinitely better than the "software engineer" who takes his good sweet time making sure everything is properly decoupled, and implements the right design pattern; the hack's code is ugly and inefficient, but it works and it's completed fast.
The sad truth is that software exists as a supplement to existing business functions, and all this talk about "craftsmanship" ignores the real goal.
I agree with the idea that software design is experimental. but this doesn't imply that you can control it. What it is needed is a different control approach, more similar to the one used in scientific research: propose an hypothesis, define an experiment, test and measure, evaluate results and go back to hypothesis.
Software engineering is not dead, it's just waiting for the next step.
When the software engineers itself.
Don't think I'm way out there in the whacked out AI world.
We are approaching sufficient horsepower where self modifying and adaptive code is goiing to become a reality. And not just the little 'toy' examples we've come up with so far.
I just hope it does not knock on my door and ask me if I am Sahra O'Conner.
The biggest difference between typical "engineering" and "software engineering" is that all the "engineering" results in automation.
Standard Template Library - Done - now I don't have to worry about how to garbage collect lists of arbitrary objects.
Visual Basic - Done - now I don't have worry about graphically coding buttons and developing sophisticated call-back mechanisms.
OpenGL - Done - Now I don't have to worry about projecting 3d space to 2d screens.
SOAP - Done - Now I don't have to worry about how to develope a platform - independent API.
Developing the tools is engineering. Using the tools is programming.
so what then is craftmanship vs. engineering. Can you explain? In some ways engineers are craftsman and vice versa.
I don't think true arty and crafty people do anything for the sole purpose of function. I think we are far closer to engineers than artists. We have a science and a toolset and a lot of knowhow, and rules/standards to follow, even if they aren't bound by law. We design and build things to order that have - first and formost -function. Making the function look good is secondary, as it is in engineering. Making it not fall down is our main priority.
Maybe 'architects' is a nice middleground as it tends to combine function with form.
First off - I've called myself a "master software craftsman" for nearly 20 years now - for much the same reasons alluded to by various posters (and TFA).
Secondly - it never has been and likely never will be engineering in the legal sense of the word - using known, repeatable processes to transform known raw materials (e.g., data) into a result provably knowable in advance. That's pretty close to the Texas definition, if I recall correctly - which is why it's /illegal/ for "software engineers" to be called such in TX and several other enlightened jurisdictions.
We're not engineers. We're barely craftfolk; we've started work on various iterations of the Cathedral at Reims any number of times - and now people think we can bring the Sydney Opera House on spec, on budget, in half the time reasonable.
We're not engineers, and we likely will be, barring a transformational tragedy roughly analogous to the New London school explosion of 1937 for which software, preferably COTS software, is left visibly holding the bag. We won't be, barring such an event, because the culture and folklore that have grown up around software development reinforce the traits that actively militate against engineering - namely the lone heroic coder, 'code complete', "don't worry, be crappy" and other such garbage. And it IS going to take an event where a sizable number of people wind up dead; we've shown great impassiveness as an industry in the face of large and expensive (but non-fatal to humans) software failures. If I had a penny for every million dollars spent on projects that ultimately were delivered but failed catastrophically, I could retire and put six kids through Harvard Law.
Back about 20-25 years ago, I used to think we were 20-30 years away from a 'true' software-engineering discipline. Now, I'd say it's going to take at least twice that long from now - even if a New London-class failure of software were to happen tomorrow. Some people call it the Microsoft Effect. I think it's just a byproduct of us as a non-profession, not having any way to overrule technical decisions made by egregiously unqualified people and making them stick.
Prove me wrong. Not with statistically-improbable events 15 or 20 sigma out from the mean - but AT the mean.
This article is great. After 15 years involved in the Enterprise Software Development field I am utterly convinced of the fact that Software Engineering has in its deepest internals noise wrapped by fancy concepts. Also it's true that many corporations are doing money selling courses, methodologies, certifications, software development based upon "state of the art software engineering techniques", but I bet for motivated people with talent AND commitment instead. I'd like to meet some day a project manager with these abilities together.
Keanu said whoa (on tape even) way more than once...
I agree, people who only program do not typically understand the engineering that goes on behind a project and are not software engineers. However, anyone who actually have to run a project, estimate a project, convince a prospective customer about the true cost and value of a project, bring the project in on time and allow it to exist in a cost effective way for a further 10 years. These people are engineers and not craftsman. It just requires hard slog and dedication and applying various sensible principles of leveraging the right people at the right time in the project. Yes it might sap some of the enjoyment out of the project for people being forced to do things they feel is ‘beneath them’ but the results are usually satisfying, if a little boring.
You misunderstand me. The software engineers are not working within the constraints of cost and value. They are the people who will say what the cost and value will be. The better they are at this then the less constrained they will be when they actually build the system that they just quoted for.
Anyway, I like your analogy of a computer programmer being a plumber. It is very fitting and it is also a job that anyone can do with the right training. A craftsman does things in a beautiful and perfect way and there are many finely crafted software components (which probably have a very low ROI).
A software engineer is neither a craftsman nor a tradesman. He is not the project manager, the sales person, the business analyst, the QA person, the programmer or the support person.
He is the person responsible for the technical delivery of a software project from start to finish.
He tells the project manager how much it will cost and he makes sure it meets the customer requirements at the end.
That's it, simple.
Anyone who is not prepared to take full responsibility for these two things is not engineering the software.
I hope its more dead than alive. I'm tired of doing everything but creating software at my company. :)
I say “craft” is still too good. I might concede “software craftsmanship” for something like building a new system with carte blanche to use the best techniques via the best tools, but that’s exceptionally rare in the corporate world (i.e. this is software craftsmanship). What we are paid to do is generally to fix other people’s programs, to glue one program to another, to massage a program into working, to decipher the mysterious behavior of badly-written software. I think better names would be software plumbing, or software carpentry.
@Glenn: craftsmen certainly know about function. Craftsman make suits, clothing, tableware, ceramics, knives, furniture. Software is really close to craftsmanship, except most craftsmen care more about the quality of tools and results than most software-oriented companies.
Architects are even further away from programmers. When was the last time you saw an architect being rated by number of blueprints produced in a month, or by the number of housing problems solved? Yet programmers are routinely rated by lines of code or bug count. Dijkstra or Knuth would perhaps rate as software architects. The industry has no jobs for Dijkstras or Knuths.
@Robin: designers and marketing people also works under the same restrictions for cost and value estimations, project maintenance, and results. Nonetheless, no one calls them «usability engineers» or «sales engineers». Go ask a real engineer (say, a civil or chemical engineer) what’s his job like, then you’ll be able to spot the huge gap between it and so-called «software engineering».
How about lean/agile software practices?
is that software engineering?
Because I think everyone starts being a "craftsman" in programming ... individuals should be craftspeople, but teams should have a structure.
The whole discussion so far (apart from a few comments) goes in wrong directions thanks to the ambiguous meaning of the term "Software Engineering".
I once took a university course called "Software Engineering", expecting to learn how to design and implement software "the engineering way". My initial fascination came from the wrong impression that I will approach building software in the same way as an engineer designs a car or a plane - picking the optimal solution given a set of constraints. I quickly got vastly disappointed and what I realised was the course should well be named "Software Manufacturing". Why I came to that conclusion? Well, the last thing that mattered in the assessment of the coursework was the structure and the design of the delivered software. Instead, a trivial kind of Java application was to be developed through a mindlessly formalized development process, producing hundreds of mindless formal artifacts irrelevant to the problem domain, totally eliminating any room for ingenuity and innovation. In short, this course just tended to reduce software to something that comes out of an assembly line being assembled by robots. Unfortunately software developers are not robots.
Indeed, trivial software can be manufactured through a formalized process, but the word "engineering" comes in play only when you have challenging constraints to consider. Fantastic pieces of software, much like some great cars and hi-tech planes, are a product of human intelligence reaching its highest levels of reasoning in the battle of struggling against constraints -- be they gravity, logic, or finite computer memory and computational power. This is in its essence a creative process that cannot be blindly formalized.
Software engineering, in the real meaning of "engineering software", is the way for the future of this field.
Read this statement "Software development is and always will be somewhat experimental. The actual software construction isn't necessarily experimental, but its conception is. And this is where our focus ought to be. It's where our focus always ought to have been " and totally agree
I love watching all these non-engineers telling me what engineering is and isn't, and how "software is special" because, like, we don't know the solution before we start.
It isn't, anymore than any other real engineering starts with solution as a given. If it is, then you don't need an engineer (except, perhaps, to check that the standards have been followed)
Engineering is about the unknown and/or hard maths. Everything else is management-guff; technical documentation or skilled-technician grade work.
It's not that SE is dead - it just was never alive. It was a name for some university course - a way for academics to scoop out some pocket money from wannabes to make them feel like professionals.
Real professionals knew for a long time there is just a hard work with a lot of thinking about every step you take.
Tom DeMarco's article is interesting, however I believe it has been made intentionally questionable. It just shows the trend in the history of SLCMs: from nothing to waterfall model, then to iterative, then to early XP and Scrum, then to modern Agile, and ... What's next? Tom suggests it's commom-sense-based anarchy. I doubt it really is. More probably it will be kind of mix: Like AUP, but better defined than it is now.
The way that a software developer wants the user to get the information is “craft” but the way that the software developer has to write the code to perform that function is “engineering”.
Software Engineering has, in general, been the futile attempt of bureaucrats to get rid (to not need) creative, irreverent programmers. It's sad that so much money got wasted in CASE, UML, and now BPM tools, instead of in workshops, study groups, mentoring, and training.
This all sounds very trendy but it's ultimately pointless, pedantic, and unhelpful.
Love this Atwood - thank you for writing exactly what I've been thinking but not knowing how to put into words. Particularly like that you called out the arguments about reliability and scale. Software is craftsmanship with some additional parameters - but the additional parameters do not imply that it is engineering. Thanks again, man.