November 26, 2007
Mistakes are inevitable on any software project. But mistakes, if handled appropriately, are OK. Mistakes can be intercepted, adjusted, and ultimately addressed. The root of deep, fatal software project problems is not knowing when you're making a mistake. These types of mistakes tend to fester into massive, systemic project failure. That's why I'm fond of citing McConnell's list of classic mistakes; I find it helpful to review every so often as a sort of triage self-check. I ask myself-- am I making any of these mistakes without even realizing it?
I suppose this could lead to a sort of project hypochondria, where you're constantly defending against mysterious, unseen project illnesses. I don't know about you, but I'd much rather be working on a project with a paranoid project manager than an oblivious one. Only the paranoid survive.
Perhaps that's also why I enjoy Brian Foote and Joseph Yoder's Big Ball of Mud paper so much. This paper was originally presented at the 1997 conference on Patterns Languages of Programs, amusingly acryonymed PLoP. It describes classic architectural mistakes in software development.
The architecture that actually predominates in practice is the BIG BALL OF MUD.
A BIG BALL OF MUD is haphazardly structured, sprawling, sloppy, duct-tape and bailing wire, spaghetti code jungle. We've all seen them. These systems show unmistakable signs of unregulated growth, and repeated, expedient repair. Information is shared promiscuously among distant elements of the system, often to the point where nearly all the important information becomes global or duplicated. The overall structure of the system may never have been well defined. If it was, it may have eroded beyond recognition. Programmers with a shred of architectural sensibility shun these quagmires. Only those who are unconcerned about architecture, and, perhaps, are comfortable with the inertia of the day-to-day chore of patching the holes in these failing dikes, are content to work on such systems.
Still, this approach endures and thrives. Why is this architecture so popular? Is it as bad as it seems, or might it serve as a way-station on the road to more enduring, elegant artifacts? What forces drive good programmers to build ugly systems? Can we avoid this? Should we? How can we make such systems better?
It's a great read. The authors enumerate seven architectural pathologies:
1. Big Ball of Mud
(a.k.a. Shantytown, Spaghetti Code)
Shantytowns are usually built from common, inexpensive materials and simple tools. Shantytowns can be built using relatively unskilled labor. Even though the labor force is "unskilled" in the customary sense, the construction and maintenance of this sort of housing can be quite labor intensive. There is little specialization. Each housing unit is constructed and maintained primarily by its inhabitants, and each inhabitant must be a jack of all the necessary trades. There is little concern for infrastructure, since infrastructure requires coordination and capital, and specialized resources, equipment, and skills. There is little overall planning or regulation of growth. Shantytowns emerge where there is a need for housing, a surplus of unskilled labor, and a dearth of capital investment. Shantytowns fulfill an immediate, local need for housing by bringing available resources to bear on the problem. Loftier architectural goals are a luxury that has to wait.
Maintaining a shantytown is labor-intensive and requires a broad range of skills. One must be able to improvise repairs with the materials on-hand, and master tasks from roof repair to ad hoc sanitation. However, there is little of the sort of skilled specialization that one sees in a mature economy.
All too many of our software systems are, architecturally, little more than shantytowns. Investment in tools and infrastructure is too often inadequate. Tools are usually primitive, and infrastructure such as libraries and frameworks, is undercapitalized. Individual portions of the system grow unchecked, and the lack of infrastructure and architecture allows problems in one part of the system to erode and pollute adjacent portions. Deadlines loom like monsoons, and architectural elegance seems unattainable.
2. Throwaway Code
(a.k.a. Quick Hack, Kleenex Code, Disposable Code, Scripting, Killer Demo, Permanent Prototype, Boomtown)
A homeowner might erect a temporary storage shed or car port, with every intention of quickly tearing it down and replacing it with something more permanent. Such structures have a way of enduring indefinitely. The money expected to replace them might not become available. Or, once the new structure is constructed, the temptation to continue to use the old one for "a while" might be hard to resist.
Likewise, when you are prototyping a system, you are not usually concerned with how elegant or efficient your code is. You know that you will only use it to prove a concept. Once the prototype is done, the code will be thrown away and written properly. As the time nears to demonstrate the prototype, the temptation to load it with impressive but utterly inefficient realizations of the system's expected eventual functionality can be hard to resist. Sometimes, this strategy can be a bit too successful. The client, rather than funding the next phase of the project, may slate the prototype itself for release.
3. Piecemeal Growth
(a.k.a. Urban Sprawl, Iterative-Incremental Development)
Urban planning has an uneven history of success. For instance, Washington D.C. was laid out according to a master plan designed by the French architect L'Enfant. The capitals of Brazil (Brasilia) and Nigeria (Abuja) started as paper cities as well. Other cities, such as Houston, have grown without any overarching plan to guide them. Each approach has its problems. For instance, the radial street plans in L'Enfant's master plan become awkward past a certain distance from the center. The lack of any plan at all, on the other hand, leads to a patchwork of residential, commercial, and industrial areas that is dictated by the capricious interaction of local forces such as land ownership, capital, and zoning. Since concerns such as recreation, shopping close to homes, and noise and pollution away from homes are not brought directly into the mix, they are not adequately addressed.
Most cities are more like Houston than Abuja. They may begin as settlements, subdivisions, docks, or railway stops. Maybe people were drawn by gold, or lumber, access to transportation, or empty land. As time goes on, certain settlements achieve a critical mass, and a positive feedback cycle ensues. The city's success draws tradesmen, merchants, doctors, and clergymen. The growing population is able to support infrastructure, governmental institutions, and police protection. These, in turn, draw more people. Different sections of town develop distinct identities. With few exceptions, (Salt Lake City comes to mind) the founders of these settlements never stopped to think that they were founding major cities. Their ambitions were usually more modest, and immediate.
4. Keep It Working
(a.k.a. Vitality, Baby Steps, Daily Build, First Do No Harm)
Once a city establishes its infrastructure, it is imperative that it be kept working. For example, if the sewers break, and aren't quickly repaired, the consequences can escalate from merely unpleasant to genuinely life threatening. People come to expect that they can rely on their public utilities being available 24 hours per day. They (rightfully) expect to be able to demand that an outage be treated as an emergency.
Software can be like this. Often a business becomes dependent upon the data driving it. Businesses have become critically dependent on their software and computing infrastructures. There are numerous mission critical systems that must be on-the-air twenty-four hours a day/seven days per week. If these systems go down, inventories can not be checked, employees can not be paid, aircraft cannot be routed, and so on.
There may be times where taking a system down for a major overhaul can be justified, but usually, doing so is fraught with peril. However, once the system is brought back up, it is difficult to tell which from among a large collection of modifications might have caused a new problem. Every change is suspect. Deferring such integration is a recipe for misery.
5. Shearing Layers
The notion of SHEARING LAYERS is one of the centerpieces of Brand's How Buildings Learn. Brand, in turn synthesized his ideas from a variety of sources, including British designer Frank Duffy, and ecologist R. V. O'Neill.
Brand quotes Duffy as saying: "Our basic argument is that there isn't any such thing as a building. A building properly conceived is several layers of longevity of built components".
Brand distilled Duffy's proposed layers into these six: Site, Structure, Skin, Services, Space Plan, and Stuff. Site is geographical setting. Structure is the load bearing elements, such as the foundation and skeleton. Skin is the exterior surface, such as siding and windows. Services are the circulatory and nervous systems of a building, such as its heating plant, wiring, and plumbing. The Space Plan includes walls, flooring, and ceilings. Stuff includes lamps, chairs, appliances, bulletin boards, and paintings.
These layers change at different rates. Site, they say, is eternal. Structure may last from 30 to 300 years. Skin lasts for around 20 years, as it responds to the elements, and to the whims of fashion. Services succumb to wear and technical obsolescence more quickly, in 7 to 15 years. Commercial Space Plans may turn over every 3 years. Stuff is, like software, subject to unrelenting flux.
6. Sweeping It Under The Rug
(a.k.a. Potemkin Village, Housecleaning, Pretty Face, Quarantine, Hiding it Under the Bed, Rehabilitation)
One of the most spectacular examples of sweeping a problem under the rug is the concrete sarcophagus that Soviet engineers constructed to put a 10,000 year lid on the infamous reactor number four at Chernobyl, in what is now Ukraine.
If you can't make a mess go away, at least you can hide it. Urban renewal can begin by painting murals over graffiti and putting fences around abandoned property. Children often learn that a single heap in the closet is better than a scattered mess in the middle of the floor.
(a.k.a. Total Rewrite, Demolition, Plan to Throw One Away, Start Over)
Atlanta's Fulton County Stadium was built in 1966 to serve as the home of baseball's Atlanta Braves, and football's Atlanta Falcons. In August of 1997, the stadium was demolished. Two factors contributed to its relatively rapid obsolescence. One was that the architecture of the original stadium was incapable of accommodating the addition of the "sky-box" suites that the spreadsheets of ÃƒÂ¢Ã¢â€šÂ¬Ã‹Å“90s sporting economics demanded. No conceivable retrofit could accommodate this requirement. Addressing it meant starting over, from the ground up. The second was that the stadium's attempt to provide a cheap, general solution to the problem of providing a forum for both baseball and football audiences compromised the needs of both. In only thirty-one years, the balance among these forces had shifted decidedly. The facility is being replaced by two new single-purpose stadia.
Might there be lessons for us about unexpected requirements and designing general components here?
The first step in dealing with a problem is to admit you have one. If you catch glimpses of any of these themes in your current software project, I encourage you to read the relevant sections in the paper, which goes into much more detail-- and provides ideas for remediation strategies.
Posted by Jeff Atwood
This essay, along with a bunch of others, is evidently a part of the book series "Pattern Languages of Program Design"
Link to Amazon seach for the five (?) book series:
Guilty of 2., 3., and 4. and hoping for a 7.
Shantytown + Urban Sprawl describes the IT dept at my last employer. PC techs were brought in with no real experience or training and expected to support thousands of PCs with hundreds of apps in the mix. IT dept on the upper end of double digits (in terms of employees) but anything that wasn't specifically requested by the CEO was on a shoestring budget. Anything not specifically owned by an in house developer was either unsupported or thrown on the PC techs.
What about "Wrap it up" as #8?
Essentially, a solid, well-designed piece of code forms the core of a business and its methodology. Over time, business needs change, but the development knowledge has mostly left. Any current efforts are merely wrapping around the old code because the business can't/won't reconstruct. To them, that's too risky. Meanwhile, the newcomers to the industry have reconstructed three times and are now kicking butt...
So, it's a bit like DC. Early on, the city was well-planned and manageable. After a while of growth without the discipline to stick to the original vision, and the entire city sprawls 45 miles in all directions in one giant, over-crowded metropolitan area. It was only a matter of time before the "diamond-shaped utopia" that L'Enfant designed and left to grow becomes wrapped and hidden under years of garbage.
Why are these supposted to be negative? I consider many of these to be sensible approaches to realistic software development. I can't believe that you don't use them yourself.
One part of shantytowns is that they are built using inexpensive tools. What's wrong with that? I regularly use open source products in my development. I shouldn't use gcc because it's free? I shouldn't use emacs because it's free? cvs and svn are evil because they can be used at no cost? Are you on drugs? A cheap tool doesn't make the end product bad. A paint brush is a cheap tool, and da Vinci has made some inspiring works with it.
2. Throwaway code.
I have no problems with throwaway code as so much of the open source world starts off as throwaway code and this seems to be a successfully approach in this arena. If you didn't have people quickly throwing their ideas together to have a look at them then innovation will die or at least crawl very slowly. Of course I judge open source projects differently from the software in a 747. Different approaches for different projects.
3. Piecemeal Growth
Or as I call it: modular design. You take a large system, like a city, and break it up into seperate and smaller areas with specialised functionality, linking them all up with roads. This is basic stuff - if a programmer doesn't know/use this then he shouldn't be programming.
4. Keep it working (Daily build)
Seriously, you're suggesting that maintance of software is bad? That daily builds are a bad thing? I'm sorry, I have to ask if you've ever worked at all in the software business? Nothing is ever perfect. As time goes on bugs are exposed and need to be addressed. This is why we write documentation and comment our code so that we can maintain it latter.
5. Shearing layers.
I'm not sure what you're trying to say here. Are you saying that software changes? If so, then why's that bad? Software has version numbers for this very reason. That software can evolve and grow is a potential force for good.
6. Sweeping it under the rug.
Again, do you have any real programming experience? How often do we reuse libraries that have bugs in them and we create a facade or wrapper to go around this library. This wrapper tries to minimise the damage these buggy libraries do. It's a practical reality. Hiding problem libraries is not a sign of bad design, it's good practice.
Here's a big WTF. So you're saying that refactoring is bad? Are you seriously saying that your code is so damn perfect the first time that you don't have to go back and improve on it? Sometimes the original code sort of almost works, sometime you just have to completely trash it and start again. Refusing to acknowledge a mistake and letting it live on in the form of crap code is unbelievably stupid.
For the record, the Chernobyl sarcophagus is not "sweeping it under the rug," it's an attempt to contain the dangerous shit within than would otherwise escape and is so dangerous that it can't be removed. I'm not sure if there's a programming metaphor there, really (it's hard to find a metaphor for code that is really analogous with radioactive sludge).
(The real Chernobyl metaphor would be management related, in my opinion—the real problem the Soviets had, aside from an inherently dangerous reactor design, was in letting vastly under-qualified people monkey with something quite powerful and potentially dangerous.)
Andrew writes, "Again, do you have any real programming experience?"
I was asking myself the same thing about you Andrew. Not only have you completely missed the point of each and every one of the pathologies, you missed the point that Jeff didn't actually come up with these ideas himself.
"Not only have you completely missed the point of each and every one of the pathologies"
Fair enough. Stupidity strikes again. Let's start with the first one. In it he quotes: "Shantytowns are usually built from common, inexpensive materials and simple tools." My question is why is the use of cheap tools a problem with shantytowns?
"you missed the point that Jeff didn't actually come up with these ideas himself."
Err. He repeats these ideas in his blog. His blog is an outlet for what he has to say even if what he writes here is a quote. Isn't it fair to consider that he agrees with the quoted text? He doesn't go: look, here's some opinions that I disagree with. Without the disclaimer one can only assume agreement.
Andrew, I think that you actually have missed the point with a lot of your commentary, especially the shantytown analogy. Using cheap tools does not make something a shantytown. The important part was this:
"There is little specialization. Each housing unit is constructed and maintained primarily by its inhabitants, and each inhabitant must be a jack of all the necessary trades. There is little concern for infrastructure, since infrastructure requires coordination and capital, and specialized resources, equipment, and skills. There is little overall planning or regulation of growth"
How did you miss that?
"I was asking myself the same thing about you Andrew."
I'm bitter, angry and I hate you all. Isn't that proof that I've worked in the software development industry for a long time?
Of course, having worked in software long enough to be completely disillusioned doesn't mean that I'm a good software developer.
I'm betting Jeff is infinitely better developer than I. At least he still has that innocent joy about the industry. That virginity will pop soon.
"How did you miss that?"
So you agree that the use of cheap tools is not a mistake? So why include that with the description of a worthy mistake? The best writing is to the point. You say what you have to and stop. Extra wordage only runs the risk of damaging your main point.
In this case a real mistake is mixed in with something that wasn't a mistake. Jeff should have cut and paste less text. Less words doesn't make a point less powerful.
Didn't Jeff have a post on this very issue just recently? Something about making the web a place full of rich information and not noise? (Maybe I'm thinking of a different blog - I'm old and my memory is going.)
I'm bitter, angry and I hate you all. Isn't that proof that I've worked in the software development industry for a long time?
Works for me. Welcome to the team.
Hey Now Jeff,
Interesting post, always good to keep in mind revisit every one in a while.
Coding Horror fan,
Andrew, I also think you are missing the point, but I am not trying to attack you -- merely to explain how I read the example.
Shantytown means (all of this is my opinion alone) that you take many separate parts, and each one is responsible for doing too many things. Like if I made an app composed of 5 different technologies. Each one had need to send an email. Instead of having one piece of code that handled all email sending capabilities, each part of my shantytown would all have to know how to create an email, communicate with a mail server, handle errors, etc.
I'm really not attacking you, I am trying to explain what I took from the Shantytown example.
Oh -- and I've been involved in plenty of shantytowns in my day.
I'd seriously like to find out from other software developers if anyone has been involved in a project that you were very proud of the code. I've been working at various companies for 10 years or so, and at every one of them the project I've been dumped into was 1, 3, 4, or 6. I have done some throwaway code things, too, but I'm more concerned with big software right now.
My first job was probably the most organized place I've worked for. They had a big (50k) PowerBuilder 5 project that was used in small co-operatives around the country helping farmers plan how to plant their fields (and therefore spend more and smarter money with the co-op). PowerBuilder MADE you develop OO code, and still there were large parts of the program that were done very poorly -- several places where code was repeated, in a slightly different manner because someone didn't have time to learn how to use another's classes correctly.
Another job had a massive health care application that all the developers on were dying to rewrite because they felt the code was a mess of spaghetti.
Another one I took over from a developer who had maybe 10k lines of code written, which were sufficient to do the task he was given. I then was asked to add more and more and more functionality, and made some bad choices so eventually I was left with 70k lines of code that made my stomach ache when a bug was found. I hated going back in and trying to piece together what to fix, how, what it would affect, etc.
At my current job, I've got 70k lines of code that I'm not proud of, but it is pretty easy to maintain. Some parts are a mess because I didn't know the .Net framework at all, and ended up doing things REALLY badly since I was under so much pressure to deliver functionality so quickly. I am working on refactoring all of this. However, I doubt that I'll ever be "proud" of the code. Who knows.
So, has anyone worked on a project that was planned (or refactored) so well that maintenance is a joy (relatively speaking), new features can be added logically without screwing up the main plan, etc?
I'm pretty sure Andrew nailed it with the Shantytown / Open Source comparison:
"There is little specialization. Each housing unit is constructed and maintained primarily by its inhabitants, and each inhabitant must be a jack of all the necessary trades."
Hrm...sounds like open source development to me...
"There is little concern for infrastructure, since infrastructure requires coordination and capital, and specialized resources, equipment, and skills."
interesting....just like open source...
"There is little overall planning or regulation of growth"
The reason "big ball of mud" approaches are so prevalent is because they *work*. I would go so far as to say it's the *only* thing that works for projects that need to evolve (read: 99% of software). Anyone who thinks their current project is not a big ball of mud, is either delusional, or simply hasn't shipped version 2 yet.
All software *wants* to become a ball of mud! It is its natural state. All emergent systems look like mud in the end, anyway.
I've spent 5 years maintaining a massive, 10-year old VB6 product for about 10 to 20 thousand customers. No automated tests, no documentation, comments you shouldn't dare trust, no consistency. None of the original developers are around anymore, and at least 40 developers have stomped around in it from around the globe. New releases are about 8 months apart.
To fix bugs and add features, I must forget everything I know about traditional software design:
- Code Duplication is king. It's the only way to prevent breaking the unknown.
- Don't comment or write documentation. Doing so will only tempt novice engineers to "fix" what is currently there.
- Only design for the immediate feature request at hand. Don't fool yourself into predicting future requirements and saving the world.
A favorite line I've heard since I started working on the project is, "Don't worry, the code won't be around for much longer". Hah! It will exist for as long as the product continues to bring maintenance revenues.
Under this system, it is expensive to add any new features that require changing the mud. So we don't change the mud. We build around it.
Embrace the beast and learn to work around it. It may be a shanty town, but it's still home to someone.
Okay maybe that was mean. But perhaps you have some advice about strategies to avoid these patterns from emerging? Certainly it's not as easy as "now that you know how to do it, don't!"
These are all very natural ways for software projects to evolve ("natural" meaning "following the path of least resistance"). After all, every programmer worth his pocket protector should be able to tell that you don't sell architecture, you sell a product. Clients (in general) don't give a crap if you spent time perfecting your design, they just want something that works. And in our capitalist system where the money goes to the lowest bidder, there is a powerful incentive to nix the designing and the refactoring and just spin out code until you've got something you can hand off.
Now you and I know that time invested in design and test driven development and prototyping and rewriting and so on can save time and money in the long run ("a stitch in time saves nine"), but to an accountant an expenditure NOW isn't the same as a possible expenditure, oh, somewhere in the future. So how do you strike the balance between "design it perfect" and "git 'er dun"? What methods do you use to keep a project from going down a dark road, or save one that has started that way?
Yeah I guess that's a tall order for a mere blog post, but that's what I'd like to read.
That's some hilarious sh@@! I'm sure you know that Indians and others added straw to the clay to make stronger bricks. You are to be commended.
Few people know that it is harder to do what you are doing than to design something from scratch, which is usually abandoned anyway.
You are are true hero. You keep a pile of crap bringing in money and employing people. Bravo!
Um, guys, are you sure you know how Big Balls of Mud are formed? Let me tell you a little story...
Boss: "Okay, we've got this new contract. It's a generic e-commerce website, except it has to have X, Y and Z. Oh, and T. And - lemme see - U, V, W... and a few other minor features. About 100 of them."
Team: "That will be long and expensive."
Boss: "I realize that. Give me an estimate, boys."
Team: "We'll need to do some speccing and planning first."
Boss: "There's no time. We've got to have a demoable product in two weeks and a production version in two months."
Team: "No way we can do it that fast. Wait, didn't you just say you want us to estimate...?"
Boss: "Actually, I've already signed the contract."
Team: (scrambles to put the finishing touches on the currently underway project)
Project manager: (scratch that, there's no project manager)
Senior programmer (let's call him Steve): "Come on, guys, put that away. We'd better get started with the new project."
Team: "Sure, let's sketch a quick plan."
Steve: "No time for that. Bob, you mock up a user interface."
Bob: "But I don't even know what will go in there!"
Steve: "Never mind, we need it quick. Marc, you get started on the database layer. You know, the usual e-commerce stuff: authentication, products, shopping cart..."
Marc: "What about those extra features the boss mentioned?"
Steve: "We'll get to them later."
I think you know (or else you can guess) the rest. There's no mistery. Greed and lack of patience, nothing else. And it happens over and over, while a lot of smart people are looking for technical causes. Can we wake up now?
You are absolutely right about the temptation to go on using a prototype and enhancing it to be good enough so we can add it to the production code. So we usually write the code on a VPC which is nothing but spaghetti... later we refactor the "POC" to be more presentable and then get it reviewed... once reviewed we throw away the VPC and start all over...
the VPC works as a good way to ascertain discipline...
Very understandable post. I was at PLoP in '97 (was a student at UIUC), and I was immediately struck by the notion of the Big Blob that Foote and Yoder presented.
In your pictorial summary, it's plain to see that analogizing the software development world to the urban planning world is helpful to the extent that we can better understand _why_ we build software shantytowns, and engage in software sprawl. These 'patterns' have worked at small scale or when under severe time pressure.
The fact that these agglomerations continue to exist in the real world gives no reason to hope that their eradication from the world of software development is at hand--but hope springs eternal because we believe that the power to transform the virtual is far more within reach than that to transform the physical.
(Jeff: would you mind fixing your "Read older entries" link at the bottom of your main page so that it goes to an earlier page of posts, rather than to a single post? Thanks.)
My project sucks.
So now what?
Its depressing to have enough experience to know what you are doing is wrong and yet not know how to fix it.
We don't exactly have defined career paths in the software industry. Its not like becoming a doctor. I would have gladly done a residency, taken a bar exam... whatever.
Reading this and other blogs as well as a multitude of Patterns and Practice book does me no good. As someone who has only been programming for 3 years, its depressing.
I wish i were ignorant of good design. Maybe then I wouldn't hate this career.
Most of my projects end up looking like the Bronx circa 1982, or some Cuban dictatorship where everyone drives around in "Classic ASP" cars. Yuck.
But don't worry folks...
VS.NET 2008 + Vista promise a "Gatacca" type of future. However I suspect it is more going to be like "Logans Run" where on the the inside it is based on fantastic and perfect ideologies (which not everyone understands or agrees on) - but when you escape that, its the same old horrible mess on the outside... based on the 1980's x86 architecture.
It's so unfortunate that most of the time we aren't choosing to make the shantytowns or the throwaway code... we're simply the building crew. With the exception of brand new projects, there's no way to see what you're walking into, and when you're there, you don't always have the authority to change large sections of code.
I think a lot of the reason why it's easy to get jaded in this field is just because management understands only immediate needs and not the necessity of clearly written, efficient code and the time it takes to write it.
Matt - "I'd seriously like to find out from other software developers if anyone has been involved in a project that you were very proud of the code." "So, has anyone worked on a project that was planned (or refactored) so well that maintenance is a joy (relatively speaking), new features can be added logically without screwing up the main plan, etc?"
Yes, but I was working for a small company that sold the software I was writing when I did it. I completely re-designed/developed a big ball of mud VB application in java swing, more or less by myself over the course of 8 months. My boss was not a technical person, and my co-worker/manager trusted me to do the right thing. If I wanted to re-factor something I didn't feel I did a very good job on I was free to do so at my own discretion. If I wanted to spend a few days to make structural changes that I knew would save me time down the road, I could just do it. If I thought of ways to make the user interface better, I could just do it (of course I would get a thumbs up or down sometimes).
I then maintained it for another year and a half. Again I had the same level of complete control over the application. Some things were not perfect, but I am still proud of about 90% of it. Everyone who did occasionally work on the project always had positive things to say about the code base.. so I know I am not just blowing sunshine up my rear. :p
#8 - Accidental Architecture
"Every interesting software-intensive system has an architecture. While some of these architectures are intentional, most appear to be accidental" Grady Booch
I would say the Accidental Architecture could encompass all of the categories above or maybe we are just stuck in the Liquid Goo phase...
My view is that we still don't know how to even design software structures yet. Heck, we are still arguing how to even draw them. At least architects have symbols that most everyone agrees upon, just like electricians have circuit diagrams and symbols. What do we have? UML stickman? It’s embarrassing.
This is in addition to the negotiating technical debt as described by others. Tim Dudra, I understand your plight, but you are still reading a software blog – you haven’t given up completely have you ;-)
copy + paste = blog!
I added photos, too! But seriously, I wanted to highlight this particular article by providing a Reader's Digest condensed version of it. It's great, but it's giant -- not everyone has time to read all of it. Hopefully they can get a good summary of it here and dig in to the sections they're interested in.
I am a bit confused about the "keeping it working" section.Jeff,are you saying that software versioning is bad practice ? Versioning is maintenance of existing software to improve the software structure,fix bugs,add new functionality etc to produce more efficient software right? If so why is that a bad practice.I may have misunderstood what you were putting across,if so enlighten me.
"Embrace the beast and learn to work around it. It may be a shanty town, but it's still home to someone."
A shantytown is a palace for someone who lives on the streets. Bad programmers love shanty towns, good programmers go there to die.
Cheap, in this context, means how much effort went into making the tools and how much they help you do your job. OSS is free, but not cheap in terms of the human capital invested in it.
They mean blunt saws and drills that have been repaired a million times that have to be borrowed from someone. I think that the article was written before OSS became so ubiquitous and muddied the metaphor.
"VS.NET 2008 + Vista promise a "Gatacca" type of future. However I suspect it is more going to be like "Logans Run" where on the the inside it is based on fantastic and perfect ideologies (which not everyone understands or agrees on) - but when you escape that, its the same old horrible mess on the outside... based on the 1980's x86 architecture."
More correct than you know, including the part where the ideal society is not ideal at all, nothing truly new is ever done, and you are killed when you get too old ...?
I agree with Andrew a little bit. These patterns can be interpreted as natural solutions to many programming problems, that arise from the need of solving complex problems.
TDD creates some shanty towns, since it builds things bottom-up
XP prototypes are some sort of Throwaway Code
Shearing Layers sounds like "structural pieces of the architecture last longer", but I also don't agree with that. Sometimes you're forced to change major pieces of the architecture due to several reasons
I think the post is still valid, but I appreciate Andrew's comment because we must always read everything critically.
Been thinking about different ways to approach architecture and came up with an idea i'd like to throw out there.
How about taking a cue from ants where each individual or team follows a common strict ruleset on how to approach the problem/solution scenario.
Consider it like the piecemeal growth model but with a strict set of rules.
These rules will obviously need to be worked out first. Maybe even a community effort to provide standards etc.
What does everyone think about this? does this already exist?
"So you agree that the use of cheap tools is not a mistake?"
Cheap != Inexpensive. Just because a tool is inexpensive does not mean it is cheap. If GCC was useless then why would every major operating system besides windows use it as it's default compiler? When Jeff is talking about "Cheap" tools, he is talking about tools that are more or less worthless. QBasic anybody?
Also, just because he cuts and pastes, does not mean he agrees. Oh yes, most of the time he does agree, but many times he pastes something just because it's interesting to think about and make your own decisions about.
Anyone who thinks Shantytown is an appropriate description for all open source software either doesn't understand what is meant by Shantytown or hasn't worked much with open source. That's not to say open source is any more or less immune to the pathologies listed, but projects like the Linux kernel are not built by unskilled laborers and do not lack infrastructure, planning and regulated growth. gcc may be free (as in beer and speech) but it is by no means a "simple tool".
I've worked on plenty of Shantytowns and believe me, they aren't pretty. I wouldn't consider myself an "architecture astronaut" (http://www.joelonsoftware.com/articles/fog0000000018.html), but software written by skilled professionals where some effort was made at long-term planning and growth management is usually far better than cobbled-together balls of mud.
It's funny, where I come from we always called this "architecture" a BIG PILE O SHIT....
Andrew: "Again, do you have any real programming experience?"
Andrew, you totally missed the point of every item mentioned in the article, and then had to resort to a personal attack? Do *you* have any real programming experience, or are you one of those self-taught VB users that just think they know how to code?
I'd suggest you re-read the original article, and then read some of the responses to the article (except yours), and then try again.
Clay: "That's not Fulton County Stadium."
So what? Does it matter? It's a picture of a stadium being taken down.
Why do people feel the need to nitpick minor details? Is it that you can't grasp the big picture being presented, and instead have to find a minor issue to quibble about?
So, has anyone worked on a project that was planned (or refactored) so well that maintenance is a joy (relatively speaking), new features can be added logically without screwing up the main plan, etc?
Just finished doing some maintenance to a project at work that I was not involved in with the writing. It was extremely well put together. It didn't take me long at all to figure out what my coworkers had done before. Several times, I was able to use methods that had already been generalized. Other times, I found code that was identical to what I needed to make embedded in another procedure. Refactor method, pull it out, pass some parameters, test the calls from both places, tweak, and check it in.
And yes, this was maintenance on a 1.0.
Another project I'm working on is a version 7 but a reconstruction of a piecemeal growth app. A lot of thought went into the rewrite also.
I used to work construction.
Now architects draw up these beautiful works of art. Everything is perfect on paper. But don't go looking behind the dry wall at most these places. Yea, the major structural parts are sometimes ok, but there is a lot of duct tape a bailing wire in there also.
Maybe in the multi-million dollar projects you have good design and good construction, but there is also tons of read tape and regulation.
Or maybe some small handcrafted projects where the designer/constructor is paid very well and has almost unlimited time you get a result to show off to the world.
The fact of the matter is our world works pretty darn good even with all the less than perfect engineering. In fact I don't know if our world would be possible if perfection and beautiful code was what we all strive for. Between the Sears Tower and a handcrafted china cabinet for the wifes special day there is much work to be done and it doesn't all have to be magnificent, just adequate.
Entertaining and sad at the same time.
I saw Brian Foote give this talk at the most recent OOPSLA in Montreal. It always surprises me how you can find metaphors for Software Engineering in disciplines that seem like they couldn't be further from it -- and the metaphors are usually pretty deep!
I'm bitter, angry and I hate you all. Isn't that proof that I've worked in the software development industry for a long time?
Probably the best comment I've seen, by someone I totally disagree with on most of his other comments. ;)
Software development can be incredibly frustrating if you care about the software. Most of the industry is in the "big ball of mud" stage of development and seems inextricably mired in their ignorance and short-term outlook. It's another aspect of the whole 80/20 thing Jeff posted about recently. (Except I still think we're lucky if it's 90/10).
On the other hand, sometimes I get to work on a ball of mud and clean it up a little, usually when no one's looking. Better to light a candle...
Frank: Where do you work? Are they hiring? ;)
A lot of the antipatterns described in Jeff's post tie into something I call the Just Barely Good Enough syndrome. I work for a small software company (5 developers), and there's always way more to do than we have time for. As long as something is just barely good enough to get the job done, even though it may be hard to use, frustrating, or downright painful, improving it usually loses out to features required by new clients or other business realities. Unfortunately, this applies to pieces of the shipped product as well as internal tools!
I'm not saying I have a good answer for this, because if we don't please our clients we don't keep the lights on. Maybe the one thing I've taken from this is to try to take the time (if possible) to implement something a bit better than Just Barely Good Enough, because I'm probably going to have to live with the results for a long time without an opportunity to improve it later.
Look at how the Victorians built bridges....
Some of the bridges designed for The Rocket are still in use by todays high speed 125mph trains....
Modern Engineering practices say they got it wrong, "over engineered".. The joke is the people saying this are not building anything that will still be standing in a 100 years time
I'd really like to thank Frank and Matt Morris for responding to my question about if there are any well architected products out there. I know it might be stupid, but it really gave me some hope.
You 2 have me so jazzed right now that I'm going to start to abstract out a bunch of stuff that was haphazardly added to some of our smaller projects and get them together...then I'll move onto the big one.
So, Frank and Matt Morris, you may not know it, but you really have made a difference to me today.
"People often talk about bad development like it's some kind of law of gravity or natural decay. But that's rubbish. Most developers ending up with a bad system have created a noose for their own necks. They want to appear helpful and to appease management. So they accumulate technical debt, and a few years later end up in a state of technology bankruptcy. To avoid this you need to grow a spine, get good at negotiating, and learn how to state the case for good development in business terms."
Mr. Ginger has it bang on. Bad software is not inevitable. Bad software is a by-product of developers lacking the spine to stand up to management, of management not understanding technical debt and its consequences, of laziness for the up front work (i.e.: lack of design) and, in general, bad software practices like "go to your cube and write code", no or superfluous code reviews, no or superfluous design reviews, etc.
My primary weaknesses as a developer turn out to be that I loathe working on big balls of mud and I can't work well on a team with developers I do not respect. These are the roots of the problem but what actually drove me out of the software industry and likely will stand in the way of me ever returning is that I could not work on a team with developers who were content to work on the balls of mud or who believed there was no way to avoid them. I could not respect those developers and would come into conflict with them as they either worked against efforts to improve the mess or implicitly encouraged management to ignore technical debt by accepting every stupid requirement and request with nary a complaint.
I'm not saying my position is correct - I could be the problem child here for all I know. However, I hope my point of view may give those who feel the ball of mud is inevitable or those who whine about balls of mud but don't proactively fight against them to pause for thought and consider that maybe, just maybe, they are the problem child.
"I'd seriously like to find out from other software developers if anyone has been involved in a project that you were very proud of the code."
I've kicked off the development of replacement subsystems a few times now. You get decent config, testing etc in from the beginning. Make sure the conditions are in place to be able to refactor and release over the longer term. Don't cave in to expedient fixes. You can keep going for years with no problem. It's possible to salvage stuff that's veered off course too, though that can be harder.
People often talk about bad development like it's some kind of law of gravity or natural decay. But that's rubbish. Most developers ending up with a bad system have created a noose for their own necks. They want to appear helpful and to appease management. So they accumulate technical debt, and a few years later end up in a state of technology bankruptcy. To avoid this you need to grow a spine, get good at negotiating, and learn how to state the case for good development in business terms.
Of course this is not particularly easy, which is why so much software is in a terrible state...
KenW: "So what? Does it matter? It's a picture of a stadium being taken down.
Why do people feel the need to nitpick minor details?"
I'm not defending the prior nitpicker, but a key detail which was ommitted from the analysis of the short life of Fulton Stadium was the fact that the 1996 Atlanta Olympics had already constructed its replacement. It cost Ted Turner a lot less to retrofit the Olympic Stadium into a baseball park (he actually demolished 1/3rd of it, it was too many seats to fill) than to build a new one from the ground up.
So part of that example should have been "if you have a huge external source of capital, that always helps".
Hopefully they can get a good summary of it here
and dig in to the sections they're interested in.
Okay, I'll buy that. Still, would be nice to have a little of the Atwood brain matter spattered throughout...
How do you not know you are creating spaghetti code? I mean, if you are not already just an all-around bad programmer, in which case you probably are not reading any of these books or blogs anyway.
I mean, if you are not already just an all-around bad programmer, in which case you probably are not reading any of these books or blogs anyway
This was exactly my point on the "Two Types of Programmers" post. We have to reach out to these folks to bring them into the fold, because they're the majority.
Quite frankly all the reach out (or around) in the world won't change this. All of these listed Architectural Disasters should really be called Management Techniques. In the world of shrink-wrap they should never happen and presumably the market would limit their occurrence naturally but for the great unwashed developing business applications we seldom have the chance to decide these issues. At this point however I can assure all of you young and frustrated programmers that with age comes the wisdom to no longer give a damn. You fight the battles you think you might win and let the Devil take the rest.
So let me ask you this.
If 99% of commercial and in-house software development projects in varying degrees of mudball-itis. Isnt' there room for competition (in any and all industries) from very well designed codebases? Ultimately won't well designed #5s truly rise to the top in the marketplace long term? My guess is that no one is really truly doing #5 in very large, commercial products, and so competiors can just "mudball a little better" to gain competitive advantage.
MY gut feel is 10-20 years from now the mudballs will succumb to better competing #5s. But maybe that's really 100-200 years from now.
My ideal utopian brain is thinking that there is room to really do #5 right on a major project and blow away the compitition who won't be able to keep up with your ability to adapt an inovate because your layers give you infinite malliability at very little cost compared to competitors. Ah to Dream. /segue And that leads right to Mitch Kapor's Outlook killing "Dreaming in Code" [insert amazon link here] where you put together a "dream team" and massive open ended budget, and try to draw in the open source community, and 5 years and millions later you've got nada.
Maybe it's not possible.
And maybe I should get back to fixing bug #9812 now.
"At this point however I can assure all of you young and frustrated programmers that with age comes the wisdom to no longer give a damn. You fight the battles you think you might win and let the Devil take the rest."
Why do so many people confuse "Wisdom" with "Apathy". Always try and fight exactly *one* battle you "can't" win. When you make any achievement there, not only is it satisfying, but it looks impressive too. But if you try to fix everything, you'll end up exhausted.
The 10 Step Programme for improving Internal Software.
1) Find the balls of mud that've been around for 40 years.
2) Pick one. No more than one. One will be enough.
3) Work out exactly what needs to be done, and what'll be affected.
4) Start work.
5) Find out what *really* needs to be done, and what'll be affected.
6) Finish the replacement.
7) Cause the original BoM to fail, and then replace.
8) Congratulate team
9) Take holiday
10) Find a new ball of mud.
This will be hard, but the alternative's "Let the Devil take the rest". And who wants to be that apathetic?
Thanks, Tom for proving my point. All the steps you outlined are management functions (except for parts of 4 and 6.) If you ARE a manager I encourage you to fight those battles you might win. If you are the coder - learn patience and improve your skill.
There are some interesting parallels here to Jared Diamond's book Guns, Germs, and Steel(didn't read the book; saw the movie), particularly about specialization. Shantytowns and their residents are something like modern day hunter-gatherers, in that there is almost no specialization.
Diamond proposes that the reasons that some people end up as "haves" and some people end up as "have-nots" are largely due to the particular geography in which they live. I wonder if there are particular aspects of the corporate environment, like the hunter-gatherer's physical environment, that lead to specialization and thus further up the ladder of software development evolution.
Maybe you'd be interested in my book of software horror stories that I personally experienced. I am hoping to do a second book that includes everyone else's worst projects; this particular story seems like a good candidate.
After reading the IE blog post, I almost feel sorry for the IE team. They have soooo much baggage now associated with the product! There are thousands of badly-written sites out there (mostly corporate) that work because the users use IE. "Stirct mode" and "non-strict mode"? What the hell are they thinking? But then, how do you improve a browser without breaking all these sites?
The only way forward I can see is to scrap it, start a new product, change the name - disassociate it from IE - make it properly standards based from the start. Just stop developing IE, IT team! Then, after a few years, all those badly-written sites would hopefully have disappeared, MS will dominate the market as usual and they will have happy developers. It'll be hard to get all those people to adapt to the new product but hey, no pain, no gain. [url=http://www.disestetigi.org ]estetik[/url]
Andrew, cheap materials have NOTHING to do with open source. What a non sense! Cheap and expensive relate to the fact to hire unskilled cheap personel, who doesn't know what to do with the tools well, instead of having a seasoned architect. Keep the fury for better moments and read carefully next time
If you are in a situation where your peers convince the management to use bad/quick architecture in software, then that is bad. If you can't convince the management otherwise, you should leave the company, but not the whole information technology business. There are lot of people who believe in good practises, and if they get into same company, then you create good momentum for better times.
We have a ball of mud. We never seriously tested it (our customer did, but their management was seriously whacked about what results were important). A few rollouts turned into several hundred.
New management came in (customer side) who had two pennies to rub together and said "these numbers are whacked - make it work accurately" (fortunately they didn't say perfectly).
The major problem is that this is an open looped design that attempts to track queued up customers using two points. However, at any time a customer can leave the queue. We get a mostly unique key from the first point, but only "customer here/isn't here" from the second point. The rest is guesswork, innuendo, and "what if's". I've tried to point out that without a way to resync our internal queue with real life, we can only have an EPIC FAIL. I've been told (in so many words) I'm depressing and we can do it... we just have to figure out how.
The text in this blog MOSTLY is a result of poor planning or design. It doesn't even address the case where the initial idea was flawed from the start. Many times I have seen clean, elegant code turn into balls of mud because people at the last minute have decided to a) make huge changes in the project or b) keep tacking on more and more crap that exceeds the original design goals.
Chernobyl was NOT an EPIC FAIL because of design - it was best practices AT THE TIME. It was an EPIC FAIL because even when it was determined it was a bad design they continued to use it (much like the reactors we have on fault lines in the U.S. or the Fukushima Daiichi reactors). When we set the parameters to meet the device or the code, not the other way around, THEN you are set for disaster.