I <3 Steve McConnell*
Coding Horror
programming and human factors
by Jeff Atwood

December 6, 2004

The Magical Build Machine

Evidently Jerry Dennany is a member of the build machine cult:

One of the golden rules of modern software development is that one should build all software on a dedicated build machine.

A build machine should:

  1. Be well documented. This includes Version of the Operating System, Service Pack level, HotFixes installed, Tools installed, along with any special installation instructions.
  2. Be easily reproducible. Anyone on your development team should be able to take the documentation, the build machine, and any required installation media, and recreate that build machine on demand. If you can't, then you don't know what exactly is in your product.
  3. Not contain a single piece of software not related to the build. For example, just because your project uses crystal reports does not mean that you need crystal reports on the build machine.
  4. Be in an area that is controlled in its access, if at all possible. If this is not possible, then you should control who may log onto the computer.
  5. Be under change control. No change to the build machine should take place unless that change is documented and approved.
A few things not to do:
  1. Never make the build computer a developer's workstation!
  2. Never do anything with the build computer except build that version of software. I strongly suggest using a disk image tool such as Ghost to re-image after every build. You don't get much more of a 'known state' than this. This was actually very important in the VB6 / COM world.
While I am all for daily-- even hourly-- builds, I strongly disagree with the perpetuation of the Magical Build Machine concept. It's a bad idea.
  • The magical build machine reinforces the disconnect between developers and users-- "us" and "them". It runs on my box! Every developer on the team should understand how to produce a reliable build from their own machine. A build that runs on the webserver. A build that runs on the end users' PC. And if it doesn't run, they should know how to troubleshoot it. It is every developer's responsibility to write responsible code, code that doesn't cause a lot of deployment problems. If you isolate developers from this process, you do so at your own risk.
  • If you use a magical build machine, you're implying that your project is so complicated to build that it takes special voodoo to get it to run. Sacrifice a chicken, sprinkle salt over your shoulder, then re-image the build machine when the stars are perfectly aligned. That's the only way to get a "clean" build! A project that is this difficult to build does not inspire confidence. It also smacks of voodoo programming or programming by coincidence.
  • Using a magical build machine perpetuates the idea that building and deployment is risky and incredibly sensitive to the exact client configuration. Jerry correctly points out that deployment was a big deal in the bad old days of VB6 and COM-- aka "dll hell". On a correctly architected .NET project, this absolutely should not be true! One of the major selling points for .NET is ease of deployment:
    1. Is the .NET runtime installed?
    2. Xcopy files to a folder.
    3. Run your app!

There are absolutely valid reasons to have a controlled build process. I'm not proposing that every developer build the project and deploy it at will. Use a build machine if it makes sense for your project, but be careful that you aren't injecting any accidental "magic" into your development process along the way.

Posted by Jeff Atwood    View blog reactions
« Universally Annoying Remotes
Gold Plating »
Comments

There's just not enough chicken sacrificing in software dev these days. Why, I remember when...

Seriously tho, nice post!

Dan F on December 7, 2004 3:55 AM

One of the points of a "magic build machine" you missed is reproducibility. Using a build machine, especially an automated one, allows very easy reproducibility: you know that all the code used was checked in, you know _when_ it was checked out (at the start of the build), and you can reproduce it locally. You can't do that from a developer's desktop machine... no matter how many chickens you sacrifice.

Robert Watkins on December 7, 2004 4:07 AM

It seems like some important points are being missed here.

I was brought up on the build machine methodology 10 years ago. The only reason anyone cared: something called "escrow".

We were a small underfunded "startup" type company with a single fortune 500 customer. That customer wanted to be able to do something (it's not necessarily important what) with our software if our company died. That's what escrow is for. After every release, we were required to basically package up our build machine and send it to a third party company for them to hold onto. If our little company went out of business, the third party escrow company would deliver the "build machine package" to the fortune 500 company.

This basically creates most of the rules Jerry Dennany is talking about. You must know how to package up the build machine.

I think escrow is the majority use case here. I think there are other user scenarios where packing up a build machine are useful, but I would hardly call it a golden rule.

I always strive to have a packagable build machine, because I have personally experienced many of the downsides of not having one. I don't make it a requirement or complain too loudly if it doesn't happen though. It's just one of many (probably more important) risks that every software project bears.

I think both Jerry and Jeff have extreme (and opposite) viewpoints on this issue that are too far outside the mainstream. Frankly, given my limited experience with the joys of VB6, I'm not even sure I understand most of Jeff's post/point, which is unusual.

Michael Maddox on December 7, 2004 7:21 AM

xcopy your files, hah! That's fine, unless you are using Serviced Components (not on XP, where you can create them without Enterprise Services) or any sort of Interop.

Even in those cases though, we'd get it down to a simple msi installer file. I've had to use classes that Microsoft didn't even document to get a serviced component installed correctly through an MSI file. I guess I should write up a post on that.

Darrell on December 7, 2004 8:54 AM

By the way, when I'm typing in this here textbox for comments, and then press tab to get to either the preview or post button, I tab up to the Coding Horror link at the top. That's like a usability issue, mmmmkay? :)

Darrell on December 7, 2004 8:55 AM

> One of the points of a "magic build machine" you missed is reproducibility. Using a build machine, especially an automated one, allows very easy reproducibility

I'm all for a build process and a build script. Just not a fan of the build machine for the reasons listed.

> That's what escrow is for. After every release, we were required to basically package up our build machine and send it to a third party company for them to hold onto.

Isn't a windows installer version of the app, plus the source tree, good enough for escrow? Say on a ZIP disk?

> I think both Jerry and Jeff have extreme (and opposite) viewpoints on this issue that are too far outside the mainstream.

Actually, aside from his "you MUST have a magical build machine" focus, we actually agree on a lot of points. The build process should be well understood and documented, and probably automated. I've seen a build machine lead to lazy developers who don't understand deployment like they should. Every piece of code should be written by a developer who understands the deployment implications of that code.. and that implies all developers know how to get the app to a user's desktop.

Jeff Atwood on December 7, 2004 9:42 AM

> xcopy your files, hah! That's fine, unless you are using Serviced Components (not on XP, where you can create them without Enterprise Services) or any sort of Interop.

True, true.. but that gets back to my "correctly architected .NET project" argument. In some cases these situations are necessary and unavoidable. In other cases, weird interop-y decisions (MS office automation, anyone?) are made by developers who are not thinking about the deployment headache they might be creating.

Jeff Atwood on December 7, 2004 9:45 AM

"MS office automation, anyone?"

cheese.

sam on December 7, 2004 1:52 PM

I was just thinking that a decent midpoint would be Virtual PC images and differencing disk images.

I agree that each developer should be able to do the build on his machine, etc., and I also agree that the build machine should be easy to recreate.

So, do your builds inside of a Virtual PC image. Create your image, set up differencing disks, and then archive the image. Once the build is done, reset the image. You can even include your automated tests in another image should the need be.

If a developer needs to reproduce an issue on the build image, he doesn't have to commandeer the build machine. He just has to copy down the image.

Admittedly, using VPC will slow down the build, but the benefits could very easily outweigh the negatives.

Michael Russell on December 10, 2004 12:56 PM

A virtual build machine is a definite improvement over a physical build machine. I still would prefer not to have one unless it's absolutely necessary.

Jeff Atwood on December 10, 2004 3:39 PM

Obviously, you guys have been dealing with this perfect build quest for a while now. I, on the other hand, have a different perspective. I'm fairly new to the "real world", and have only worked for a company that had the isolated build machine. I never know much at all about what went into deployment; only that it was complicated. Too complicated for a neophyte like me. However, it worked and it worked well. QA built. Anyway, now I'm here at my second place of employment and lo and behold, they don't have a build machine. Heck, they don't even know who built the last release. Or, what went into it. OR, what version of Delphi the code was built with, and in some cases, where that code is. (We currently have 4 programmers and three different versions of Delphi.) Also, VSS was apparently abandoned long ago because basically nothing ever builds straight from "get latest". I always have to ask someone, "Hey, what files do I need?" which is followed with "Oh, here, you need this. I didn't add it to sourcesafe yet. " Jeesh. And, I've been given the wonderful opportunity to develop SOP's for our little shop, including pulling everyone together and defining some sort of work flows. Basically, my boss wants me to get us all on the same page. Honestly, I'm not sure where to start. I figured I'd start with standardizing the build process. If anyone has any insights, I'm all ears. For now, I'll keep trying to talk everyone into at least using their shiny new DElphi 2005 that they've had for about 5 months.

Mary Prochaska on September 6, 2005 2:44 PM

> Anyway, now I'm here at my second place of employment and lo and behold, they don't have a build machine. Heck, they don't even know who built the last release. Or, what went into it. OR, what version of Delphi the code was built with, and in some cases, where that code is

LOL! Sounds like you have your work cut out for you..

My general advice is, for what it's worth, is to make things simple. Remove any barriers or dependencies that prevent the build from being dead easy. XCOPY deployment is the holy grail:

http://www.codinghorror.com/blog/archives/000243.html

Jeff Atwood on September 6, 2005 3:28 PM

Is XCOPY for .NET technology only? Our products are Win32 projects.
I have the network guy setting up a machine with XP and I've ordered the Microsoft Virtual PC software - should be here tomorrow. I'll set up a couple different build environments (one module is built in delphi 6 and another in delphi 7) and apparently Delphi has some differences there, per my boss. As for eliminating dependencies, I have a big one! Microsoft Office - we use Access for our database. Ugg. I've suggested XML but that was quickly shot down with complaints of poor security. I'll have to research that more. I can't imagine that MS Access is so secure.

Mary Prochaska on September 8, 2005 11:29 AM

Where can I find out how to write a build script without using Ant??

Mary Prochaska on September 8, 2005 12:33 PM

> Is XCOPY for .NET technology only?

Pretty much, since the .NET framework is the only significant dependency.

> I've ordered the Microsoft Virtual PC software

Sounds like you're on the right track..

> Where can I find out how to write a build script without using Ant??

Ant is java-oriented, just like MSBuild (soon to be released with VS.NET 2005) is .NET and C++ oriented. I am not sure if there is a Delphi specific build script language or not.. I found this thread for Delphi, though:

http://www.cmcrossroads.com/ubbthreads/showflat.php?Cat=0&Number=47826&page=0&view=collapsed&sb=5&o=&fpart=1

Jeff Atwood on September 8, 2005 1:25 PM

Nice essay Jeff. I would have trackbacked it, but didn't see here the url was. Looks like I'm facing the same thing others have in this comment trail. I found a Delphi Ant clone called WANT on sourceforge, but noticed it appears to be abandoned :(. I guess I'll leave a note here if I come up with a build script. I'm hoping to get my team off VSS and onto CVS at the same time too. Wonder if this task will doom me.

here's a link to my trackback since I couldn't find the url to use: http://patf.net/blogs/index.php?blog=2&title=magic_builds&more=1&c=1&tb=1&pb=1

Patrick Farrell on January 2, 2007 2:18 PM






(no HTML)


Verification (needed to reduce spam):


Content (c) 2009 Jeff Atwood. Logo image used with permission of the author. (c) 1993 Steven C. McConnell. All Rights Reserved.