programming and human factors
by Jeff Atwood
Hacknot's If They Come, How Will They Build It? is a harrowing series of 29 emails sent over a two week period.
To: Mike Cooper
From: Ed Johnson
Mike,
I finally got CVS access today from Arnold. So I've checked out the AccountView module OK, but it won't compile. The Eclipse project has dependencies on about five other projects. I tried checking those dependent projects out as well, but a few of them won't build at all? How are you managing to develop this thing when the dependent projects don't build?
Ed
From: Mike Cooper To: Ed Johnson
Oh yeah - I forgot to tell you about the dependent projects. I always forget about them. I'm not so surprised some of them don't build for you. I've got versions on my machine that build OK but I haven't checked them in for a while. Gimme about 15 minutes and I'll check them in, then you should be right to go.
M.
It's a cautionary tale about a serious software project pathology: the pain of getting a new developer up and running on an existing software project. It's startlingly common.
This points us to one of the most important health metrics on a software development project. How long does it take for you to get a new team member working productively on your project? If the answer is more than one day, you have a problem. Specifically, you don't have a proper build process in place.
I've talked before about the importance of a build server as the heartbeat for your project. A sane software development project has automatic daily builds, performed on a neutral build server. If your team is in the habit of producing those kind of daily builds, it's difficult to accumulate the deep technical debt enumerated in all those emails. If the build server can do it, so can your newly hired coworkers.
But based on the development practices I've often seen on site with customers, I think setting up a build server might be an unrealistic goal, at least initially. It might not get done. We should shoot for a more modest goal to start with.
Here's how most clients I work with build a project:
If your "build process" is the F5 key, you have a problem. If you think this sounds ridiculous-- who would possibly use their IDE as a substitute for a proper build process? -- then I humbly suggest that you haven't worked much in the mainstream corporate development world. The very idea of a build script outside the IDE is alien to most of these teams.
Get your build process out of the IDE and into a build script. That's the first step on the road to build enlightenment.
The value of a build script is manifold. Once you have a build script together, you've created a form of living documentation: here's how you build this crazy thing. And naturally this artifact is checked into source control, right alongside the files necessary to build it (and even the database necessary to run it, too). From there, you can begin to think about having that script run on a neutral build server to avoid the "Works On My Machine" syndrome. You can also consider all the nifty ways you could enhance the script with stuff like BATs, BVTs, and Functional Tests. Your build server can become the heartbeat of your project. There's no upper limit on how clever you can be, and how many different build scripts you can come up with. Build scripts can be incredibly powerful-- but you'll never know until you start using them.
The F5 key is not a build process. It's a quick and dirty substitute. If that's how you build your software, I regret that I have to be the one to tell you this, but your project is not based on solid software engineering practices.
So, if you don't have a build script on your project, what are you waiting for?
Back to the tools:
I would definitely suggest Maven2 as a build system in Java. The difference between Ant and Maven is that Ant does not protect developers against themselves. It is script after all and is fairly easy to mess up.
On the other hand Maven is not that powerfull in terms what you can do during the build, but you get a huge benefit in standardization.
Maybe you have noticed that many open source projects leave Ant and start building with Maven;-)
"How long does it take for you to get a new team member working productively on your project? If the answer is more than one day, you have a problem. Specifically, you don't have a proper build process in place."
I totally agree that F5 is not a build process. Actually it's never meant to be one. But what a build process has to do with a new guy in the team? The new guy may be a trainee, a fresh college graduate. Lots of software houses don't allow newcomers to commit code in the source repository for some time.
Jeff - please spent some more time reading your draft posts. Those two lines spoiled the entire post... Don't lower your standards.
I think the take-away here is that:
(a) all and any member of the team should be able to retrieve a current, working copy of the code in a simple manner.
(b) all code should be checked against a "clean" system to ensure it compiles without undocumented/unscripted tweaks.
(c) checking in code is the equivalent of saying "this portion is done" - if it doesn't compile, you're not done, and don't check it in.
this is just one point of view, all it depends on size of your project, number of contributors and dependencies. adding proper build process into your project will require at least one more new dependency (http://www.codinghorror.com/blog/archives/000497.html) and add some more new code (http://www.codinghorror.com/blog/archives/000878.html) which have to be maintained and that's not a good thing. so really, we're talking here more about trade-off than requirement.
lubos on November 1, 2007 3:40 AMAre there still people that have code in their source repository that doesn't compile? That's probably the most strictly enforced rule around here.
Jasper Bekkers on November 1, 2007 3:42 AMI've been meaning to get build scripts setup since I started the last project but I haven't gotten around to it because as you say, F5 is just easier. How do you build and run / debug an application? Or do you hook up F5 to just run your build script?
Mark Ingram on November 1, 2007 3:57 AM@Jon - I have to assume that either you only ever work on one-man projects, or you hate your colleagues. ;)
Ciaran on November 1, 2007 4:00 AMJon,
Not everyone who builds software continually checks in perfectly working code. If you are working on something that you aren't going to get done in a working day and will break the build if checked in, branch that development and merge it into the main trunk once you are done.
Peter
Peter on November 1, 2007 4:01 AMGenerally in the open source world things are better in this regard, for a couple of reasons. The obvious one is that most people would like contributions from others, and the non obvious one is that there is not a standard IDE that one can hit F5 in :). Unfortunately in my experience some projects are still very difficult to build which is a huge barrier to entry for most.
Hmm, I should add a "ease of build" metric to my "open source openness score":
http://www.pixelbeat.org/docs/oss_project_quality.html
cheers.
Pdraig Brady on November 1, 2007 4:10 AMKeeping code that doesn't compile in source control shouldn't really be a problem if your source control system supports something similar to Team Foundation Server's "shelving". It lets you push checked out files up into a temporary storage area that is still in source control, but not in the main branch. Then your existing checkouts get undone. When you are ready, you can come back and pull down the shelved files and your checkouts get redone.
Admittedly I don't have much experience with source control outside of Source Safe, TFS, and Vault. Other products may support something similar.
kettch on November 1, 2007 4:10 AMTwo years ago we tried to make a dedicated build server, it was a total failure for our middle-sized Windows C++ project.
To make the build outside the IDE, you have to create your makefiles and a few scripts for other jobs, like invoking 7zip to create the binary data file. We wasn't lucky with our project leader, he was a python guru, and he insisted that our build script would be made in python. So it was. The build system emerged, and everything was working.
A few days later a pile of new source files were added to the project. The scripts had to be modified to include those files in the build. Here came the first problem: it turned out that only 2 of 8 programmers have knowledge in makefile and python syntax. Modifying the project content would need one of these two guys.
We also struggled to synchronize the scpits with the IDE settings. There were times when the project built finely through the IDE but took hours to find why the script won't work. Lots of hours.
Two months later the "build server" idea was dropped. It didn't help us to find any problem, but took much work to maintain. I don't doubt that in some places the build server is a good idea, but for us it was a waste of human resources.
Peter Juahasz on November 1, 2007 4:11 AM"The F5 Key Is Not a Build Process"
Of course not. It's F7, dummy. :-)
Seriously, with IDEs (eq. VisStud) there are multiple issues. You need to get the script to use the same options to the compiler as VC does, and when the build script yields results different from the IDE the blame is always with the build script writer. :-(
We went as far as to generate the .dsp files from a description, so not everyone has to patch them themselves.
Andreas Krey on November 1, 2007 4:12 AMWell, just to remind you, Visual Studio project file is MSBuild script file and so each project can be built using "MSBuild MySuperProject.csproj" command. So most people have build script already, but they never use it outside Visual Studio, which is where you are completely to the point.
Oleg Tkachenko on November 1, 2007 4:30 AM@Jon
If you really don't want to branch and merge, which would be the best way to do it, you could always check in more frequently, and immediately revert the changes. Everything is source controlled and daily build doesn't break. /hack
Hear hear!
One thing that's often overlooked are database builds. Even when a project has a centralised build process, it's often hampered by all the develpers working against a central development database, not under source control, just backed when someone wants to deploy to test.
I wrote about it here:
http://mikehadlow.blogspot.com/2006/09/how-to-do-database-source-control-and.html
And Secret Geek has a good discussion here:
http://secretgeek.net/dbcontrol.asp
Within our company, most projects are a one-person deal. So, the F5 key very much becomes the build process.
As others have mentioned, there are times when I am making a change which may not be completed by the end of the day. I still check them in. Sure, I could branch and merge, but since I'm the only one working on it, that's overhead that gains me (or the company) nothing.
caloggins on November 1, 2007 4:42 AMEasier build processes would help many open source projects, I'm sure.
Ever tried to compile Thunderbird or Firefox on Windows? I wanted to contribute to TB, add some desperately needed features like default templates, but couldn't build the damn thing. I set up a fresh VM and followed the instructions, only to get build errors so gave up after hours of trying. Now TB doesn't have default templates, a really basic feature.
Someone suggested that a Linux based VM should be made for TB, which would could be downloaded and run on any platform. That way, the build environment would be set-up for developers already and they could just start coding. I think the biggest issue was licensing though, since on Windows TB only builds properly in VisualC. VC is free but probably not free enough to include in a VM.
MoJo on November 1, 2007 4:49 AMI go the other way: the IDE is my automated build. That means that when you debug, you're using the exact same build instructions that you'll be using to produce your final released code. Going the other way, trying to get the debugger to debug something you built with a different build process, is always an exercise in frustration because the IDE wasn't really designed to do this. The makefile project is a complete cop-out.
To do a command-line build, you can still use Visual Studio - devenv /build (or if still using VS6/eVC, msdev /make). The problem with this approach for a build server is of course that you need a copy of the development environment on the server. For .NET 2.0 projects, MSBuild can handle .csproj and .vbproj files, but can't do C++ projects - for that you need VCBuild which as far as I know only comes with Visual Studio (not the .NET Framework SDK).
I should note that I mainly work with mobile devices running Windows Mobile or OEM Windows CE platforms. Traditionally, with eVC 3.0 and 4.0, it was such an utter pain in the backside to get a command-line environment set up to build, assuming you could then get a makefile. Why? Because the platform-specific defines were configured in the registry!
Mike Dimmick on November 1, 2007 4:49 AM@Ben Moxon my two main languages are PHP and C++ both of which I don't want syntax errors in when they are in source control. They might not always be tested, but having code that compiles makes merging conflicts a lot less painful, it also makes deployment to the test environment less painful (no more, "what was the latest revision that *did* work?") and it's easier to add new people.
Jasper Bekkers on November 1, 2007 4:55 AMSome of the comments on here so far are really scary. Having a build script should be one of the very first things on a project. Admittedly, it is much more work to build one for an existing project, but it is absolutely well worth the time to do so for the reasons Jeff mentioned in the article - getting a new developer going and getting a CI build going.
People who say that CI servers (and build scripts) add no value (or worse - reduce value) must not have worked on a project where they have been used properly.
The project I'm working on at the moment has had a build script and CI build from the start. Our checkin process is:
- developer gets new functionality going (with appropriate tests) in their ide
- they run a pre-checkin target in the build script which runs all of the unit tests (around 2500) and selected integration and distribution tests
- if these all pass, they checkin and the CI server does a full build which runs all the tests and publishes the generated artifacts.
Granted, keeping the build scripts up to date requires more work than pushing F5, but having a repeatable build process more than makes up for it.
I get the feeling that some languages provide better tools than others for writing/maintaining these scripts, but that shouldn't be a reason not to use them.
(This feels like a bit of a rant - but I know my life is much easier with all these things set up)
mg on November 1, 2007 5:05 AMHi,
Build process is not just about F5. I totally agree. And I cannot rant more about an automated build process and continuous integration and build scripts. I agree to that also completely. But can the new employee ramp up be resolved by continuous integration?
So what I am saying is a new employee gets a new laptop. He gets a URL to his source code repository. He fires up his source code repository client, links to the URL, gets the latest source code, opens it up in Studio and starts enhancing. the moment he checks in, the code is re-built and all the integration procedures of running unit tests, applying standards and coding rules, checking for code coverage, etc is triggered. does this happen?
There are a no of other parameters. A codebase might fail because of a digital certificate not installed properly, or proper authorization not given to the database server. We tried to tackle this problem using Virtualization. The code base is installed on a Virtual PC and is given away to a new developer with all environment variables set. The only thing that remains is to give him proper authorization. Even after all this effort we are struggling with the standardization procedures. And still the ramp up procedure takes over a week.
Of course there are soft reasons like non-availability of the right people to help you with it, etc. but even so...
We are not planning to prepare a webcast specifically designed to ramp up the new intern. And a no. of such initiatives.
Do let me know what you think...
Nilotpal on November 1, 2007 5:10 AMAUGH! Jeff, you just hit my Big Issue At Work.
I agree with you wholeheartedly. I'd been working with a client whose build and SCC process is SourceSafe, F5, and ZIP files for packages (all manual, mind you). And there was talk of "evaluating TFS" for its "Project Management features".
My response to that:
"FUCK NO".
Why? Because you're not there yet. Before you can/should work on creating a heavyhanded TFS development "process" (not that I believe you even should), you need to be a couple steps higher in maturity:
Level 1: SOURCE CODE CONTROL
You can't do JACK unless your projects are ACTUALLY in SCC. That means not in VSS. Don't have any money? Learn and use Subversion. Wait 2 months for the "new" SVN way of doing SCC to sink in for your development team. Make sure everyone on the team really knows the ins and outs of SVN. Don't proceed to any other step until you've done this. This is the bedrock of all non-joke software development.
Level 2: A BUILD FILE
Make a build file. Have it MSBuild your .sln file initially, so it works. Choose MSBuild or nAnt. They're both good. nAnt is a bit more mature, MSBuild is getting there, but if you're a pure MS shop it's probably the one for you, since you'll be able to learn some of the ins and outs of MSBuild by studying your .csproj and .sln files. Put your documentation, deployments, everything you need in the MSBuild file. Make sure it builds on couple of peoples' machines.
Level 3: A BUILD SERVER
Use CruiseControl/CC.NET or equivalent. Get your build server up. If you can convince the Man, put it on a REAL server with a LOT of CPU, disk, and memory. It should reference your SCC provider and make your builds. You're now at the level of maturity Jeff's talking about. Wait 2 months. Get the whole team up and on the build server. You'll never go back.
Level 4: TESTS! (Yes, this is Level 4).
If you've been using F5 and VSS, I'm willing to bet a $20 that you don't have any unit tests written. Work on them. Add the tests to your build process, and make sure your build server runs them with each build. Soon, maybe in a few months, you'll discover that you need to write your tests FIRST. Start going the way of TDD.
Level 5: POLISHING UP
Since you're using VSS and F5, you don't have a serious deployment plan. Refine your build file. Cut out as many manual steps as possible to DEPLOY your app. Use WiX. Use InstallShield, I don't care. But XCOPY deployment is not a real, viable deployment strategy. (Does it create your event logs for you? Does it upgrade your databases? Does it roll back in the middle of an install if something goes wrong? Didn't think so)
Level 6: MORE ADVANCED PROJECT MANAGEMENT
Now you're ready to add tools that integrate estimation techniques into your project. Your project build and deployment process is now repeatable enough that your estimates are no longer a joke.
Don't have a big project to do this on? A lot of "one man" projects? Guess what. You need this even more. What happens when you go on vacation and there's a critical bug? What happens when you quit? How do you know that someone has all of the files they need to completely deploy your app when you're gone?
It is real shame the coding horror author always gives advice about how to things, but he never gives any advice on tools. I would really like it if he would once on a while blog about essential tools which everyone is on the look-out for.
:-)
Steve on November 1, 2007 5:15 AMAt the end of the day every software development team needs a way to produce "official" builds for regression testing, deployments to Test group, and deployments to manufacturing/GA release team.
Official builds are "official" since they are stamped with a version number, compiled with fixed build/configuration flags, tag/checkpoint via source control with a unique build tag, and always compiled with fixed external library versions.
So it's not important whether you use "make", a batch script, or a GUI button, what's important is that you need the PROCESS of developing a official build with a CONTROLLED build environment.
Because if you leave it to some "developer" who hits "F5" to generate a new build and put it on the "file server", there is no way you can accurately verify that the "developer" is using pristine source code from source control (e.g. he could have modified a file and no one knows about it), no way to verify if he is building with the right library versions, no way to verify what build flags are being used.
I'd really like to see IDE packages (I'm talking to YOU, Eclipse) be able to use a build file as the build process instead of it's own build process. You'd be guaranteed that the build-script that is part of the solution/project/whatever is the same in the IDE as on the build server.
Eclipse/ANT would be good.
Visual Studio/Whatever Scripting Language would be good, too.
g
Garret on November 1, 2007 5:28 AMWhile we use a build script to handle all builds for deployment to our test or production servers, we don't use the script to handle builds on our local development machine.
We are building a very large ASP.NET application where relying on a build script to do a complete build every time we want to test changes to a single page would be a major headache. Luckily for us Visual Studio lets us build just that one page and the assemblies that have been modified by our changes. This results in a much shorter build time to do our testing and a lot less wasted time during our development.
I hope you don't mean you should never use your "F5" key to run in a development environment. I would think it would be enough to just have the build script be used for deploys. This will help try to keep the process cleaner and more reliable. I am not a fan of the often practiced Visual Studio build then xcopy the results deploy method.
John Chapman on November 1, 2007 5:36 AMWe use f5 to build, and have ccnet do an automatic build on every checkin using a giant solution file which includes every project we have. We unload projects we are not working on as needed so f5 is still zippy. Running the whole build script would take far too long to do on each build, so f5 suits us perfectly - in fact I would adovocate its use as that was VS provides a convenient way to select only the projects you are working on.
This article is falsely accusing the f5 key, when the problem is people not running a ccnet instance.
Harry M on November 1, 2007 5:41 AMOne thing overlooked in the article yet painfully noticed by some commenters: a build script requires a build script maintainer. A build server requires a build server administrator.
I have been that guy for several years in several companies. Observations from the trenches:
- Doing this kind of thing in a simple manner is relatively straightforward. At the first sign of scaling up, issues become rather more involved, quickly surpassing the amount of time any individual developer can reasonably be expected to spend 'not developing'.
Therefore: don't keep it under the radar. Make a case for it and get it into whatever process you have that decides on effort allocation. Preferably (imho) argue for someone to gain centralised expertise to avoid wheel re-invention.
- Many competent developers don't get source control. Or branching. Or deployment to anything beyond 'another directory on the localhost server'. I find that this is disturbingly common amongst even otherwise rather capable smart people. In other words: one can follow a learning path towards Capable Developer without ever noticing the side street labeled 'development infrastructure'.
- Build test environment deployment etc activities do not exists within the development team perse. In many environments, this is where you will need to get to know the sysadmin folks beyond getting your desktop working and telling them to punch a hole in the firewall for your Voip toy. Seriously, there are considerations in deployment to live environments that are alien to many a dev team that has led a sheltered life.
Point to make: if you create a position (even part time) with responsibility for the development/deployment infrastructure, be aware that it needs to be within throwing distance of both dev and sysadmin people.
- Once you have infrastructure to build automatically, deploy automatically, run a test suite, etc, you might find that there are more processes that might be extracted (or even consolidated) from other places to be handled centrally, by the same folks. Examples:
= coordinate documents produced by technical writers and bundle those into a documentation product to go with the builds. Automate that.
= align the requirements of projects' building steps and test suite (with their release frequency) with provisioning of server environments. (rather than sysadmin working on their own schedule and budget). Incorporate virtualisation in this concept. Let sysadmin folks worry about network, wider hardware issues, whatever, rather than installing identical stuff over and over again.
= provide a whole new dimension of capabilities for your QA people by giving them the opportunity to order testable (virtual) machines with very specific code content loaded at specific times.
(mind you: these are applicable to larger teams, rather than three-man shops)
Schmolle on November 1, 2007 6:01 AMYour nightly build has to be complete. This means not just the source code but any custom tools that are used in the build process. As said in other posts, the build process' output should be your installation package and any documentation that you build.
If you are using a good source control system than use a private branch if you are going to work on something that will take you days or weeks and that is incompatible with the rest of the project.
We have a product with around 100 VS projects. We have a tool that steps through a list of projects and spawns off a task to build each project from the command line. This calls devenv directly. This way we are using the setting in the .vsproj and .csproj files and we get the same build from the script as we get from the IDE.
Our nightly build script starts by deleting all of the files that are not in source control (in ClearCase terms view private files). It updates the view, builds the tools, builds the product, and creates the installation package. Our .vsproj files have 3 configurations: debug, release and non-optimized release. The non-optimized release is the same as the optimized release but with optimization turned off. We use non-optimized release when we have a crash that we can duplicate because it gives us performance closer to the product and can be debugged because the source code matches the machine code.
We have two branches: A release branch and a working branch. This minimizes the chances of a file check in breaking the build.
asm on November 1, 2007 6:06 AMWe use the 'team project set' within Eclipse to get a new developer started. You can export this file from Eclipse, give it to the new developer, and have him import the project set. Eclipse will then pull all the projects in the set out of CVS for him. It's really handy and can be done within minutes.
I think focusing on the build step is a mistake. If you are really looking at what it takes to make a developer productive there are many more steps:
- installation of the dev environment (IDE, runtimes)
- installation of dependent libraries and packages
- installation and configuration of source control client
- how to check out the repository
- installation and configuration of other subsystems (database, web server, etc.)
- local configuration (DNS, hostnames, logins)
- building the system
- running the system to verify everything worked
In our world - Python web applications - the build step doesn't exist. But all the others do, as they likely do in every project. Creating a checklist on a wiki that walks someone through the drudgery of getting everything setup properly is a big win in my eyes.
At the project I am currently working on, we do have a server that every night checks out all code from CVS, does a full build, sets up the test databases and runs the complete set of unit tests with code coverage measurement. Every morning someone looks at the report from the nightly build to see if there were any problems (compile errors or if there are failed tests).
Having a nightly build process on a server is important, and it should also include automatically running unit tests.
We're using mainly Java.
Garret said: I'd really like to see IDE packages (I'm talking to YOU, Eclipse) be able to use a build file as the build process instead of it's own build process. You'd be guaranteed that the build-script that is part of the solution/project/whatever is the same in the IDE as on the build server. Eclipse/ANT would be good.
- There has been Ant integration in Eclipse for a long time, didn't you know that?
Someone asked about tools to do builds. In the .NET world you could use NAnt.
Tero Vaananen on November 1, 2007 6:49 AMAs I read "The F5 Key Is Not a Build Process" ... I thought this post would be about usability ... because in most applications (at least on Windows) F5 is the refresh key.
Who put that thing on "Start Debugging" ...
Hinek on November 1, 2007 6:54 AMYour post basically says: Build outside the IDE, Build outside the IDE, Build outside the IDE, Build outside the IDE, Build outside the IDE
Please give some good reasons too. You know, one could just run the build from command line even if the configuration is made in the IDE.
You need an argument on November 1, 2007 7:00 AMI've never worked on a project with daily builds but I've always worried about the idea.
Afaik, in most places there isn't a "daily build", there's a build server and it starts a build as soon as there's a commit.
It may take me several days to make the changes and get them all working. With daily builds I have to keep these changes checked out so they don't break the build
Or you make your changes incrementally so that at each step the whole system compiles cleanly (and passes the test).
The other alternative is to use DVCS or patch-handling tools (see Quilt) to have a local history on top of the remote one
I would much prefer to check in my non-working code as regularly as I can (hopefully several times a day) so that I can see at a lower level how my changes happened and where a problem might have occurred.
Does this mean you have no problem with breaking the build for all your coworkers? Whoa...
A build server is merely another coworker's working copy. If you break the build, you break it not only for you and for the server, but for every coworker who's had the misfortune to update recently.
Is there a way round this problem when running a daily build?
Only commit working code.
adding proper build process into your project will require at least one more new dependency (http://www.codinghorror.com/blog/archives/000497.html) and add some more new code (http://www.codinghorror.com/blog/archives/000878.html) which have to be maintained and that's not a good thing. so really, we're talking here more about trade-off than requirement.
Don't you think a dependancy on the closed inscrutable build artifacts of an IDE is a bit more worrying than on a plaintext script of a (usually) open-source build tool?
Within our company, most projects are a one-person deal. So, the F5 key very much becomes the build process.
yeah but... what happens if the dev in charge of a project leaves and someone else has to take over?
But can the new employee ramp up be resolved by continuous integration?
I think jeff's point was more along the lines of: an external build tool/script gives you a central place where *everything* necessary to build a project is/should be visible in plain text.
Thus, a new employee wastes no time trying to track down deps and maybe-working code, check out, build, done.
And if he wants/needs more knowledge of the organization of the code and build process, he can just open the build file (and, often, learn the build tool's syntax)
There are a no of other parameters. A codebase might fail because of a digital certificate not installed properly, or proper authorization not given to the database server.
This should be handled by one of the build's targets.
A complete build system/script should allow you to take any clean machine (with all the pre-requirements, usually the build tool itself and the compilers/runtimes your language-of-choice uses), checkout the code, launch the build, launch the application.
If there's any other step involved, then the build system is incomplete.
It is real shame the coding horror author always gives advice about how to things, but he never gives any advice on tools. I would really like it if he would once on a while blog about essential tools which everyone is on the look-out for.
That's because there are dozens of tools for each task. Furthermore, Jeff is a windows developper, so the tools he suggests usually wouldn't work for people e.g. developping Erlang software on Solaris.
I'd really like to see IDE packages (I'm talking to YOU, Eclipse) be able to use a build file as the build process instead of it's own build process. You'd be guaranteed that the build-script that is part of the solution/project/whatever is the same in the IDE as on the build server.
Are you sure Eclipse can't yet? I'm pretty sure IntelliJ allows you to specify Ant or Maven files to build the project through, so if it isn't in base eclipse I bet there's a plugin that adds that.
We are building a very large ASP.NET application where relying on a build script to do a complete build every time we want to test changes to a single page would be a major headache.
No build script should rebuild *everything* unless you explicitely *ask* it to.
The partial rebuild features has been available for years, make had it before I was born or something.
Many competent developers don't get source control. Or branching. Or deployment to anything beyond 'another directory on the localhost server'.
Then they're not competent.
I find that this is disturbingly common amongst even otherwise rather capable smart people.
and they're not smart either, sorry.
installation of the dev environment (IDE, runtimes)
usually manual. Better, as a new employee your machine should be preconfigured with everything you'll *need* the moment you sit at your new desk.
installation of dependent libraries and packages
the dependant libraries are in the source tree, thus part of the build system
installation and configuration of source control client
it's part of the dev environment, and was thus preinstalled at step 1
how to check out the repository
get the repository URL, checkout the project, done.
installation and configuration of other subsystems (database, web server, etc.)
should be configurationless or handled by e.g. an "install" target in the build file
local configuration (DNS, hostnames, logins)
there is no reason why a dev should have to do that, it's part of the dev environment and should thus have been done at step 1
building the system
and running the test, all handled by the build system
running the system to verify everything worked
yep.
In our world - Python web applications - the build step doesn't exist.
That's because you restrict the word "build" to "compile", which Python doesn't need.
The concept of building goes far beyond the mere act of compilation, or it would be called "compile".
Masklinn on November 1, 2007 7:01 AMHey Now Jeff,
Good Post. After reading this I understand that a build script is much better practice than F5. You've taught me when building a doghouse the F5 key may work fine, however on a skyscraper a script is needed if not a dedicated server.
Coding Horror Fan,
Catto
Day one, line one, must compile.
How does it compile ?
Write a one or two line script...
Continue to develop, make as few assumptions on build environment
If a developer wants to use an IDE, hotwire a key to call the script from within that IDE or Editor.
Jeff is 101% right, F5 is not a makefile.
David Ginger on November 1, 2007 7:06 AMI love build scripts. I am a Linux guy though, however I started in the world of IDE development in Windows. I am not sure what a Visual Studio build script would look like in comparison to a Makefile.
I also have grown somewhat fond of cmake. I hated it at first, but it now seems great for cross platform solutions.
Though, none of that matters for interpreted languages. How do these concepts change for a system that lacks a distinct build step? I have been doing PHP for a while now, and don't have a concept of a build verification /authentication test.
Setting up a developer is as straight forward as making a new database and creating a virtual directory with a source copy on the test server. Is this a benefit for languages like PHP or Perl/Python etc. or am I doing something wrong?
Great post though Jeff.
I wish I could find a way to make your blog mandatory reading for undergraduates at my university(I am a graduate student).
Do apps need an overnight compile nowadays?
The last app i worked on that required a long build was in Powerbuilder 6 which had a bug in the compiler where it ignored incremtal builds and did a full each time!
And
What does/should the script do when widgetA.dll doesent compile? The whole nights build is wasted as changes to TheApp.exe probably relied on the updated widgetA.dll, and a prior suscessfull build of widgetA.dll is probably useless.
Adrian on November 1, 2007 7:12 AMHow do these concepts change for a system that lacks a distinct build step? I have been doing PHP for a while now, and don't have a concept of a build verification /authentication test.
run unit, functional and acceptance tests, build documentation (from docstrings/doccomments), run a lint-type tool, run other kinds of static analysis tools, create a bundled app (Py2App/Py2Exe) and test that, ...
All of these are as many build targets, some (building the app, running long)ass functional and acceptance tests) shouldn't be in the default build, others (unit tests, building documentation, running a lint) should be part of the default pipe.
Masklinn on November 1, 2007 7:18 AMI can't agree more - I've worked in so many environments where it took days just to get the project up and running. And when it came time to build a "release" for production, there was always one guy who we would let do it, because no one else really knew what to do with the installer, or he was the only one with the installer on his machine, etc.
I've only been at one place really that saw the value of a fully set up build server. Unfortuanetly they decided to write all of their own custom batch files and C# code to achieve this (instead of CruiseControl or something similar), but in the end, it was really sweet to be able to check in your code, go to lunch, and when you came back, there'd be a file sitting on the server called "Version2Installer.exe" all ready to go.
Great article!
Sam Schutte on November 1, 2007 7:19 AMA testimonial for a solid build process:
In my shop we use NAnt for builds and Cruise Control running on a dedicated server to automatically build everything, then run unit tests. The automated build typically runs several times per day. We've been using that system with great success on all of our projects including our main product (version 5.0 just went GA, thank you), for about four years. This project is a suite of Windows services and apps consisting of a tens-of-thousands of lines of C#, C++, and Java.
The system works very well. The only time there is a problem is if someone checks in code they've built only as Debug that does not compile or run correctly under Release, or checks in code that they have not run the unit tests on. When that happens, the group is notified by email. The offending party must, of course, quickly fix the problem, but also must put a dollar in the build jar.
The build jar, literally a cookie jar, is emptied every year or two and the money donated to charity. It's a terrific incentive to run those unit tests before check-in. The disincentive created by the build jar is not, by the way, so much the dollar you lose as the walk of shame as you walk over to the jar and put in your dollar.
The system is WELL WORTH the time and effort that goes into set-up and maintenance. I would never give it up.
Dave C. on November 1, 2007 7:49 AMI know this is a not typically a Java oriented blog, but can I take the pulse of people's opinion of maven?
I thought at first it was an over-engineered non-solution for non-problems that wouldn't work behind a firewall, but I'm slowly beginning to come around. I see its usefulness but I'm just not sold that the effort (its not trivial to understand) is worth it.
Jim on November 1, 2007 8:09 AMThe state of the art goes one step further: continuous integration. When you check in code, it's immediately run through unit tests, and you're contacted immediately if it fails to pass the tests (or fails to compile).
Note that build processes are not relevant for some forms of software.
iI propose an alternative: a 1-page document (it's almost never longer than this) detailing the prerequisites for a particular build and deployment. Keep it in SCC. If your entire dev team quits, all necessary information is there for another one to pick it up. If the document becomes longer than one page, or if you find that it has to be updated more than once a week - then consider moving to an automated build process/i
In my opinion documentation is useless unless it is executable. Specs and tests are great documentation, because you can prove them true or false simply by running them. Continuous integration is great documentation as well, for the same reason. Documentation is for computers to read.
OK, that "I propose" bit was a quote from a previous comment. It's 7:24 AM. Software shouldn't work only when its users are fully awake, fully caffeinated, and highly intelligent. This thing stripped my HTML instead of complaining. That was lame. Anyway, just read the quote as a quote and there you go.
Giles Bowkett on November 1, 2007 8:27 AMAs a few people have pointed out, Visual Studio solution/project files are build scripts. Admittedly you may have the limitation of being a bit more restricted on what tools you use for a build, but there is a great deal you can do there.
What's wrong is if the IDE (given the current state of tools) is you're build designer. For MSBuild all of the advanced stuff is only available through a text editor. But once you write it, it works in Visual Studio just as well as on your build server or from the command line.
So, to counterpoint. I think the goal should be to have a build process that is so good it is your IDE. What better way to make sure the developers and the build server are on the same page? Why have two different sets of files that describe how your projects are built? Why have the developers work with one script half the time, and then submit their changes to a totally different script they haven't tested with?
On another note, source control really needs to expand it's horizons beyond the mainline "working code" branch. I think it's getting there, but the more I look at the problem the more I'm convinced source control needs to be more than a repository alone. It needs to be the communication tool. It needs capabilities to allow you to shuffle non-working code around between developers without resorting to zip files and email. Of course, it has to do this without breaking the initial mission too.
Ryan Baker on November 1, 2007 8:31 AMI use PHP most of the time. What build? :)
Gregory on November 1, 2007 8:32 AMA++ for Jeff Atwood! Excellent post.
Except F5 doesn't do anything in my IDE...okay, so I'm using Eclipse and it's F11....potato/potatoe.
We have a 12 person development team and were using CVS and did nightly integration (don't remember what he was using...maybe CruiseControl). All Java code had to build every night off of CVS Head. Branches in this environment were the way to do alot of the things that have been mentioned. We're all using Eclipse and have chosen to do the Ant integration outside of Eclipse. Eclipse does support Ant build but the way we're doing things in Ant, it kind of "conflicted" with Eclipse. So we have a server build and deploy step that is outside of Eclipse and we have to manage our Eclipse configs slightly differently...a bit of a hassle but worth the trade off. Our legacy applications (also Java) would literally take days to get a new developer up and running. What was in source wouldn't build (we were a VSS shop back then) and it was impossible to recreate the production build. Now we can pull latest, which includes the necessary instructions for Eclipse configurations and have a new person (or rebuild production) in an hour! Well worth the pain.
In a small, 1 to 3 person shop it is admittedly tougher. It "seems" like a hassle but Jeff's statement about it being a living document is dead on. So many little projects that I've either worked on or inherited suffered from the "oh yea, and then we do that...forgot about that".
Unfortunately, our group refers to this in past tense...since our corporation "upgraded" our source control from CVS to something called Harvest from CA. Branching is a thing of the past and basically everything get's slammed into the trunk. Not a good system really. And our CM guy split and went to work somewhere else and the management isn't replacing him. So we're back in the dark ages really......
Great post Jeff.
Mike Shaffer on November 1, 2007 8:50 AMI recently worked (after six months I moved on) at a company that used build scripts but it took me days to get the bare bones of the application setup and compiling. I was told not to worry about the components that wouldn't compile because I wouldn't be working on those anyway.
Build scripts are no cure-all. As with most software development, it's about using a well thought out process, sticking with it, and improving it over time.
Gunther on November 1, 2007 8:56 AMIn our team, whoever breaks the daily build becomes the 'beer bitch' for the following Friday!
john on November 1, 2007 9:10 AMHow long does it take for you to get a new team member working productively on your project? If the answer is more than one day, you have a problem. Specifically, you don't have a proper build process in place.
The complexity of your code base is low enough that being able to build makes a team member productive in one day?
JR Tipton on November 1, 2007 9:16 AMGood god. This is exactly this predicament I'm in now.
Started at a new client (I'm a consultant)
- Got the source from VSS (which sucks already)
- CTRL + SHIFT + B = Failed Build.
- Me: ??????
- I then go hunt down all the dependencies only to find out that they are managed by people "emailing" binaries around the office and they then sit on a certain area of your folder.
NOW, if a binary changes and then no one tells anyone, EVERYONES code breaks because they no longer have the proper binary. SO now the ENTIRE #@$@#!#%@!!!'n team is running around wasting hours trying to find out where this new binary is.
Hours wasted = INSANE AMOUNT OF TIME.
The good news is that we're implementing TFS for them and fixing their dev tree structure so that a single check out will get them up and running.
Previously I ran on Subversion and as long as you got the recent version of the trunk, IT WOULD BUILD. Mainly due to Continuous Integration and a proper build process.
Donn Felker on November 1, 2007 9:22 AMGood post, thank you.
Sam Jones on November 1, 2007 9:22 AMSurely if you are using VS then you can just use MSBuild on the same solution file? Then you don't have to worry about the make file and the solution diverging.
CCNet and MSbuild + NUnit + FXCop = Build-Bliss
(We've not integrated FXCop yet but it's on my task list)
Omar on November 1, 2007 9:35 AMYup. Welcome to the REAL world Jeff. You're not in blabosphere anymore...
What this really gets to, is the fact (as Rocky pointed out) that business software development is just way too painful and difficult. Things unfortunately aren't getting easier with each .NET release, they're only getting harder and more complicated to be a productive and cost effective citizen. It's saddens me to think that .NET is on the same projectory as Java. Look where that is now. Lot's of Java folks defected to .NET for it's "initial" simplicity. But now, with all the Java-wannabees in .NET (like Palermo), it's headed down the same dark path. And this is "kewl" with everyone?? Pehaps if developers had to pay for the development process, it wouldn't be so cool anymore. Seriously.
You should read Paul Vick's latest post too Jeff. Sorry to see you compromise your standards and defect. Too bad for you.
VBMan on November 1, 2007 9:41 AMF5 IS a build process. Thing is, F5 can be replaced with
msbuild solution.sln
on the command line.
Dmitry on November 1, 2007 9:43 AM"If your "build process" is the F5 key, you have a problem. If you think this sounds ridiculous-- who would possibly use their IDE as a substitute for a proper build process? -- then I humbly suggest that you haven't worked much in the mainstream corporate development world. The very idea of a build script outside the IDE is alien to most of these teams."
"The F5 key is not a build process. It's a quick and dirty substitute. If that's how you build your software, I regret that I have to be the one to tell you this, but your project is not based on solid software engineering practices."
Unfortunately I've been a solo developer in my time with .NET (for the last six years). Every project I've been on, the process to get a working copy of the project has been to check it out from VSS and then hit the F5 key. I've always wanted to move into team development in a large corporate environment, however from your article it sounds like I'm not ready to do so until I can automate the build. I really enjoyed reading Hacknot's "If They Come, How Will They Build It?" blog post, and I was impressed that he suggested having documented procedures for setting up a new employees system. Can you provide me a process for "scripting" my projects? For what it is worth, I haven't scripted anything since a few .bat files for MS Dos 6.22, so bear with me and provide some detail in the required way to write and execute the scripted build process. By the way, I'm using VS 2005 and have both ASP.NET and .NET windows applications.
Thank you.
Solo Developer looking for a team environment on November 1, 2007 9:55 AMTo:AaronG
I worked in a shop where if you broke the build, you were in charge of the build server. Attaching a "punishment" to breaking the build gently encourages that people not break the build.
Now that I'm working somewhere else, I wish that I did break that build so I would have learned how to babysit the build server. My current shop does not have a build server; and I'm thinking of ways to propose bringing one in.
wes on November 1, 2007 9:58 AMExcellent post as usual, Jeff! I almost always learn something from your entries.
Cal Jacobson on November 1, 2007 10:08 AMWe're slowly moving toward a better build process here. I'm loving FinalBuilder + FinalBuilder Server for avoiding the angle-bracket tax in MSBuild, and for those special touches like emails and deploying to a client "demo" site.
Daniel on November 1, 2007 10:23 AMThank you Aaron G. You just saved me having to type that myself. Jeff asks "What are you waiting for?"
1. Management to see tangible benefits ("it's worked fine for 10 years as is")
2. Competent IT department to implement it
3. Time in my schedule not devoted to actually shipping the product
4. The Great Pumpkin?
At my gig we have a system that has not been able to be fully rebuilt for about 2 years. And, true to the corporate IT Dilbertism, the management hasn't allowed us to work on resolving that issue.
Having said that: one of the first utilites I had our team work on (okay: sort of a side project, as IT-mgmt would never let us spend good time on things like *that*) was a dedicated build capability that can roll-out deployables to both windows and unix. Sweet capability.
I hope we get a chance to use it when/if we are allowed to fix the software-based Sword-of-Damocles we find ourselves living with.
SurfnFL on November 1, 2007 10:40 AMAny professional software developer who is not:
1.) starting a software project based upon a design.
2.) developing software using source code control system
3.) using some form of built-in unit testing
4.) commenting in source files
5.) relying upon an IDE for anything other than code editing, GUI building and debugging
is simply not acting in a professional manner. These are not niceties, they are basic principles that anyone who claims to be a professional should be following.
Smitty on November 1, 2007 10:42 AMNice idea Smitty...
Pray you work for a company that has an IT and management team that recognizes that those "basic principles" that you mention are worth spending time and money on.
The sad truth is that many professional (and not-so-) software developers work in places that that is not the case.
Most of the time we were just invited to the dance, and others pick the music. And yes: this means that sometimes you just need to go find another dance.
SurfnFL on November 1, 2007 10:48 AMI don't really like the one-shoe-fits-all approach a lot of developers take. Yes, certain things may be necessary for certain people in certain situations. However, you can't prove that your solution is going to work for _everyone_ every time. There are almost surely edge cases that you haven't considered. If you think you've found an exception to this rule, you should seriously doubt yourself.
(The above is applicable for (almost) every all-sweeping argument.)
GWiney on November 1, 2007 11:03 AMVisual Studio actually lets you build a SLN or CSPROJ (or VBPROJ of course) file from the command-line, just run "devenv.exe /?" to see how. So there really isn't any excuse! We used this in Paint.NET for awhile, along with an intricate perl script, to do builds and send e-mails to volunteer testers on a daily basis whenever there were any changes (no changes, no e-mail). And of course to e-mail us if the build broke. Sadly I have not set this up again, but mainly because I'm the only developer now so there's no need for this type of coordination.
You might also add that automation of generating the installer is also super important (if relevant -- any client app will need this). Visual Studio makes it easy to generate an MSI, but if you use anything more complicated than what it allows and have an NSIS or other custom EXE wrapper, it's imperative that you automate it. I rolled the first few installer EXE's by hand and it's way too easy to fumble it.
Rick Brewster on November 1, 2007 11:42 AMWell I could not resist giving my 2 cents worth here. For starters, I just want to say it is an eye opening article for some, and some of the comments will definitely be appreciated by some.
I do agree with John Chapman's response and just wanted to elaborate on Schmolle's response.
For large projects with multiple development teams there is actually a professional CM (Configuration Management) Engineer. I actually managed a number of such teams, for a premier IBM partner building large bank systems that had 20+ developers broken down into different teams by layers, like the Presentation Layer, the Business/Data Layer, Integration Layer and the Database Layer etc.
Working with the IBM tools, we had the luxury of working with IBM Rational ClearCase, which solves all the problems people did not really find elegant and straight forward solutions for. ClearCase has been reviewed as the most sophisticated tool of its kind, much more advanced that VSS and CVS. I loved using these other tools in the past, espacially CVS, and would use again for small to medium sized projects meaning less than 10 developers.
Well besides nicely integrating with ClearQuest for automatic bug code tracking and WebSphere IDE integration, it has a solution designed specifically for the issues much talked about here, which is promoting partially functional code. Well it introduces Views, so a developer sees a particular view with certain files and commits to that view partially completed code as often as he/she wants. Then the next level up would be the streams, there can be multiple streams one for individual teams say per layer and unit testing and then an integration stream for when the different teams want to combine code, and run functional tests, be it automated or manual tests. What is great about ClearCase is that it allows every project to do its own CM design to fit the prject team and size's needs as well as gives you out of the box solutions called UCM. It also allows you to to do baselines from which different teams can start of, besides branching management which is really for the purpose of have having multiple projects off of a product line. I just want to say I actually never owned IBM stock, and really am a big fan of open source and use it every chance I have, and hence in the Java and Linux/Unix world. I use Eclipse most of the time in the past few years, and yes Eclipse has had a CVS and Ant plugin come standard with it for the past 5 years.
Moe Badawi on November 1, 2007 11:43 AMDefinitely have to agree. I created the build process at my work. My coworkers were a bit concerned over it, but they have all come around now.
This post inspired me to finally write my own post on the subject (I promised it in my blog awhile ago). http://brewder.blogspot.com/2007/11/creating-build-process.html
Brian Brewder on November 1, 2007 11:56 AMAs with testing, or even source control, people that haven't tried it judge it to be more complicated or time consuming that it really is.
In practice I found that the ability to check out a project and build it with a single command is invaluable. And I work with a small team, with 5 developers, normally distributed by 2/3 projects.
The time to get someone to compile one of our projects, from scratch? 1 hour! There are only 2 dependencies, java and ant (we are mostly a Java shop). Everything else is part of the build process.
And the time to maintain it? Well, the initial version is simple enough, when the project starts, and every time some of the devs adds a new dependency, it's his responsibility to include it in the build. Who better to do it? He/She has just added it to his/hers environment. Every step is fresh and clear.
"The value of a build script is manifold"
Would that be the intake manifold, or the exhaust manifold?
Eric on November 1, 2007 12:35 PMI strongly disagree. Your build process should be F5! It justs needs to be done properly. I have personally designed systems with several types of clients and several types of servers, plus a dozen shared libraries. All you need to do is check out the tree from SVN, open Visual Studio, pick the project (think executable) you need to run and run it. Two more clicks and you can instantly push a new build to all users and servers. It just works.
Furthermore F5 is a build script. Eclipse creates an Ant file for you, Visual Studio uses MSBuild, which is comparable to Ant. In both cases, the IDE writes the build script for you and gives you enough options to tweak it (including a text editor), if you want to (but ideally you don't). Therefore, I can take any VS or Eclipse project (probably the same for other IDEs) and compile it from the command line.
Although writing your own build scripts might give you geek credit, it is in no way effective, it's just an example of hubris overtaking reasoning.
Alek on November 1, 2007 12:37 PMWhen I worked at IBM in 2006, it took over two weeks to get a new employee access to five different AFS cells spread across three different sites. This was required to compile our application. The good news is, they had a build process for that project. The bad news: it took over seven hours to compile on 500MHz AIX machines.
Anonymous on November 1, 2007 12:46 PMAh thank you Jeff... one of my many battles... At least we're using the leading edge .NET tech at my current office, but this whole Build Server, TDD, Multiple Deployment environments, etc. stuff is still lost on my 12-man outfit.
Mind you, we only have 2-3 on any given project, so it doesn't take more than a day to get anyone on-line. But we learn so many new practices on each of our projects that we can't really seem to make a set stick.
*sigh* we'll get there.
Gates VP on November 1, 2007 12:46 PMI don't understand. If you automate the effects of pressing F5 (or whatever) by using "devenv [solution] /Rebuild" and specify the environment, is that a build process?
Or are you arguing that it's not a build process unless it involves a separate set of files? No, obviously not, that would be stupid.
I'm sorry, but your concentration on the keyboard shortcut completely hides any point that you're trying to make.
James S. on November 1, 2007 1:06 PMRegarding the "only check in code that compiles" discussion: I prefer to check in code that has only a relatively small number of changes from the previous version. There are at least two benefits: 1) you can roll back to any point without undoing as many "good" changes, and 2) you can accurately describe the changes in a short change comment, and build a good change log. So while it's nice if the code compiles, I think the "small number of changes" rule is more important. In a group environment, you can use labels or promotion groups to identify the last compilable version.
A. Lloyd Flanagan on November 1, 2007 1:53 PM"The value of a build script is manifold"
Would that be the intake manifold, or the exhaust manifold?
Eric on November 1, 2007 11:35 AM
No... just manifold. Buy a dictionary and try again.
Matthew Wilkes on November 2, 2007 2:09 AM"I use PHP most of the time. What build? :)"
To this and other similar comments about interpreted languages: building is not (just) compiling. In the (only) slightly advanced example that you are not developing PHP with Vi on the web server itself, but rather on a different machine, possibly using a localhost webserver of sorts, your build process might serve no other function than making sure you haven't forgotten about dependencies when deploying to another box.
I've defended somewhat involved build processes (with seemingly frivolous bells and whistles) on this basis: it allows you to do the work involved in deploying at the start of your project, rather than at the point where people are putting up the office decoration for the Grand Going Live of your precious app (when you find out that the live box for some obscure security-related reason does not allow the file system permissions you founded your architecture on).
Anecdotal counter-evidence: I once found myself (as the CM guy/deployment mule) painstakingly explaining to a Perl web app developer that 'no, really, even though it works happily on his local machine, it is not a good design to have his app write state stuff to the file system on a 15 machine distributed load-balanced server herd. If only because it will functionally break within seconds.'
Usable-towards-management-point in a nutshell: a seperate build/deployment process is a test in itself (of the infrastructural capabilities) which allows for significant end-of-project risk reduction.
Schmolle on November 2, 2007 2:35 AMDoesn't it heavily depend on the language? Pascal, C# and perhaps even some Java projects (or in that case Eclipse makes the build-script) don't need to make your own build scripts. Build-scripts became popular because of C++. But you do need to write dependencies and version numbers in the developers readme. BAT files are the worst things ever to have in a cross platform project. Might work in a corp. where everyone has the same OS (winXP sp2?).
Daniel Swe on November 2, 2007 3:00 AMJust finished building our build server...so far it:
-builds the complete dependancy tree (if needed)
-updates assembly versions with the build date and svn revision number
-tests
-perfoms code coverage (fails if less than 90%)
-generates NCover NDepend reports
-runs FitNesse tests
-creates documentation
-updates Sharepoint with docs reports
-creates a new branch and deploys to a dev server
-logs to a DB
cant do without it :)
I tend to use TestDriven.Net for local builds and leave the rest to the build server which runs on commit using CC.Net
John Shawcross on November 2, 2007 6:33 AMFunniest "F5" build process I ever used involves building twice, then cleaning, then building again. Absolute corker it was. Best thing is that the devs *still* commit code that breaks the build for a new starter.
What makes it totally chav-tastic is that the build server never bothers to get a new source tree from the source repository each night (or indeed ever). Therefore any build errors that result purely from never having built before will routinely get missed.
Yup, you guessed it, the build process uses post-build steps to copy things around that other, previously-built projects depend upon being there in the first place. What makes it even funnier is the same project will build subtly differently until its post-build-steps are run. Utter quality. An obvious sign of both intelligence and care.
This isn't a flame, but a request for information - could someone please explain Dave Markle's statement that Visual Source Safe isn't really source control (maybe Mr. Markle)? The comment seems to have passed unremarked, as if everyone already knows VSS is no good.
We use VSS at work - when you check stuff in it seems to be there, and when you check it out, it comes out as expected. What's missing?
At some point we'll want to upgrade, and I'd like to know why I shouldn't stick with VSS, since we're a Microsoft only shop.
Thanks for any response.
Steve on November 2, 2007 6:58 AM@ jon's: "I would much prefer to check in my non-working code as regularly as I can"
You can check in non-working code all you want. It code that must *compile* you must maintain all the time with your compiles.
If you really need to go off into lala land, use a branch. If your company disallows them, who ever did that is mistaken, for this is the *exact* thing they are best for.
--Michael
Most Build processes I've worked on use labelling - that is only files that are correctly labelled (usually with a CR number) will go into the build. This allows developers to check in unfinished code at will, safe in the knowledge that the Build Manager won't include it because it won't have a label.
Scripts scan through the source repository and create lists of labels so the BM can let everyone know what exactly went in. Sometimes this list is sent out before the build starts so people can approve it etc.
Jaded on November 2, 2007 7:24 AM@Steve: "We use VSS at work - when you check stuff in it seems to be there, and when you check it out, it comes out as expected. What's missing?"
VSS has had issues for years.
1. If your network/the internet is not swift, it's near impossible to use. Errors abound in this situation, and you can't always check out/in the code you need all the way. To put it on the internet at all, you have to set up tunneling, which frankly is not always a skill that goes with the target audience (it's more of a hardcore unix user thing than a pure VS shop type of thing). Visual studio can die if you use the integration while checking in code. This means you then lose the code that's not checked in (sometimes). That's exactly what I look for in a source control system.
2. Its damn useless at tracking externals. By externals I mean you can't reasonably check in multiple versions of an external library or a changing art pack developed elsewhere. It doesn't handle files that are removed between versions, and it doesn't make it easy to add them.
3. You can't check out build test editions. One good practice to make sure a dangerous change or large change can build for everyone is to check out the thing you just checked in to a /tmp folder then compile it there. If you can compile there in the tmp folder, there is a much higher chance you didn't break the build. VSS can't do this, because its one checkout per user. You actually need multiple users per people to do this with VSS, and you need to have your devevn setup on multiple machine users.
4. At least when I used it, history sucked. It took for-ever. Therefore, no one used it unless they catastrophically screwed something up. It should be used for all sorts of debugging.
5. Database corruption happens all the time. Oh god that's scary.
6. Oh yeah, renaming and deleting totally screws with your history. WTF mate. I want to be able to build version X easily, and version A too, even if I've renamed everything.
7. LOCKING ONLY DEVELOPMENT
This one hurts: Concurrent development is the only way to go for any project where people have to work together. Locking development takes *forever* to coordinate any project. "Hal can you unlock this, amy needs it, oh crap hal left for the day, amy wait until he gets back". With concurrent version control, the problem goes away. Sure, you occasionally have to use a merge tool with concurrent versioning systems, but you have 1/15th the coordination about working on things. People just can check in things much more often so there is often very little to merge. Just update whenever you get there, and merge as you go and you don't have to depend on anyone to get your job done.
Michael Langford on November 2, 2007 7:32 AM@Steve:
We have been using VSS at my previous job and now we use SVN + Ankh + Tortoise SVN and I must say that it is much better.
The thing I hate about VSS is locking. The most frustrating is locking of project files when someone adds a file to project - noone else can add to project until checkin. I've also seen issues when developer went on holiday without checking in so noone could work.
With SVN you can work freely and occasionaly when there is a conflict, you resolve it with your teammate. Just remember to UPDATE and resolve conflicts(if any) before every COMMIT.
I've read some other critique on VSS, try google:
- it is very slow over VPN (I can confirm this)
- it can corrupt files (did't happen to me)
@Jon,
If you run into this issue a lot. I would look at version control systems that let you commit to a local (on machine) repository and push up to the main one. This doesn't necessarily have to be a distributed version control system, but that's what they were designed for.
Cezar on November 2, 2007 7:58 AMI read the "If They Come, How Will They Build It" at the very beginning of the post.
So you suggest a build server with everything necessary installed.
I still don't understand how a developer can debug the program on his local machine without having all the necessary libraries / databases installed himself.
I'm sorry, but I'm working on a small project and we build from Visual Studio (yes, everyone who wants to build needs to have Firebird database and libraries installed, but that's it).
Inv on November 2, 2007 8:04 AMI've done a bunch of build systems for clients. Often it was like, "Well, we *do* have a build system, but it doesn't work all that well, so Sean over there does the compiles, and copies the folders to make the setup programs...". All by hand.
At one company where I had just installed a daily build, test and release system, "Sean" raised a hand in a meeting and asked, "Now, this is great, but where do I find the files that I need to copy to the releases folder?"
[buried head in hands on that one]
Builds are all about repeatability. Years later, if necessary. I've done whole-machine snapshots (tools, databases, all) and had to go back to them long after re-creating the build environment would have been impossible (e.g., the Really Smart install for Frobozz Compression Tools wouldn't install, because Frobozz had gone out of business and its license server was dead). If you are dependent on software packages with wacko licensing, with expiration dates, etc., just go buy something else; if your product lasts more than a few years, you'll be thanking yourself.
Being able to cut a release from a specific point in time (or branch) is critical. You should be able to take your build system and easily clone it on another machine. This is good for disaster recovery, and helps to ensure that you're building what you think you are (it sucks to ship a patch to a customer, only to find that it contains "extra" stuff from Joe Developer's random post-release checkins, or that critical files aren't getting properly rebuilt. Happens).
Any shop of more than a handful of developers needs someone working at least half time on build systems and similar infrastructure. If you aren't doing this, you'll regret it when your org gets larger, and you *are* planning to succeed, yes?
Not a comment on the build process etc, but on the pain of getting a new developer up to speed.
1) If the Ed is a contractor, then he's been paid for two weeks work for nothing. The company should be wondering why.
2) If the Ed is an employed person, then why hasn't he documented what he has had to do? Straight away Mike is on his side - he just saved him a shed load of work by Mikes own admission he doesn't have time to do?
But I suppose, "It's not my job to do that" comes into play.
Tubs on November 2, 2007 8:43 AMDid I really just read the words, "the luxury of working with IBM Rational ClearCase" further up on the page? Astounding. I wonder if that's anything like the luxury of drinking month-old milk.
Anyway, it astounds me how many comments above say that a build server is too expensive or requires IT support. Our build server started its life as a stolen-ish spare machine jammed under someone's desk with a KVM switch. IT doesn't even know it exists -- well, except when we're slow to patch it.
As far as maintenance effort goes, you can start simply and just use batch files. We happen to use a scripting language, but it doesn't really matter that much. Add a simple Web server to make the entire history of the build server available over HTTP, and you've got yourself a decent little utility.
We use it for a couple of VS.NET and eVC projects in the 1-to-10-man range. It takes maybe a few hours a month to maintain, usually something like, "Can you make it do _this_, too?"
undees on November 2, 2007 9:51 AMThose who don't understand unix are condemned to reinvent it -- badly.
On another note, source control really needs to expand it's horizons beyond the mainline "working code" branch. I think it's getting
there, but the more I look at the problem the more I'm convinced
source control needs to be more than a repository alone. It needs
to be the communication tool. It needs capabilities to allow you to
shuffle non-working code around between developers without resorting
to zip files and email. Of course, it has to do this without breaking
the initial mission too.
Something like git?
We take this even one step further and generate all vcproj/sln files. They're as temporary as .obj files for us :). I made a quick blog post about this on my site.
bullocks on November 2, 2007 12:34 PMHmm, first you check out your code using a private branch. Public daily build uses submitted code or, if no branch, you tag submitted code and build the tagged code. And what happened to make files. We build every day on Sun using the Sparc compilers and while we use various IDE's, we only use them as editors. Of course for source control, Clearcase gives you a lot of options.
dweeks on November 2, 2007 1:36 PMBeing very new to the industry (graduate in a few weeks, actually) everything I read here is so valuable! I've experienced a lot of this in my limited professional experience and have seen the bad habits showcased here destroy many a project.
I'm glad to know that it dosn't have to work the way I've seen it play out so many times before and that there really are "better ways" of building products where the "better way" is more than a religeous argument.
lord_humungus on November 3, 2007 2:31 AMyeah - but who cares? F5 saves time until a 'build script' becomes necessary.
jim on November 3, 2007 3:06 AM'Did I really just read the words, "the luxury of working with IBM Rational ClearCase" further up on the page? Astounding. I wonder if that's anything like the luxury of drinking month-old milk.'
I'm still laughing over that one....nice work undees
JosephCooney on November 3, 2007 4:50 AMF5 was run long before it was refresh, long before there *was* an IE that made F5 popular for refresh. I wonder if in fact F5 was chosen by the developers as "rerun this HTML."
Joshua on November 5, 2007 1:04 AMHey, don't laugh to long it might hurt. I did not mean it in the way you read it, as in luxory after all it is just work.
I just meant that ClearCase gives you so many more options that VSS, CVS and other toold don't, like building different models for the different projects, and developer views customized for each, and different streams for instance for developers committing all the time to, and other streams like integration streams, besides ofcource branching, which I mentioned, I think is for project branchings off of product lines to customize different clients' needs.
If you are going to laugh then at least explain why you are laughing.
Moe Badawi on November 5, 2007 1:49 AMThe comments to this entry are closed.
Subscribe in a reader
Subscribe via email
|
|
Traffic Stats |