programming and human factors
by Jeff Atwood
Hacknot's If They Come, How Will They Build It? is a harrowing series of 29 emails sent over a two week period.
To: Mike Cooper
From: Ed Johnson
Mike,
I finally got CVS access today from Arnold. So I've checked out the AccountView module OK, but it won't compile. The Eclipse project has dependencies on about five other projects. I tried checking those dependent projects out as well, but a few of them won't build at all? How are you managing to develop this thing when the dependent projects don't build?
Ed
From: Mike Cooper To: Ed Johnson
Oh yeah - I forgot to tell you about the dependent projects. I always forget about them. I'm not so surprised some of them don't build for you. I've got versions on my machine that build OK but I haven't checked them in for a while. Gimme about 15 minutes and I'll check them in, then you should be right to go.
M.
It's a cautionary tale about a serious software project pathology: the pain of getting a new developer up and running on an existing software project. It's startlingly common.
This points us to one of the most important health metrics on a software development project. How long does it take for you to get a new team member working productively on your project? If the answer is more than one day, you have a problem. Specifically, you don't have a proper build process in place.
I've talked before about the importance of a build server as the heartbeat for your project. A sane software development project has automatic daily builds, performed on a neutral build server. If your team is in the habit of producing those kind of daily builds, it's difficult to accumulate the deep technical debt enumerated in all those emails. If the build server can do it, so can your newly hired coworkers.
But based on the development practices I've often seen on site with customers, I think setting up a build server might be an unrealistic goal, at least initially. It might not get done. We should shoot for a more modest goal to start with.
Here's how most clients I work with build a project:
If your "build process" is the F5 key, you have a problem. If you think this sounds ridiculous-- who would possibly use their IDE as a substitute for a proper build process? -- then I humbly suggest that you haven't worked much in the mainstream corporate development world. The very idea of a build script outside the IDE is alien to most of these teams.
Get your build process out of the IDE and into a build script. That's the first step on the road to build enlightenment.
The value of a build script is manifold. Once you have a build script together, you've created a form of living documentation: here's how you build this crazy thing. And naturally this artifact is checked into source control, right alongside the files necessary to build it (and even the database necessary to run it, too). From there, you can begin to think about having that script run on a neutral build server to avoid the "Works On My Machine" syndrome. You can also consider all the nifty ways you could enhance the script with stuff like BATs, BVTs, and Functional Tests. Your build server can become the heartbeat of your project. There's no upper limit on how clever you can be, and how many different build scripts you can come up with. Build scripts can be incredibly powerful-- but you'll never know until you start using them.
The F5 key is not a build process. It's a quick and dirty substitute. If that's how you build your software, I regret that I have to be the one to tell you this, but your project is not based on solid software engineering practices.
So, if you don't have a build script on your project, what are you waiting for?
In my work as a consultant specialized in CM and build systems Iv'e encountered alot of different, both good and bad solutions. One of my favourites, which we designed for a customer was based around CM Synergy. You see, Synergy projects have rules for which object versions they do and do not include during updates. All checkouts are connected to a task (which in turn are connected to a Change request), and once you 'complete' the task, all checkouts associated to it are checked in.
This means that you can have an integration project with rules specifying that it should only consider versions which are in the 'completed' state for the integration builds, which allows developers to work on tasks for extended time periods without interrupting nightly (or hourly etc) builds. You can also have a project for system testing, which only include tasks associated to a CR which are set to testable etc. On the bonus side it's almost completely invisible for developers for day to day work: They get (or create, depending on process) their tasks and work until happy. They complete the tasks when done, and the next morning they get a report from the nightly build and test detailing the results. Delivering into system test is as easy as pressing a button.
My opinion is that most companies spend far to little time on theese issues. There needs to be specified build processes and tools, and preferably people with expertise in theese areas. The time and quality benefits can be quite huge if this is done well, while at the same time management get's better overview of projects _and_ developers experience less frustration. I can't see why anyone but an imbecil would not want to do it.
Jens J on November 6, 2007 5:21 AMIn response to the comments on VSS
----------
" 1. If your network/the internet is not swift, it's near impossible to use. Errors abound in this situation, and you can't always check out/in the code you need all the way. To put it on the internet at all, you have to set up tunneling, which frankly is not always a skill that goes with the target audience (it's more of a hardcore unix user thing than a pure VS shop type of thing). Visual studio can die if you use the integration while checking in code. This means you then lose the code that's not checked in (sometimes). That's exactly what I look for in a source control system. "
This is simply not true; if your network is so slow it cannot handle a few k being passed around then you should not be developing on it. If you do decide to VPN in to use SourceSafe then you can (but i grant you it performs slower), however it does indeed support web based usage (Which is the prefered method of external access as this performs very well).
As an aside there are also countless third party tools that handle external VSS access very well.
" 2. Its damn useless at tracking externals. By externals I mean you can't reasonably check in multiple versions of an external library or a changing art pack developed elsewhere. It doesn't handle files that are removed between versions, and it doesn't make it easy to add them. "
Utter rubbish, souresafe supports multiple chekouts, code merging and bracnhing and it works a treat.
" 3. You can't check out build test editions. One good practice to make sure a dangerous change or large change can build for everyone is to check out the thing you just checked in to a /tmp folder then compile it there. If you can compile there in the tmp folder, there is a much higher chance you didn't break the build. VSS can't do this, because its one checkout per user. You actually need multiple users per people to do this with VSS, and you need to have your devevn setup on multiple machine users. "
Yes you can - why cant you? set a new working folder check it out and away you go. Wheres the issue?
" 4. At least when I used it, history sucked. It took for-ever. Therefore, no one used it unless they catastrophically screwed something up. It should be used for all sorts of debugging. "
History is fine, I use it very often. See point one about your own slow network or PC.
"5. Database corruption happens all the time. Oh god that's scary."
Not had a single issue with corruption in two years, the server has tools for tuning up the database if you do have an issue (not that I ever have).
"6. Oh yeah, renaming and deleting totally screws with your history. WTF mate. I want to be able to build version X easily, and version A too, even if I've renamed everything."
No it does not, renaming and deleting works perfectly, VS prompts you and warns you if you are about to do something that is not supported and prompts you to check in first, it works fine.
"7. LOCKING ONLY DEVELOPMENT"
You do not understand VSS at all. It is a prefernece, turn on single file chekc outs or enable multiple checkouts. Your comments seem to come from a lack of knolwedge more than valid arguments.
Anyone know of any real technical reasons why not to use VSS?
Dave on November 6, 2007 7:24 AMIn response to the comments on VSS (please delete previous post it ate my quotes)
----------
" 1. If your network/the internet is not swift, it's near impossible to use. Errors abound in this situation, and you can't always check out/in the code you need all the way. To put it on the internet at all, you have to set up tunneling, which frankly is not always a skill that goes with the target audience (it's more of a hardcore unix user thing than a pure VS shop type of thing). Visual studio can die if you use the integration while checking in code. This means you then lose the code that's not checked in (sometimes). That's exactly what I look for in a source control system. "
This is simply not true; if your network is so slow it cannot handle a few k being passed around then you should not be developing on it. If you do decide to VPN in to use SourceSafe then you can (but i grant you it performs slower), however it does indeed support web based usage (Which is the prefered method of external access as this performs very well).
As an aside there are also countless third party tools that handle external VSS access very well.
" 2. Its damn useless at tracking externals. By externals I mean you can't reasonably check in multiple versions of an external library or a changing art pack developed elsewhere. It doesn't handle files that are removed between versions, and it doesn't make it easy to add them. "
Utter rubbish, souresafe supports multiple chekouts, code merging and bracnhing and it works a treat.
" 3. You can't check out build test editions. One good practice to make sure a dangerous change or large change can build for everyone is to check out the thing you just checked in to a /tmp folder then compile it there. If you can compile there in the tmp folder, there is a much higher chance you didn't break the build. VSS can't do this, because its one checkout per user. You actually need multiple users per people to do this with VSS, and you need to have your devevn setup on multiple machine users. "
Yes you can - why cant you? set a new working folder check it out and away you go. Wheres the issue?
" 4. At least when I used it, history sucked. It took for-ever. Therefore, no one used it unless they catastrophically screwed something up. It should be used for all sorts of debugging. "
History is fine, I use it very often. See point one about your own slow network or PC.
"5. Database corruption happens all the time. Oh god that's scary."
Not had a single issue with corruption in two years, the server has tools for tuning up the database if you do have an issue (not that I ever have).
"6. Oh yeah, renaming and deleting totally screws with your history. WTF mate. I want to be able to build version X easily, and version A too, even if I've renamed everything."
No it does not, renaming and deleting works perfectly, VS prompts you and warns you if you are about to do something that is not supported and prompts you to check in first, it works fine.
"7. LOCKING ONLY DEVELOPMENT"
You do not understand VSS at all. It is a prefernece, turn on single file chekc outs or enable multiple checkouts. Your comments seem to come from a lack of knolwedge more than valid arguments.
Anyone know of any real technical reasons why not to use VSS?
Dave on November 6, 2007 7:25 AMThat technical enough for you?
VSSSucks on November 9, 2007 6:08 AMexcellent article ...i am convinced. :) Does anybody have any tips or recomendations for what software (and hardware) you need for this (with .Net)? Jeff ,maybe you can write up another article about your recommendations
Supriya on November 9, 2007 8:26 AMMaven2 has a nice dependency management feature that addresses the issue Jon has outlined. It allows you to "baseline" a module and publish it to your remote repository that typically hosts all of the project dependencies. You can then force all of the dependent modules to reference the baselined version of this module rather than the snapshot. Therefore, you can continue checking in your updates to this module and it won't break the build. Finally, once you've refactored all of the dependent modules locally, you re-associate them with the snapshot and check them in.
The NMaven project supports building .Net assemblies as Maven projects, although I've never used it ;)
http://incubator.apache.org/nmaven/
Pete on November 12, 2007 12:25 PMThat's a reason why VMWare sometimes useful for a new developer :)
Mike on December 29, 2007 11:32 AMAll well and good unless you have the kind of environment where...
a) Multiple seperate apps talk to the same database, build process or no build process, if you break the database, some other developer or production user is going to break you.
b) development timelines ranging from 2 hours to 3 weeks, if you've got less time (including working late) than you actually need to just write basic procedural code, how do you fit in build scripts?
c) new projects appearing on the radar on a weekly basis rather than working on large systems.
d) production fixes needed "now", not in a day, not even in an hour as without them the whole shop does nothing.
In my experience in organisations running daily production on inhouse developed software, the F5 approach has allowed us to run successfully with a handfull of developers. The 'do it properly' approach has, at times, taken 8-10 developers more than a year and more than a million dollars to produce what the F5 developers could replicate in a month. "Proper" procedures have their place, but the difference between 1 month for 2-3 developers and a year for 8-10 can be the difference between profit or failure.
I've never worked on a project with daily builds but I've always worried about the idea.
I'm making some fairly serious changes to an assembly, which will involve changing several files. It may take me several days to make the changes and get them all working. With daily builds I have to keep these changes checked out so they don't break the build, but this means that I have several days' work that hasn't been source controlled. I would much prefer to check in my non-working code as regularly as I can (hopefully several times a day) so that I can see at a lower level how my changes happened and where a problem might have occurred.
It would be possible to branch for the development (or use a local source control database) but I'd prefer to be able to see the version history in one place.
Is there a way round this problem when running a daily build?
Jon on February 6, 2010 10:13 PMI have loads of code in my source repository that doesn't compile- comes with the territory if you're using scripting languages most of the time.
Ben Moxon on February 6, 2010 10:13 PMMeh.
In what you call "mainstream corporate development", there is usually just a handful of developers (less than 5), and very few resources allocated to the software team.
Your suggestion - nay, insistence - has the following prerequisites:
- A modern build server;
- Someone to administer/maintain the build server. This person must work closely with IT, but cannot officially be IT because they'll be swamped with helpdesk issues every day;
- Someone to maintain the build scripts. This is a full time job. This person must work very closely with both the build server admin and the developers;
- A business process around the build. What happens when it breaks?
- Someone to oversee this business process. Do you really want your lead developer wasting his time trying to figure out why the build broke? Or are you naive enough to believe that individual developers will all stick around long enough at the end of the day to make sure that all the tests succeed?
- A method of mirroring this build process on individual developer machines. How else are people supposed to efficiently go through the debug-edit-test cycle?
And so on, and so forth...
In short, this is an utterly ridiculous amount of overhead for your typical 3-man corporate dev team. The build process you describe is well-suited for a software shop of at least 10 people, although IMO the number is probably closer to 25 before the F5 process really starts to break down - assuming that the team is competent.
So, while I understand that your post is being written from the perspective of someone who works in an average-sized dev shop, the fact that you specifically singled out an entirely different industry with an entirely different business model (internal corporate RD) as "violators" makes me wonder if you've given any thought to context, or are merely recommending this as a "best practice".
Source control is a piece of cake. It takes 5 minutes to set up, doesn't require an expensive server, and any idiot can do check-ins and check-outs without interrupting their more important tasks. This is not the case for a build server. There are legitimate trade-offs here that can't simply be dismissed with hand-waving.
I propose an alternative: a 1-page document (it's almost never longer than this) detailing the prerequisites for a particular build and deployment. Keep it in SCC. If your entire dev team quits, all necessary information is there for another one to pick it up. If the document becomes longer than one page, or if you find that it has to be updated more than once a week - then consider moving to an automated build process.
Aaron G on February 6, 2010 10:13 PMThe IDE/pressing F5 or F7 thing is a red herring. The real issue that you start off with is REPRODUCIBILITY. Can you reproduce the build environment without dependencies on another machine...or on your own machine but in a different directory. And if there are external dependencies, how well are they managed. Are they obvious?
Having a build file that parallels/is parallelled by an IDE solution is crazy - it means you're changing thiongs in two places. What are we always told, peeps? DON'T REPEAT YOURSELF - that applies to build/configuration as well.
Personally, I use Visual Studio 2003. External dependencies (such as include and library directories - and basically, that's just Boost, because it's big enough that I don't want to duplicate it) are defined using environment variables - if the environment variables aren't defined, it doesn't build, but it tells you what's missing. In addition, I use SOlution Build Environment (Google it), which allows the required environment variables to be gathered in a single, per-solution location, meaning that they aren't hidden away in your machine's registry or whatever.
I also try to pull as many things into the configured source tree. For example, ANTLR - I configure the ANTLR run-time and the particular version of ANTLR I'm using.
Does it work? For me, yes. I'm able to throw zipped configurations to other people and expect them to be working with the code in a minimal amount of time - an hour at most.
Stuart Dootson on February 6, 2010 10:13 PMUsing CruiseControl here, for build and test of our product - each distinct area tested separately. Can't imagine working without something like that anymore....
Simon on February 6, 2010 10:13 PMGreat post Jeff!
RE: New developers. We follow the single solution model here: http://msdn2.microsoft.com/en-us/library/ms998208.aspx
this makes things a lot easier for new developers.
A new team member simply
1) installs visual studio
2) syncs source code
3) opens and builds the solution
This works well with 6 small applications for us. The new team member can be up and running within half a day.
The other advantage of the single solution is that the build machine is simpler to maintain - we build a single solution using MSBUILD/TFS instead of several.
Stuart Eggerton on February 6, 2010 10:13 PMI think there are two parts to this:
1. Build scripts to make sure all the moving parts of creating a release are completed.
2. Work hard to simplify your architecture so build scripts are unnecessary. If you need a script that explains "here's how you build this crazy thing," maybe you need to take another look at why your application is hard to build.
If an application:
- is kept simple enough that you really can check it out and run it
- is tested via IDE integration (e.g. Visual Studio Code Analysis)
- takes care of build related process via the IDE (e.g. MSBuild Build Events)
- doesn't have a build server / continuous server
...then what purpose does a build process serve?
If you can't identify any practical advantages of a build process for a specific project, what purpose will it serve? At that point it's just another moving part that can break.
Jon Galloway on February 6, 2010 10:13 PMThe .sln and the .??proj files are MSBuild scripts. The IDE doesn't make it easy or obvious that you can extend the power of these but there is no reason why you can't add the database deployment and test run tasks in the right place in the project files so that F5 can do exactly the same build process that you also do on your build server.
I believe in having a complete build process on a build server but I also believe the IDE build experience should be identical and ideally they should be maintained in a single location.
Regards,
Jason Stangroome on February 6, 2010 10:13 PMno its not, the Ctrl-Shift-B is :)
Ion Todirel on February 6, 2010 10:13 PMThe comments to this entry are closed.
Subscribe in a reader
Subscribe via email
|
|
Traffic Stats |