September 13, 2006
For the last few days, I've been surveying a software project. Landing on a planet populated entirely by an alien ecosystem of source code can be overwhelming. That's why the first first thing I do is bust out my software tricorder -- static code analysis tools.
The two most essential static code analysis tools, for .NET projects, are nDepend and FxCop. Like real software tricorders, they produce reams and reams of data -- lots of raw metrics on the source code.
Even basic metrics can identify potential trouble spots and/or areas of interest in the code, such as..
- Methods that are too large or too small.
- Classes that are too large or too small.
- Methods that are too complex (as measured by cyclomatic complexity).
- Methods with too many parameters (more than 7 plus or minus 2).
- Methods with too many local variables.
- Classes with an excessively deep inheritance structure.
- Types that are excessively large.
These simple metrics are already quite valuable. You can imagine how valuable more advanced software metrics could be, such as code coverage. Or how quickly you're finding and fixing bugs. And more advanced static analysis tools can offer literally hundreds of recommendations, ranging from mundane to mission-critical.
Having more data about your software development project can never be bad. The real trick, of course, lies in interpreting all that data, and deciding how to act on it. There's a huge temptation to become a metermaid-- to use the metrics as a reward or punishment system.
If Joe wrote a method with a cyclomatic complexity of 52, then he better get slapped with a complexity ticket, right? No excess complexity in the simplicity zone, you idiot!
Not necessarily. Responsible use of the metrics is just as important as collecting them in the first place. Gregor Hohpe elaborates:
Some of the most hated people in San Francisco must be the meter maids, the DPT people who drive around in golf carts and hand out tickets to anyone who overslept street cleaning or did not have enough quarters for the meter. On some projects, the most hated people are the metric maids, the people who go around and try to sum up a developer's hard work and intellectual genius in a number between 1 and 10.
Many managers love metrics: "You can't manage it if you can't measure it". I am actually a big proponent of extracting and visualizing information from large code bases or running systems (see Visualizing Dependencies). But when one tries to boil the spectrum between good and evil down to a single number we have to be careful as to what this number actually expresses.
Martin Woodward calls this the measurement dilemma.
The reporting aspects of Team Foundation Server are a new, more accurate instrument to take measurements inside your software development process. But you need to be wary about the things you measure. The metrics need to mean something useful rather than just be interesting. The effect of taking the metric should be carefully considered before taking it. This is not a new problem. But because Team Foundation Server makes it so easy to get data out of the system, the temptations are greater.
Martin also references the Heisenberg Uncertainty Principle, which states that you can't measure something without changing it. I believe this is true for software development metrics only if you are using that metric to reward or punish.
Recording metrics on your project can be beneficial even if you don't explicitly act on them. Having a public "wall of metrics" might be a better idea. It can be a focal point for discussion about what the metrics mean to the team. This gives everyone on the project an opportunity to discuss and reflect, and act on the metrics as they deem appropriate. Maybe the team will even remove a few metrics that are of no value.
What metrics do you find helpful on your software projects? What metrics do you find not so helpful? And if you have no project metrics to talk about, well, what are you waiting for?
Posted by Jeff Atwood
One fine day the upper management at my company decided they wanted ISO certification. Immediate effect - we are supposed to ensure we follow coding guidelines.
While coding guidelines are good to ensure consistency across the teams, often the even more important issues like code correctness, not breaking abstractions or encapsulations are most important.
I personally think the single most important metric for any commercial software developer is how well he meets his specs.
ASML is a way to programmatically write specs so there will be a way to check if code meets specs or not. TDD is another way.
This is an excellent post. I need to show it to everybody at work. I'm trying to improve our software production process. I'm adding source control (and trying to get comments on the commits), creating an automated build, and trying to get some standards implemented. I'm getting resistance because the head developer thinks it's impeding his ability to code, but I'm really just trying to make everything easier for the whole team. Perhaps this type of thinking will help him move towards a more team approach without feeling his personal space is being violated.
We've recently started doing code coverage of a very large code base that has never used those metrics before. We have a series of automated tests (mostly UI and output tests - not so much unit tests), and it's been interesting to see how much of the code is covered by the tests.
Unfortunately, the instrumented code writes its results out at application exit. Most of our tests involve starting the application, performing one small operation then quitting. So a set of tests which normally takes about 2 hours to run now takes over 24 hours! So, yeah, measuring it does change things. If we make it perform more tests at a time, then we run the risk of running into memory patterns that might cause a code path to not be tested. (Not that we're worried about that, but it just occurred to me it could happen.)
But overall, it's been a pretty cool experience. Now it's a question of sitting down with QA and automation and telling them, we need to add a test to do A, B, and C to cover this code, etc.
from the wikipedia article you linked to http://en.wikipedia.org/wiki/Uncertainty_principle :
Common incorrect explanation of the uncertainty principle
The uncertainty principle in quantum mechanics is sometimes erroneously explained by claiming that the measurement of position necessarily disturbs a particle's momentum.
In fact Heisenberg's Uncertainty Principle means (in supersimple terms) you can know either the exact velocity of a particle or its exact location -- but not both!
Interesting (p)article nontheless.
You're certainly right that it's very easy to abuse metrics to micromanage. One reason which I think your alluding to is that by applying metric maids, you end up with bad data.
Truth is, you have to be careful not to follow the adage of "You can’t manage it if you can’t measure it”, because it's very misleading as well.
I wrote a post about this second thought last month, which I think relates.
Find it at: http://ryan-technorabble.blogspot.com/2006/08/rule.html
It seems to me that the real value of metrics is predictive -- if you can find data to measure whose quantity you can predict for future development, you can use actual historical data to get a handle on how long something takes to create. Alas, there are many if's in there -- IF you can find data to measure, and (implicitly) IF you can predict this quantity for future work.
I don't know about coding, but for writing and editing you can take a gross measure of how much documentation was created between time A and time B, from draft to published doc; if you do this over the course of several projects, you can get a somewhat accurate estimate about how much documentation a team can create in a given time period. If you can then (somehow) estimate the size of a new project, you've got a handle on how long it might take.
An important point, it seems to me, is to look at the metrics in a kind of gross way and to NOT qualify them with "well, this was a different kind of project" or "well, three of our folks went on vacation on that project" or the like -- because every project introduces some sub-optimal elements like learning curves, unexpected difficulties ("our build server crashed four times"), personel turnover, etc.. By essentially ignoring all these, you smooth them out over the course of multiple projects.
Note that I am not a statistician.
I very much agree that metrics can be useful and in the ideal case (as described in Jeff's post) can help you find places to improve your process, and that using metrics to measure individual productivity is a much more difficult task. Getting overall team productivity numbers is good; beating an individual on the head about stats, while perhaps ok in some schools of management, seems fraught with para-productivity issues, like whether your people quit in disgust.
Why does a 'too small' method is considered 'bad'?
Is having a method that contains only one or two lines considered as a violation of some FxCop rule? If so, why?
i think the value in knowing the rules given to you by fxcop etc is the ability to know when to break them.
tho why it's a bad thing having a method with 7+-2 parameters is anybodies guess.
7+-2 is the rough number of distinct attribute you can remember about an object, tasks you can keep track of etc. Long since forgot where I read this stat. But I think it is referenced in Steven Covey's books on time management.
Metrics are useful for showing areas that can be improved in code. Any large scale project should use metrics to see if the code base is becoming uncontrollable. A piece of code with a complexity rating of 52 will still work, but it is tougher to maintain, fix bugs, etc. than a piece of code with a complexity of 8. Also, highly complex systems tend to have more bugs in them and are more difficult to test than simpler ones because of the higher number of logic paths.
Anyone can write code, but can you write code that is easier to maintain and update. The metrics should show how you are doing.
Metrics should be a part of a code review process. But most times its not like that because of time crunches etc.
I ran some complexity tools on a project I am working on and many classes and pages over 100 for cyclomatic complexity, grand prize being 382!
Nobody likes a meter maid, but nobody likes people parking for free either.