Agile Metrics Design:

Workshop Presentation

This workshop has been run, with the invaluable input of Duncan Pierce from agile consultancy Amarinda, at the XPDay05 conference in London, and SPA2006 at St Neots in Bedfordshire. It has also been run successfully for clients, involving people at all levels from CTO to junior programmers.

The format of the workshop is simple, and we strongly encourage you to try it for yourselves.

The underlying premise couldn't be more straightforward:

When you measure what people do, people do what the measure tells them to do

Or, in even simpler terms:

Be careful what you wish for!

If you're thinking of using metrics to help manage your projects and programs, or to guide you in software process improvement - and we strongly recommend you use metrics in any SPI initiative - then you need to take care over the design and application of those metrics.

How Does Agile Metrics Design Work?

We are great believers in Test-driven Development, and this workshop applies TDD principles to metrics design.

In TDD, we have a functional goal for our code. In other words, we kind of, sort of know what we want the code to do. We write a test that we think will force us to write that code. Then we write the code specifically to pass the test. We don't write the code we think we ought to write. We write the code that the test tells us to write. We do the simplest thing possible to pass the test. If the test tells us to write the wrong code, then we wrote the wrong test.

In Agile Metrics Design, we apply exactly the same technique to metrics. We kind of, sort of know what we want - or what we think we want. We have a high-level goal (or, more often, a set of goals) that we want our teams to achieve.

So we try to express our goal(s) in a testable way - using metrics.

Let's take a simple example:

The Goal: Reduce bugs in each release

The Metric: Bugs captured in Bugzilla / release

So far, so good. Except, what is that metric telling us to do? What's the simplest way of achieving this goal, as described by the metric? What would take the least effort?

We want the number to go down, and we can most easily do that by releasing software more often. If we introduce 20 bugs in 10 weeks, we might only introduce 2 bugs in one week. Releasing software every week would give the appearance of an improvement in quality. It could fool us into believing that we've achieved our goal.

This is what performance measurement experts call a Gaming Scenario. A gaming scenario is a strategy for cheating on a metric, giving the impression of improved performance when no actual improvement has occurred.

By exploring gaming scenarios, we can test and evolve the design of our metrics in much the same that, by writing tests, we can evolve the design of our software.

The Agile Metrics Design Process

Agile Metrics Design is an evolutionary approach to metrics design and metrics management. The trick is to iterate. Design a metric. Test it by gaming. Then redesign the metric and test it again. Use this iterative metrics design approach to converge on the optimal design, just as you use TDD to converge on the optimal software design.

The gaming scenario warns us that releasing more frequently could fool us into believing we've achieved our goal. We can make gaming much harder by modifying the design of the metric:

New Metric: Bugs captured in Bugzilla/ Week

I'm sure you can already think of other gaming scenarios. The Agile Metrics Design process iterates through each one until the metric is so difficult to game that there would be little value gained by even trying.

Striking A Balance

We are great believers in balance. All work an no play makes Jack a dull lad, the saying goes. And so too with software projects. All deadline and no quality makes Jack 2.0 a bad release!

The most effective way to avoid falling into the "Be Careful What You Wish For" trap is to create a balanced suite of metrics that will hopefully prevent your project tipping too far in one direction.

Experience has informed us that there are 4 Pillars of Software Team Performance:

• Time

• Scope

• Cost

• Quality

Time and again we've seen teams focus too much on one aspect of performance at the expense of the others. Most usually, time is the "big thing" teams are being measured on. It's not unusual for managers to say "cut corners if you have to, spare no expense - just hit that deadline!"

As we all know, they will live to regret it...

(Can you think of a 5th pillar? We can...)

Establishing the relationship between these elements of performance is the key to effective process improvement. Contrary to our intuition, buggy and unmaintainable software costs more and takes longer to deliver.

The Balanced Scorecard is a widely-used performance measurement framework that allows you to balance your goals over different performance perspectives. We highly recommend you use a scorecard to help articulate and manage your goals.

Just Do It

Of course, your boss probably won't believe you when you tell him that. Which is where metrics can be of enormous benefit. Like TDD, teams can implement a suite of metrics without having to buy expensive "lifecycle" tools or hiring in teams of high-priced consultants. You can just do it.

A team doing eXtreme Programming, for example, probably already has considerable data from which useful metrics can be calculated . Those story cards are a wealth of data. And how about those acceptance test scripts? Free defect tracking tools can be downloaded for pretty much every programming platform we can think of. And what about the code? Isn't code just like a big semi-structured database? Couldn't we query the code and found out all sorts of useful stuff? Open source code quality metrics tools for languages like Java and C# are also out there for the taking. The commercial tools aren't that expensive, either.

To run the workshop, you will need:

• The right people - key stakeholders whose interests need to be represented

• Plenty of paper and pens - ideally flip chart paper and marker pens, so people can read your metrics from a distance

• Several tables - you will be split into groups

• At least 2 hours  - ideally a whole day

The process is simple:

• Identify your scorecard perspectives - whose points of view do you need to take into account?

• Identify your goals, and place them in the most appropriate perspectives

• Sanity check - have you covered all 4 pillars (time, scope, cost, quality)? Have you figured out what the 5th pillar is yet?

• Split into groups - you will need at least 2 groups for the exercise to work

• Each group takes one goal and designs a metric that they believe will be a good indicator of progress towards it. NOTE: Do this quickly - take no more than 20 minutes to come up with your first metric design. It's okay if the design isn't perfect - after all, that's the whole point of iterating!

• One person from each group takes their metric design to another group (e.g., moving to the next group in a clockwise direction - it helps enormously if you number each group and if the facilitator writes those numbers on cards placed prominently on each table, so we can all see who we're referring to).

• The other group attempts to come up with at least one gaming scenario for the metric. The most obvious way to game a metric is to do the most obvious thing. Do what the metric tells you to do. Is that the same thing as the goal? Is in the spirit of the goal? Is there something wrong with the goal itself? (Yes, Agile Metrics Design is also a really effective way of pinning down exactly what you it is you really want...)

• The person who jumped teams then goes back to their original team, who redesign the metric taking the gaming scenarios into account.

• Rinse and repeat...

There is some supporting presentational material here, and we thoroughly recommend you look into Agile Software Process Improvement, which is an evolutionary and metrics-driven approach to SPI that provides a healthy, low-fat alternative to rich, stodgy SPI methods like CMMi.

Who's Doing It?

If you've tried Agile Metrics Design, and want to share your experiences, please let us know so we can list you here:

• XP Belgium - run by Pascal Van Cauwenberghe

If you are thinking of running a workshop at a conference, or as part of a non-commercial educational course, please contact us to obtain permission first, as both the material and the format are copyrighted.