Is Your Metrics Database Write-Only?

Goal-Question-Metric to the rescue!

A "Write-Only" database?

Yes, I have seen them, and many of you probably have seen them as well! Those are the cases where organizations are constantly collecting metrics and writing them to a database, but not using that data.

There is no question that a write-only metrics database is unhealthy for the organization. But collecting no metrics at all is also unhealthy. Of course, the ideal would be for us to collect metrics that we actually can use - and then to actually use them! And GQM (Goal-Question-Metric) can get us there!

Why "Write-Only"?

Building a Write-only database isn't easy. Lots and lots of data must be put into it on a regular basis. Megabytes, gigabytes, terabytes of data goes in ... and is never seen again! What makes the database "write-only" is the fact that the data in it is rarely used for anything. It is collected for the same reason that people will climb a mountain - "Because it is there."

Often, metrics are collected because at some point in the past someone found them to be useful. But that person may be long gone, and the reason has been forgotten. Unfortunately, tradition kicks in, and the metrics continue to be collected in spite of the fact that they aren't used by anyone.

Paradoxically, the other main cause of write-only metrics databases is metrics programs! "Metrics are good", it is declared, "and so we should collect them!" A project is initiated, and a database is established. Of course, a database presupposes that there will be data in it, so the search is on for things to measure. Usually, metrics are chosen because other companies are collecting them. If they find a particular metric to be important, then we should collect it as well! After all, it must be important!

Choosing the Right Metrics

Naturally, "because it is there" is a poor reason to measure something. But how can we decide what we should be measuring? Out of all of the things that we could measure, which metrics will actually be worth collecting?

There is no universal answer to that question. The metrics that will be useful to different groups, projects and organizations are as varied as the entities themselves. Indeed, it is by understanding your own organization and its needs that you can identify the metrics you should be collecting.

The fact that someone else needs to collect a specific metric does not mean that you should as well. It is not in other organizations' needs that you will find the guidance you seek. That guidance is within your immediate reach; it resides within your own organization!

GQM, the Goal-Question-Metric paradigm is designed to provide the guidance you need.

GQM Step 1: Goals

Your starting point in identifying the metrics you should collect is your goals. What is important? What are you trying to achieve? You have goals at many levels. Your company has goals, your division has objectives, your project has success criteria, your team has deliverables, and you have personal goals. All of these goals are the signposts that point you to the metrics that you should collect!

Goals must be your starting point because they embody what is important; to you, to your team, and to your organization. For metrics to be useful to you, they must tell you something that you care about. And your goals are precisely the things that you care about!

GQM Step 2: Questions

Once you know what your goals are, you need a tool to convert those statements into a list of metrics. There are times when the metrics that relate to a specific goal may be obvious, but not always. So, when the metrics you need are not immediately obvious, you need a tool to find them. That tool is questions.

Since your goals are important to you, you will want to know if you are progressing toward achieving them. That is the basis for framing your questions. What will I what to know on a regular basis? What questions should I be asking myself? What questions am I likely to be asked by other people? What will my boss want to know? Or my customer? Or my spouse?

The questions you need to ask and answer are the keys to your goals. They are your tool for understanding how you are doing, and if you are likely to reach success. That is why they are the stepping-stones to identifying useful metrics.

GQM Step 3: Metrics

Having defined your goals and listed the questions you need to answer, the last step is simply to identify what data you need to answer those questions. What facts and figures will give you insight into your status and progress against your goals? Those are the things that you should be measuring. Those are the metrics that will not be write-only, because you will find them to be useful; even important to your success.

Some goals are stated in terms of metrics. In many of those cases, you don't really need step 2, because your goal tells you what to measure. For example, if my goal is to "Expand code-review coverage to 75% of all code written", then clearly, I must be measuring the amount of code that is reviewed.

But there will be cases where a goal that is stated in terms of metrics doesn't point us in the right place. For example, if my goals is to "Reduce post-release defects by 25% compared to our last product", then it is obvious that I must be counting defects. But that won't help me to know if I am tracking toward my goal, because I can only measure post-release defects after the fact. These are cases where the questions become important. In this example, the kinds of questions I will be asking are "How many defects do I expect are in the product?" and "How many have we removed so far?" Now the questions tell me what I need to measure!

Finally, there will be cases where neither the goal nor the question is state in terms of metrics. In those cases, it requires a bit of creativity and insight to turn the goal and question statements into metrics. For example, if my goal is to "Define project scope in a way that prevents disagreements about scope late in the project.", then I may ask questions like "Have we defined scope well enough?" and "Are we having significant disagreements about project scope?"

In this case, it takes some thought to convert the questions into metrics. In this example, I may decide to count the number of In-Scope and Out-Of-Scope items in my scope description, the number of disputes about scope that come up, and maybe the number of person-hours that are spent on resolution of those disputes.

Getting Started With GQM

For many of us, GQM is easy to use because we already have organizational and project goal statements to work with, and often those statements are already in terms of metrics! But even if your goals are not so clearly defined, you still know what you are responsible for and how your work will be judged. So you can easily step through GQM to identify the metrics you care about.

The other thing that makes GQM easy to implement is that you don't need anyone else's cooperation to get started. You can use GQM for your own individual responsibilities without involving anyone else. Of course, as you see the benefits of using this simple paradigm, you may well spread its use to others, and may even influence your organization to use it.

Here's to converting our write-only metrics databases into storehouses of valuable information. Or, for those of us who have no metrics database, here's to getting started with metrics in a way that will provide real payback for the investment. Name it. Measure it. Achieve it!