Investing in Quality: When is "Enough" Enough?

There is no doubt that we need to invest in producing a quality product. But it is not clear how much we should invest! Too much and we are wasting precious resources. But if we invest too little, our product will be judged as poor-quality, and our project may be labeled a "failure". (Yikes!)

So, how do we find that sweet spot? How do we determine the "right" amount of quality investment?

Surprise! You already have the information you need to answer that question. It is in the records from your prior projects. Both the projects that produced "good" quality, and those that did not set the stage for you to make the right quality investment decisions this time around.

Harvesting Your Quality Data

We start by going back through our recent projects. Sort them into three piles: Sufficient Quality, Insufficient Quality, and Exceptional Quality. If you are lucky, you will have at least one project in each category. In this case, finding the sweet spot is a matter of comparing what we did on the projects that fell into the different categories, and using the differences as our guide.

But most of us are not lucky. We will find that the majority of our projects fall into only one of these three piles. And, unfortunately, it may be the "Insufficient" category that is over-populated. Although this situation gives us much less information to go on, it still provides us with a good starting-point. After all, if we know that the quality investments we made on those projects were insufficient, we know that we must invest more (or at least smarter)!

Look back at the projects that you put into each category. What kinds of quality investments did you make on each of them?

Counting the Cost of Quality

As we look back at each project, we must be careful to fully count the costs of quality on those projects. Some of those costs come to mind immediately. But other quality costs are less obvious. So it is important to take a few minutes to look at each.

Testing -- How much testing was done? And more importantly, why types of testing did we do? On software projects, several types of testing are likely. "Unit" testing of the individual components that the programmers write establishes a good basis for building a high-quality product out of those components. "Software Integration" testing verifies that the components work well with each other and that interfaces work as designed. "System" testing verifies that the whole software product actually satisfies its requirements. "System Integration" testing checks to be sure that the software product can coexist with the other products, devices and databases in its environment. And "Acceptance" testing ensures that the end-users can actually use the product to do real work.

If your project is building a service, a physical thing, or anything else other than software, you can probably identify a similar set of tests that may be called for. Regardless of the nature of the product of your project, investing in the right kinds of testing is important.

Preparation for testing -- What kinds of investment was made in testing before testing started? For each of the types of testing that you did, how much time and effort went into planning it, defining a testing strategy, and mapping out the specific tests that would be run?

Random, flow-of-consciousness testing can rarely be expected to produce consistently good results. The testers may stumble across serious defects, but they could just as easily miss them. Well-planned testing that follows a defined strategy will have a much greater likelihood of finding the important problems sooner, rather than later. (This goes for Unit testing as well as for any of the others!)

Review and Analysis -- Testing (using the product to observe its behavior) is not the only way to find defects. In fact, for many types of defects, visual or automated analysis of the product is a much more effective way to identify problems for removal.

It is obvious that using a tool to analyze work products will be very efficient. But many organizations had reported that having another person or a few of them (usually peers of the person who created the product) perform a technical review of the work is a very good use of the engineers' time, because it is so efficient at finding many kinds of problems.

Are you making use of these types of evaluations on your projects? And if so, what percentage of the product is being analyzed?

Defect Detection -- For the kinds of testing and reviews that you have employed, how effective have they been at finding defects? How many defects are being detected for each person-hour spent doing it? We want to ensure that our people's time is being well spent. So we may want to invest in the more efficient defect-detection activities. Of course, we may also find that we can improve the efficiency of those activities (for example, by planning testing instead of doing it randomly).

Defect Correction -- For each kind of testing and review that we employ, how much effort does it take to correct the defects that are being found? Organizations that measure the engineering time that goes into defect correction have found that the defects that are found earlier in the project tend to be less expensive to fix than those found later in the project.

If you find that this is true on your projects, then you may want to shift more of your quality investment to activities that find defects when they are cheaper to fix.

Where Is This Data?

Many of us will find that we don't have good data about the hours that were invested in specific activities on our prior projects. We know that our testers reported a certain amount of time against the project, but we don't know the specifics of how each of those hours was spent. By the same token, when know how many engineer hours were billed, but we have no idea how many of those hours were spent fixing defects.

This is not an uncommon problem. If this information had not previously been identified as being important, then it is unlikely that if would have been measured and documented. But that does not mean that your hands are tied. There are two things that you can do about it.

First, you can estimate the missing information based on the data that you do have. You may have to ask people to try to remember how much of their time went into which activities, and then use their best recollection to estimate the numbers. But that is a good starting point. It may not be terribly accurate, but it is better than no information at all.

The second thing you will want to do is to start collecting the pertinent information today. Even if you are in the middle of a project and have no detailed data about what has gone on to date, there is no reason not to start collecting useful data right now. That data will very quickly become a goldmine for you. You will discover that any data (even only a little data) is valuable. And, of course, the sooner you start collecting data, the sooner you will have a trust-worthy collection of data you can use for making important project decisions!

Using This Information

As you look back over past projects, you can learn from your experiences. Projects that produced acceptable quality can be your template for appropriate quality investments. Projects that produced unacceptable quality can serve as a baseline that you must exceed on future projects in order to have a chance of producing better quality.

And your data can help you to pick out the quality investments that produce the best returns for the lowest investment. Imagine! Instead of investing more, we may be able to simply invest smarter! That that is a recipe for wildly successful projects!