Testing Is NOT Quality Assurance

When the Capability Maturity Model (CMM)(R) came out, it confused a lot of people. It had a Key Process Area (KPA) named "Software Quality Assurance", but that KPA didn't have the first thing to do with testing, or with peer reviews of technical artifacts. Did the CMM not address these topics? It most certainly did; and it did so in KPA's that don't mention Quality Assurance!

Wait a minute! What's going on here? Why is this getting so confusing?

It is confusing because we in the software industry have gotten into a bad habit of using the term "Quality Assurance" to refer to project activities like testing and technical reviews that are actually Quality Control activities. We've been do it this for so long, and so consistently that most of us don't even know what true Quality Assurance is!

Assuring Quality vs. Controlling It

What is the difference between Quality Control and Quality Assurance? Most of us have heard the truism that quality cannot be tested into a product; it must be built in, or it will not be there. This is the heart of the difference between these two concepts.

Quality Control is evaluative. It consists of activities that are designed to determine if quality was built into the product, and if not, where the deficiencies reside (so they can be corrected). These activities are done after the product has been built. Clearly, testing and reviews fall into this category. You can't test a program until after it has been built. And you can't review a document until after it has been written. And the purpose of both of these sets of activities is to evaluate the product to see how well it was built. In software, as in manufacturing, both testing and reviews are Quality Control activities.

Quality Assurance is proactive. It consists of activities that are designed to assure that quality will be built into the product. These activities either precede the building of the product, or happen concurrently with it. Which are the proactive activities that we engaging in to assure the quality of our software? Unfortunately, in too many projects, there are few (if any) Quality Assurance activities. Indeed, this presents a wonderful opportunity for us to improve the quality of the software we build, and make our projects run more smoothly at the same time!

QA: Process Assurance

In the CMMI (Capability Maturity Model Integration), the "Process and Product Quality Assurance" Process Area focuses on the processes and standards being used by the project. Its intent is to ensure that the processes and standards support the needs of the project, and that they are used properly and consistently. Appropriate processes and standards (if consistently used) will provide an environment in which our programmers can do their best work.

Is anyone doing process assurance on your projects? This may be an opportunity for you! The CMMI recommends that an independent oversight group do these things. The Agile methods and a few others like the Team Software Process (TSP)(SM) include the role of a process coach who operates as a member of the development team.

QA: Continuous Learning

Beyond the base of solid processes and standards, Quality Assurance also includes learning from past projects. Holding a project retrospective (or lessons learned or postmortem) session after each project (or phase) give the team an opportunity to step back from the busyness of the day-to-day work and look for ways to improve. The key questions in these sessions include:

What worked particularly well? How can we make those successes the norm on our projects?
What went wrong? How can we make those problems less likely, or eliminate them altogether?

Of course, these sessions are only worthwhile if we actually do something with the information. Based on what we learn, we should make changes to our processes, procedures, methods, tools, or standards to make the next project or phase better. Then at our next retrospective, we can ask if those changes had a positive impact, and if we should adopt them as the norm, or if we should abandon them.

QA: Avoiding Errors

Finally, QA includes learning from our defects. All of those defects that were identified by our Quality Control activities should be analyzed periodically to look for patterns. Are there certain types of defects that we inject regularly? Or that are inordinately expensive to diagnose and fix? If so, the key questions we want to ask are:

Is there a way to avoid these defects altogether? Would it help to adopt a different method or standard, or to change the way certain things are done? For example, are we investing sufficiently in software design? Would a different design method be more appropriate?
If we can't see how to avoid these defects, how can we detect and remove them more cheaply? Can we find them in earlier testing (e.g. Unit instead of System test)? Or could we tailor code or design reviews to catch them before testing even starts?

Affording QA

But how can we afford all of this extra QA work that we don't do today? This is the great part! Quality Assurance activities pay for themselves! Most QA activities are a modest investment (e.g. a 2-hour retrospective session), but the benefits they can yield in improved productivity and better quality are likely to far surpass those costs.

Integrating Quality Assurance activities into our project will assure that we build better products. Then our Quality Control activities will be less a matter of finding so many problems, and more a matter of confirming that we did, in fact, build quality in!

(R) "CMM", "Capability Maturity Model", "CMMI", and "Capability Maturity Model Integration" are registered in the US Patent and Trademark Office by Carnegie Mellon University.

(SM) "TSP" and "Team Software Process" are sales marks of Carnegie Mellon University.