Assuring Quality: Beyond Testing & Reviews

"Test? Of course we test! It seems like we spend way too much time testing. And we review designs and code, too. But all we are doing is finding defects. Lots and lots of defects! Isn't there more to QA than finding all of these defects?"

Yes, there is much more to QA than that. In fact, testing and reviews should really be called "Quality Control", or QC for precisely that reason. They don't really assure quality; they merely check the quality and keep the product from going into production if it is too bad.

Quality Assurance involves the proactive activities that can actually assure that we will build better quality than we have in the past. The bottom line on QA is paying attention to how we do the work we do, and how effective that work is.

QA = Process Assurance

It is instructive to note that two of the most authoritative sources in the software industry agree with each other that QA does not deal with testing. They both tell us that QA means Process Assurance. Consider these quotes:

"The ... Quality Assurance process area supports the delivery of high-quality products and services by providing the project staff and managers at all levels with appropriate visibility into, and feedback on, processes and associated work products throughout the life of the project." -- Capability Maturity Model Integration (CMMI)^®

(The CMMI does address testing and technical reviews, in the "Verification" and "Validation" process areas.)

"Quality Assurance (QA) is the application of planned, systematic quality activities to ensure that the project will employ all processes needed to meet requirements." -- Guide to the Project Management Body of Knowledge (PMBOK)^®

(The PMBOK does address testing and reviews, in the "Perform Quality Control" Knowledge Area.)

"Why would QA be more about processes than about any technical activity?" That question belies an important fact. Ensuring that our technical processes are serving us well is very much a technical activity. It takes strong technical knowledge and experience to judge if technical processes are appropriate to the task, and if they are being used properly.

Quality (or Process) Assurance includes answering three closely related questions:

Are we using the appropriate processes?
Are we consistent in how we use those processes?
Are our processes effective and efficient?

Are We Using the Appropriate Processes?

What are the technical and non-technical challenges involved in the projects we undertake? What are the types of problems do we encounter on our projects? What opportunities do we need to be able to capitalize upon?

The processes that we use must be tuned to the needs of our projects. Projects with more dynamism need more Agile processes. Projects with bigger risks need more deliberate processes. Projects with technical challenges need processes that meet those challenges head-on. And, no, those three are not mutually exclusive! If you have dynamic, risky and technically challenging projects, then you need processes that are Agile, deliberate and head-on.

These processes must address all of the activities that we engage in to complete a successful project. How do we determine our customers' requirements? And how do we manage changes to those requirements as the project progresses? How do we make architectural and design decisions throughout the project? And how do we keep the technical design resilient as things change? How do we code and test? How do we identify and fix defects? And how do we ensure that changes don't cause regressions? And, of course, how do we plan and manage all of these technical activities?

In too many organizations, the work is done the way it has always been done, just because of history. QA means taking a critical look at the processes that we use, and deciding consciously if they are right for the projects we undertake.

Are We Consistent in How We Use Those Processes?

The best of processes won't do us much good if we are inconsistent about them. If some people do things one way, and others do them another way, we are likely to have confusion, problems, and wasted effort. If I do the same job a different way each time, I am likely to produce inconsistent results. Sometimes I will do really well! But other times, I will make mistakes or miss steps.

QA includes ensuring that after we decide which processes we should be using, we actually use them consistently. There are a variety of ways that we can do that.

The CMMI talks about QA audits, where someone (the "process-police") checks over people's shoulders to be sure they are following the appropriate processes, and writes them up if they are not. This is usually implemented as a punitive approach (although the CMMI does not intend for that to be the case).
The Agile methods employ a process coach who works with the team members to ensure the processes are used effectively. This is more of a collaborative approach that is consistent with Agile principles.

Whether you choose one of those two extremes, or some other method, the important part, from a QA perspective, it to ensure consistency. There are times when people circumvent a process because it doesn't work well in a specific situation. But the next question is designed to address that.

Are Our Processes Effective and Efficient?

We chose to adopt each of our processes because we expected to get some benefit from it. So it is important to take a step back from time to time and to ask if the process is effective; that is, if it really provides the benefit we expected. Often our processes don't operate as effectively as we expected. And sometimes, changes in our work environment can render a formerly good process ineffective.

Also, when we adopted each process, we expected that it could be performed without adding unduly to our costs. Of course, we must check periodically to ensure that the process is as efficient as we expected it to be. And, as with effectiveness, we could find that a process that used to be efficient is now rendered inefficient because the work environment has changed.

QA includes evaluating our processes on a regular basis to be sure that they are effective and efficient, and either improving or replacing them if they are not. Inconsistency in execution of a process is often a symptom of inefficiency or ineffectiveness. When people consciously choose to deviate from the prescribed process, it is usually because of some efficiency or effectiveness problem.

Assuring Quality

If we are content with finding and fixing defects, then we do not need to practice Quality Assurance. Quality Control will suffice. But most organizaitons aspire to produce better quality products than they have produced in the past. Quality Assurance activities will assure that we can achieve the levels of quality to which we aspire.

^® Capability Maturity Model Integration, and CMMI are registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.
^®PMBOK and Project Management Body of Knowledge are registered trademarks of the Project Management Institute.