Measuring Quality in Software

“If you can’t measure it, you can’t improve it.” – Peter Drucker

“Testing gathers information about a product; it does not fix things it finds that are wrong.” – Gerald M. Weinberg

I’ve been giving some thought as to metrics in software testing, as a way of measuring quality. This may be a bit of Frequency Illusion on my part, but I’ve noticed recently that a lot of conversations have come up around me on that particular topic.

I’ve been grappling with the question of, how (if at all) can you measure quality in software, and how can you communicate that information? It’s a question that tends to generate a fair bit of friction in the software testing community, and some pretty heavy-hitters have weighed in on the subject over the years. In this post, I’m going to share some of my thoughts on possible answers.

What is Quality?

In order to figure out if we can measure quality, we must first understand what quality actually is. This can be a bit of a zen koan, because quality is a subjective experience, highly dependent on the context of the subject, but at the same time groups can sometimes look at something and agree that it is of good quality without explicitly discussing what they mean.

I tend to rely on the definition pioneered by Gerald Weinberg, and expanded on by James Bach and Michael Bolton:

“Quality is value to some person (who matters)”

If we start from that definition, then this gives us a place to start asking somewhat less abstract questions. Which people matter, and why? What do those people value, and how do we know? In my view, these questions are ones that need to be considered by the entire business, and some degree of agreement needs to be reached. This questioning-confirming-agreement process will likely need to be revisited periodically, since systems (both electronic and human) are always changing, and as we gather new information on our own efforts to deliver value to people that matter.

How Do We Measure Quality?

In my view, the short answer is: You don’t.

The longer answer is, I don’t think you can’t measure quality directly, because it’s so completely subjective. That being said, you can measure value, and if you have enough of the right data, you may be able to infer a sense of the quality. At the very least, you have a common place to start the conversation.

Let’s look at a hypothetical situation: Your company is building a web-based platform for enterprise customers, who want to be able to easily share larger files with each other, without relying on insecure methods (eg. SMB) or systems outside the business’ control (eg. Dropbox).

Since this is a for-profit business, we want to make enough money to cover our costs, and make a reasonable profit to help fuel future growth. So, who matters? Well, paying customers matter. More specifically, the decision-makers matter, since they’re the ones who will have the authority to give you money in exchange for your service. What do they value? Maybe they’re looking to improve the efficiency of their staff, in which case performance is probably of value to them. Maybe they’ve had a bad result from a security audit (or worse, experienced a breach), in which case they’re probably looking for a highly secure solution.

Who else might matter? It seems to me that you probably want to make sure that the people who influence those decision-makers have a positive view of your platform; maybe those are internal IT staff, or maybe those are front-line workers of some kind. What do they value? Likely, they want to reduce the friction of their job, so ease-of-use is probably of high value to them. You might even consider your customer’s customers, to better understand why your customer values the things they do.

Another angle to consider, is what reduces the value of your product to the people who matter? For example, if you’re selling software to the Government of Ontario (a province in Canada), you must meet some fairly stringent accessibility requirements. If your software doesn’t meet those standards, then some of the people who matter (decision-makers in the bureaucracy) will see your product as lower quality, in comparison to applications that more closely meet the accessibility requirements they set.

Closing Thoughts

Ultimately, I can’t tell you what metrics to use to measure value; it’s something that you have to work out in your own context, and that context needs to be developed by representatives from across the business. It may take longer, and it may not be comfortable. As Patrick Lencioni has said, “Every endeavor of importance in life, whether it is creative, athletic, interpersonal, or academic, brings with it a measure of discomfort,”