The Quality Constant: Think and Act Experimentally

I was asked what one piece advice I have that I would give to testers starting out in the industry. It had to just be one piece of advice. It’s an interesting challenge.

What Survives?

Before I get to any possible advice, I’ll say that there’s an interesting, and perhaps, related challenge about if there was an apocalypse and only one scientific statement could be retained, what would it be? The idea being the one statement should allow as much of a rapid re-advancement of science to its pre-apocalypse level.

I should note I’ve approached this apocalypse idea before when talking about a test apocalypse and a test snap.

The usual answer here is that “Matter is made up of discrete units called atoms.” My argument has always been that a more interesting idea is: “The mediating influence between space and time is the finite speed of light.” There are various reasons why I think it could be a strong contender.

• This statement ties directly to Einstein’s theory of relativity, which revolutionized our understanding of space, time, and gravity. The finite speed of light forms the basis for both special and general relativity, two theories that have been essential to the development of modern physics. If you preserve the relationship between space and time through the speed of light, you essentially hold on to a cornerstone of the theory that allows the rest of physics to build on it.
• While the idea of atoms and matter is fundamental to understanding the physical world, understanding the nature of space and time might allow you to more rapidly advance theories about energy, motion, and gravity. It’s not just a statement about a specific material thing but about the fundamental relationship between how the universe is structured. Preserving that could help scientists rebuild both the conceptual framework and the mathematical tools needed to advance other areas like quantum mechanics, electromagnetism, and eventually, atomic theory.
• The notion of light speed being a constant mediating influence opens doors to other key concepts: time dilation, the relationship between mass and energy (E=mc²), and the limits of causal interactions. These concepts shape our understanding of everything from the cosmos (black holes, gravitational waves) to subatomic particles (relativistic particles, quantum field theory).
• In some sense, a statement about the speed of light also gives a kind of “unit of measure” for scientists. The speed of light isn’t just abstract; it’s something that can be experimentally verified and connected to every corner of scientific endeavor — whether that’s in the form of understanding distances in space or recalculating energy equations.

Sounds great! However, my approach does come with challenges. Atoms and matter are immediate and more intuitively grasped concepts, which could more directly tie into day-to-day technological applications, like electronics or material science. The speed of light, though foundational, might seem more abstract at first glance, requiring further development in mathematics and a deeper conceptual leap to re-establish much of what is built from it. That said, I think my suggestion is a good one for re-advancing science because it offers a broad, underlying framework for understanding the universe. If the goal is to re-establish both our understanding of physics and the tools for discovery, I’d say it’s definitely one of the strongest candidates.

Advice to Testers

Okay, so let me apply that to testing. Here’s my advice:

Build up your skills to think and act experimentally. Understand what it means to treat everything as the smallest possible experiment you can do that will tell you what your next experiment should be. Understand how to recognize when an experiment is going poorly (and thus correct for that) and when an experiment is going well (and thus amplify for that). And always, always, always listen to the results of your experiments. Decide how, whether, and to what extent those experiments are guiding you toward changing your thinking and your behavior. Apply this to everything you encounter in your career.

I did talk about the basis for this idea previously when I talked about my role as a quality and test specialist Now, sticking with my mediating influence bit above, I might have a mantra like this:

• The mediating influence between risk and confidence is the testing mindset.
• The mediating influence between software and value is the act of testing.
• The mediating influence between testing and insight is thinking and acting experimentally.

I think I’ve beaten that analogy to death by now so let me shift focus a bit or, rather, let me break down what I said a bit more.

The Nature of Experimentation

There are often debates about the “tester mindset” or the “developer mindset” or the “engineer mindset.” I think all of that can be subsumed with what I say above. Or, rather, the distinction about who is doing it matters much less.

First, let’s consider the basis of any experiment. It’s the hypothesis, right? Any hypothesis must be stated in such a way that it can be tested by experiments. “Can be tested” inherently suggests that there’s some mechanism to observe a cause-and-effect relationship. A hypothesis needs to be structured so that an experiment can either support or refute the relationship it proposes. If the hypothesis isn’t testable, you can’t determine whether the cause-and-effect link you’re investigating exists. By designing the hypothesis to be testable, you’re essentially saying: “I believe this cause (independent variable) will lead to this effect (dependent variable), and I can set up an experiment to observe and measure that relationship.”

What I just said is the basis of experimentation: a mixture of confirmation, falsification, and implausification. Applied to testing (a form of experimentation), this means considering the context in which something can be used and the conditions operative in that context. Being reductionist, those conditions are test conditions and data conditions. We can further break those down into valid conditions and invalid conditions. There can be complexity in the conditions, but fundamentally (atomically, if you will), everything breaks down into this.

The mindset comes in by how much and to what extent people can think and act experimentally. Some people do this very simplistically. Others have a much more nuanced approach. This also takes us into the standards of evidence that people apply. This also takes us into people’s theory of error. For some, this theory is very shallow. For others, it is very broad. Those theories of error and those standards of evidence are a large part of what determines someone’s mindset about what it means to experiment (test).

All of this is the case in any domain where experimentation applies.

So, just as the statement “the mediating influence between space and time is the finite speed of light” may cause us to think of a lot of other things, my hope is that the notion of “think and act experimentally” does likewise.

Experimentation, Techniques and Mistakes

The above was an expansion of my advice. So, now let me expand on that expansion. I suggest that effective testing is about finding the most impactful risks in a timely enough manner so people can make decisions about those risks. This is about the cost-of-mistake curve. In this context, costs are always cheaper if you can find your mistakes as quickly as you make them.

To find your mistakes as quickly as you make them, you need to put various test techniques where you are likely to make the most mistakes. We know what those places are. Those are when we elaborate (specify what to build) and then when we actually build it. This is the case even in non-software. Architecture has long dealt with this, as just one example.

Those test techniques I mention will differ. They will operate at various levels of abstraction. Sometimes, those techniques will rely on tools to scale human activity (but not, currently, human thought). This requires the democratization of testing and the distribution of quality assurance. This recognizes that people’s domain of competency needs to be aligned with their sphere of authority, such that they are best situated to find the most impactful risks using those test techniques and backed with a good understanding of what it means to think and act experimentally.

The -ilities

I argue that testability is the primary internal quality attribute. That quality leads us to controllability and observability. I can’t control what I can’t observe. But if I can’t observe, controlling doesn’t matter much. If we have those aspects in place, that can help us towards reproducibility. That might even lead us towards some measure of predictability. Those “-ilities” are crucial qualities for me.

Notice that someone might argue that “testability” as a quality has little value because customers don’t see it. Directly? No, they don’t. Indirectly, they feel it. If we don’t have testability — which is an internal quality — then that likely means we are sacrificing other internal qualities: extensibility, maintainability, scalability, and so on. Lack of those can impact our ability to deliver what customers do actually see, which are external qualities. At the very least, lack of the internal qualities might lengthen the time before we can deliver to the customer. What that impacts is our feedback loop.

Being able to deal with internal qualities via a fast feedback loop is what I think is crucial for keeping the cost-of-mistake curve tight. And since I believe that testers are, in part, “mistake specialists” (helping to find mistakes as early as we make them), the intersection between development and testing is very clear to me in this context.

A few related ideas popped up here (just like they would with my apocalypse-surviving science statement): there is not Quality (capital Q, singular). There are qualities (lowercase q, plural). Any of those can degrade or amplify. Further, those qualities can be broken down into internal qualities and external qualities. A failure to handle the former compromises the ability to deliver the latter. The former being compromised tends to be reflected as a lack of a primary quality attribute, which is testability. When that happens enough, you lose another primary quality attribute, which is trustability.

The Epistemic Basis of Testing

When I promote the above advice that I suggest, I work with people to frame operational questions that help stem what I see as the epistemic drag that is becoming a blight on our industry. Specifically:

• How do you gather evidence?
• How do you reason through a problem?
• How do you decide what evidence matters and what does not?
• How do you conduct experiments to get observations?
• How do you decide what observations matter and which do not?
• How do you decide where to focus your time when time is limited?
• How do you reconcile differences of opinion regarding evidence or observations?
• How do you reconcile different interpretations for the same evidence?
• How do you avoid soft reasoning and shallow questions?
• How do you balance exploration of the unknown with exploitation of the known?
• How do you use an always/never heuristic in testing?

Oh, and …

• How do you articulate all of the above?

Think about how many interviews for testers out there actually cover any of that. Next to none, I would argue. And that’s pretty sad for a lot of reasons but one of them is because all of this requires certain attributes that can easily be tested for.

• Be thorough in reading.
• Be patient in listening.
• Be careful in questioning.
• Be thoughtful in evaluating.
• Be precise in checking responses.

An easy framing question when looking for testers: How do you think and act experimentally? This is one of the biggest areas I see as being problematic, it’s one of the areas I least see asked about, and it’s one of the least I see being engaged with in our industry. Consider what engaging with a tester on this topic can show you.

• Testers need to show how they determine if the experiment is going well.
• Testers need to show how they determine if the experiment is going poorly.
• Testers need to show how they amplify the parts that are going well.
• Testers need to show how they dampen the parts that are going poorly.

The whole basis of how experimentation works — investigation, exploration, discovery, forensics, statistical thinking — seems to be getting more and more lost. Testers either lose it or don’t have it. The industry doesn’t seem to require it. Therefore, school settings don’t teach it. Thus, none of this really gets promoted as a way to help construct a society that is more and more being determined by the intersection of humans and technology.

This story isn’t being told.

Maybe testers need also to be storytellers?

Well, yes, I would argue they do. I explored that idea a tiny bit when I talked about the craft of testing as well as when I talked about how testing is like writing fiction. By “storyteller,” we don’t have to mean fiction, of course. My broader focus was on the idea of framing a narrative.

The Quality Constant

Just as the finite speed of light acts as a fundamental constant mediating space and time in the universe, perhaps we can think of a “quality constant” within software development. This isn’t a fixed value, but rather the inherent potential for quality that exists within a system at any given moment. Similarly, this constant isn’t directly measured but is revealed through the various mediating influences I’ve discussed: the testing mindset between risk and confidence, the act of testing between software and value, and the experimental approach between testing and insight. Our testing efforts, much like scientific experimentation, act as the mediating influence that allows us to observe, understand, and ultimately realize that potential. And perhaps, just as the context of the universe can reveal different facets of fundamental constants, the specific context of our software and its use can illuminate different expressions of this underlying quality potential.