AI-Assisted Testing – The Rules and Roles

Let’s talk about AI and testing. And not just because everybody does. Let’s talk about what we never have enough.

Time.

No matter what we do, we’ve got more work than we have time. So we try to save time. But saving time is not really what we want. What we want is more time. More time for testing.

We test in many different ways. Manual testing, scripts, automation and exploration. While the first ones are about things that we already thought about, exploration is about what we haven’t yet. At least partially – that’s how learning is done.

Testing Can Definitely Use Some AI

Our job as testers is to collect as much information as we can, and learn about the system we test. And our systems are complex, so we need a lot of time for testing. Time we probably will not get.

We say we’re after coverage. But, that’s not some percentage out of a test plan. Real coverage cannot be calculated, because it includes what we thought about, what we explored, and what we haven’t thought about yet. All these scenarios are in the system. We just didn’t go through all of them yet. Or found them.

We always want more of that coverage. And, we want it in the limited time we have, by optimizing the tasks we do, without compromising on quality.

Remember that compromise, we’ll get to that later.

We can get more time by investing. Automation, for example, is an investment that helps us save time later. We write automated tests now, and they continue to run later, freeing us to test something else. Another example of investment is in our new companion and soon-to-be-overlord: AI.

AI cannot replace the human-in-the-loop in testing. At least, not yet. We still need a human to look, notice, respond, approve, wonder and try. These things are not going away.

But if we break Testing into smaller activities, we’ll see that some of them can be optimized. Like, coming up with test case ideas. Or generating automated tests for a path we’ve explored, and confirmed works. Or, preparation of data, to get us to the point of where we want to explore. Many of our daily tasks can be delegated to our favorite LLMs.

And it’s completely free! Well, almost.

And I’m not talking about paying your LLM masters for calling their APIs.

Is It The Quality We Want?

Remember that I said we want to optimize tasks, but without compromising on quality?

There’s a bit of a problem with something (I hope that something won’t be offended), that not only thinks it’s never wrong, but may also invent stuff, just to make us feel better about ourselves.

And if we let our genie do whatever it thinks, without supervision – we’re going to lose that quality.

In order to keep quality at the same level we want, the humans-in-the-loop (that’s us) get a new job. We’re not just offloading work any more. We’re delegating.

Real delegation means being responsible to the quality of delegated work.

We need to be the gatekeepers. We are the verifiers and approvers. If before we were the quality assurers of our sysetm, based on our work, now we’re responsible also for the work result of our prompt minions.

We need to evaluate and confirm the generated cases, and data and code. If we just accept what it spits out (and that’s a lot), it’s a recipe for disaster.

AI vs APIs

Here’s something that you might have seen before, if you’ve asked AI to generate a test for an API. And who of us didn’t try? Most self respecting LLMs (that means all of them, they don’t do anything wrong, remember?), will give you a test that checks that the returned status code is 200. Which is ok, right?

But should it be a 200?
And is that really what tells us that the call succeeded?
That the API does what it needs to do?
That the feature works?

What AI tools have not managed to get yet, is context. The human-in-the-loop has that context. And while we try to give our genies as much context as we can, they just don’t understand…

It’s up to us to look at their suggestions, apply our knowledge, our experience, and judgment to accept what we’ve been given, fix it, or even throw the answers away.

Or we can ask again in a different way. That may work too. Because even if we won’t gen an the answers we seek the first time, we still save time if we get it on the second.

Can AI replace the tester? At this point, not without compromising on quality. We’ll explore more way of using AI to help us with our testing activities, in the upcoming post. But remember, we need to approve everthing.

That’s a big task by itself.

Did you ever trust your wizard too much? What were the consequences?