What does AI say about software testing?

Studies show that people now trust AI more than they trust other humans when it comes to their careers. So what does this mean for us? It means that people will be strongly influenced on topics (like Software Testing) based on what a basic prompt tells them, this can have massive impacts to our careers as testers:

• A CTO asking whether they need software testers in an organisation.
• Software Engineers trying to understand what modern testing looks like.
• AI used in hiring processes (to create questions, model answers or as the interviewer).
• Testers trying to solve the testing problem and explain what and how to test.

If people trust the AI responses more than a human response, there’s a risk that even a seasoned professional will be told they’re wrong if they don’t adhere to what the AI response says.

Asking what is software testing

With some basic prompts you can get a pretty good basic understanding of software testing, to my eye these descriptions seem to be a little biased towards confirmatory testing and engineering (automation) style testing.

The initial responses from ChatGPT that I got were very focused on confirming requirements, it talked about comparing actual results with expected results and bug identification. The types of testing it brought up were focused on passing/failing requirements and scripted testing (both automated and manual).

There wasn’t an explanation covering critique of the item under test, uncovering the unknown or risk analysis; things that push testing way beyond confirmatory testing and provide a lot of value. To me, it read like a very simple and surface level of testing that didn’t add a new understanding or provide a “hey, did you know that testing also provides THIS?” view of things.

The surface level responses were also really visible when I asked about Agile development software testing; it explained things from a confirmatory point of view with a brief mention of early collaboration. There was nothing about testing releases to learn from customers in production (which is THE WHOLE POINT OF AGILE), shifting testing left to derisk things and build the right thing first time around or testing to a view of good enough to go faster.

I also noticed a lack of explaining why things might be different. The testing we do in Agile is all predicated around making it quick to release things (to learn from the customer) and provide confidence quickly to avoid stalling the many iterative releases we want to make. This means pragmatism, testing to good enough, faster automated tests and shifting both left and right to support these things.

A more specific prompt for asking about quality engineering gives a bit more context around holistic testing. Here it mention shift left, shift right, continuous testing, being data driven and team collaboration; but it doesn’t mention why we need them or what we’d use them for (there’s no reference to QE being a facilitator for Agile).

Something I’m noticing is that these explanations are telling me HOW but not a lot of WHAT and WHY when it comes to testing. If I wanted to learn about why I’d need different testing styles then I’d have to know to prompt for those. Even when asking what type of software testing would I need the responses are limited in scope: code needs unit tests, behaviour needs functional tests. There’s not a lot here to help someone understand that the philosophy of testing might be different based on context.

Asking what does a tester do?

Basic prompts seem to treat what a tester does in the same way as it did testing: with a focus on confirmation and meeting requirements. Whilst this does form a part of testing, it’s a limited view that might not show people all the value a tester can bring to an organisation.

To my eye, when asking about what a tester does or asking the AI to create a job spec the results are very aligned to a regulated industry like banking. They don’t align as well to engineering organisations or startups (arguably the environments who need the most help working out how a tester can help them).

A quick refinement of the ask “create a job spec for a software tester in a startup or engineering organisation” does refine this much further. It speaks much more to non-functional, setting up frameworks and shift left. Interestingly the job spec that it generated for me asked for ISEB certifications and strong SQL skills; things that speak more to banking than a startup.

Going further and asking for what a Quality Engineer does, gives a view of more holistic testing and describes shift left testing, shift right testing, processes and collaboration. It also seems to pit QE against QA as different things, as opposed to giving a view that QA is something that happens within QE (or that QE might just be a renaming of QA in a lot of organisations). Weirdly it’s telling me that QE is different because it’s embedded in a team where QA is not, which doesn’t align with my experiences.

Asking for model answers for testing in interviews

Once again, basic prompts seem to generate questions and answers that lack nuance and relate back to traditional / regulated industries. There’s a lot of focus on HOW rather than the WHAT and WHY of testing too, focusing on types of testing and not the philosophy that underpins them.

Whilst a lot of answers (like the one above) are technically correct, I don’t think they show much practical understanding of how testing works in the real world. Like how, for many people, testing involves verification and also critique of a product; we’re actually seeing a lot more of the latter expected these days in teams.

When asking about what tools software testers use, most responses provide the same answers:

• Selenium / Cypress / Playwright
• Postman / RestAssured / SoapUI
• JMeter / k6 / LoadRunner
• Jira + Zephyr / Xray / TestRail / Quality Center

These cover the biggest tools, but again these are probably more used and seen in big organisations and banking (because they cost money). Smaller, more niche, tooling isn’t being referred to in standard queries.

When asked about the characteristics associated with a good tester the response provided some good stuff (Curious, Attention to details, Organised, Communicative, Passionate) for doing the basics of the job. It missed out on some key things like influence, pragmatism, resilience, self starting and change agency, all things that a tester would need to sell testing into a team or build a culture of quality.

I refined the prompt to ask specifically about the characteristics of the first tester in an organisation and the response was very similar, with nothing about being able to sell ideas or self start to identify gaps in quality processes and run initiatives for them. It did reference building a quality culture though.

So again, like tools, some key characteristics could not be seen as valuable in answering interview questions based on ChatGPT prompts. Things that’d make a candidate stand out and thrive at an organisation (especially in senior roles) would fall between the gaps or wouldn’t be seen as valuable in testing roles.

Conclusions: What does AI say about software testing?

Basic prompts really seem to paint a picture of testing as a validation exercise based on scripts, overly focused on automation and not showing the pragmatism and influence that a quality professional brings. I guess this is to be expected since the majority of testing content on the web is probably focused towards big organisations (like banking and finance) that have a very specific vibe to their testing.

Where using AI to learn about testing really falls down is where we need a view for testing in smaller engineering startups or at a more senior level. Again, this is to be expected since there’s not huge amounts of super-senior testers building testing capabilities from scratch. Plus when people post it can be about different topics each time, meaning there’s not a lot of reenforcement of those messages.

Maybe there’s something to be said for trying to standardise how we talk about what testers provide in 2025, reenforce the messaging around Quality Engineering, pragmatism, looking for what’s good enough, collaborating with engineers and influence. By doing this, maybe we can shift AI to start to talk about all the additional ways that a tester helps a team beyond “checking requirements and finding bugs”.

That’d also mean challenging outdated stereotypes of dev vs. tester adversity or just trying to break the system. That content just reenforces the messages that don’t help the people we need to outreach to in the industry: the people who if they knew how we can hep them, would want to hire more of us.

More Blogs

LinkedIn