How I do exploratory testing

Wayne Roseberry posted the other day about how few testing blogs are about testing. I mean there are loads of other skills around testing that need to be explored, but I take his point about core testing skills related blogs. I just have to look at my own blog.

I really like doing exploratory testing. I know that sounds simple, but its the doing I like over the planning part. Perhaps this approach leads me to leap before I look sometimes, but I’m also a great believer in testing something as soon as possible and getting feedback into someones who matters hands. And I don’t mind if I’m wrong, I’m happy for the developer to chuckle as I firmly grasp the wrong end of a problem or feature. Time to get started is really important in testing, the longer you wait, the feedback tends to go stale and everyone moves on to the next thing.

To counter this, I have tried to develop my technique to allow for the immediate feedback, with a stopping point to allow me to wind it back in and change direction. I don’t think this is ground breaking or anything, its just the way I’ve adapted exploratory testing techniques to suit me.

Single broad charter, relaxed constraint exploratory testing

This charter I create straight away, after a walkthrough of a pull request and demonstration of working software is purposefully left broad to allow me leeway to test widely. Take this (changed slightly to protect the somewhat innocent) example from my last role:

Explore the ability to generate PDF reports in the iOS app
Using the Follow the Data heuristic
To discover how accurately content from the app is shown in the PDF report

To break it down a bit:

Explore

I picked the iOS app, as that was far more popular in terms of usage and paying subscribers. Android could come later on.

Using

The Follow the Data (The Heuristic Cheat Sheet is your friend) heuristic doesn’t place too many constraints on your testing, allowing you to go with the energy. Plus the data was entered into the app, uploaded then retrieved by a web service, making data integrity important. The web service itself had been tested previously also to some extent.

Discover

Then the ‘to discover’ part speaks to the content of the report, rather than the mechanics of report generation. But you get some of the mechanics implicitly anyway and I asked the developer to demonstrate the basic communication between app and service working.

Get testing started

I would give myself an hour or so with this charter and provide my initial feedback. If the developer wants to hang around test together, then thats better. Or just on an open call while I test, I’m happy with that too.

Debrief

Once the charter is done, back on the call with developer, notes at the ready. Notes are taken in markdown, behaviours, problems and questions the usual structure, with the charter at the top. The debrief is such a crucial part, but it can take some developers a little while to adjust. Some prefer to see your notes first and then discuss, others happy to jump in. Good written, testing notes are important in remote teams. Basically, ask what the developer prefers, rather than just assuming.

Multiple specific charters, tighter constraints exploratory testing

Depending on the results of the above charter and debrief, I would generate a maximum of 3 further charters. In the example above, PDF reports containing none or many images, PDF reports with sections spread over more than one page and App handling of PDF Report Service errors interested me most.

I have deep respect for the number 3, wide enough to give you more options, not so wide to overwhelm:

Explore the ability to display images in PDF reports in the iOS app
Using the Goldilocks (Too Big, Too Small, Just Right) heuristic
To discover if there are any limits to the number of images that can be rendered

Explore the ability to render non-image data over multiple pages in a PDF Report in the Android app
Using the Position (Beginning, Middle, End) and Selection (Some, None, All) heuristics
To discover if the PDF report maintains its structure when some sections have multiple pages

Explore the ability of the app to handle errors from the PDF Report service
Using a mocked version of the PDF report web service
To discover the extent to which the app fails in favour of the user

If I find the results of those charters useful, I generate another set of charters. I might not add more charters for areas that I feel have enough depth. I keep my eye on how many charters I now have. I test based on what catches my interest and follow the energy. If I limit the charters it can bring me back to the core mission more easily.

Bonus: Add a ‘Hard to Test’ section to your exploratory testing notes

Exploratory testing helps to make the application more testable. This is one of the cool things about testability, if you are testing, you can move the testability needle just a little bit. Add a unit test, you understand the code and its dependencies better. Pair with developers or debrief them after a session, everyone knows a little more about the system.

You might also fix some bugs with your developers, or a developer you pair with might add a utility endpoint to reset some data to a known state. All these things help testability, but can often get lost in the aftermath of an exploratory testing session. I have a free form notes area in my session notes to detect possible testability gains.

If, at any point in your testing session, you feel:

• There is something out of your control happening, there are side effects of testing.
• There is something important happening in the application, but there is no record of it in logs/events/analytics/etc.
• There is something you think would be useful to isolate and test but you can’t.
• There is something to test that needs lots of things to go right for it to work.

Add these to your debrief with your developer.

Closing remarks

Approaching exploratory testing with intent is really important. Its also a lot of fun and helps you to exercise your creativity. Its part of learning a system, deciding what and when to automate, providing feedback to developers and a host of other activities. From my own early experience of following a test script in HP Quality Center I was still be exploring around the script, but using the script as a guide. Exploratory testing is a skill that testers need to nurture, you do it regardless, even if you call yourself an SDET III, QA Automation Tester, Manual Tester or whatever.