Test Cases – a partial exploration

A 45 minute blog installment.

Note: I’m using the term “test case” but am really talking about the “test script” (the details/steps that describe the test execution). The terms seemed to be mixed up quite a bit so I’m declaring my usage up front.

There are a lot of thoughts I’ve had while writing this blog post. They might form a later post as covering them would put me outside the 45 minute limit.

What is a test case?

I think, once upon a time the, almost universal, answer to this question would have been (something like) a detailed series of instructions around data and inputs to test an idea. In my mind I see lots of writing as a “guide” accompanied by an expected outcome. I also tend to associate these with “testing the specification” – that is, if it’s not in the specification, it’s not a test case. When you have a space for an expected outcome, and leaving it blank is not an accepted practice, then it can narrow your view.

I think that now, at least for me, the answer is that a test case could be a lot of things. Sources of inspiration for great testing ideas are everywhere. You need to choose to engage with the discovery and curation process (because not all your test ideas will be executed). It could be an idea you have from early investigation of the changes. It could be an idea you generated from discussions with a programmer or business analyst. It might be supported with some notes about key aspects of the scenario or variations of it. I think of these as “test ideas” but they are a form of test case. This approach, in my mind is about having enough documentation to help keep your testing focused but not so much it gets in the way of performing testing.

The context in which you work will no doubt have a large influence on whether you must document test cases in detail or if you can take a lighter approach. Of course, the approach you take might be neither as there is a spectrum of possibilities.

I’ve tested using both approaches. I can write detailed, structured test cases. When the understanding of testing is akin to an activity that “ensures” some documented aspect is checked, I can see why many might default to the detail and even see it as an input to the production of a quality product. If I’m asked to write detailed test cases, I would if that is part of the engagement context. Is it the best use of my time? Maybe not.

The idea that more detail makes a test “highly repeatable”, “clearer” or “suitable for anybody to execute”, in my experience is flawed. The more focused you are on following details in a test case the less time you can spend observing the software you are testing. That’s simply an outcome of how our brain is wired. You’re also somewhat focused on looking for that “expected outcome” which brings an unintentional level of tunnel vision when you really want to be more holistic in your observation of behaviours.

“Clearer” is an attribute that is a relationship between the test case and the user of it. When somebody writes detailed steps they are writing it from their perspective. Knowledge, both explicit and tacit, system experience, writing style, assumptions, specific use of words, and other factors all contribute to “it’s clear to me” and that’s where the clarity warranty expires.

I remember reading once (and I cannot remember who wrote it) “that a test case does not find bugs, but it does help lead to you places where you might find them”. I think there is merit in that idea. How much information do I need to help keep my testing on track, focused on finding important risks? Where do I need to “shine my testing light”? I quite like a “minimal documentation” approach because it allows me to focus heavily on exploring the software, observing behaviour, posing new hypothesis, running new tests that have emerged from my interactions with the software. This is, at least for me, the real excitement of testing. Observing, thinking, challenging, exploring, posing “what if’s” as I look for new information about the system.

So, what is a test case? It can be a variety of things but, at the heart of the concept, is something that will help you test with focus. Documentation that will help you to “check” specific claims made in documentation and test in all those spaces in which the documentation is silent.