Automation State – Answer Me, These Questions Three
“Stop. Who would cross the Bridge of Death must answer me these questions three, ere the other side he see.”
Most of us nerds (and non-nerds alike) have seen Monty Python and the Holy Grail. The Bridge of Death scene is one of the most quoted and paraphrased. The three-question concept (not to be confused with the three seashells concept) got me thinking about test automation.
My good friend and colleague, Joe Colantonio, the creator of Test Guilds and the associated conferences, has a new newsletter called “Are You Automated?”, the name having been inspired by the Jimi Hendrix album “Are You Experienced?”. This reminded me of my thoughts on the binary nature of automation. Whether something is automated or not is a topic that I wrote about a few years ago, but Joe’s newsletter title got me thinking a bit deeper.
As both a consultant and practitioner, a portion of my job is to assess the state of an organization’s automation endeavor. I’ve been privileged to see automation endeavors in many states of construction. I’ve helped organizations start from scratch, rehabilitate endeavors that were not producing sufficient value, and evolve their automation when additional value was appropriate. Part of how I do this is I ask a lot of questions; that’s how I learn what can be appropriate for a specific organization. I can learn a lot in a short amount of time by asking three high-level questions; I then help organizations answer them as appropriately as possible. What I’ve seen can often be generalized by answering: “Do I have the right automation?”, “Do I need more automation?”, and “Do I have too much automation?”. Once I have an idea of the state of the organization’s automation, I can begin making recommendations on appropriate ways forward for them.
Do I have the right automation? I previously wrote about why we should automate and why we must take care to create automation that is appropriate to meet our business goals. We should frequently ask ourselves “Why should we automate this?” and “Which business goal will this help us attain?”; these are questions not asked often enough. The answer always seems obvious: “because we need it” or “our process dictates that we have it”. But this isn’t always the case; sometimes we can meet our goals without automation, especially if some specific implementation of automation will cost more than the value we will gain. The reality is that most teams will benefit from the automation of some portion of their testing work, though it may not always be the traditional “test case-based” automation; non-traditional automation can provide similar or more value in many situations.
What do I do if I’m in this state? If you determine you need automation and you don’t know how to start or if you have automation and it’s not meeting your needs, you might need help. You could hire an experienced testing and automation leader, you could rent one (i.e., hire a consultant to get you started), or you can acquire the expertise yourself. There are no wrong answers, but if you decide to acquire the expertise yourself, your leadership must be tolerant of the time investment and iterative changes that will occur while the teams are learning.
Do I have enough automation? This question also goes back to value. The main question we should be asking ourselves here is “Can computers help us be more effective or more efficient than they already are?” We’ve automated our smoke testing suite, it now runs in 8 minutes instead of 30, and the team doesn’t have to wait for one of us to run it. Huzzah! In that suite, would you like to have more checking on each build or deployment? Or did you have to limit the suite’s content due to its duration? Since your new automation execution requires a shorter duration, is there additional automation you’d like to add? Does your product have new or changed features that could use some automation for testing? Asking and answering, these questions are key to maximizing your automation’s value.
What do I do if I’m in this state? Assess where you spend the bulk of your testing effort; those areas can be good targets for automation. Also, look at activities that are iterative or repetitive; computers are typically good at performing those kinds of tasks. Finally, look at areas that are “programmatically straightforward”, meaning areas like API testing that don’t always spring to mind. Automation in these areas can be less effort to create and maintain over time. Additionally, API-based automation typically runs significantly faster than UI-based automation.
Do I have too much automation? This is one of my favorite automation topics. How can we have too much automation, you ask? It’s very simple: we forget to pay attention to the value we’ve already built. If we aren’t attentive to our value, we can easily Yes, automation is very much a “what have you done for me lately” endeavor and the easiest way to show what we’ve done lately is to add more automation. And more, and more. Until our automation effort outstrips the value (and there’s that word “value” again) it’s providing; this part is the bad part. I won’t belabor this topic since I’ve written on similar topics such as automation value, Rube Goldberg machines, and automation maintenance in general.
What do I do if I’m in this state? If you think you’re spending more on your automation than the value you are realizing from it, the first thing I recommend is looking at all of your automation to see what of it is producing value and which might not be. Sometimes, some of our automation can go stale; stale areas are ripe for removal or combining with value-added areas.
Ok, yeah, I’ve listed a lot more than three questions. The reality is that, although the three questions can help us categorize our automation state, we must be vigilant with respect to our automation in order to create and maintain value. If there’s no value, we’re just wasting our time.
Like this? Catch me at or book me for an upcoming event!
