The Efficiency and Deficiency of Automated Tests

Automated tests are a fundamental pillar of agile software development. But are they the holy grail or are there aspects of manual testing that cannot be automated? In this article you will learn about the hidden risks when relying too much on test automation and how to handle them.

Introduction

For many years automation has been spreading into the testing domain, replacing hordes of testers and allowing more flexibility in the delivery process of modern software development. But can manual testing be replaced without leaving gaps? The advantages test automation brings are manifold: Tests can be executed on demand at any time without having to allocate human resources. Tests can be executed extremely quickly by leveraging parallel execution, limited only by available computational power and without relying on the availability of a large group of testers. But do these advantages outweigh possible disadvantages?

The Inefficiency of Automated Tests

Many have experienced the advantages of test automation and have them deeply rooted within their test strategy. But have its inefficiencies been accounted for? Even the most inexperienced tester will look beyond the written test steps and listed expected results. A tester, being a human being, has a subconscious expectation. When executing a test case, which follows the AAA (Arrange, Act, Assert) design principle, a tester’s expectation is additionally influenced by experience, habits and customs. It might be something the tester would not be able to name or explain upfront, but it is something to which they will react when confronted with a triggering situation. So, although there are many different types of test automation, covering various aspects, such as functionality, usability, design, performance, etc., the “human touch“ is not replaceable. Automated tests verify, but they do not validate.

Definition of Test Automation

The term “test automation“ has many facets and can be defined depending on experience and point of view. Using a nightly import to reset test data can be described as partial test automation, as it replaces manually fulfilling preconditions which are a fundamental part of a test case. Also, executing a http request using a tool such as Postman or even Curl can be defined as test automation, even though the result still needs manual attention for verification. In this article, test automation is defined as replacing both the execution and the verification of test steps as well as fulfilling preconditions. This article does not promote test automation as a specific type of testing, but rather as a type of execution.

Closing the Gaps

The risk of neglecting the gaps left by solely relying on test automation can be mitigated. But guess what? The solution cannot be automated. Manual testing must be employed. But how can this be done without the usual disadvantages of manual testing? An agile software development process relies on a lean testing strategy which distributes testing activities across the whole process, relying heavily on automation to minimise or even avoid potential bottlenecks. To leverage the power of manual testing without establishing bulky process steps manual testing must be embedded in a lean way. The following two approaches can be integrated leaniently into the development and release process.

Exploratory Test

Functional testing is often accompanied by exploratory testing. This becomes even more important when replacing manual execution of functional tests with automation. Exploratory testing harnesses the human traits that test automation obviously lacks. It can be potentially applied across the whole process, but it is best executed specifically when testing user stories. As the aim of an exploratory test does not rely on professional testing abilities but rather on experience, it can be performed by any member of the project team. Especially business near functions, such as Product Owners and Business Analysts, are good candidates to execute exploratory testing, avoiding it becoming a bottleneck represented by testers. Best case would be having a potential user execute exploratory testing, similar to a user acceptance test. Unfortunately this involves resources usually not available within a development team and thus represents a dependency, which should generally be avoided to ensure a lean development stream. This brings us to a second option, leveraging canary releases.

User Test

A canary release utilises user traffic splitting in a production environment and allows to direct a controlled number of users onto a new or changed functionality to observe its performance. It can also be used to grant access to new or changed functionality to specific user groups, such as beta testers. These can be internal employees, crowd testers or volunteers from the respective customer base. It is important that this group is aware of their role and that there will not be any negative reaction to potentially poor perceived quality, but only constructive feedback, which can be used to improve the experience finally released to actual customers.

Wrap up

Just like a gambler, who does not put all his money on one horse, the inefficiencies of test automation should be mitigated by applying specific manual testing activities across the development and release process. The manual execution of test cases includes implicit expectations, based on human traits, such as empathy, comprehension and interpretation, which go beyond the predefined expected results. It is essential to reduce the risk of poor quality deliveries by introducing both exploratory testing within the development process as well as lean user acceptance testing in the release process. This helps to ensure the limits of test automation are adequately compensated, without introducing bottlenecks and external dependencies.