The Hidden Costs Of Conditional Testing In UI Automation
Conditional testing, at its core, seems like a straightforward and logical approach to automating tests: write conditions, and based on those conditions, execute specific actions. This method promises a level of adaptability and intelligence in handling various scenarios in automated tests. However, this common test pattern often conceals a tangled web of challenges.
Despite the apparent simplicity of conditional testing they often lead to unpredictable outcomes and complex test maintenance. What’s more, conditional tests can mask these problems until they appear much later, leading to costly and time-consuming fixes.
In this article, we’ll peel back the layers of conditional testing in UI test automation, revealing why it’s often considered an anti-pattern and why steering clear of it may lead to more robust and maintainable test suites.
What are conditional tests?
Conditional tests use conditional logic to test diverging pathways in software applications. You will know them by the use of familiar if-else statements:
if x then y else z
In UI automation, they are commonly applied to initialise the state of the application, as well as for testing diverging pathways. But as we’ll soon discover, they can also lead to some unintuitive consequences.
Shown below is a basic test scenario that uses conditional testing to validate an app’s dark mode. This is an oversimplified example, but it should help us understand some key aspects of conditional testing:
1. Select a random user from the test database.
2. Navigate to the 'Settings' page.
3. Check that 'Dark mode' is enabled:
A: If dark mode is enabled, skip to step 4.
B: If dark mode is disabled, toggle on dark mode.
4. Verify dark mode.
The test design is also illustrated below. The decision point to check whether dark mode is enabled is depicted in yellow, and the two logical pathways are depicted in blue and green. This design helps ensure 1) the required software state is initialised for each path to execute predictably, and 2) diverging pathways for either state can be tested.
At first glance, this design seems effective for initialising the application state and testing diverging pathways. However, the very nature of diverging logic can produce some unintuitive issues which we’ll turn to next.
Problems with conditional tests
The diverging logic in conditional testing can be particularly problematic when paired with randomised data to drive the test. In most applications, the shape and distribution of each dataset will likely be different, which can lead to the following:
- Unpredictable test frequency: The number of times each pathway is tested is dependent on the test data distribution, which can vary greatly. For example, if only 1 in 5 users have dark mode disabled as illustrated below, the blue path for enabling dark mode might only execute 20% of the time.
- Flaky edge cases: The unpredictable schedule of tests can naturally lead to flaky tests. If changes in the application result in test failures for any conditional pathways, the test failures might show up as infrequent edge cases, leading to unreliable test results. This is particularly problematic for test failures caused by bugs, which may not be picked up immediately.
While the above issues tend to occur with randomised data, conditional tests can introduce additional layers of complexity even with non-randomised data:
- Synchronisation and awaits: Conditional tests often require additional handling to ensure that the application is in the expected state before proceeding. If not carefully implemented, this can lead to increased test execution time or race conditions that cause intermittent failures and flaky tests.
- Difficult to debug: Due to the tangling logic of multiple pathways and diverging application states, conditional tests tend to be unintuitive and difficult to interpret. This can make debugging tedious and time-consuming.
- Reproducibility: A conditional test may take a different pathway each time it is run. Hence, reproducing errors in a test can require multiple test runs, manual manipulation of application states or mocking of variables to trigger specific test pathways.
These pitfalls can produce unreliable results and excessive maintenance overheads. What’s more, due to the unpredictable test frequencies and edge cases, these problems could appear much later, leading to costly and time-consuming fixes.
Now that we have an understanding of the problems that conditional tests introduce, let’s look at some solutions for navigating conditional tests.
Solving for conditional tests
To avoid the pitfalls of conditional testing we have to write more deterministic tests. In practice, this means designing tests with setups for initialising application states while using linear logic.
Let’s revisit our dark mode test as an example. In the dark mode test, we started by fetching a random user from the database, which left us at the whim of random application states (either dark mode or light mode). This is what the query for the fetch would have looked like:
SELECT user_id FROM users
ORDER BY RANDOM()
LIMIT 1;
Instead of using conditional logic to set the application state via the UI, we can initialise predictable application states right at the start of our tests. We’ll ensure only data for users that have dark mode disabled are fetched at the start of the test (alternatively, we could set the dark mode for any user to true):
SELECT u.user_id, us.notifications_enabled
FROM users u
JOIN user_settings us ON u.user_id = us.user_id
WHERE us.notifications_enabled = true
ORDER BY RAND()
LIMIT 1;
Now that we know that the dark mode toggle will always be off, we can remove the conditional pathways in our script. This allows for a linear test flow:
It is worth noting that we could use other setup methods to initialise the application state early, such as seeding the application with data before each test. In either case, removing conditional statements in test design simplifies test logic, making tests more consistent, more reliable, and easier to maintain. However, it’s important to recognise that conditional statements aren’t inherently problematic and can be useful in some occasions.
When should you use conditional statements?
Of course, conditional statements can be useful in automation tests but their use should be carefully considered. When needed, it’s crucial to implement them in a way that minimises their impact on test predictability and maintainability:
- Use conditional logic sparingly: Reserve conditional logic for scenarios where it’s absolutely necessary. For example, handling different user roles that have access to different features in the application.
- Isolate conditional logic: When possible, isolate conditional logic to specific parts of the test, particularly the setup phase. This should make it easier to identify and manage conditional logic.
- Document thoroughly: Ensure that any conditional logic within your tests is well-documented to explain why it’s needed and how it affects the test flow. This documentation will be invaluable for maintenance and debugging.
With test automation, and especially UI automation, conditional statements should be the exception, not the rule. The key concern is not to trade off determinism in a test unnecessarily.
Final Thoughts
Conditional testing in UI automation presents a paradox: while seemingly useful and flexible, it introduces complexity, unpredictability, and maintenance challenges. By recognizing the hidden costs associated with conditional tests, automation testers can take proactive steps to minimize their use, simplifying test logic and improving reliability.
Adopting more deterministic testing practices, such as writing linear tests and initializing the application state early, can significantly reduce the reliance on conditional logic. When conditions are necessary, applying best practices to manage their complexity ensures that tests remain as predictable and maintainable as possible.
Ultimately, the goal of any testing strategy should be to create a suite of tests that are reliable, easy to understand, and straightforward to maintain. By carefully considering the role of conditional logic in your tests, you can achieve a balance between flexibility and predictability, leading to a more predictable and reliable testing process.
Liked this article?