Testing Basic: Fragile Tests

Fragile tests are a scourge on technology. And testing technology. Sometimes they pass, sometimes they fail, and to make life interesting, sometime they just stop, and on the odd case, come up with a very unusual result.

“Fragile tests” is what we call tests that we can’t trust. And we very much would like to trust our tests.

It all comes down to time and attention. When a test passes, and we trust it, that means that we can ignore that specific functionality we just tested, and instead we can now focus on that new failure over there.

Instead, fragile tests are not to be trusted. They spread doubt. When are they right? When they pass, or when they fail?

That doubt now translates, again, into time and attention. We go and debug, and try to reproduce the faulty behavior. And when that doesn’t work, we just say “some voodoo must have happened” and move on.

But that doesn’t help, of course, because if we can’t trust that test (now that we have proof!), can we trust the other tests that always (seem to) pass?

It is possible, that the tests fail because of bugs in them. The more complex the system we’re testing, the more complex the setup for both the environment and the tests. And complexity breeds bugs.

If that’s the case, we need to either get rid of these tests, replace them or fix them. However, we’re focusing on the tests, but are they the major cause?

Fragile Tests: Mystery Solved

While we blame the tests, we turn our focus away from the real culprit. Most of the time, it is not the tests that break. It’s the code they test.

If your tests are complex, I guarantee the code is even more so. After all, we fix a lot more bugs in production code than in tests.

Code that hasn’t been tested enough by the developers, would be buggy. And not just those easy-to-find bugs, the more illusive ones. Those that throw the blame on the so-called “fragile tests”.

So what do we do about this? First we need to measure. We need to know if we’re blaming the right piece of code. Because, if those tests break because of the code, it’s not that the tests are fragile, the code is not robust enough.

And to get “robust” code, we don’t need to just “unit test” it. As a team, we need to make sure that the devs know the expected quality of their output. How they are expected to test it, and how the testers are planning to test it.

The more the developers know, in advance, how they and the testers, are going to test, we minimize the risk of bugs. And by that, the risk of having fragile tests.

That means collaboration in test planning and design. Because quality is a team effort.

Next time you see a test you don’t trust, ask not how you can fix it. Ask how code of that quality came through the pipes, and plug that leak.

Want to make your tests, and code, more stable? Check out the “test automation” workshop for developers.