E2E vs UI testing

In my previous post I mentioned that I purposely used UI to indicate what it should go on top of the test pyramid, when other times the top is called E2E. I feel there is room for an extended explanation about that, which I intend to do in this post.

UI stands for user interface. UI testing refers to testing performed over the UI.

E2E stands for end to end testing. These are tests performed from an entry point to an exit point of an application.

Now, the vast majority of applications have UI providing the entry and exit points, which then form part of E2E and that is why tests for UI and for E2E are sometimes used equivalently.

However, there are different nuances between the two.

Ideally, the tests on top of the test pyramid should check an end to end behavior that is as close as possible to the user’s behavior. This is known as “user flow”.

On the other hand, UI tests tend to focus more on the look and feel of the final product. Typically, we can find visual validations (manual or nowadays automated) and CSS validations as part of these tests. Here is when things get a little bit complicated..

Since we need the UI to be ready in order to test it, we tend to perform UI tests as part of E2E testing, at the very top of the test pyramid. However, being at the top of the test pyramid means there should be less of these tests than the other levels.

The right number of UI tests

One of the best practices of testing is to have small, independent tests so that the issues are easier to debug and we do not need multiple re-runs to get all the information (as in, test cases tend to fail when they find the first error and stop going on). At the same time, we want the least tests at the top of the test pyramid as possible. How can we achieve both things?

Visual validations and css comparisons are generally done as part of E2E, but could end up changing the test pyramid proportions if we want those tests to be nuclear. Luckily, nowadays, having visual validations tools can provide all errors separately with only one check.

There are also ways of proceeding with the tests even if a failure is found. Check your testing tool to validate if this is a possibility, so one test could be “check looks and feels of page X” and within that test we should be able to get all potential issues on the page.

This gives us the benefits of nuclear testing, that is, not having to re-run after an issue gets fixed only to find another issue on the same test and getting all the information of all failures in one test run. While at the sametime we are following the test pyramid recommendation.

Dynamic content

Although the tests we mentioned before could work out for a static website, most times we find ourselves with dynamic content. On these cases, a screenshot comparison will not work out for us to keep the tests nuclear, as you will need different screenshots.

Sometimes we need an action before the content appears. For example, if when you press a button a list appears, we might want to check that list to have the correct content. That said, the content of such a list, with all the combination of possibilities, should already be covered by integration tests, E2E tests should only check that a list is there.

What happens if the pagination is incorrect, or the list does not fit into the given space with the given resolution or some other UI part covers the results? All these are valid concerns that we cannot know during integration testing and that might require more flow and more checks, and therefore more tests.

On top of that, tests like this are very hard to maintain. Here is when we need to start getting creative with testing, finding or creating tools that could help us achieve more with less.

Front-end teams

Sometimes it might feel a the test row in the test pyramid can be skipped as we can cover it with the row above. We could, but it is more expensive, harder to maintain, longer to run… Which is the reason the test pyramid does not recommend it.

This is commonly the case for developers that prefer to avoid mocking connections and do unit tests as part of the integration tests. One problem here is that integration tests are often less detailed and also, thinking about unit testing can help the developers create better code to begin with.

Another common case is when teams are split into back-end teams and front-end teams, which don’t get exposure to the integration tests of the features they receive.

Some features might be exclusively front-end, as in, change the look and feel of a button. Does that mean that there should be no integration test for this? Most times there should.

Front-end integration tests

Let us consider the case of a button that changes appearances. The back-end has not changed so there are no back-end integration tests.

Chances are that working on the front-end you are dealing with a widget, a storybook or something on the like. These are parts of a website that combine together at some point. Testing these requires that you have your feature running in some environment and a browser is up and running too. These can totally be considered integration tests, as you need several pieces together in order to perform the verifications, even though you are verifying the UI.

At times these tests are not logged anywhere and they could even be done manually, which creates the false sense that they don’t exist.

To test this from an E2E perspective, you would need the entire application to be up and running, with all widgets already combined and connected, which would be more expensive to set up in order to perform the test.

If the button also creates new functionality, then you should also have some API calls (which requires integration tests) and UI checks might be part of the overall UI checks. There might be a new user flow, but generally it will require less checks than the integration test would require.

Case by case

As you can see, there are different details depending on the specific case and decisions need to be taken accordingly. I could even thing of weird scenarios where integration tests and E2E tests end up being equal in number.

Consider the test pyramid to be a guideline to help you get faster and better testing, not some sort of set in stone requirement. At the end of the day, you need to do whatever makes more sense and covers best the application balancing out with development speed with quality, and always with costs in mind.

That said, remember that the test pyramid exists for a reason, as if we are not careful and aligned between all the teams, we might end up repeating tests and even structuring logical tests, but that’s well..another story.