GitLab Feature Flags in UI Test Automation_

Implementing feature flags in UI testing frameworks can be challenging, but in terms of time, the payoff is significant! In our recent project, we faced the exciting challenge of conditionally executing UI tests based on GitLab feature flags. Using Maven, TestNG, Java, Selenium, and GitLab, we designed an elegant solution that dynamically filters test execution depending on the active feature flag detected from our branch names.

How we did it!

Imagine you’re developing multiple features simultaneously. Each feature requires specific tests, but running all tests every time is inefficient. What if tests could automatically run based only on the features actively being worked on? And what if we could filter tests dynamically based on these set of features?

In this project we needed a way to conditionally run only certain tests when a specific feature flag is active. Using TestNG’s interceptor mechanism, along with a custom annotation and utility class for parsing the branch name, we were able to filter tests dynamically at runtime.

Implementation

Create a Custom Annotation

We started by defining a custom annotation (@FeatureFlag) that can mark tests that require a specific feature flag. We made sure the annotation is retained at runtime so that reflection can pick it up.

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface FeatureFlag {
    String[] value();
}

Develop FeatureFlagUtils Utility

Next, we built FeatureFlagUtils, a utility class that inspects the Git branch name to determine active feature flags:

public class FeatureFlagUtils {
    public static Set getActiveFeatureFlags() {
        Set activeFlags = new HashSet<>();
        String branchName = System.getProperty("branchName", "").trim().toLowerCase();
        if (branchName.contains("featureflag1")) activeFlags.add("featureflag1");
        if (branchName.contains("featureflag2")) activeFlags.add("featureflag2");
        if (branchName.contains("featureflag3")) activeFlags.add("featureflag3");
        // Fallback regex detection
        if (activeFlags.isEmpty()) {
            Matcher matcher = Pattern.compile("featureflag\\d+").matcher(branchName);
            while (matcher.find()) activeFlags.add(matcher.group());
        }
        return activeFlags;
    }
}

This utility class fetches the branch name from a system property and then determines which feature flag should be active. For example, if the branch name includes “featureflag1”, the utility returns that flag.

Implement FeatureFlagInterceptor

The heart of our solution is FeatureFlagInterceptor. It filters tests dynamically based on the active flag:

public class FeatureFlagInterceptor implements IMethodInterceptor {
    @Override
    public List intercept(List methods, ITestContext context) {
        String activeFeatureFlag = FeatureFlagUtils.getActiveFeatureFlag();
        if (activeFeatureFlag.isEmpty()) return methods; // Run all tests if no flags
        return methods.stream().filter(m -> {
            FeatureFlag flag = m.getMethod().getConstructorOrMethod().getMethod().getAnnotation(FeatureFlag.class);
            return flag != null && Arrays.asList(flag.value()).contains(activeFeatureFlag);
        }).collect(Collectors.toList());
    }
}

The core of the functionality is handled in a TestNG interceptor. This class implements IMethodInterceptor so that it can inspect every test method during suite execution. When an active feature flag is detected, the interceptor filters out any tests that are not annotated with @FeatureFlag containing that flag.

Configure TestNG Interceptor

We updated our testng.xml to register the interceptor:

To ensure that the interceptor is applied to every test within the suite, we registered it in the testng.xml configuration file. This file is placed in the project root (or referenced in your Maven configuration) so that every test class in the specified package is processed by the interceptor.

Update the Base Test Class

To handle specific login flows for flagged tests, we adjusted our base test initialization:

if (method.isAnnotationPresent(FeatureFlag.class)) {
    loginHelper.signInExistingFeatureFlagUser();
} else {
    loginHelper.signInDefaultUser();
}

When a test method is marked with @FeatureFlag, the base class changes the login routine to use the appropriate user (for instance, by calling signInExistingFeatureFlagUser()). This ensures that tests that pass the interceptor are initialized correctly using the account that was enabled to see and access the feature.

Annotate Tests

Test methods declare their feature flag requirements:

@FeatureFlag({"featureflag1"})
@Test
public void checkNewFooter() {
    agreement.verifyNewFooter();
}

Finally, we updated our UITestSuite to mark only those tests that should run when a specific feature flag is active. For example, a test like checkNewFooter() is annotated with @FeatureFlag({“featureflag1”}) so that when the branch name indicates “featureflag1” (via -DbranchName parameter), only that test will run.

Updating .gitlab-ci.yml to Pass the Branch Name

script:
     - mvn clean test -DbranchName="${ORIGIN_BRANCH_NAME}"

When running your tests on the GitLab CI pipeline, you want to pass the branch name to Maven so that the feature flag logic in FeatureFlagUtils works correctly. This is done by updating your .gitlab-ci.yml file to include the branch name as a system property in the Maven command.

The Maven command is executed with the -DbranchName option, which takes the value of the GitLab environment variable ORIGIN_BRANCH_NAME (the name of the branch the pipeline ran on). As a result, FeatureFlagUtils reads this system property, and if the branch name includes “featureflag1”, “featureflag2”, or “featureflag3”, it returns the corresponding flag. The FeatureFlagInterceptor then uses this value to filter tests accordingly.

Run and Verify

Executing the tests:

$ mvn clean test -DbranchName="${ORIGIN_BRANCH_NAME}"

Active Feature Flag: featureflag1
Filtering tests based on feature flag: featureflag1
Skipping test: ConsumercheckViewAgreement because it does not match active flag: featureflag1
Skipping test: Homepage because it does not match active flag: featureflag1
...
Including test: checkNewFooter with matching feature flag: featureflag1
...

Wrapping

Instead of repeating or summarizing all of the above, let me leave you with this quote:

“Embrace the challenge, for within every struggle to learn lies the power to transform who you are into who you aspire to become.”

Best