Intercept Requests & Responses with Selenium WebDriver BiDi Protocol

Automate Traffic Interception with BiDi Network Domain

Why Network Interception Is Important

Network traffic interception is an essential skill for modern testers, enabling us to monitor, manipulate, and validate API calls directly within our browser tests. Selenium WebDriver’s BiDi (Bidirectional) Protocol provides a powerful mechanism to achieve this. This article explains how to intercept both requests and responses, with detailed code explanations.

Intercepting network traffic allows us to:

• Debug: Monitor outgoing requests and incoming responses to detect issues.
• Validate: Ensure APIs are called with correct parameters and headers.
• Mock: Simulate failures or edge cases by modifying responses.
• Measure Performance: Analyze request/response timings.

Setup and Teardown Boilerplate — Common for Both Responses and Requests

Setting Up Selenium for BiDi

Before we dive into interception, we need to configure Selenium WebDriver for BiDi. We’ll use Firefox as an example, as it natively supports the BiDi protocol.

var options = new FirefoxOptions();
options.enableBiDi();
WebDriver driver = new FirefoxDriver(options);
Network network = new Network(driver);

𝑬𝒙𝒑𝒍𝒂𝒏𝒂𝒕𝒊𝒐𝒏

1. FirefoxOptions options = new FirefoxOptions(): Creates an instance of Firefox-specific browser options.
2. options.enableBiDi(): Enables BiDi support in Firefox.
3. WebDriver driver = new FirefoxDriver(options): Initializes the WebDriver with BiDi-enabled options.
4. Network network = new Network(driver): Provides access to the network domain for intercepting traffic.

Synchronizing with CountDownLatch

To ensure the program waits for network events before exiting, use CountDownLatch.

CountDownLatch latch = new CountDownLatch(2);
network.onBeforeRequestSent(event -> {
    latch.countDown();
});
latch.await(5, TimeUnit.SECONDS);
assert (countdown);

𝑬𝒙𝒑𝒍𝒂𝒏𝒂𝒕𝒊𝒐𝒏

1. CountDownLatch latch = new CountDownLatch(2): Initializes a latch with a count of 2, which ensures that the main thread waits until two network requests are intercepted before proceeding.
2. latch.countDown(): Decrements the latch count by 1 after processing each intercepted request. This signals the main thread that a request has been handled.
3. latch.await(5, TimeUnit.SECONDS): Blocks the main thread for up to 5 seconds or until the latch count reaches 0.
4. assert (countdown): Asserts that the latch count reached 0 within the timeout, indicating successful interceptions.

Running the Test with Requests/Responses

Now, we can launch the browser and capture outgoing requests and incoming responses.

driver.get("https://www.selenium.dev/selenium/web/bidi/logEntryAdded.html");
Thread.sleep(3000);
network.close();
driver.quit();

𝑬𝒙𝒑𝒍𝒂𝒏𝒂𝒕𝒊𝒐𝒏

1. driver.get(): Navigates to the specified URL.
2. Thread.sleep(3000): Pauses the browser some time for demo only.
3. network.close(): Terminates the BiDi network module and releases associated resources.
4. driver.quit(): Closes the browser and ends the WebDriver session.

Intercepting Network Requests

Here’s how we can capture and log details about outgoing network requests.

Adding a Network Interceptor

network.addIntercept(new AddInterceptParameters(InterceptPhase.BEFORE_REQUEST_SENT));

𝑬𝒙𝒑𝒍𝒂𝒏𝒂𝒕𝒊𝒐𝒏

1. network.addIntercept(): Sets up an interceptor that triggers before a network request is sent.
2. AddInterceptParameters: Specifies the phase to intercept (BEFORE_REQUEST_SENT).

Handling Request Events

network.onBeforeRequestSent(beforeRequestSent -> {
   String requestId = beforeRequestSent.getRequest().getRequestId();
   FetchTimingInfo timings = beforeRequestSent.getRequest().getTimings();
   String url = beforeRequestSent.getRequest().getUrl();
   String method = beforeRequestSent.getRequest().getMethod();
   List cookies = beforeRequestSent.getRequest().getCookies();
   List
 headers = beforeRequestSent.getRequest().getHeaders();
   Long headersSize = beforeRequestSent.getRequest().getHeadersSize();

   System.out.printf("%nRequest method %s %n. "
     + "Sent to URL %s %n. "
     + "Timing info %s %n. "
     + "Cookies %s %n. "
     + "Headers %s %n. "
     + "Headers size %s %n.", 
     method, url, timings.getRequestTime(), cookies, headers, headersSize);

   network.continueRequest(new ContinueRequestParameters(requestId));

𝑬𝒙𝒑𝒍𝒂𝒏𝒂𝒕𝒊𝒐𝒏

1. onBeforeRequestSent: Registers a listener that executes whenever a network request is about to be sent.
2. Inside the callback:

• requestId: Captures the unique ID of the request for further manipulation.
• timings: Retrieves timing information (e.g., request start time).
• url: The destination URL of the network request.
• method: The HTTP method (e.g., GET, POST).
• cookies: List of cookies sent with the request.
• headers: List of headers sent with the request.
• headersSize: Size of the headers, useful for debugging or optimization.

4. continueRequest: Proceeds with the request after capturing its details.

Note

Ensure that chosen parameter(s) are supported by the browser under test.

Full Code and Test Run

InterceptRequest.java:

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.bidi.module.Network;
import org.openqa.selenium.bidi.network.AddInterceptParameters;
import org.openqa.selenium.bidi.network.ContinueRequestParameters;
import org.openqa.selenium.bidi.network.Cookie;
import org.openqa.selenium.bidi.network.FetchTimingInfo;
import org.openqa.selenium.bidi.network.Header;
import org.openqa.selenium.bidi.network.InterceptPhase;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxOptions;

import java.io.IOException;
import java.lang.Thread;
import java.util.List;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;

public class InterceptRequest {

 static protected WebDriver driver;
 static protected Network network;

 public static void main(String[] args) throws InterruptedException, IOException {

  var options = new FirefoxOptions();
  options.enableBiDi();
  driver = new FirefoxDriver(options);
  network = new Network(driver);

  network.addIntercept(new AddInterceptParameters(InterceptPhase.BEFORE_REQUEST_SENT));

  CountDownLatch latch = new CountDownLatch(2);

  network.onBeforeRequestSent(beforeRequestSent -> {
   String requestId = beforeRequestSent.getRequest().getRequestId();
   FetchTimingInfo timings = beforeRequestSent.getRequest().getTimings();
   String url = beforeRequestSent.getRequest().getUrl();
   String method = beforeRequestSent.getRequest().getMethod();
   List cookies = beforeRequestSent.getRequest().getCookies();
   List
 headers = beforeRequestSent.getRequest().getHeaders();
   Long headersSize = beforeRequestSent.getRequest().getHeadersSize();

   System.out.printf("%nRequest method %s %n. "
     + "Sent to URL %s %n. "
     + "Timing info %s %n. "
     + "Cookies %s %n. "
     + "Headers %s %n. "
     + "Headers size %s %n.", 
     method, url, timings.getRequestTime(), cookies, headers, headersSize);

   network.continueRequest(new ContinueRequestParameters(requestId));

   latch.countDown();

  });
  
  driver.get("https://www.selenium.dev/selenium/web/bidi/logEntryAdded.html");

  boolean countdown = latch.await(5, TimeUnit.SECONDS);

  assert (countdown);

  Thread.sleep(3000);

  network.close();
  driver.quit();
 };
}

When we run this code, we get the following output for debugging and analysis:

Intercepting and Modifying Network Responses

Next, we can capture and analyze incoming responses to ensure they contain the expected data.

Adding a Response Interceptor

network.addIntercept(new AddInterceptParameters(InterceptPhase.RESPONSE_STARTED));

𝑬𝒙𝒑𝒍𝒂𝒏𝒂𝒕𝒊𝒐𝒏

• addIntercept: Configures the network module to intercept responses when they are about to be sent to the browser (RESPONSE_STARTED phase).

Response Handling Logic

network.onResponseStarted(responseDetails -> {
   String responseId = responseDetails.getRequest().getRequestId();
   FetchTimingInfo timings = responseDetails.getRequest().getTimings();
   String url = responseDetails.getRequest().getUrl();
   
   System.out.printf("%nResponse sent for URL %s %n. "
     + "Timing info %s %n", 
     url, timings.getRequestTime());
     
   if (url.contains("selenium.dev")) {
    ContinueResponseParameters responseParams = new ContinueResponseParameters(responseId).statusCode(500);
    network.continueResponse(responseParams);
   } else {
   network.continueResponse(new ContinueResponseParameters(responseId));
   }

𝑬𝒙𝒑𝒍𝒂𝒏𝒂𝒕𝒊𝒐𝒏

1. onResponseStarted: Registers a callback when a response is received.
2. Inside the callback:

• responseId: A unique identifier for the response, needed to continue or modify it.
• url: URL of the resource being requested.
• timings: Timing information about when the request was made.

3. continueResponse: Conditional logic on how to proceed with the response.

• If the URL contains selenium.dev, the response is intercepted and a 500 status code (Internal Server Error) is sent.
• Otherwise, the response is allowed to continue unmodified.

Note

Full Code and Test Run

InterceptResponse.java:

import org.openqa.selenium.WebDriver;
import org.openqa.selenium.bidi.module.Network;
import org.openqa.selenium.bidi.network.AddInterceptParameters;
import org.openqa.selenium.bidi.network.ContinueResponseParameters;
import org.openqa.selenium.bidi.network.FetchTimingInfo;
import org.openqa.selenium.bidi.network.InterceptPhase;

import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.firefox.FirefoxOptions;

import java.io.IOException;
import java.lang.Thread;

import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;

public class InterceptResponse {

 static protected WebDriver driver;
 static protected Network network;

 public static void main(String[] args) throws InterruptedException, IOException {

  var options = new FirefoxOptions();
  options.enableBiDi();

  driver = new FirefoxDriver(options);
  network = new Network(driver);

  network.addIntercept(new AddInterceptParameters(InterceptPhase.RESPONSE_STARTED));

  CountDownLatch latch = new CountDownLatch(2);

  network.onResponseStarted(responseDetails -> {
   String responseId = responseDetails.getRequest().getRequestId();
   FetchTimingInfo timings = responseDetails.getRequest().getTimings();
   String url = responseDetails.getRequest().getUrl();
   
   System.out.printf("%nResponse sent for URL %s %n. "
     + "Timing info %s %n", 
     url, timings.getRequestTime());
     
   if (url.contains("selenium.dev")) {
    ContinueResponseParameters responseParams = new ContinueResponseParameters(responseId).statusCode(500);
    network.continueResponse(responseParams);
   } else {
   network.continueResponse(new ContinueResponseParameters(responseId));
   }
   
   latch.countDown();

  });

  driver.get("https://www.selenium.dev/selenium/web/bidi/logEntryAdded.html");

  boolean countdown = latch.await(5, TimeUnit.SECONDS);

  assert (countdown);

  Thread.sleep(3000);

  network.close();
  driver.quit();
 };
}

When we run this program, we get the following output:

Conclusion

Selenium WebDriver’s BiDi protocol opens a new frontier for automated testing, offering unparalleled insights into network traffic. By following these step-by-step examples, you can integrate request and response interception into your test suites, improving debugging, validation, and performance monitoring capabilities.

𝑪𝒐𝒎𝒎𝒐𝒏 𝑼𝒔𝒆 𝑪𝒂𝒔𝒆𝒔 𝒇𝒐𝒓 𝑩𝒊𝑫𝒊 𝑷𝒓𝒐𝒕𝒐𝒄𝒐𝒍:

1. API Testing: Validate that the front-end makes the correct API calls.
2. Performance Monitoring: Track request/response timings.
3. Error Handling: Simulate and handle network errors.
4. Security Analysis: Inspect cookies, headers, and payloads.

Go beyond UI testing — harness the power of network interception!

𝓗𝒶𝓅𝓅𝓎 𝓉𝓮𝓈𝓉𝒾𝓃𝓰 𝒶𝓃𝒹 𝒹𝓮𝒷𝓊𝓰𝓰𝒾𝓃𝓰!

I welcome any comments and contributions to the subject. Connect with me on LinkedIn, X , GitHub, or Insta. Check out my website.

If you find this post useful, please consider buying me a coffee.