How to Manipulate INI File Variables

A step-by-step tutorial using Python and Pytest

Situation

As a quality software engineer on a recent project, I tested transformed data via Python, Pyspark, and Pytest hosted in AWS S3 buckets. The S3 bucket names and path variables were in an INI configuration file within the repository.

The project was architected first to develop the work in the dev environment and then later progressed into the QA and prod environments, which is common. However, the variable names were dependent on the environment.

For example, the S3 bucket for the dev environment was dev.financial.records , and then for the QA environment, it was qa.financial.records , and so on for prod. To execute tests in different environments, these constraints required me to manually alter the INI variables based on the environment in which I intended to execute the tests.

I started pondering the possibility of controlling the variable values from the terminal rather than manual changes that can be error-prone. The hard-coded configuration of variable values was not malleable or ideal, so I considered the question:

How can I manipulate the INI file variables from the command line with each test session based on the environment of my choice?

Behaviour

The solution was two-fold.

Spoiler: the solution also wasn’t a huge lift!

First, I leveraged the pytest.addoption function to pass the environment -E flag along with the Pytest command in the command line. The conftest.py file at the root of the repository now held a function that looked like this:

import pytest

def pytest_addoption(parser):
    """
    Configuring a custom var E (env variable): dev, qa, prod, "", testing
    Throws an error if you pass in a wrong value to 'E' arguments
    """
    print("conftest arguments")
    avail_envs = ["dev", "qa", "prod", "", "testing"]
    parser.addoption(
        "-E",
        dest="environment",
        action="store",
        metavar="NAME",
        default="",
        help="Test environment name.",
    )

As you can see, the -E flag and the Pytest command were now available to execute tests. The command could vary based on the environment as follows: pytest -E=dev or pytest -E=qa or other based on the list in the avail_envs variable from the code snippet above.

But this addition alone did not solve my problem of changing the actual variables in the INI file based on the -E flag passed in. The next step will take a snapshot, as the article photo suggests, of the INI file as a template to be reused for any desired environment and shift the values of the variables as indicated.

Next, I added functions to the conftest.py file inside the tests directory of the repository to initially grab the value from the -E flag by a fixture function and then use that value to dynamically change the variable values in the INI file for each test session. Let’s first look at the fixture function in the conftest.py file inside the tests directory.

"""Conftest file to define pytest fixtures for test data"""
import os

import pytest

@pytest.fixture(scope="session")
def environment(request):
    """
    Get the value of the "environment" command-line option from the pytest 
    configuration.

    Args:
        request: Pytest request object.

    Returns:
        str: The value of the "environment" option.

    """
    return request.config.getoption("environment")

The fixture function in the code above grabs the value indicated by the -E flag. It’s important to note that the scope is shown as ‘session’ because I only want the variable to be altered for the duration of each test session. The idea is that I can execute in the dev environment and then follow up the next execution in the QA environment just based on the -E flag choice in the terminal.

The next few functions in this same file handle the INI file like a template and manipulate the INI variable values based on the -E flag value. This update requires changing the existing variable values in the INI file to have curly braces in the places where the hard-coded environment was indicated initially.

If you remember from the beginning, the value for the dev environment would be something like this: dev.financial.records. The value now needs to look like this: {environment}.financial.records and the {environment} part will dynamically change due to the following functions:

import configparser
import shutil

def backup_ini(config_file):
    """
    Create a backup of the original INI file.

    Args:
        config_file (str): The path to the configuration file.

    """
    backup_file = config_file + '.backup'
    shutil.copyfile(config_file, backup_file)

def restore_ini(config_file):
    """
    Restore the INI file from the backup.

    Args:
        config_file (str): The path to the configuration file.

    """
    backup_file = config_file + '.backup'
    shutil.move(backup_file, config_file)

def pytest_sessionfinish(session, exitstatus):
    """
    Pytest hook to restore the INI file to its original state after 
    the tests finish.

    Args:
        session: Pytest session object.
        exitstatus: Exit status of the pytest session.

    """
    config_file = 'data/data.ini'
    restore_ini(config_file)

def update_ini(config_file, environment):
    """
    Update the values in a configuration file based on the specified 
    environment.

    Args:
        config_file (str): The path to the configuration file.
        environment (str): The environment to substitute in the 
        configuration values.

    """
    config = configparser.ConfigParser()
    config.optionxform = str
    config.read(config_file)

    for section in config.sections():
        for key, value in config.items(section):
            updated_value = value.replace('{environment}', environment)
            config[section][key] = updated_value

    with open(config_file, 'w') as configfile:
        config.write(configfile)

def pytest_configure(config):
    """
    Pytest configuration hook to update the configuration file based 
    on the specified environment.

    Args:
        config: Pytest config object.

    """
    environment = config.getoption("environment")
    config_file = 'data/data.ini'
    backup_ini(config_file)
    update_ini(config_file, environment)

The first three functions above are how the INI file is now handled like a template — altered during test execution and returned to its original, template state upon test completion. The bottom two functions are how the {environment} values in the INI file are changed during the test session based on the -E flag.

While the tests are running, you will be able to see a .backup INI file temporarily. The .backup file is the template used during the test session, and the INI file holds the altered values. Once the test session is completed, the INI file is restored to the template state and the .backup file is removed.

If you are more of a hands-on learner and would like to get a jump start on trying this concept out for yourself, feel free to clone my POC repository and follow the README directions to get a better view locally:

GitHub - taylorwagner/pytest-variable-manipulation

Please note: the order of the functions in the conftest.py file from the tests directory was changed for the article explanation. Please review the file in the POC repository for clarification on the exact order of these functions.

Impact

This small lift to the testing framework revealed many benefits and advantages. Here are a few:

• No longer needed to remember to manually change the INI variable values based on the desired environment
• Ability to alter several variable values at once
• Lower chance of user mistakes — typos
• Could execute tests in different environments back-to-back…faster!
• Did not require additional outside libraries or dependencies
• POC that INI variable values can be altered via test command —could this concept be utilized for other use cases?

The impact of the changes provided a higher level of quality for the overall product and more flexibility for the automated tests — ensuring that the transformed data landing in the S3 buckets was what was intended and expected. The results were dealt out faster, allowing the team to address concerns more quickly. This improvement also no longer limited the project scope to one environment at a time — the project could move along as different environments could be tested at the switch of an -E flag.

Resources

• Configuring Python Projects with INI, TOML, YAML, and ENV files
• Command Line Flags for Pytest
• Pytest Fixtures
• POC GitHub Repository

How to Manipulate INI File Variables was originally published in Code Like A Girl on Medium, where people are continuing the conversation by highlighting and responding to this story.