Dynamic Test Suite Generation for More Flexible Releases

After publishing my previous article on Medium, I started thinking about what else I could share to make life easier for QA engineers and other people working on large projects.

Every project is different. Teams are structured differently, release processes vary, and testing strategies evolve over time. But all of us usually share the same goal: stable and fast releases with fewer bugs.

At some point, our existing regression approach stopped scaling.

During a major refactoring phase, our regression suite had grown to more than 1500 functional tests of different complexity and execution time. At the same time, 8 different teams were working on the project, and release queues started stretching into weeks.

CI/CD is obviously much more than testing alone, but in this article I want to focus specifically on what we changed on the QE side to improve execution flexibility and reduce maintenance overhead.

This is the Part One.

Managing Test Metadata in a Single Source of Truth

Our main functional automation project is built with Java and TestNG. For test management, we use TestRail.

TestRail stores:

• references to Jira stories
• user acceptance criteria
• test execution steps
• keywords for filtering and categorization

At the same time, our Java + TestNG project used a static master suite configuration that controlled how tests were executed.

Something like:

<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Master Suite" verbose="1">
  <suite-files>
    <suite-file path="parallel-suite.xml"/>
    <suite-file path="non-parallel-suite.xml"/>
  </suite-files>
</suite>

We started asking ourselves:

What if all test execution metadata could be managed in a single source of truth?

This approach could help with:

1. More flexible child suite generation and execution
2. Centralized test management
3. Easier onboarding for new engineers
4. Better visibility into regression structure

The Core Idea

Instead of maintaining execution logic inside static XML files, we decided to store execution-related metadata directly in our Test Management System using additional keywords.

Introducing keywords: Parallel and NotParallel

This allowed us to:

1. Configure child suites dynamically
2. Control execution behavior from one place
3. Reduce maintenance of static TestNG XML files
4. Simplify regression management for engineers unfamiliar with the project

If the project already contains many tests, only the smaller set of non-parallel tests needs to be marked manually.

Execution Flow

The execution pipeline became the following:

Jenkins Pipeline
↓
Fetch execution metadata from Test Management System
↓
Generate JSON execution model
↓
Create dynamic TestNG child suites
↓
Execute regression tests

Jenkins Pipeline

The pipeline consists of two main stages.

pipeline {

  parameters {
    string(name: 'THREAD_COUNT')
    string(name: 'TESTS_VERSION')
    string(name: 'ENV_NAME')
    string(name: 'KEYWORD')
  }

  stages {

    stage('Prepare Test Suite') {
      steps {
        script {
          env.TEST_CASE_FILE_PATH = "/testCasesFile-${BUILD_NUMBER}.json"

          sh """
            gradle :integration-job:run --no-daemon \
              -PutilName="CreateTestCasesList" \
              --args="rex_intesting ${env.TEST_CASE_FILE_PATH}" \
              -PthreadCount=${params.THREAD_COUNT} \
              -Pkeyword=${params.KEYWORD}
          """
        }
      }
    }

    stage('Run functional tests') {
      steps {
        sh """
          gradle test \
            -PcasesIdFile=${env.TEST_CASE_FILE_PATH} \
            -PenvProfile=${params.ENV_NAME} \
            -PTHREAD_COUNT=${params.THREAD_COUNT} \
            -Pbranch=${params.TESTS_VERSION}
        """
      }
    }
  }
}

Stage 1 — Preparing Test Suite

This stage:

The CreateTestCasesList does the following:

• fetches tests from TestRail
• filters them by keywords
• creates execution models for parallel and sequential suites
• exports the result into a JSON file

Main Logic

private static List<SuiteTemplate> prepareSuite(List<TestRailCase> testRailCases) {

   List<SuiteTemplate> suites = new ArrayList<>();

   List<TestRailCase> testRailParallelCases = testRailCases.stream()
       .filter(p -> !p.getKeywordsAsString().contains("NotParallel"))
       .toList();

   List<Integer> parallelCases = filterByKeyword(testRailParallelCases);

   List<TestRailCase> testRailNonParallelCases = testRailCases.stream()
       .filter(p -> p.getKeywordsAsString().contains("NotParallel"))
       .toList();

   List<Integer> nonParallelCases = filterByKeyword(testRailNonParallelCases);

   suites.add(new SuiteTemplate(
       parallelCases,
       "ParallelSuite",
       Integer.parseInt(CreateTestCasesList.threadCount),
       XmlSuite.ParallelMode.TESTS
   ));

   suites.add(new SuiteTemplate(
       nonParallelCases,
       "NonParallelSuite",
       1,
       XmlSuite.ParallelMode.TESTS
   ));

   return suites;
}

Generated JSON

[
  {
    "tests": [1, 2, 3],
    "suiteName": "Parallel",
    "threadCount": 40,
    "parallelMode": "TESTS"
  },
  {
    "tests": [4, 5],
    "suiteName": "NotParallel",
    "threadCount": 1,
    "parallelMode": "NONE"
  }
]

Stage 2 — Running the regression based on the prepared metadata

It is started almost the same as the usual test flow.

tasks.named('test') {
   useTestNG() {
       suites "src/test/resources/testng/regression.xml"
   }
}

CustomListener includes logic related to the whole suite.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="rex_intesting_threaded" verbose="1" parallel="tests" thread-count="10">

    <listeners>
        <listener class-name="com.project.tests.listeners.CustomListener"/>
        <listener class-name="com.project.tests.listeners.CreateXmlSuiteForRegression"/>
    </listeners>
</suite>

Dynamic Suite Generation

In our case, CreateXmlSuiteForRegression implemented IExecutionListener and dynamically generated child suites.

The process looked like this:

1. Read JSON file
2. Create XmlSuite
3. Find tests by @TestCaseId
4. Build dynamic child suites
5. Execute them via TestNG

public class CreateXmlSuiteForRegression implements IExecutionListener {

 private static final Logger log = LoggerFactory.getLogger(CreateXmlSuiteForRegression.class);

 @SneakyThrows
 @Override
 public void onExecutionStart() {
   String casesIdFile = System.getProperty("casesIdFile");
   List<SuiteTemplate> suiteTemplates = null;

   List<XmlSuite> suites = new ArrayList<>();

   if (casesIdFile != null && !casesIdFile.isBlank()) {
     try {
       suiteTemplates = getSuiteTemplatesFromFile(casesIdFile);
       log.info("Cases size from file : {}", suiteTemplates.size());
     } catch (IOException e) {
       log.error("Can not get testRail tests list from the File:", e);
     }
   }

   suiteTemplates.forEach(suiteTemplate -> {
     XmlSuite childSuite = createChildTestSuite(suiteTemplate);
     suites.add(childSuite);
   });

   TestNG testNG = new TestNG();
   testNG.setXmlSuites(suites);
   testNG.run();
 }

 List<SuiteTemplate> getSuiteTemplatesFromFile(String filePath) throws IOException {
   ObjectMapper objectMapper = new ObjectMapper();
   return objectMapper.readValue(new File(filePath), new TypeReference<>() {
   });
 }

 private static XmlSuite createChildTestSuite(SuiteTemplate suiteTemplate) {
   XmlSuite childSuite = new XmlSuite();

   childSuite.setName(suiteTemplate.getSuiteName());
   childSuite.setParallel(suiteTemplate.getParallelMode());
   childSuite.setThreadCount(suiteTemplate.getThreadCount());

   List<String> rawTests = suiteTemplate.getTests();

   List<XmlTest> xmlTests = findTestsByCaseId(rawTests);
   childSuite.setTests(xmlTests);

   return childSuite;
 }

 /**
  * This is an example of how tests can be searched and filtered by @TestCaseId.
  * I believe each project has its own unique implementation
  */
 public static List<XmlTest> findTestsByCaseId(List<String> testCaseIds) {

   List<XmlTest> result = new ArrayList<>();

   Reflections reflections =
       new Reflections("com.example.project.functional");

   Set<Class<?>> testClasses =
       reflections.getTypesAnnotatedWith(Test.class);

   for (Class<?> testClass : testClasses) {

     List<XmlInclude> includedMethods =
         Arrays.stream(testClass.getMethods())
             .filter(method ->
                 method.isAnnotationPresent(TestCaseId.class)
             )
             .filter(method ->
                 testCaseIds.contains(
                     method.getAnnotation(TestCaseId.class).value()
                 )
             )
             .map(Method::getName)
             .map(XmlInclude::new)
             .toList();

     if (!includedMethods.isEmpty()) {
       XmlClass xmlClass =
           new XmlClass(testClass.getName());
       xmlClass.setIncludedMethods(includedMethods);

       XmlTest xmlTest = new XmlTest();

       xmlTest.setName(testClass.getSimpleName());
       xmlTest.setXmlClasses(List.of(xmlClass));

       result.add(xmlTest);
     }
   }

   return result;
 }

 @Data
 public static class SuiteTemplate {
   private List<String> tests;
   private String suiteName;
   private int threadCount;
   private XmlSuite.ParallelMode parallelMode;

   @JsonCreator
   public SuiteTemplate(
       @JsonProperty("tests") List<String> tests,
       @JsonProperty("suiteName") String suiteName,
       @JsonProperty("threadCount") int threadCount,
       @JsonProperty("parallelMode") XmlSuite.ParallelMode parallelMode) {
     this.tests = tests;
     this.suiteName = suiteName;
     this.threadCount = threadCount;
     this.parallelMode = parallelMode;
   }
 }
}

Final Result

This approach gave us significantly more flexible regression management from a single centralized source.

On our project, TestRail effectively became not only a test management system, but also a lightweight execution orchestration layer.

This was especially useful for:

• engineers unfamiliar with the project
• newcomers joining the team
• release managers
• cross-functional teams

It became much easier to understand:

• what regression contains
• how tests are grouped
• which tests run in parallel
• which tests require sequential execution