The Automated Tests of OS Deployment Using Jenkins Pipelines: Architecture, Flow, and Execution Model — Part 4

In earlier parts of this series, we explored the foundational elements of our automation platform: Jenkins as the CI/CD backbone, Docker as the execution engine, and Git/GitHub as the version‑controlled structure for managing pipeline definitions. Right now, we bring all those components together and walk through the complete execution model used to automate Operating System Deployment testing across our store infrastructure.

What distinguishes this solution is its scale, flexibility, and ability to test production‑grade MCM deployment flows end‑to‑end without manual intervention. The automation framework provisions Proxmox virtual machines, PXE boot, interacts with Windows PE and Task Sequence wizard, and executes a comprehensive health‑check suite on fully deployed systems. All tests run in parallel across multiple Server at Store (S@S) environments.

Architectural Overview

At the top of the stack is the Jenkins Controller, which acts as the orchestration hub for the entire pipeline. It coordinates every stage of execution, manages secrets and credentials, and loads all configuration from GitHub‑hosted Jenkinsfiles.

To actually run the workloads, Jenkins relies on Docker‑based Jenkins agents, which are provisioned dynamically through the Docker Engine API. Each agent is launched in a clean, isolated container. These images are pre‑configured with the tooling required for our environment (including sshpass, PowerShell components etc.).

Supporting this execution layer is the Jenkins Shared Library, located in our GitHub repository. The shared library provides a standardized, reusable foundation for pipelines by centralizing utility logic. By encapsulating these repetitive tasks, the shared library keeps Jenkinsfiles concise, makes the system easier to maintain, and ensures consistent behaviour across all automation workflows.

Finally, the execution reaches the Proxmox Store Servers (S@S), which form the backbone of our on‑site virtualization environment. Each store operates a Proxmox hypervisor contains Windows and Linux virtual machines. These VMs serve as fully isolated test targets for MCM Task Sequence execution. The pipeline can simulate physical keyboard input — allowing automated progression through early Windows PE stages and Task Sequence selections that normally require manual intervention.

Repository structure explained (why and how we organize the project?)

Before diving into the execution model of the Jenkins pipeline, it is essential to understand why our project is structured the way it is and how each component contributes to the automation lifecycle.

A predictable, modular, and clearly‑separated repository layout is critical in large IT Operations environments. This is why we keep all Pipeline logic, shared library functions and supporting PowerShell/Bash scripts inside a unified GitHub repository. Every change that affects OS deployment tests — from VM initialization to Task Sequence verification — is committed, reviewed, and merged using the same standards as any modern CI/CD project.

This structure reflects a clean separation of:

• Source code (Groovy pipeline definitions)
• Shared library files (vars/)
• External scripts (resources/)
• Documentation

All files are stored in GitHub, allowing change tracking, code reviews, and auditable updates.

How we handle the credentials inside Jenkinsfile?

A recurring challenge in any automation pipeline that interacts with remote infrastructure is how to handle authentication securely. Hard-coding usernames or passwords into pipeline scripts is a well-known anti-pattern — it exposes secrets in version control, build logs, and console output. In our framework, this problem is solved through a dedicated Jenkins Shared Library function that centralizes all credential injection into a single, reusable component.

Every stage that touches a remote host wraps its logic inside Credentials { c -> ... }. Credentials are never hard-coded; Jenkins injects them at runtime and they remain masked in logs. The closure pattern means each call site simply declares what it needs without repeating withCredentials blocks.

// vars/Credentials.groovy — Shared library function for credential injection
def call(Closure body) {
    withCredentials([
        usernamePassword(credentialsId: 'T3_CICD',        usernameVariable: 'T3_USER',       passwordVariable: 'T3_PASS'),
        usernamePassword(credentialsId: 'CICD',            usernameVariable: 'SSH_USER',      passwordVariable: 'SSH_PASS'),
        usernamePassword(credentialsId: 'CICD_BOOT_IMAGE', usernameVariable: 'SSH_BOOT_USER', passwordVariable: 'SSH_BOOT_PASS'),
        usernamePassword(credentialsId: 'T3_CICD',        usernameVariable: 'SSH_USER2',     passwordVariable: 'SSH_PASS2')
    ]) {
        body([
            t3User: T3_USER, t3Pass: T3_PASS,
            sshUser: SSH_USER, sshPass: SSH_PASS,
            bootUser: SSH_BOOT_USER, bootPass: SSH_BOOT_PASS,
            sshUser2: SSH_USER2, sshPass2: SSH_PASS2
        ])
    }
}

How we remotely execute all of the commands through the SSH by Shared Library functions

Every remote operation in the pipeline — whether it targets a Proxmox hypervisor, a WinPE boot environment, or a fully deployed Windows POS system — flows through two shared library functions defined in sshOps.groovy. These functions provide a unified interface for SSH command execution and SCP file transfers, while enforcing a critical security pattern that prevents credential leakage.

The withEnv + single-quoted shell block pattern is a deliberate security measure — it prevents Groovy's GString interpolation from leaking credentials into the build log. Every remote operation in the entire pipeline (Proxmox commands, WinPE scripts, post-deployment checks) flows through these two functions, giving a single point of control for timeouts, logging labels, and credential handling.

// vars/sshOps.groovy — SSH and SCP helpers using sshpass
// Single-quoted shell blocks prevent Groovy from interpolating secrets
def scp(String host, String localPath, String remotePath,
        String user = null, String pass = null, String port = '22', String label = null) {
    withEnv([
        "SSH_HOST=${host}", "SSH_LOCAL=${localPath}", "SSH_REMOTE=${remotePath}",
        "SSH_PORT=${port}", "SSH_USER_EXPLICIT=${user ?: ''}", "SSH_PASS_EXPLICIT=${pass ?: ''}"
    ]) {
        sh(label: label ?: "SCP ${host}:${port}", script: '''
            set +x
            sshpass -p "${SSH_PASS_EXPLICIT:-$SSH_PASS}" \
              scp -P "$SSH_PORT" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
              "$SSH_LOCAL" "${SSH_USER_EXPLICIT:-$SSH_USER}@$SSH_HOST:$SSH_REMOTE"
        ''')
    }
}
def ssh(String host, String command, String user = null, String pass = null,
        String port = '22', String label = null) {
    withEnv([
        "SSH_HOST=${host}", "SSH_CMD=${command}", "SSH_PORT=${port}",
        "SSH_USER_EXPLICIT=${user ?: ''}", "SSH_PASS_EXPLICIT=${pass ?: ''}"
    ]) {
        return sh(returnStdout: true, label: label ?: "SSH ${host}:${port}", script: '''
            set +x
            sshpass -p "${SSH_PASS_EXPLICIT:-$SSH_PASS}" \
              ssh -p "$SSH_PORT" -T -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
              "${SSH_USER_EXPLICIT:-$SSH_USER}@$SSH_HOST" "$SSH_CMD"
        ''').trim()
    }
}

Pipeline Execution Flow

The automated OSD testing pipeline is built as a carefully orchestrated sequence of operations designed to simulate, validate, and verify a complete MCM Task Sequence deployment as it would occur on real store hardware. The pipeline flows through three major phases: provisioning the virtual machines, monitoring the Task Sequence execution in real time, and validating the resulting Windows installation.

Phase 1 — VM Provisioning and PXE Boot Automation

Phase 1 prepares the environment in which the OS deployment will run. Without a reproducible and controlled baseline, any test of the MCM Task Sequence could produce inconsistent or misleading results. To prevent this, the pipeline rebuilds every VM from scratch on each execution and guides it through the earliest boot‑time interactions.

First view before we “build” our automated test

Step 1: Host Normalization

The pipeline begins by interpreting the user’s input. Testers typically provide simple, store‑friendly identifiers such as: D010, 5150, which are store numbers across all the organization.

However, Proxmox expects the full, structured hostname format used across our distributed S@S environment. This conversion happens in the Jenkins shared library function which resolving the hostname based on provided store name convention which is placed in /resources catalogue.

Why this matters:

• Different countries use different naming patterns (e.g., D010, 5150).
• Jenkins must convert ambiguous input into deterministic identifiers.
• Avoids typos or inconsistent naming between testers.
• Enables a single pipeline to work globally across all stores.

// Shared library helper to normalize hostnames
import java.util.regex.Pattern
def call(String rawHostsInput) {
  def cleanedHosts = rawHostsInput?.trim()?.toUpperCase()?.replaceAll(/[\,\s]+$/, "")?.split(/[\,\s]+/)?.findAll { it }
  if (!cleanedHosts) {
    return []
}
// Ordered rules to transform host tokens into fully qualified hostnames
Map<Pattern, Closure<String>> rules = [
  (~ /^P-.*-SAS01$/): { h -> h },
  (~ /^D\d{3}$/) : { h -> "P-DK${h}-SAS01" },
  ….. // any other instances
  (~ /^M\d{3}$/) : { h -> "P-MA${h}-SAS01" },
  (~ /^T0\d{2}$/) : { h -> "P-TR${h}-SAS01" }
]
return cleanedHosts.collect { host ->
  rules.findResult { pattern, transformer ->
    pattern.matcher(host).matches() ? transformer(host) : null
  }
}.findAll { it != null }
}

Step 2: VM Reset and Re‑Creation

To guarantee that every Operating System deployment test begins from a clean and reproducible baseline, the pipeline completely reinitialize the virtual machine on the Proxmox S@Ses. This ensures that no outdated or incompleted installations can influence the results of the new test.

This reset process is performed via initialization VM script on S@Ses, which:

• Forcefully stops the VM, destroys all VM artifacts
• Builds a new VM definition with controlled CPU, RAM, network, and storage parameters
• Re‑creates EFI and OS disks
• Tags the VM for identification
• Starts the VM and trigger PXE boot.

#!/bin/bash
PATH_ENV="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"; export PATH="$PATH_ENV"

sudo qm stop "${VM_ID}"; sleep 10; sudo qm unlock "${VM_ID}"; sudo qm destroy "${VM_ID}" --purge
sudo mkdir -p "${PATH_VM}"; sudo chmod 740 "${PATH_VM}"
sudo qemu-img create -f qcow2 "${PATH_VM}/vm-${VM_ID}-disk-"{0..0}".qcow2" 128K
sudo qemu-img create -f qcow2 "${PATH_VM}/vm-${VM_ID}-disk-"{1..1}".qcow2" 150G

sudo qm create "${VM_ID}" \
  --agent 1 --cores 2 --efidisk0 local:${VM_ID}/vm-${VM_ID}-disk-0.qcow2 \
  --virtio0 local:${VM_ID}/vm-${VM_ID}-disk-1.qcow2,aio=native \
  --bios ovmf --cpu x86-64-v3 --kvm 1 --memory 4096 \
  --name LOAD-TEST-${VM_ID} --net0 virtio,bridge=vmbr0 --numa 0 --onboot 0 \
  --ostype win10 --scsihw virtio-scsi-single --sockets 2 --vga virtio

sudo pvesh set /nodes/"$(hostname | xargs)"/qemu/${VM_ID}/config --tags load-test
sudo qm rescan; sudo qm start "${VM_ID}"

for i in {1..60}; do sudo qm sendkey "${VM_ID}" kp_enter; sleep 1; done; sleep 60

Step 3: VM IP Discovery

Once the VM boots into WinPE environment, it receives a new IP address from the store’s DHCP server. But because this environment is dynamic, Jenkins cannot assume what the IP will be.

Another function coming from resources written in bash solves this by:

• Scanning the store’s subnet with nmap
• Extracting the MAC address from Proxmox config
• Mapping MAC → IP
• Returning a single, clean IP address

Why this matters:

• Allows parallel testing across multiple stores
• Ensures the pipeline always connects to the correct VM

# Find Proxmox host IP range prefix
SUBNET_PREFIX=$(hostname -i | cut -d '.' -f 1,2,3)
# Extract the MAC address of the VM network interface
VM_MAC=$(sudo qm config "$VM_ID" 2>/dev/null \
         | grep 'net0:' \
         | cut -d '=' -f 2 \
         | cut -d ',' -f 1)
# Scan local subnet and match VM MAC to IP
sudo nmap -sP ${SUBNET_PREFIX}.10-100 2>/dev/null \
    | grep -i -B 2 "$VM_MAC" \
    | head -n 1 \
    | cut -d '(' -f 2 \
    | cut -d ')' -f 1 \
    | tail -n 1 \
    | sed 's/[A-Za-z]//g' \
    | xargs

Step 4: WinPE and Task Sequence State Detection

Once the pipeline successfully discovers the VM’s IP address, the next challenge is determining what stage of the MCM Task Sequence the device is currently in. At this moment, the VM is running WinPE hosts Microsoft’s Task Sequence engine (TSCore.exe, TSLauncher.exe, and associated SMSTS processes).

Unlike a fully booted Windows OS, WinPE:

• does not have WinRM
• does not support remote PowerShell sessions
• does not expose standard remote APIs
• can only be accessed via PowerShell local execution over OpenSSH through raw SMS TS log monitoring

Why SMS TS Logs Are the Key to Reliable Detection

The SMS TS log (smsts.log) is written from the moment WinPE starts to load the Task Sequence engine. It records every state, every script, every wizard screen, and every failure.

This enables your pipeline to:

• Detect when WinPE is fully initialized
• Detect when the Task Sequence Wizard is visible
• Detect when UI prompts (like computer name) appear
• Detect failures early
• Progress the automation by reacting to specific log entries
• Avoid fragile timing-based automation entirely

To automate the early stages of OSD, the pipeline relies on a small but effective set of PowerShell scripts. Each one serves a clear purpose and enables Jenkins to understand exactly what is happening inside WinPE — without requiring GUI automation or manual intervention.

Step 5: Detect When WinPE Is Ready

This script checks for the presence of the SMSTS log directory:

X:\Windows\Temp\SMSTSLog

If the directory exists, it tells Jenkins that:

• WinPE has fully initialized
• The Task Sequence environment is active
• Task Sequence logging has started
• IP address will be stable during the whole OSD process

// part when the Jenkins is waiting for WinPE starts, getting the proper IP address and establish SSH connection
mounts[host] = "1" // self-check mechanism which retrying PXE boot if deployment fails
while (mounts[host] != "0") {
   // uploading scripts to WinPE environment
   sshOps.scp(host, 'initLoadVm.sh', '/tmp/initLoadVm.sh', env.SSH_USER, env.SSH_PASS, '22', "Upload initLoadVm script to ${host}")
   sshOps.scp(host, 'probe_vm_ip.sh', '/tmp/probe_vm_ip.sh', env.SSH_USER, env.SSH_PASS, '22', "Upload Probe VM IP script to ${host}")
   sshOps.ssh(host, "PATH_VM=${PATH_VM} VM_ID=${params.VM_ID} bash /tmp/initLoadVm.sh", env.SSH_USER, env.SSH_PASS, '22', "Run initLoadVm script on ${host}")
   // probing IP address of WinPE environment
   def VM_IP = ""
   for (int probeAttempts = 1; probeAttempts <= 8 && !VM_IP; probeAttempts++) {
    VM_IP = sshOps.ssh(host, "bash /tmp/probe_vm_ip.sh ${params.VM_ID}", env.SSH_USER, env.SSH_PASS, '22', "Run probe VM IP script, attempts: ${probeAttempts}")
    if (!VM_IP) sleep 15
   }
   if (!VM_IP) { mounts[host] = "1"; echo "VM IP not found after 8 attempts, retrying PXE boot..."; continue }
   vmIPs[host] = VM_IP
   echo "The value of VM_IP is: ${vmIPs[host]}."
   def vmReachable = false
   // making sure that IP address is correct in fully initialized WinPE state
   try {
    sshOps.ssh(vmIPs[host], "echo ok", env.SSH_BOOT_USER, env.SSH_BOOT_PASS, '22', "SSH check on ${host}")
    vmReachable = true
   } catch (Exception ignored) {
    for (int reprobe = 1; reprobe <= 5 && !vmReachable; reprobe++) {
     def reprobedIP = sshOps.ssh(host, "bash /tmp/probe_vm_ip.sh ${params.VM_ID}", env.SSH_USER, env.SSH_PASS, '22', "Re-probe VM IP on ${host}, attempt ${reprobe}")?.trim()
     if (reprobedIP) {
      vmIPs[host] = reprobedIP
      echo "Re-probed VM IP is: ${vmIPs[host]}."
      try {
       sshOps.ssh(vmIPs[host], "echo ok", env.SSH_BOOT_USER, env.SSH_BOOT_PASS, '22', "SSH re-check on ${host}, attempt ${reprobe}")
       vmReachable = true
      } catch (Exception ignoredAgain) {}
     }
     if (!vmReachable) sleep 10
    }
   }
   if (!vmReachable) {
    mounts[host] = "1"
    echo "VM IP ${vmIPs[host]} is still not reachable after 5 re-probes, retrying PXE boot..."
    continue
   }
   // uploading scripts to temp location on WinPE
   sshOps.scp(vmIPs[host], 'check_mount.ps1', "${winTempDir}/check_mount.ps1", env.SSH_BOOT_USER, env.SSH_BOOT_PASS, '22', "Upload mount check script to ${host}")
   sshOps.scp(vmIPs[host], 'get_smsts_tail.ps1', "${winTempDir}/get_smsts_tail.ps1", env.SSH_BOOT_USER, env.SSH_BOOT_PASS, '22', "Upload SMSTS tail script to ${host}")
   sshOps.scp(host, 'send_ts_keys.sh', '/tmp/send_ts_keys.sh', env.SSH_USER, env.SSH_PASS, '22', "Upload TS keys script to ${host}")
   sshOps.scp(host, 'send_name_prompt_keys.sh', '/tmp/send_name_prompt_keys.sh', env.SSH_USER, env.SSH_PASS, '22', "Upload name prompt keys script to ${host}")
   // check the SMSTS logs to be mounted on S@S instance
   for (int i = 0; i < 30 && mounts[host] != "0"; i++) {
    mounts[host] = sshOps.ssh(
     vmIPs[host],
     "PowerShell -NonInteractive -NoProfile -ExecutionPolicy Bypass -File ${winTempDir}/check_mount.ps1",
     env.SSH_BOOT_USER,
     env.SSH_BOOT_PASS,
     '22',
     "Check if the drive is mounted on ${host}, attempt ${i + 1}"
    )
    if ((i + 1) % 2 == 0) { VM_IP = sshOps.ssh(host, "bash /tmp/probe_vm_ip.sh ${params.VM_ID}", env.SSH_USER, env.SSH_PASS, '22', "Re-run probe VM IP on ${host}, attempt: ${i + 1}"); vmIPs[host] = VM_IP }
   }
   if (mounts[host] != "0") { mounts[host] = "1"; echo "Load initialization failed, retrying PXE boot..." }
}

Step 6: Read the Latest SMS TS Log Entries

Scripts tails the last lines of smsts.log, allowing the pipeline to detect key Task Sequence states such as:

• “Loading bitmap” sentence → it marks when Task Sequence selection wizard is shown
• “no_MDT.ps1” name of the script → Wizard of computer name prompt is displayed
• *“Setting wizard error..”* starting sentence → An error occurred during wizard initialization

Step 7: Automated Keyboard Input

Certain steps in the MCM Task Sequence still require manual interaction when running inside WinPE — such as selecting the Task Sequence from the wizard and entering the computer name. Since WinPE has no remote desktop, no agents, and no UI automation capabilities, we inject keystrokes directly through Proxmox’s hypervisor using qm sendkey method.

To achieve this, the pipeline uses two lightweight shell scripts.

Task Sequences list

Selecting the Task Sequence

#!/usr/bin/env bash
VM_ID="$1"
sudo qm sendkey "$VM_ID" alt-tab
sleep 1
sudo qm sendkey "$VM_ID" kp_enter
sleep 1
sudo qm sendkey "$VM_ID" s
sleep 1
sudo qm sendkey "$VM_ID" kp_enter

What it does:

• Brings the Task Sequence wizard to the foreground
• Confirms the wizard dialog
• Press certain letters to select the correct Task Sequence
• Confirms the selection

Entering the Hostname

#!/usr/bin/env bash
VM_ID="$1"
for _ in {1..10}; do
sudo qm sendkey "$VM_ID" tab
sleep 1
done
sudo qm sendkey "$VM_ID" kp_enter

What it does:

• Tabs through fields until the computer name input is reached
• Confirms with ENTER

With this step, the pipeline completes all user‑interaction requirements. From this point onward, the Task Sequence executes autonomously, and Phase 1 of the automated OSD process is fully complete.

(Extra) Step 8: Alternative method of automatic pick of task sequence

There is another alternative approach for this automation step, which is based not by pressing certain keys in the virtual machines, but trough injecting the pre-start command as variable in WinPE MCM environment. It can be done by uploaded script and execution on early stage of WinPE:

# this needs to be done as PowerShell Script
regsvr32.exe /s X:\sms\bin\x64\tscore.dll
regsvr32.exe /s X:\sms\bin\x64\tsenv.dll

@'
Dim env
Set env = CreateObject("Microsoft.SMS.TSEnvironment")
env("SMSTSPreferredAdvertID") = "<Task Sequence Deployment ID>"
'@ | Out-File -FilePath start_ts.vbs -Encoding ascii; cscript .\start_ts.vbs

.\start_ts.vbs

This simple few lines of code does:

• Registers the MCM Task Sequence DLLs (tscore.dll, tsenv.dll) to make the TS environment available in WinPE.
• Generates a VBScript on the fly that creates a Microsoft.SMS.TSEnvironment COM object and sets the SMSTSPreferredAdvertID variable — this programmatically selects the desired deployment, bypassing the manual Task Sequence wizard entirely.
• Executes cscript, which triggers the task sequence to start automatically with the pre-selected deployment ID. By setting SMSTSPreferredAdvertID before the TS wizard loads, the script tells the MCM client which task sequence deployment to run, eliminating the need for a human to select it from the wizard UI.

Phase 2 — Task Sequence Monitoring

Once the Task Sequence has been initiated through WinPE, the pipeline shifts from “bootstrapping” into continuous monitoring mode. Unlike traditional automation that relies on static wait times or assumptions about how long an installation “should” take, our pipeline monitors the Task Sequence using its real log output.

This script reads the tail of the post-WinPE SMS TS log located in:

C:\Windows\CCM\Logs\smsts.log

How Completion Detection Works

The script searches for a specific completion phrase:

“Successfully finalized logs to SMS client log directory”

This message is written only when the Task Sequence has completed all of the actions planned in whole Task Sequence process. That means - this is the earliest moment when the OS is officially “ready” for validation.

Phase 3 — Post‑Deployment Validation Suite

Once the Task Sequence has fully completed and Windows has rebooted into a freshly deployed system, the pipeline enters its final stage: post‑deployment validation. This phase determines whether the OS installation is not only complete but also correct, functional, and ready for store operations.

Today, the validation suite contains all of the automated checks listed below, and it continues to grow with every production step being added. The framework is intentionally designed to be expandable — the PowerShell data-collection script outputs a single compressed JSON object, and the Groovy validation logic Jenkins Shared library functions simply reads each key and compares it against expected values. Adding a new check requires only appending a new entry in the PowerShell script and a matching stage in Groovy, without disrupting the existing logic.

Below we highlight the checks currently performed during this stage:

Enactor POS & PDC Validation

One of the most important checks is verifying that the Enactor POS and Payment Device Controller services are running and match the expected country-specific versions. The pipeline validates four properties in a single stage:

$enProc = (Get-Process -Name EnactorP*Service*).Count
$enJava = (Get-Process -Name javaw).Count
$pos = ([xml](Get-Content 'D:\Enactor\pos\manifest.xml')).manifest.version |
       Where-Object { $_.applicationId -eq "Enactor Pos" } | Select-Object -ExpandProperty "#text"
$pdc = ([xml](Get-Content 'D:\Enactor\pdc\manifest.xml')).manifest.version |
       Where-Object { $_.applicationId -eq "Enactor Payment Device Controller" } | Select-Object -ExpandProperty "#text"

The expected POS version is looked up from a per-country map, so the same pipeline handles every market without modification. Additionally, the pipeline queries an external API to cross-validate that Enactor reports healthy from the server side.

Security Configuration Checks

Security baselines must be consistent across all POS systems. Even a small deviation can lead to compliance issues or operational risk.

BitLocker Encryption Status:

(Get-BitLockerVolume -MountPoint C).VolumeStatus   # expects FullyEncrypted
(Get-BitLockerVolume -MountPoint C).ProtectionStatus # expects On

UAC and Windows Update Policies:

(Get-ItemProperty 'HKLM:\...\Policies\System').EnableLUA         # expects 0 (disabled)
(Get-ItemProperty 'HKLM:\...\WindowsUpdate\AU').NoAutoUpdate      # expects 1 (disabled)

LAPS Account:

Get-LocalUser -Name "<administrator name>"
Get-LocalGroupMember -Group "<group of administrators>" | Where-Object { $_.Name -like "*administrator_name*" }

Confirms the Local Administrator Password Solution account exists and is a member of the local Administrators group.

Firewall Rules:

Get-NetFirewallRule -DisplayName "OpenJDK Platform binary"
Get-NetFirewallRule -DisplayName "Snarl"

OS Configuration & User Experience

These checks verify the aspects that, while sometimes overlooked, have direct impact on store usability and performance.

Missing Drivers:

Get-CimInstance Win32_PnPEntity | Where-Object { $_.ConfigManagerErrorCode -ne 0 }

When failing devices are found, the script logs the device name, device ID, PNP ID, hardware ID, and error code — critical for POS hardware stability (scanners, EFT terminals, printers, etc.).

Time Zone Validation:

The pipeline maps the two-letter country code from the hostname to the expected Windows time zone identifier (e.g., Romance Standard Time for DK/ES/FR, Turkey Standard Time for TR, FLE Standard Time for BG/FI/UA — covering 29 country codes in total).

Host Result Aggregation

Each host receives a final status:

• SUCCESS — all critical checks passed
• UNSTABLE — minor deviations detected (non‑blocking)
• FAILURE — deployment incorrect or unsafe for production

Jenkins aggregates results into a JSON map and renders them into a color‑coded HTML table included in an automatic email summary. This gives the team immediate visibility into which hosts passed validation and which require review.

The validation results are serialized into currentBuild.description as JSON during the test phase, then deserialized in the post block. This decouples data collection from reporting, and means the email summary is generated even when individual stages fail. Each host's issues are listed explicitly, giving the team instant actionability.

// Post-build reporting
post {
    always {
        script {
            def parsed = new groovy.json.JsonSlurperClassic()
                .parseText(currentBuild.description)
            def hostResults = parsed.results ?: [:]
            def issuesByHost = parsed.issues ?: [:]

            def overallStatus = "SUCCESS"
            if (hostResults.values().contains("FAILURE"))  overallStatus = "FAILURE"
            else if (hostResults.values().contains("UNSTABLE")) overallStatus = "UNSTABLE"

            def tableRows = hostResults.collect { host, status ->
                def color = status == "SUCCESS" ? "green" :
                           (status == "FAILURE" ? "red" : "orange")
                def issues = (issuesByHost[host] ?: []).join(', ') ?: "-"
                "<tr><td>${host}</td>" +
                "<td style=color:${color}><b>${status}</b></td>" +
                "<td>${issues}</td></tr>"
            }.join('\n')

            emailext(
                subject: "Jenkins Pipeline" +
                         "Completed (Build #${env.BUILD_NUMBER})",
                body: """
                    <b>Status: <span style=color:${color}>${overallStatus}</span></b><br/>
                    <b>Per-host results:</b><br/>
                    <table border="1" cellpadding="4" cellspacing="0">
                        <tr><th>Host</th><th>Status</th><th>Issues</th></tr>
                        ${tableRows}
                    </table>
                """,
                mimeType: 'text/html',
                to: "mailto@mail.com"
            )
        }
    }
}

Parallel Execution Model

Each resolved host gets its own Jenkins agent (node('NODE')), meaning every store is tested simultaneously in its own Docker container. The failFast = false setting ensures that a failure on one store does not abort tests running on other stores. The catchError wrapper lets the pipeline continue to the validation phase even if provisioning fails on some hosts — partial results are still valuable.

// Dynamic parallel stage generation
def parallelStages = [:]
def sshHosts = sshHostsResolver(params.SSH_HOSTS)

sshHosts.each { host ->
    parallelStages[host] = {
        node('TEST') {
            stage("Initialize Task Sequence on ${host}") {
                Credentials { c ->
                    // ... provisioning, PXE boot, IP discovery, TS automation
                }
            }
            stage("Wait for finish of Load@Store") {
                Credentials { c ->
                    // ... SMSTS log monitoring until completion
                }
            }
        }
    }
}

catchError(buildResult: 'UNSTABLE', stageResult: 'FAILURE') {
    parallelStages.failFast = false
    parallel parallelStages
}

Ongoing Development & Future Tests

It’s important to note that this validation suite is still being actively expanded. New store requirements, application updates, and infrastructure enhancements regularly introduce new validation needs.

Our long‑term goal is to build a full, automated compliance suite that can validate:

• software versions,
• POS hardware configurations,
• integrations,
• and regional rules.

This phased build-out approach ensures that tests evolve at the same pace as the infrastructure, and the pipeline remains maintainable, scalable, and future‑proof.

Summary

This part demonstrated how Jenkins, Docker, GitHub, Proxmox, and a well‑structured shared library form a cohesive, end‑to‑end automation ecosystem for Operating System Deployment testing. By combining controlled VM provisioning, fully automated PXE boot flows, SMS TS log‑driven state detection, hypervisor‑level UI emulation, and an extensive post‑deployment validation suite, we created reproducible testing framework that mirrors real store conditions. The result is a pipeline that eliminates manual testing effort, reduces deployment risks, and dramatically increases our confidence in every imaging update delivered to stores.

Importantly, this platform is not static. It is the foundation on which future automation capabilities will be built. Over the coming months, we plan to extend this framework to validate Proxmox ISO builds for our in‑store servers — a process that is currently performed manually and would greatly benefit from the same automation orchestration, shared library logic, and parallel execution model used in OSD testing. In addition, we intend to integrate automated testing for Windows POS backup and recovery workflows, allowing us to verify not only that POS systems can be deployed successfully, but also that they can be reliably restored when needed.