Standard Build Environment #2

Part 2: The Engine — Automating the Environment

In Part 1, we defined the “What” and the “Why.” In this second installment, we move into the “How.” As a technical lead, your goal is to move away from manual “sysadmin” tasks and toward a fully orchestrated “Cold Start” and ensuring that the foundational services are stable before the specialized platforms are layered on top.

This is the process of taking a bare-metal Proxmox node and transforming it into a functional environment holding:

• Horizontal Components: PostgreSQL, Redis, Infisical and Keycloak
• Build Components: jenkins, sonarqube, Nexus
• Project Workers: Go and Quarkus

The following diagram illustrates the “Factory Build-Up.” It considers the creation of the virtual hardware (the shell) and the installation of the professional-grade software inside it.

Factory Build-Up

Infrastructure creation is a phased process. Each phase requires a successful OpenTOFU provision, followed by Ansible installation and configuration tasks.

Detailed Task Execution

Installing servers and software requires knowing passwords; we want to have passwords as secure as possible and, for that reason, we decided to centralize password management into a secret’s vault, Infisical.

However, Infisical installation and deployment also requires passwords and we find ourselves in a catch-22 situation: during Infisical vault installation, we cannot conceal passwords in the vault because, to do that, it requires the vault itself.

So, in first, we need to deploy Infisical and have it up an running. Only then we can safely proceed for the next components.

During “1. Vault Foundation”, additional care must be taken to do not compromise or give extra visibility to passwords or any other secret.

1. The Vault Foundation

• Provisioning (OpenTOFU): create Vault infrastructure. This is, deploying LXC containers for:

• PostgreSQL (the data engine), * Redis (the cache engine)**,
• Infisical **(the secrets Vault)

• PostgreSQL installation (Ansible): Deploy PostgreSQL with a hardened baseline configuration.
• Redis installation (Ansible): Deploy Redis with a hardened baseline configuration.
• Database preparation (Ansible): Infisical requires a database. Ansible connects to the PostgreSQL instance to provision a dedicated user and schema for it.
• Infisical installation and Configuration (Ansible): installs Infisical, configures the persistence data layer to use the postgres database previously created, configures the caching layer to use Redis previously created and ensure that operational configurations are ready to accept secrets.

2: IdP (keycloak Identity Provider)

• Provisioning Keycloak (OpenTOFU): deploy deploying LXC container for keycloak
• Database Preparation (Ansible): Keycloak requires access to the postgreSQL database. Ansible connects to the PostgreSQL instance to provision dedicated users and schemas for those systems.
• Keycloak Identity Provider Installation and Configuration (Ansible): installs keycloak, configures the persistence data layer to use the postgres database previously created and ensure that master and one operational realms are created and configures as well as any application OIDC client is created and configured.

3: Build Foundation

• Provisioning (OpenTOFU): define the “Shells” for the primary infrastructure. This is, deploying LXC containers for: * Jenkins* (the orchestrator), Nexus** (the registry), * SonarQube (the Quality gate)
• Database Preparation (Ansible): SonarQube requires access to the postgreSQL database. Ansible connects to the PostgreSQL instance to provision dedicated users and schemas for those systems.
• Jenkins installation (Ansible): Install Jenkins
• Nexus installation (Ansible): Install Nexus
• SonarQube Installation (Ansible): Install SonarQube (using the previously created schema). This ensures all scan history and metrics are persisted in our central database.

4: The Worker Tier (Execution Environments)

• Provisioning the Worker Shells (OpenTOFU): define the “Shells” for the execution infrastructure
• Preparing the Runtimes (Ansible): Ansible hardens the OS and installs the necessary “Execution Context” for both golang and Java Quarkus workers

It’s a common misconception that “100% Automation” means “Zero Human Input,” but in a professional setting, the last 5% is where tools are actually tailored to our objectives. While automation handles the heavy lifting of installation, hardening, and foundational setup, it is vital to acknowledge the ‘Final Mile’ of configuration. Once the automation engine hands over the systems, it remains a necessary phase of manual finalization to align each tool with the specific DNA of the organization. Whether it is defining internal naming conventions, fine-tuning complex access control policies, or structuring team hierarchies within each tool, these steps represent the human oversight required to transform a generic technical environment into a bespoke operational asset. Automation provides the bedrock; manual finalization provides the purpose

Implementation insights

There are countless ways to architect a solution. Technical implementation is rarely about finding the only way, but the right and balanced way to make it work with quality and stability and easy to maintain (never forget that you will need to maintain your stack). We avoided getting bogged down in implementation details. Rather than providing a line-by-line manual, we will provide insights into our specific implementation, highlighting issues and critical friction points — and their resolutions — that we encountered during the build.

In summary, we believe and acknowledge that, while many technical paths exist to reach the final goal, the value of the article lies in the “lessons learned from the hands-on” rather than just the syntax of how we did it.

OpenTofu tips

• Two Tofu/Terraform providers are required: one for Proxmox interaction and another for Infisical interaction. For example:

...

required_providers {
  proxmox = {
    source = "Telmate/proxmox"
    version = "3.0.2-rc07"
  }
  infisical = {
    source = "infisical/infisical"
    version = "~> 0.15.0"
  }
}

...

• Organize your project considering that you have to manage two sets of containers: one, where the passwords are not secure (hard coded/Plain text) and another where the passwords are fetched from Infisical vault.
• Define your containers and characteristics in a variable array of objects. Each object represents a container with its characteristics. For example:

...

variable "lxc_configs" {
  description = "LXC containers to be managed"
  type = map(object({
    vmid        = string
    hostname    = string
    rootfs_size = string
    use_infisical = bool
    infisical_secret_name = optional(string, "")
  }))
  default = {
    "machine1"  = { vmid = "001", hostname = "machine1", rootfs_size = "30G", 
                    use_infisical = false }
    "machine2"  = { vmid = "002", hostname = "machine2", rootfs_size = "30G",
                    use_infisical = true, infisical_secret_name = "INFISICAL_SECRET" }
  }
}

...

• Deploy LXC containers based on the array variable and for each machine, decide if the password is fetched from Infisical vault. For example:

...

# Passwords from Infisical
# Fetch secrets only for machines that have use_infisical = true
data "infisical_secret" "lxc_passwords" {
  for_each = { for k, v in var.lxc_configs : k => v if v.use_infisical }

  name        = each.value.infisical_secret_name
  project_id  = var.infisical_project_id
  env_slug    = var.infisical_env_slug
  folder_path = var.infisical_folder_path
}

resource "proxmox_lxc" "containers" {
  for_each = var.lxc_configs

  # Logic: Use Infisical secret if enabled, else use a default fallback
  password = each.value.use_infisical ? data.infisical_secret.lxc_passwords[each.key].value : "HARDCODE_PASSWORD"

...

• Be exhaustive in your analysis of the tofu plan output. During implementation, we were confronted by seemingly innocuous code changes that triggered a 'Destroy and Create' action rather than an 'Update.' Because OpenTOFU treats the LXC container as the unit of state, an overlooked plan output can result in the immediate deletion of an existing container, leading to the total loss of all local configuration and local persistent data. Combine the analysis with the usage of the lifecycle block:

lifecycle {
  prevent_destroy = true
}

• Use the best practices to store and protect your state file. Remember that state files can store confidential and critical data that should not be available to all eyes.

Ansible tips

• Leverage the active Ansible community. The Ansible community is highly active, and you will likely find a role or collection for almost any use case. However, evaluate community contributions carefully; some may be overly complex for your specific needs or contain undetected and unresolved bugs. In our case, we took advantage of:

# Collection used to handle keycloak installation tasks
$ ansible-galaxy collection install middleware_automation.keycloak

# Collection used to handle nexus installation tasks (warning about bugs)
$ ansible-galaxy collection install cloudkrafter.nexus

# Collection used to handle postgreSQL installation tasks
$ ansible-galaxy collection install community.postgresql

# Colection to deal with infisical secrets
$ ansible-galaxy collection install infisical.vault

# This collections requires infisical SDK. To install
$ sudo pip install infisicalsdk --break-system-packages

• Don’t depend directly on external roles. By creating internal versions, you gain the flexibility to customize logic and respond quickly to changing infrastructure needs. But you will have extra effort to keep them up-to-date.
• Always wrap community roles into your own internal roles. This ensures you can adapt them to your specific requirements and provides the flexibility to changes as your infrastructure needs evolve.

• Make sure that you identify common tasks and isolate those in a common role. For example, some components rely on JAVA; therefore, create a common role to install JAVA and reuse it:

• Carefully define and organize your inventory. Consider to segregate by environment and organizing servers/containers by function groups. This will bring flexibility when scaling is needed. For example:

## hosts for environment test 
[pgsql_servers]
pgsql03.lan

[redis_servers] 
redis01.lan

[infisical_servers] 
infisical01.lan

[keycloak_servers] 
keycloak01.lan

[jenkins_servers] 
jenkins01.lan

[nexus_servers] 
nexus01.lan

[sonar_servers] 
sonar01.lan

[go-app-workers] 
goworker01.lan

[jquarkus-app-workers] 
jqworker01.lan

• The rule: all secrets shall be fetched from Infisical Vault. It is therefore important to define and standardize the way the team accesses Infisical. As example, consider the snippets below that are part of the sonarqube installation project:

...
## Variable to group postgres attributes needed for the sonarqube installation
postgres:
    url : "jdbc:postgresql://pgsql03.lan:5432/sonardb?currentSchema=sonar"
    user : sonar
    password : "{{ database_password.value }}"
    schema : sonar
...

...
## Create a file to contain variables for the Infisical vault lookup plugin.

infisical:
  client_id: "INFISICAL_CLIENT_ID"
  client_secret: "INFISICAL_SECRET"
  project_id: "INFISICAL_PROJECT_ID"
  url: "http://infisical01.lan:8080"

database_password: "{{ lookup('infisical.vault.read_secrets', 
                        url=infisical.url,
                        auth_method='universal_auth', universal_auth_client_id=infisical.client_id, universal_auth_client_secret=infisical.client_secret,   
                        project_id=infisical.project_id, env_slug='dev', path='/databases', secret_name=postgres.user) }}"

• The exception: secrets that are used during the deployment of the Vault Foundation phase (see above); usage of ansible vault is a fair solution to increase security during this phase. For example:

...
keycloak.db_password: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          66613562383335356465376364313938303363366666396232323033373037346232396538643132
          3635663137393533323730333964366632623463626330380a643430653236343230623832646430
          34393538343131303363383730346334653663306630313963366234323532306263313062396534
          3438386233393338380a323763343966323363353763396164623963653135353339306262386562
          3562
...

The “Ready-to-Receive” State

By the end of Phase 4, your infrastructure is in what is called a “Ready-to-Receive” state.

1. The Shell exists (OpenTOFU).
2. The Runtime is installed (Ansible).
3. The Horizontal Components are set (Ansible).

The stage is now set. The “factory” is running, the quality gates are standing, and the worker containers are waiting. In addition it sets important principles:

• The team has a “Full-Fledged” build stack that they can immediately rely on without loosing time and effort in reinventing the wheel.
• The build stack contributes to standardize the work across the team.
• The developers don’t need to worry about “How do I get a database?” or “Is Java installed?” The infrastructure has been provisioned as a service for them.
• The build stack can easily be scaled up (horizontally or vertically) to respond to additional requirements.
• The build stack can easily be duplicated to respond to projects with critical segregation requirements.

️A Note on Production Readiness

To maintain the clarity of this architectural model, certain high-level production requirements have been abstracted. In a final professional enterprise deployment, the following “Hardened Layers” must be implemented:

• Protocol Encryption: While this model demonstrates internal HTTP communication for simplicity, HTTPS (TLS/SSL) is mandatory for all traffic. All endpoints must be secured with valid certificates (e.g., via Let’s Encrypt or an internal PKI).
• Edge Security & Reverse Proxy: No platform component (Jenkins, Infisical, Keycloak) should be exposed directly. A secure Reverse Proxy (such as Nginx, HAProxy, or Traefik) must act as the single entry point, handling SSL termination and header hardening.
• Root-context per application: always use a different root-context per application. This brings segregation of namespaces leading to better isolation between the different application, facilitates Reverse Proxy configurations, increases security, facilitates authorizations based on URLs and improves observation and maintainability.
• Zero-Plaintext Mandate: Any “Bootstrap Secrets” (the initial keys used to setup the Phase 0 above or to unlock the Vault itself) must never exist in plaintext on disk. For example, use Tofu Ephemeral concepts to reduce exposure (Ephemeral variables are not stored in the state file) and Ansible Vault for encrypted local variables.
• Network Segmentation: In a production Proxmox environment (or any other environment), shared platforms and workers should reside on isolated VLANs, with traffic strictly governed by firewall rules (ACLs) to ensure the “Principle of Least Privilege.”
• Data & Backups: Infrastructure as Code (IaC) allows us to easily recreate the tools, but the data require a rigorous backup strategy. Always remember Infisical and Keycloak; since these services store the ‘keys to the kingdom,’ extra care must be taken to ensure data is protected, secure, and highly available for recovery. Fortunately, because we use IaC, we have the perfect environment to regularly test data restoration, ensuring that if the ‘kingdom’ falls, we can rebuild it — and its identity — in minutes.

In Part 3, we will cover the final mile:

• The roles of Jenkins, Nexus and SonarQube
• How Jenkins triggers the build for Golang and Java Quarkus projects.
• How SonarQube acts as the final “Technical Lead” by blocking poor code.
• How Ansible performs the actual deployment by fetching the immutable binaries from Nexus and injecting them into our prepared Workers.