From Frustration to Automation: Our Journey to Unattended Tooling Upgrades

Photo by Derek Thomson on Unsplash

Motivation

One of the key missions of the Platform Engineering team is to provide toolset that simplifies the day-to-day operations and workflows for product teams across the organization. These tools span a wide range of areas within the tech ecosystem, including CI/CD, observability, computing, networking, infrastructure provisioning, and security.

Like any software, these tools need regular updates for several reasons:

• Addressing security vulnerabilities with patches.
• Fixing bugs to ensure reliability.
• Resolving compatibility issues.
• Enhancing performance.
• Introducing new features that benefit product teams.

As the number of tools in the platform grows, so does the time required to keep everything up to date. This presents a significant challenge for platform teams.

In this series of articles, we’ll walk you through the technical steps our team took to reduce the time spent managing tool upgrades. Ultimately, this process evolved into an almost entirely unattended experience. Keep reading to learn more about it!

Context

At seQura Tech, all transversal third-party tooling and applications are defined as code in a monorepo. We use Helm charts to define how these applications are deployed into our Kubernetes cluster. However, all operations over these applications were manually managed from our local environments, which introduced inefficiencies and consumed valuable time.

Each application definition required maintaining multiple files:

• **values.yaml (Helm values configuration files):** These define the application’s configuration and Kubernetes resources/components. Since we deploy this apps in multiple Kubernetes clusters, there are also values files specific for those clusters such as values.dev.yaml , values.prod.yaml and so on.
• Deployment script: A shell script designed to simplify maintenance tasks. This script included a $CHART_VERSION variable to specify the chart version to deploy and supported commands like template, diff, and upgrade, depending on the input flags provided on runtime. This helps us defining the helm commands for each application depending on the existing values files.

This is an example of the deployment script for opentelemetry-collector :

#!/bin/sh
NAMESPACE=tracing
CHART_VERSION=0.107.0
APPLICATION="opentelemetry-collector"
ENVIRONMENT=`kubectl config current-context | cut -d- -f1`
DEPLOY=false
TEMPLATE=false

while getopts "dt" arg; do
  case $arg in
    d)
      echo "-d was triggered, Deploying"
      DEPLOY=true
      ;;
    t)
      echo "-t was triggered, Rendering template"
      TEMPLATE=true
      ;;
    *)
      echo "Showing differences"
      ;;
  esac
done

echo "Environment is ${ENVIRONMENT}"

if [ ${DEPLOY} == "true" ]
    then
        ACTION="Upgrading"
        COMMAND="upgrade"
        FLAGS="--atomic --wait --timeout 300s --install ${APPLICATION}"
    elif [ ${TEMPLATE} == true ]
    then
        ACTION="Rendering template"
        COMMAND="template ${APPLICATION}"
        FLAGS=""
    else
        ACTION="Showing differences of"
        COMMAND="diff upgrade --install ${APPLICATION}"
        FLAGS="-C 1"
fi

echo "${ACTION} ${APPLICATION} in namespace ${NAMESPACE}..."

# Adddd helm repository
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

# install opentelemetry chart
helm ${COMMAND} \
    ${OTEL_PROCESSOR} \
    --namespace=${NAMESPACE} \
    ${FLAGS} \
    -f values.yaml \
    -f values.${ENVIRONMENT}.yaml \
    open-telemetry/opentelemetry-collector --version ${CHART_VERSION}

Whenever an application needed an upgrade, the process followed this workflow:

Team member A:

• Notices (or remembers) that an upgrade is needed.
• Reviews the application’s releases on GitHub or another repository.
• Checks the changelog, often including multiple patch, minor, or major updates.
• Ensures there is a corresponding Docker image tag for the desired release version.
• Updates the $CHART_VERSION in the application’s definition.
• Adjusts the Helm values files, if necessary.
• Deploys the application to the development environment for testing.
• Creates a pull request with all the changes and relevant details, like ChangeLog notes or other considerations.

Team member B:

• Reviews the pull request.
• Conducts additional checks if there are concerns.
• Approves the changes.

Team member A:

• Merges the pull request.
• Manually deploys the updated application.

As you can see, this process was time-consuming and often involved back-and-forth discussions during the pull request review phase. Ultimately, it didn’t add much value to the organization — it was more about maintaining the status quo than driving meaningful progress. To address this, we decided to rethink the workflow and free up our time to focus on what truly matters and delivers real value.

Additionally, relying on an upgrade calendar or constant reminders — our best option at the time — only added to the frustration. It made the process feel tedious and easy to procrastinate, leading to a vicious cycle where applications fell multiple versions behind. Catching up became even more challenging, as it required reviewing and reconciling numerous version releases to bring everything back up to date.

Workflow breakdown

To understand and improve our workflow, we broke it down into distinct steps:

1️⃣ Releases Discovery

• Identify the need for an upgrade.
• Review the available releases.
• Check the requirements and conditions that need to be met.

2️⃣ Create the Pull Request

3️⃣ Pull Request Review

4️⃣ Deployment & Testing

We took a pragmatic approach, focusing on automating every step where human intervention was not adding any value, while considering our current experience and capabilities. This meant delivering a solution that addressed the problems we could solve effectively, without investing excessive effort or overcomplicating the process.

Conclusions and Next Steps

Upgrading third-party tooling and applications can consume a lot of valuable time — time that could be better spent on tasks that deliver meaningful business impact. This challenge only grows as the number of applications requiring maintenance increases.

By rethinking and breaking down our workflow, we’ve set the foundation for a more streamlined process. But what if we told you there’s a way to fully automate the discovery of new releases and the creation of pull requests with minimal effort?

In the next article, we’ll reveal how we leveraged powerful tools to turn this vision into reality. Get ready to dive into the technical details and see how we automated everything from Releases Discovery to Pull Request Creation saving countless hours and headaches.

Stay tuned!