“Smart” CI/CD systems — improving overall efficiency

Improving CI/CD efficiency

Introduction

It is a common practice nowadays for CI/CD system to be as extensive & as thorough as possible, as part of the “Shifting-left” mindset. This usually includes, but no limited to:

• Unit-testing
• Building artifacts
• Scanning artifacts (dependency graph & SPBOM creation)
• Integration-testing
• Publishing + Signing artifacts
• Creating a Pre-release
• Deploying to a internal environment
• Performing End-to-end testing + QA validation
• Creating an official release

To each team their own unique quirks / jobs, but these are usually the building blocks to make your projects bullet-proof (as far as we can control! See recently discovered Trivy’s supply chain attack).

The Problem

Having a robust system makes the pipeline’s overall timing exhausting for the entire team (starting with the developers that wish to add more features, the devops the wishes to improve pipelines coverage, the project manager that wishes to generate a new hotfix release quickly, and so on) as 6+ jobs must be performed of every successful merge commit of a new PR to the main branch. Take in to consideration that the average build time per job in around ~5 minutes, we’re looking at about half an hour for a developer to get a sense of his recent changes and for the next CD to kick in.

And as projects grow and new features are added, our service repository become sort-of similar to a large monorepo, with different programming languages & tooling, that require their own custom steps for building, testing, etc.

before — over an hour per pipeline

The “easy” solution

Here are some useful tricks and pattern to improve the overall timing:

1. Networking — Use fast, private connections, as public traffic usually traverses a greater distance, has lower bandwidth, and greater latency.
2. Adding parallelism — Leverage matrix’s where possible, is sequential really necessary?
3. Caching — Most build systems are “smart” / “remember where they left off” by using hashes and other nifty tricks. Having incremental builds by leveraging those caching capabilities can possible speed build time by a great factor.

efficiency improved — steps 1–3

A little more about caching…

Make sure to be aware about remote caching capabilities / solutions that may fit your needs. The ups & downs of each and how selecting one of the other could effect the overall performance. For example, my recent shift to usage of S3 bucket as remote cache store for docker/oci builds instead of the traditional registry cache store has reduced our build time by 30–35%, as s3 allows more api calls in parallel than ecr, greater chunking & multi-part upload capabilities.

The well-known actions/cache could also play the part, however the free plan offers a very limited storage for caching, followed up by an expansive bills for an actual real-world project. In such cases hosting your own GitHub Actions Cache Server could be your best bet.

Still too slow! what’s next?

Our project grew big, more than 20 artifacts are to be built, scanned, pushed, signed, pulled & deployed. Even after lowering the pipeline runtime to 15–20 minutes, there are still some redundant jobs, and we can do better. We do wish to have a unified version for all of our artifacts, but that does not mean we must build EVERYTHING for every merged pull-request… are we?

The Cherry on Top 🍒

By identifying what has changed, and detecting which artifacts are effected, and adding some logic to our pipelines, we can perform actions in a “smart” way, making our CI/CD system incremental, like most modern build system do. There are several actions presented in Github Actions Marketplace that can perform the required identification, such as tj-actions/changed-files (also had a supply-chain attack about a year ago..), however I ended up using paths-changes-filter by dorny which seems to fit my needs better. I’ve leveraged the usage of AI (Opus is my favorite) to generate a dependency graph in a single filters.yaml file, where each key is an artifact name. e.g:

action usage example call w/ filters file

# filters.yaml
zmynxx-containers-oci/aws-lambda-calculator: &zmynxx-containers-oci/aws-lambda-calculator
- src/aws-lambda-calculator/**
- lib/**

zmynxx-containers-oci/sentor: &zmynxx-containers-oci/sentor
- src/sentor/**
- lib/**

zmynxx-charts-oci/aws-lambda-calculator:
- *zmynxx-containers-oci/aws-lambda-calculator
- charts/aws-lambda-calculator/**

Thanks to YAML capabilities, I could easily leverage anchors & expansions, making my filters.yaml file DRY.

The next step was to leverage this actions outputs combined with logic:

path-filter outputs

path-filter outputs

For example, after altering main.py of src/aws-lambda-calculator , this action’s output would become:

zmynxx-containers-oci/aws-lambda-calculator: true
zmynxx-containers-oci/sentor: false
zmynxx-charts-oci/aws-lambda-calculator: true
changes: ['zmynxx-containers-oci/aws-lambda-calculator', 'zmynxx-charts-oci/aws-lambda-calculator']

Now by adding if: contains(fromJSON(steps.path-filter.outputs.changes,'zmynxx-containers-oci/*') to my build-containers re-usable workflow call, I would only build containers if required, and to make it complete I would build my container’s matrix accordingly, so I won’t build all of my containers for such change (for example, my zmynxx-containers-oci/sentor should not be built). In a similar fashion I’d perform the build for all of my charts, libraries, CDK and the rest of my artifacts.

Keep the ball going

Improve efficiency — leverage the full power of “path-filter-ing”

In the previous step I’ve shown how to perform incremental builds to artifacts according to detected path changes, but I did (and you also should!) keep using the same pattern to perform the rest of your CI/CD jobs, only effected artifacts should be built, then scanned, etc. What about unified tagging? tags are immutable, however multiple tags can be used to reference the same artifact (SHA), so a simple re-tagging is all that is needed to keep you organized.

⚠️ WARNING ⚠️

Build are now incremental, and developer may tend to ignore pipeline failures, which is now crucial. To avoid such cases, first increase developers awareness, and second, add a nightly trigger to build EVERYTHING (by adjusting the skipping logic), just to be on the same side.

BONUS #1: “Smart” Deployments

Our product is part of a Multi-tenant SaaS and contains several components on different deployment levels ( sub-environment, tenant ) and stages ( pre , onboarding , post ). Deployment used to be inclusive and took around 15–30 minutes overall. There are many benefit for using filters.yaml file with this action, but a major one is that you have have filter-keys according to your wishes. I had leveraged that capability to add deployment level & stage specific filter-keys ( I even used specific .github/actions and .github/workflows paths!) and once again filtered the calls for deployment as needed.

The result:

Improved efficiency — “Smart” builds & deployments using path-filter

“Smart” deployments — detecting effected phases

“Smart” deployments — tigger only effected phases, reduce overall run time

BONUS #2: PR Auto-labeler

If you’re not using actions/labeler already, you are missing out on all the fun. But if you do, you can leverage the same filters.yaml file from before to automatically add labels to your pull-requests, adding observability to your developers PR, so they can clearly see what artifacts are effected by the changes, and will be built upon merge.

example usage of auto-labeler

FIN; as always, had a great fun writing this! If you'd like to further discuss, have a challenge for me or a new field to explore, let me know! See ya next time! 😄