One line in SBT 2— How Remote Caching Cut Our CI in Half

Thank Gemini for the nice infographic here

When every CI build starts from scratch, your feedback loop dies. Here’s how we fixed it.

Every engineer on our team knew the ritual: push a branch, open YouTube, come back when the build was done — and your focus was gone.

This is the story of how we buried that ritual — with one Kubernetes pod, a gRPC cache server, and SBT 2’s built-in Bazel protocol. From 25 minutes to ±11 minutes of total build time.

I’d like to thank Avri Chen Roth for his DevOps engineering expertise, patience, and optimism — without his help none of this could have happened. I would also like to thank Eugene Yokota for all of his hard work on the SBT project.

This article became larger than I initially thought — please don’t hesitate to skip to the section that interests you the most and if you just need the gist — you can find it here:

• SBT 2 AI model migration .md file
• Bazel-remote Docker image setup

The Problem: A Slow-Burning Build

Most of our Scala backend is built as a mono-repo consisting of roughly 35 sub-projects: a core commons library, a dozen-plus connector SDKs (Google, Office365, Slack, Salesforce, and many more), and seven deployable services assembled into fat JARs.

It is a cross-cutting foundation: nearly every other service in the platform depends on it, and internally its sub-projects form a deep dependency graph of their own.

When you have that many sub-projects, build time compounds quickly. A feature branch CI run that touched a mid-level SDK would recompile not just that SDK, but every downstream module that depends on it — even if the changes were trivially small, as small as a one-character typo.

On a modest build node, a full compile-and-test cycle took almost 25 minutes of pure Zinc compilation and unit test execution (even when forking JVMs and parallelizing test runs). Multiply that by a dozen engineers pushing branches throughout the day — each one killing focus while you reminisce about the good old days of downloading files on a 56k modem — and it starts to cost real calendar time.

We had already tried to address this with a remote cache. The approach used a third-party sbt plugin — fm-sbt-s3-resolver — to push and pull compiled class directories to an S3 bucket. It worked, mostly, but it brought its own set of problems:

• High latency: Every cache read and write was an HTTPS round-trip to S3. Even a cache hit could take several seconds per sub-project.
• Credential sprawl: The plugin depended on AWS SDK v1. Every build node needed AWS credentials wired up specifically for this.
• Plugin fragility: fm-sbt-s3-resolver sat between SBT's task engine and S3 in a way that was hard to debug when things went wrong — and it still hasn't been built for SBT 2.

Additionally, it never really improved performance significantly (over 3–4 minutes) and when trying our own custom solutions they constantly required maintanance and fixes.

Enter SBT 2

SBT 2 solved this in one step — by making remote caching a first-class, built-in feature with a well-specified protocol. Here is how we migrated, what broke, what we fixed, and how you can do the same.

This is no small feat: including remote caching via the gRPC Bazel remote execution protocol is a major undertaking under the hood, and the contributors to the SBT 2 project deserve significant credit for it.

Remote Caching — The General Concept

At a high level, the idea is to use remote, content-addressed build caches to turn large, incremental builds into fast, mostly-lookup operations.

Instead of recompiling or rebuilding everything on every CI run, the build system fingerprints each unit of work (sources, dependencies, flags, environment), checks a shared cache, and only executes the actions whose inputs truly changed. In the case of unit tests — only those that need to run due to code changes, either in the tests themselves or in their corresponding code.

That pattern shows up across ecosystems: Bazel and Pants for polyglot monorepos, Xcode build caching, container build layer caches, JavaScript monorepo tools, and more. In all of them, remote caching moves teams from “every build is a fresh start” (hence the title) to “most builds are cache hits” — which is often the only way to keep feedback loops fast as repos, teams, and CI matrices grow.

Under the Hood — A Crash Course: Merkle Trees and Content-Addressed Storage

Before explaining how the SBT 2 remote cache works, it helps to understand the underlying data model: Content-Addressed Storage (CAS) backed by a Merkle tree structure.

Content-Addressed Storage (CAS)

In a conventional file store, you give an object a name and look it up by that name later. In a content-addressed store, the name is derived from the content itself — typically its SHA-256 hash. There are no user-chosen filenames. Two identical objects always have the same key; two different objects (even differing by one byte) always have different keys.

This property is immensely useful for build caches:

• Determinism: Given the same inputs, the same hash is produced. The cache does not care which machine produced the artifact or when — only what went in. (This still requires the same Java version and build parameters.)
• Deduplication: If two branches produce the same compiled output — common for sub-projects that were not touched — they share a single cache entry.
• Immutability: You never update a cache entry. You only ever write new ones. Old entries are evicted by LRU when the cache is full.

Merkle Trees in Builds

A Merkle tree takes CAS one step further: a parent node’s hash is derived from the hashes of its children. In a build context this maps perfectly onto the dependency graph:

Hash(service-jar)
                 │
         ┌───────┴───────┐
    Hash(commons)    Hash(sdk-google)
         │                   │
      Hash(src)          Hash(src)

The hash of service-jar depends on the hash of commons, which depends on the hash of its source files. Change one source file in commons and the entire chain above it gets a new hash — triggering a cache miss only where it is needed, and a cache hit for every unaffected sibling branch.

SBT 2 implements exactly this model. Each sub-project is identified by a hash of:

• Its source files (content-hashed)
• Its classpath (hashes of dependency JARs)
• The compiler flags in effect

If any of these change, the hash changes and the cache misses. If none change, the cached .class files and Zinc incremental analysis are pulled from the server and SBT skips compilation entirely.

This also means cache invalidation is automatic: bump a library version, change a compiler flag, or upgrade Scala — the affected hashes change, those sub-projects get cache misses, and everything else stays cached. There is no manual “flush” step.

The Protocol: Bazel Remote Execution API over gRPC

SBT 2 does not invent its own cache protocol. It implements the Bazel Remote Execution (RE) API — the same open protocol used by Google’s Bazel build system, defined at bazelbuild/remote-apis.

The protocol is built on gRPC (HTTP/2 + Protocol Buffers). The two relevant services are:

• ContentAddressableStorage (CAS): Read and write blobs by their SHA-256 digest. This is where compiled class files and Zinc analysis files live.
• ActionCache: Map a hash-of-inputs → a set of output blob digests. “For this exact combination of sources and flags, the outputs were these blobs.”

The SBT 2 remote cache interaction looks like this:

SBT 2                             bazel-remote (gRPC server)
  │                                       │
  │  1. Hash(sources + classpath + flags) │
  │──────── ActionCache.GetActionResult ─►│
  │                                       │
  │      [Cache HIT] ◄──────────────────  │  Return blob digests
  │         │                             │
  │  2. Fetch each blob                   │
  │──────── CAS.BatchReadBlobs ──────────►│
  │◄───────────────────────────────────── │  Stream .class files
  │                                       │
  │      [Cache MISS]                     │
  │  3. Run Zinc compiler                 │
  │  4. Upload results (master only)      │
  │──────── CAS.BatchUpdateBlobs ────────►│
  │──────── ActionCache.UpdateActionResult│

A cache hit on a local-network gRPC server takes under 100ms for a typical sub-project. Compare that to several seconds for a Zinc compilation or an HTTPS S3 round-trip.

The gRPC Cache Server: bazel-remote

[bazel-remote](https://github.com/buchgr/bazel-remote) is an open-source, lightweight Bazel-compatible cache server written in Go. It implements the full Bazel RE API and stores artifacts in a local directory (or optionally S3/GCS as a backend). Two ports:

• HTTP :8080 — legacy Bazel HTTP cache protocol (not used by SBT 2, but useful for Bazel itself)
• gRPC :9092 — the RE API, which is what SBT 2 connects to

We deploy it as a dedicated Kubernetes pod in the same namespace as our Jenkins build agents. Because it runs inside the cluster, every build pod can reach it by DNS name — no egress or VPN required. The full Kubernetes manifest (Deployment + Service) is available as a gist.

The key points of the manifest:

• A single-replica Deployment running buchgr/bazel-remote-cache:latest with --max_size=50 (GB — tune to your disk).
• Ports 8080 (HTTP, legacy) and 9092 (gRPC, used by SBT 2).
• A Service named bazel-remote-cache in the jenkins namespace — this is the DNS name that appears in build.sbt.
• Storage starts as emptyDir (ephemeral); swap for a PersistentVolumeClaim once validated.

Every Jenkins build pod resolves this DNS name automatically. Note that while bazel-remote has a local disk cache, we also opted for an S3 backend as a persistent layer.

Tip: Start with emptyDir (ephemeral) for the first deployment to validate the setup. Once you are confident, swap it for a PersistentVolumeClaim so the cache survives pod restarts and you avoid cold-start misses.

Wanna try it yourself without having to deal with setting up S3 or a K8s pod on your EKS? just run this locally and you’ll have your bazel-remote cache running locally for testing. Remember that if you don’t want a persistent cache between pod restarts / you can remove -v $HOME/.bazel-remote-cache:/data

When setting the remote cache on build.sbt — just refer to grpc://0.0.0.0:9092

docker run -d \
  --name bazel-remote-cache \
  -p 9092:9092 \
  -p 8080:8080 \
  -v $HOME/.bazel-remote-cache:/data \
  buchgr/bazel-remote-cache \
  --grpc_address=0.0.0.0:9092 \
  --dir=/data \
  --max_size=50

The SBT 2 Migration

A detailed breakdown of how we upgraded our existing code and infrastructure. The migration touches three areas:

1. Scala build configuration (build.sbt, plugins.sbt, .sbtopts)
2. Jenkins pipeline code
3. Kubernetes bazel-remote cache server

Step 1 — Pin SBT 2 in project/build.properties

- sbt.version = 1.9.0
+ sbt.version = 2.0.0-RC9 (hopefully 2.0.0 by the time you're reading this)

That is the entire change that selects the new build tool version. The SBT launcher reads this file, downloads the specified runtime if it is not already cached, and starts up. On CI we pre-install 2.0.0-RC9 on the build agent so there is no cold-start download.

Step 2 — Rewrite project/plugins.sbt

Before:

addSbtPlugin("com.github.sbt"       % "sbt-javaagent"      % "0.1.8")
addSbtPlugin("com.frugalmechanic"   % "fm-sbt-s3-resolver" % "0.21.0")  // old S3 cache
addDependencyTreePlugin

After:

addRemoteCachePlugin
addSbtPlugin("com.github.sbt" % "sbt-javaagent" % "0.2.0")

What changed and why:

Removed fm-sbt-s3-resolverEntirely replaced by the built-in gRPC cache — no plugin version or AWS credentials needed.

Removed addDependencyTreePluginSBT 2 has improved built-in dependency inspection. No longer needed.

Added addRemoteCachePluginNot an external artifact with a version number. It is a built-in SBT 2 directive that activates the entire gRPC remote cache subsystem, wiring pull/push hooks into the compile and test tasks automatically.

In the assembly.sbt file — upgrade to

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "2.3.1")

Step 3 — Wire the Cache in build.sbt

Remove (S3 approach):

// These are gone entirely
import com.amazonaws.auth.profile.ProfileCredentialsProvider
import com.amazonaws.auth.{AWSCredentialsProviderChain, DefaultAWSCredentialsProviderChain}
s3CredentialsProvider := { (bucket: String) =>
  new AWSCredentialsProviderChain(new ProfileCredentialsProvider("us1"), DefaultAWSCredentialsProviderChain.getInstance())
}
pushRemoteCacheTo := Some("S3 Remote Cache" at "s3://my-bucket/path/snapshots")
ThisProject / Compile / pushRemoteCacheConfiguration := (ThisProject / Compile / pushRemoteCacheConfiguration).value.withOverwrite(true)

Add (gRPC approach):

// Detect CI environment
val isJenkins = sys.env.contains("JENKINS_HOME") || sys.env.contains("BUILD_NUMBER")
// One line to rule them all
Global / remoteCache := (if (isJenkins) Some(uri("grpc://bazel-remote-cache:9092")) else None)

• grpc:// — plain gRPC (no TLS). Safe inside the private Kubernetes cluster network. Use grpcs:// if you expose the server outside the cluster.
• bazel-remote-cache — the Kubernetes Service DNS name from the manifest above.
• 9092 — the default gRPC port for bazel-remote.
• isJenkins — local developer builds silently skip the cache. The DNS name bazel-remote-cache only resolves inside the Jenkins namespace, so you do not want to try connecting from a laptop.

That is the complete remote cache wiring. No AWS credentials. No per-project configuration. No explicit pullRemoteCache or pushRemoteCache commands in the pipeline. SBT 2 handles it all automatically.

Why master-only push? In our setup, only master builds push results to the cache. Feature branches pull from the cache but never write to it. This prevents half-finished or experimental branch artifacts from polluting the shared cache. The result: the cache always contains artifacts produced by a known-good, full build.

Step 4 — Fix packagedArtifacts for SBT 2

This is one of the subtler SBT 2 API changes that will bite you if you have a root aggregate project:

// SBT 1 — works but behaves unexpectedly in SBT 2
packagedArtifacts := Map.empty
// SBT 2 — correct
packagedArtifacts := Def.uncached(Map.empty)

In SBT 2, the task engine memoises task results more aggressively. Returning a bare Map.empty allows the engine to cache this value across evaluations in ways that can cause the root aggregate project to accidentally publish artifacts it should not. Wrapping it in Def.uncached(...) signals that this value should never be cached — it is a constant that means "publish nothing."

Step 5 — Fix --add-exports Flags for fork := false

This one caused a subtle class of failures that were hard to diagnose. Here is what happened.

SBT 1 builds in cap-commons were using ThisBuild / fork := true, which meant each sub-project's tests ran in a child JVM. The javaOptions setting (including --add-exports flags for accessing Java internals needed by BouncyCastle, Nimbus JOSE+JWT, and Azure XML parsing) was applied to those forked JVMs.

SBT 2’s setup that works best for our large mono-repo is Global / fork := false — all sub-projects share the same JVM as SBT itself. This is faster (no JVM startup cost per sub-project), but it also means javaOptions inside a sub-project's .settings(...) block has no effect.

The fix: move the flags out of build.sbt entirely and into the SBT launcher's own JVM options:

project/.sbtopts (local development):

-J-Xmx16g
-J-Xms8g
-J-XX:MaxMetaspaceSize=1g
-J-XX:ReservedCodeCacheSize=1024m
-J-XX:+TieredCompilation
-J-XX:+UseG1GC

SBT_OPTS environment variable in the Jenkins pipeline (sbt.groovy):

export SBT_OPTS="--add-exports java.base/sun.security.pkcs=ALL-UNNAMED \
                 --add-exports java.base/sun.security.x509=ALL-UNNAMED \
                 --add-exports java.base/sun.security.util=ALL-UNNAMED \
                 --add-exports java.base/sun.security.rsa=ALL-UNNAMED \
                 --add-exports jdk.crypto.ec/sun.security.ec=ALL-UNNAMED \
                 --add-exports java.xml/com.sun.xml.internal.stream=ALL-UNNAMED"

The SBT launcher reads .sbtopts and SBT_OPTS before starting, applying the flags to its own JVM. Since all sub-project code runs in that same JVM (under fork := false), the flags are in effect everywhere. If you have other Java exports or other options, just follow the logic suggested here.

Rule of thumb: If you use fork := false globally, any JVM flags you need for your code must live in .sbtopts or SBT_OPTS, not in javaOptions inside build.sbt.

Step 6 (optional) — Remove assembly / test := {}

In SBT 1, the sbt-assembly plugin ran all tests before packaging by default. It was common practice to disable this with:

// SBT 1 — needed to avoid running tests twice
assembly / test := {}

In SBT 2, sbt-assembly no longer runs tests before packaging. The setting is harmless but dead weight — removing it keeps the build definition clean and honest.

Step 7 (optional) — Register rootPaths for Readable CI Logs

ThisBuild / rootPaths += "COMMONS" -> baseDirectory.value.toPath
ThisBuild / rootPaths += "COURSIER"    -> csrCacheDirectory.value.toPath

This is a purely ergonomic SBT 2 feature. Instead of CI log lines like:

[error] /home/jenkins/....

You get:

[error] $COMMONS/project/discord/....

When you have 35 sub-projects all printing absolute paths in their error messages, this makes a meaningful difference to readability.

here is an migrate-to-sbt-2.md file that you could provide your AI agent to execute most of the above steps. Just use it as a baseline for your mono-repo.

CI — Jenkins Pipeline: Two Changes That Compound the Speedup

You might be using GitHub Actions or another build platform. If so, take the concepts below and translate them to the syntax and workflow that suits your tooling.

Parallel Fat-JAR Assembly with all

With SBT 1, building eight fat JARs was sequential — each assembly command blocked the next. The pipeline looked roughly like:

sbt pullRemoteCache compile testQuick pushRemoteCache assembly
                                                      ^
                                     8 assemblies run one at a time

now replaced with (pullRemoteCache ,testQuick and pushRemoteCache are gone)

sbt compile test 

and for the assembly, SBT 2 introduces all — a first-class parallel task scheduler.

The new assembly stage:

sbt "all <service_1>/assembly <service_2>/assembly <service_3>/assembly ..."

All assemblies run concurrently in one SBT session, bounded only by available CPU. This turns a sequential fan-out into a parallel one.

Why not just use ; or a loop? The ; separator in SBT is strictly sequential. A shell loop calling

sbt service/assembly eight times launches eight separate SBT sessions — each one paying the full build load cost. all shares one loaded session and one Zinc compiler instance.

A Jenkins shared library function auto-discovers which services need assembly so the list never goes stale:

def assembleInParallelForCapCommons() {
    sh(script: '''#!/bin/bash
    set -e
    # Auto-discover services that enable AssemblyPlugin in build.sbt
    PROJECTS=$(grep -B5 '\\.enablePlugins.*AssemblyPlugin' build.sbt \
               | grep -E '^lazy val' \
               | sed -E 's/^lazy val ([a-zA-Z_]+).*/\\1/')
    SBT_CMD="all $(echo "$PROJECTS" | sed 's/$/\\/assembly/' | paste -sd ' ')"
    sbt "$SBT_CMD"''')
}

When a new service is added to build.sbt with AssemblyPlugin, it is automatically included in the assembly run — no pipeline change required.

Parallel Docker Image Builds

The Docker build stage was rewritten from a sequential loop to a fully parallel block:

// Before: sequential, one shared builder
finalServicesList.each { service ->
    containerImageTools.buildContainerImageWithBuilder(builderName, service, ...)
}

// After: parallel, per-service builders, fail-fast
def branches = finalServicesList.collectEntries { String service ->
    def svc = service
    ["Build ${svc}": {
        def svcBuilderName = containerImageTools.createBuilder(svc, ARTIFACT_VERSION, "linux/arm64")
        try {
            containerImageTools.buildContainerImageWithBuilder(svcBuilderName, svc, ARTIFACT_VERSION, [...], "linux/arm64")
        } finally {
            containerImageTools.removeBuilder(svcBuilderName)
        }
    }]
}
parallel branches + [failFast: true]

Notable improvements:

• Per-service builder isolation: A failure in one service does not corrupt others.
• failFast: true: The pipeline fails immediately on the first error rather than waiting for healthy builds to finish.
• linux/arm64: Our build nodes are ARM-based; building native-arch images avoids emulation overhead and produces images that run natively on ARM deployment targets.

As we’ll see in the next section, the Cap Build Service Selector plugin takes this further — letting developers choose which services to assemble on feature branches, skipping the rest entirely.

pigz for Parallel Compression

# Before
tar zcvf "../${service}.tgz" .

# After
tar -I pigz -cvf "../${service}.tgz" .

pigz is a parallel gzip implementation. On a multi-core build node it compresses fat JARs (200 MB+) using all available cores simultaneously, cutting the tar step from seconds to sub-second.

The Developer Tooling: Build Service Selector

One side effect of having a large mono-repo is that engineers sometimes need to rebuild only a subset of services — say, only the two services their branch actually affects. We built a small IntelliJ IDEA plugin (service-selector) that lets developers check off which services to include in the CI assembly step before committing.

The plugin:

• Auto-discovers all AssemblyPlugin-enabled services by parsing build.sbt at project open time.
• Provides a sidebar tool window with checkboxes and a pre-commit dialog that appears automatically before every VCS commit.
• Writes a buildservicelist file at the project root (one service directory name per line).

The Jenkins pipeline reads buildservicelist in the "Discover Services" stage. If the file is present and non-empty, only the listed services get assembled and pushed as Docker images. If it is absent or empty, everything builds — the default for master.

Another way of saving additional time — only assemble and package what you modified

This is a developer convenience feature rather than a core part of the remote cache story, but it compounds the benefit: a focused feature branch that touches two services skips the assembly of the other five entirely.

If you should need the source code please get in touch.

The End-to-End Build Flow

Putting it all together, a CI build on a feature branch now looks like this:

Jenkins (sbt2-java17 pod)
│
├─ Stage: Prepare Environment
│   └─ rm -rf target/out/.cache  ← wipe stale local Zinc state
│
├─ Stage: SBT Compile + Test  (remote caching transparent)
│   └─ sbt compile test
│       │
│       └─ For each sub-project (in parallel, up to CPU count):
│           1. Hash(sources + classpath + scalacOptions)
│           2. gRPC → bazel-remote-cache:9092 → ActionCache.GetActionResult
│              ├─ HIT:  stream .class files from CAS, skip Zinc
│              └─ MISS: run Zinc, upload results (master pushes; branches skip push)
│
├─ Stage: Assembly
│   └─ sbt "all paginator/assembly enricher/assembly ..."
│       └─ All 7 fat JARs assembled concurrently
│
├─ Stage: Archive  (tar -I pigz per service)
│
├─ Stage: Publish Artifact  (master only → Artifactory)
│
└─ Stage: Build & Push Container Images
    └─ parallel { per-service Docker builds, failFast, linux/arm64 }

Known Limitations and Follow-Ups

• SBT 2 is still in RC, but a final version should be published in the near future. Some ecosystem plugins still haven’t been published in a version compatible with SBT 2.
• Flattening repetitive layers of our Docker images that add unnecessary size to the final image.
• Re-enabling the scoverage plugin once it is working in our mono-repo context.

Contributing Back: Fixing a Cache Resilience Bug in SBT 2

Running SBT 2’s remote cache at scale on a real mono-repo inevitably surfaces edge cases that smaller setups never hit. We found one — and fixed it upstream.

The Bug

After deploying the full pipeline, builds started failing intermittently with:

[error] java.io.FileNotFoundException:
  .../target/out/value/sha256-16b04cae.../48.json (No such file or directory)

The failure was random — retrying the same build with zero code changes succeeded. The pattern: the Action Cache reported a hit (“I have results for these inputs”), but the corresponding CAS blob had been evicted or corrupted on the bazel-remote server. SBT tried to read the blob from a local path that syncBlobs was supposed to populate, found nothing, and threw a FileNotFoundException that propagated all the way up and killed the build.

By definition, a broken cache entry should mean “recompute” — not “fail.” A cache is an optimization; it should never reduce reliability.

I was able to find the root causes for the issue when corrupt / evicted / non existing “cache hit” were found and a patch was introduced into SBT’s code.

The Takeaway

If you are adopting SBT 2’s remote cache, this fix will ship in the next release after RC9. Until then, the intermittent FileNotFoundException on evicted CAS entries is the most likely failure you will encounter at scale — and a retry will succeed. Once the fix lands, the build will silently recompute instead of failing. I had the privilidge of contributing back to the project as part of the huge gains we’ve obtained thanks to the new capability.

Closing Thoughts

The headline change is simple: one line in build.sbt replaces a third-party S3 plugin and its entire credential apparatus.

But what makes this migration genuinely useful is the combination of changes discussed above, especially incrementally building and testing using remote caching.

None of these changes is revolutionary in isolation. Together, they compound into a build pipeline that is measurably faster, simpler to reason about, and free of the credential and plugin-compatibility debt we were carrying.

The bazel-remote server is the only piece of infrastructure to add. It is a single container, stateless (until you add a PVC), zero configuration beyond a disk path and a size limit, and it speaks a well-specified open protocol. If SBT ever moves away from the Bazel RE API, any other server that implements the same spec will work as a drop-in replacement — including remote services like BuildBuddy or EngFlow if you want a hosted option.

If you are running a large Scala mono-repo on SBT 1.x and grinding through slow CI builds, the migration path is well-defined and the payoff is immediate.

The bottom line: Our full CI pipeline went from ~25 minutes to ~10 minutes on a warm cache. On a focused feature branch with the service selector limiting assembly to two services, end-to-end time drops to about 7 minutes.

Remote caching isn’t just a performance optimization — it’s a shift in how CI works. Instead of rebuilding the world on every change, you reuse it — sometimes, that shift really does start with just one line.

Thanks for taking the time to fully read the article. My hope is that you can take both new theoretical knowledge and implement the same changes to save time.

If you have questions or have done a similar migration, the comments are open.