Short-Lived Application Testing Environment (SLATE) on EKS

If you’ve ever shared a single DEV environment with multiple teams, you know the pain. A new feature is deployed to the development environment, and unexpectedly, another team’s API tests start failing. Soon after, a different team deploys their branch, causing upstream integrations to break. What was once a smooth CI/CD pipeline has now turned into a daily cycle of trial and error. — “who broke DEV this time?”

As our engineering teams grew, this chaos only amplified. Developers were constantly stepping on each other’s toes, waiting for their turn to test, or trying to debug issues that weren’t even theirs. Having one shared environment wasn’t cutting it anymore.

The obvious fix — giving each team their own dedicated DEV environment — looked great on paper, but in practice it is too expensive, too hard to maintain, and does not scale for a fast growing org like Quince. For larger teams with multiple developers working on different features simultaneously, the problem only intensified. Each developer often needed their own environment to test and take multiple features live in parallel, further increasing infrastructure costs and operational complexity. We needed a solution that could provide each pull request with an isolated sandbox environment, without the overhead of managing dozens of long-lived environments.

That’s when at Quince we decided to build SLATE — lightweight, short-lived ephemeral environments that can spin up on demand for every feature and tear down just as easily.

Developers can preview features, run integration tests, and validate releases in true production parity — without waiting on shared infra or impacting anyone else.

The result? Faster feedback loops, stable environments, and a developer experience that actually scales with the size of our teams.

Challenges

Before diving into implementation, it’s worth recapping the pain points that pushed us here:

• Shared environment instability made feature testing slow and unreliable.
• Parallel development within the same team and across multiple teams forced sequential testing and coordination delays.
• Limited infra capacity meant we were always balancing stability vs. speed.

We needed a model that allowed true parallelism, production parity, and self-serve provisioning — without blowing up costs or cluster complexity.

Design Requirements

Our design goals were shaped around developer autonomy and operational safety. We wanted something that just worked — predictable, governed and easy to use.

• On-demand creation: PR-tied environments that can be spawned up manually, with strict TTL-based auto-cleanup. On-demand provisioning environments per PR gives developers fast, isolated feedback loops (tests, manual QA, demos) without waiting for shared env availability. TTL-based cleanup prevents resource leakage and keeps cloud costs and cluster complexity under control by ensuring environments are short-lived unless explicitly extended.
• Isolation: Each environment runs in its own namespace, argo-cd project, ingress, and shareable feature url. Also, parity with production reduces “works on my machine” incidents by validating the feature against the same configuration, routing, and security constraints. Isolated namespaces and Argo CD projects limit blast radius and make rollback/teardown deterministic. A shareable URL enables reviewers and automated tests to interact with the exact running system.
• Dependency-aware fan-out: A dependency graph defines relationships between services. Developers can choose which dependencies get deployed and even pin versions. Many features require multiple services to run together. A dependency graph automates the correct set of services, avoiding manual, error-prone wiring. Selective fan-out and version pinning let developers test with realistic dependency combinations (e.g., a stable backend with a feature branch frontend) and isolate failures to the intended change.
• Quota governance: Each team has a resource quota enforced before provisioning, based on estimated fan-out. Quotas prevent a single team or runaway PRs from exhausting cluster capacity and impacting others. Evaluating quotas up front enforces fair resource allocation, enables capacity planning, and protects production/critical workloads from noisy neighbors.
• Cost awareness: All environments are tagged for cost attribution and expire aggressively by default. Tagging enables chargeback/showback and helps teams understand the cost of CI/CD practices. Aggressive expiration limits idle spend and encourages responsible resource use; teams can request extensions when genuinely needed.
• Simple update loops: Developers can refresh or modify their environments directly from Bitbucket pipelines after code changes. Seamless update paths keep environments synchronized with the latest code without manual intervention. Integrating with the existing CI (Bitbucket Pipelines) keeps the process familiar, reduces context switching, and enables deterministic, repeatable environment updates for iterative development and testing.

These goals laid the foundation for a system that could scale seamlessly across teams and support parallel development workflows, while remaining fully automated and low-touch.

Implementation

Our ephemeral environment platform is built around a modular control plane and a convention-driven workflow that balances developer flexibility with operational safety.

Control Plane

At the heart of the system is a Flask-based microservice that acts as the control plane. It exposes APIs for all major operations:

• Creating new environments
• Generating dependency YAMLs
• Syncing and enforcing quotas
• Cleaning up expired or idle resources

This service is the single source of truth for environment orchestration, coordinating provisioning across Kubernetes, ArgoCD, and AWS infrastructure components.

Dependency Modeling

We introduced a canonical dependency.yaml to define service-to-service relationships. It captures both direct and transitive dependencies, allowing the system to construct a complete dependency graph for any given service. This flexibility lets developers selectively enable or disable subsets of upstream services as needed based on the product feature that needs to be tested.

When a developer requests an environment, the control plane expands that graph into a normalized YAML. Developers can:

• Toggle which dependencies to include.
• Pin builds or versions for specific services.
• Based on the direct dependencies enabled, transitive automatically gets enabled.

main: service A
branch: feature/<jira-ticket-id>
pull-request: pr-88
dependencies:
- name: service B
 branch: master
 build: latest
 enable: true
- name: service C
 branch: feature/<jira-ticket-id>
 build: build-22
 enable: true
- name: service D
 branch: master
 build: latest
 enable: false
transitive-dependencies:
- name: service E
 branch: master
 build: latest
- name: service F
 branch: master
 build: latest
.... and so on

This gives each environment deterministic reproducibility with just enough flexibility to test real-world scenarios.

Provisioning Workflow

The provisioning workflow is fully automated and follows a predictable sequence:

1. Namespace creation: Each product feature gets its own Kubernetes namespace for isolation.
2. Helm + ArgoCD integration: Helm values are customized per service, and ArgoCD Application specs are generated dynamically.
3. Infrastructure bring-up: Ingress (ALB + TLS), unique application URL, Route53 DNS records, ESO, and a dedicated ArgoCD project are provisioned.
4. Metadata persistence: Each environment’s details — status, TTL, owner, team, and “quota used” — are stored in DynamoDB.

This approach ensures every ephemeral environment is complete, isolated, and fully auditable.

Quota and Cleanup

To prevent runaway provisioning, we built a quota and cleanup system on top of DynamoDB.

• Preflight checks: Before provisioning, the control plane computes the estimated service count from the dependency graph and validates it against the team’s quota. For example:

1. A standalone service with no downstream dependencies might have a quota of 2, meaning only 2 SLATE environments with one deployment per SLATE can exist at the same time for that service.
2. A service that depends on 10 downstream services can have a higher effective quota say “20” — to accommodate the additional resources needed for all dependent services to run.

• Developers can also selectively enable or disable specific downstream services, which adjusts the total number of SLATE environments provisioned. Depending on the combination of services enabled, a team might have anywhere between 2 and 20 SLATE environments running in parallel at any given time for this example service. Key takeaway: The quota isn’t fixed per service; it scales based on the number of dependencies and which services are enabled, ensuring realistic testing environments without over-provisioning resources.
• Scheduled cleanup: A background job periodically scans for expired environments and cleans them up — deleting ArgoCD Applications, DNS entries, and Kubernetes namespaces, before marking them expired in DynamoDB.

This blend of automation and governance keeps our clusters lean and predictable without human intervention.

Developer Workflow

The developer experience was intentionally kept very simple — no YAML wrangling, no kubectl gymnastics. Everything starts from a self-service developer portal that abstracts the complexity behind a clean UI.

Self service developer portal

Dependency Information Generation

Developers fill out a short form specifying:

• Repo name [Dynamic Dropdown]
• Service name [Dynamic Dropdown]
• Feature branch [Dynamic Dropdown]
• PR number [Dynamic Dropdown]

Based on the above provided information, dependency yaml is generated and post customizing it with the available specification, it is ready to feed in the next step.

Environment Provisioner

Developers fill out a short form specifying:

• Team Name [Dropdown]
• Dependency File Upload [File Input]
• Time to Live [TTL] [Dropdown]

Once submitted, the portal calls the control plane microservice, which:

1. Validates the requested input.
2. Checks for available quota for the team.
3. Generates Kubernetes Manifests for Namespace, ArgoCD Project and ArgoCD Applications.
4. Applies the generated Kubernetes Manifest, in the order of generation.
5. Configures ingress, DNS, and relevant dependencies.
6. Returns links to the ArgoCD project and a host url.

SLATE user flow

Within a few minutes (< 10mins), developers get a fully functional environment — isolated, production-like, and ready to test.

Updating Environments

After pushing code changes, developers simply trigger a Bitbucket pipeline, with few custom fields, to re-sync their environment. This updates both the main service and any impacted dependencies automatically, ensuring the environment always reflects the latest commit state.

Teardown and Expiry

Every environment is ephemeral by design — it only lives as long as it needs to.

• Automatic cleanup: Once the TTL expires, a scheduled job decommissions all resources and updates the record in DynamoDB.
• Manual teardown: Developers can also delete environments on demand directly from the portal.

This self-regulating lifecycle keeps clusters clean, costs predictable, and the developer experience frictionless.

Conclusion

Building SLATE fundamentally changed how we develop and ship software. What used to be a bottleneck is now a superpower.

• Decreased lead time for changes: Teams ship faster by eliminating shared-env contention.
• Lower change failure rate: Production-parity testing catches integration issues earlier.
• Independent, parallelized testing: Every PR gets its own isolated sandbox.
• Governed, cost-aware scale: Quotas, TTLs, and tagging keep usage predictable without sacrificing autonomy.

By investing in SLATE , we didn’t just improve our CI/CD pipeline — we made our development process scalable, reliable, and fast enough to keep up with our teams.

Roadmap

Our next goals focus on extending SLATE beyond Kubernetes resources to fully provisioned environments for each PR:

1. Ephemeral AWS resources: Automatically create and tear down cloud resources (DB, S3, REDIS etc.) per PR to achieve end-to-end testing parity using Crossplane+Terraform
2. Fresh data seeding: Provide each feature with an isolated set of stateful data, ensuring realistic test scenarios without interfering with other environments.

Author: Vatsal Dhar