Winning the Super Bowl: How We Built Our Load Testing Platform @ Underdog

One good boy hard at work

Suffering from Success

The first week of the NFL season is an exciting time at Underdog, and one of the busiest weeks of the year for us. We experience some of the highest levels of traffic across our platform during this period, and it’s crucial that our systems perform well so we can deliver a stellar experience for our users.​

As the Underdog platform has grown in popularity year over year, so has the load on all of our systems. It’s a great problem to have. But when our core monolithic API service experienced performance issues during the first NFL game of the year as users flocked to our app, we realized we didn’t have a reliable way to determine how much load our services could handle. With the Super Bowl (our busiest day of the year) on the horizon, we needed a way to simulate traffic to our API so we’d be ready.

Load-Testing-as-a-Service

The first choice we had to make was the “build vs. buy” decision. Given our time constraints and what we had on the line, “buy” initially seemed like the better option. But what we quickly realized is that load testing is highly specific to your company and your product. Load testing tools need to be flexible enough to handle nuanced use cases, like endpoints which require user-specific authentication, deciding how and when to bypass HTTP request caching, and emulating request patterns that happen during large traffic spikes.

These kinds of unique situations are common in load testing, and SaaS load-testing products struggle to be flexible enough to meet them all. The “build” option was the clear alternative, but it came with its own set of challenges:

• How do we generate traffic to our service?
• How do we host a traffic generation service in a central location?
• How do we ensure this service has enough compute resources to generate the load we need?
• How do we make creating and running load tests simple for product engineers?

Ultimately, we decided to build. Next, I’ll walk through how we answered each question.​

How do we generate traffic?

Many great load testing tools exist, including Locust, JMeter, Taurus, k6, and even Postman. Each tool has its strengths and trade-offs. For our use case, k6 was the best fit.​

k6, an open-source load testing tool that was acquired by Grafana in 2021, gives us the tools to define tests that simulate the exact HTTP traffic patterns our API sees in real-world scenarios, like when a user opens the Underdog app. Tests are written in JavaScript or TypeScript, which are common languages for our platform engineering team. Under the hood, k6 uses goja to execute JavaScript in a Go runtime, which delivers significantly better performance than Node.js.​

How do we host our traffic generation service?

k6 tests are normally executed via the k6 CLI, often from individual developers’ machines. This presented a couple of problems for us:

• The load we need to generate often exceeds the CPU and memory available on a single engineer’s machine.
• Our goal was to eventually hand off this tool to our product engineers to use, so the experience needed to be as frictionless as possible.

Fortunately, k6 provides a Kubernetes operator. That gave us the ability to run tests in EKS rather than on local laptops. It also answered our third question about resource-scaling: in AWS, we can scale vertically by increasing pod resources and horizontally by splitting runs across multiple pods.​

How do we make creating and running load tests easy?

While the k6 operator solved most of our core technical challenges, it wasn’t something we could directly hand off to product engineers. Test runs are defined with the operator’s TestRun CRD, and new runs are triggered by applying that CRD to the cluster. If we wanted product engineers to adopt this tool, we needed an abstraction that shielded engineers from Kubernetes and let them focus on what they really care about: writing and running tests.​

Building that abstraction was the fun part. Given our time constraints, we decided to start small and iterate. Our first version would lean on an existing tool both our platform team and our product engineers were already very familiar with: GitHub Actions. Using the workflow_dispatch trigger, engineers could kick off workflows directly in GitHub, giving us a basic UI. In the workflow, we stitched together bash scripts to read user inputs, create TestRun CRDs, and apply them to the cluster. We also created a simple Datadog dashboard so engineers could observe load test behavior in real time. This was our first “minimal viable product”. We called it Dogpile.

Introducing: Dogpile

Once we proved the feasibility and desirability of our load testing platform, we continued iterating on it throughout the NFL season. Our first users were on our infrastructure team; they wanted to load test scenarios in our production environment to ensure their database configuration changes wouldn’t impact our ability to handle Super Bowl traffic. Along the way, their feedback helped us improve the Dogpile developer experience.​

Let’s take a quick tour of what Dogpile looks like today.​

Writing load tests

Engineers write k6 tests directly within the Dogpile repository under /tests. Because k6 tests can be run locally with the k6 CLI, the inner loop is straightforward: write tests, run tests locally, and iterate.​

Once the engineer is satisfied with their test, they push their code and open a PR. They can also run tests from their branch in Dogpile to verify its behavior in the cloud environment. After testing and code review, they merge. Dogpile’s CI/CD pipeline then makes the new tests available for the k6 operator in-cluster.​

If you’re a product engineer, this flow should look familiar. We intentionally designed Dogpile’s SDLC to closely mirror a normal product SDLC, making it easier for engineers to contribute to our growing catalog of load tests.​

Running load tests

Once a test has been created, how do engineers run it? We eventually outgrew the GitHub Actions UI and created a small web interface where engineers can trigger test runs.

Triggering a load test through our web UI

We still use GitHub Actions as our orchestration layer, but we’ve added some quality-of-life updates to the test-run workflow, like converting our bash scripts to TypeScript, adding an output summary, and streaming pod logs to make failed test runs easier to debug. Because test runs happen in GitHub Actions, engineers also have the ability to trigger runs via the GitHub CLI.​

Our primary UI for viewing test runs and results is still a Datadog dashboard, and it has improved over time as the platform has matured.​

Snapshot of production load testing we performed prior to this year’s Super Bowl

​How’s it going?

We used Dogpile to measure critical parts of our API infrastructure leading up to the Super Bowl. Load testing helped us evaluate database connection pooling changes by targeting endpoints that drive a large share of our query volume. We also used it to simulate load from common app interactions, such as opening the app and viewing the main lobby. Most importantly, we achieved our goal of delivering a smooth Super Bowl experience for our users.​

The biggest long-term value may be cultural: Dogpile has dramatically lowered the barrier for engineers to create and run load tests. In doing so, it has helped make load testing — including production load testing — a normal and valuable part of how we ensure reliability during our busiest moments.​

We’re excited to see how Dogpile evolves and what problems engineers solve with it next.