Why Performance Monitoring Should Come Before Performance Testing

When it comes to building reliable, scalable applications, performance testing is crucial, but it’s only part of the equation. Performance testing helps us measure how systems behave under certain conditions, but if you don’t know the baseline performance of your application, these tests can lead to inaccurate conclusions. That’s where performance monitoring comes in.

Performance monitoring is the ongoing process of observing your application’s behavior in a live environment. It’s not just about catching issues post-release ; it’s about gathering the right data before you even start writing performance tests. Without monitoring, you’re effectively shooting in the dark, unable to determine if your tests are measuring the right things.

Example of a Performance Monitoring dashboard in Datadog

In this article, I’ll walk you through why monitoring should be the first step in your performance strategy, how to set meaningful metrics, and what tools you can use to make the process as smooth as possible.

The Case for Starting with Performance Monitoring

Before you write a single performance test, you need to understand how your application currently performs in the wild. Many teams jump directly into testing without knowing their system’s baseline, leading to blind spots. This can result in over-testing areas that don’t need it, or worse, under-testing critical parts of your application.

Monitoring gives you insights into:

• Response times for key user interactions.
• Error rates that could be indicative of system instability.
• System resource usage (like CPU, memory, and network bandwidth) under typical workloads.

These insights guide your testing by showing you where your application needs improvement. For example, if monitoring reveals that your system consistently struggles with long-running queries during peak traffic hours, this is a clear indicator that you should focus your tests on database performance under heavy load.

An APM (Application Performance Monitoring) dashboard in New Relic

Let’s consider a team that began testing a web application’s front-end performance without monitoring it first. They ran automated Lighthouse audits and received poor performance scores, triggering panic and a major refactor. However, if they had first monitored their system, they would have seen that the real issue wasn’t the code but the third-party scripts slowing down the page. Tools like Datadog could have flagged this early, allowing them to address the real problem before any major code changes were even necessary.

Finding the Right Metrics to Measure

To conduct effective performance monitoring, you need to decide what metrics actually matter for your application. Too often, teams are overwhelmed by the sheer number of available metrics — leading to “monitoring noise” where important signals get drowned out by irrelevant data.

Here are a few key metrics that tend to be critical across most applications:

1. Latency: How long does it take for your system to respond to a request? This is often the most direct indicator of performance issues.
2. Throughput: How many requests can your system handle per second? Tracking throughput alongside latency gives you a clear picture of how your application scales.
3. Error Rates: What percentage of requests result in errors? If you’re seeing a spike here, you may need to look deeper into error logs or exception handling.
4. Resource Utilization: CPU, memory, and disk usage under typical workloads. High resource utilization without corresponding throughput gains could indicate inefficiency.

By defining thresholds for these metrics early on, you create a performance baseline. This baseline helps you set realistic goals for performance testing later.

Choosing Your Performance Monitoring Tools

A range of tools exists to help you capture performance metrics effectively, both for back-end and front-end monitoring. Let’s look at a few that stand out:

• Datadog: One of the most popular choices, especially for cloud-based environments. It provides monitoring for everything from server resource usage to real-time request traces. Datadog’s log management and dashboard features make it easy to track key metrics and detect performance bottlenecks.

Datadog’s front-end monitoring highlighting the performance impact of third-party scripts over time.

• Lighthouse: A great tool for front-end performance monitoring, Lighthouse can audit your site’s performance in terms of speed, accessibility, SEO, and best practices. It’s especially useful for single-page applications (SPA) where load times and interactivity can make or break the user experience.

• Sentry: Often thought of as just an error tracking tool, Sentry also has performance monitoring capabilities, especially around request traceability and identifying slow or failing parts of your system.

These tools help you collect real-time data that will inform your testing strategy. But before you dive into writing tests, it’s essential to define the thresholds you’re aiming for.

Setting Realistic Metrics and Thresholds

After you’ve monitored the performance of your application for a while, you’ll have data that shows your system’s normal behavior. Now, it’s time to define the metrics and thresholds that will guide your performance testing.

Start by looking at your baseline. For instance, if your average response time is 400ms under normal load, but it spikes to 800ms during peak traffic, you may want to set a threshold at 500ms to aim for in your tests.

Here’s an example of how you can set meaningful thresholds:

• Latency: 90% of all requests should be processed within 500ms.
• Error Rates: Should remain below 1% at all times.
• CPU Utilization: Should remain under 70% during peak load.

Datadog dashboard providing insights into both API response times and front-end load times, allowing teams to monitor performance across the entire stack.

These thresholds not only give you measurable goals for your tests but also provide a way to alert your team when real-world performance deviates from expectations.

Writing Performance Tests with Data-Driven Insights

Once you’ve established your monitoring baseline and set your thresholds, you can start writing performance tests that target real-world scenarios. At this point, testing becomes much more focused and meaningful.

Let’s take an example using k6 for load testing:

import http from 'k6/http';
import { check, sleep } from 'k6';

export let options = {
  stages: [
    { duration: '30s', target: 50 },
    { duration: '1m30s', target: 100 },
    { duration: '20s', target: 0 },
  ],
};

export default function () {
  let res = http.get('https://test-api.example.com');
  check(res, {
    'status was 200': (r) => r.status === 200,
    'response time was < 500ms': (r) => r.timings.duration < 500,
  });
  sleep(1);
}

This k6 script defines a scenario where we ramp up traffic over time, testing how well the system holds up as the number of requests increases. The script checks for response time (under 500ms) and the success of the requests (status 200), aligned with the thresholds set during monitoring.

You can also use k6 for front-end performance testing. For example, if monitoring with Datadog RUM has shown that page load times should stay under 2 seconds, you can create tests that simulate user behavior, navigating the site and ensuring load times stay within this limit.

import { sleep } from 'k6';
import { chromium } from 'k6/x/browser';

export const options = {
  scenarios: {
    browsing: {
      executor: 'constant-vus',
      vus: 10,
      duration: '1m',
    },
  },
};
export default async function () {
  const browser = chromium.launch();
  const page = browser.newPage();
  await page.goto('https://yourapp.com');
  await page.waitForSelector('#element');  // wait for an element to be visible
  await page.screenshot({ path: 'screenshot.png' });
  await browser.close();
  sleep(1);
}

This script uses k6’s browser extension to simulate a user visiting the front end, interacting with a page, and ensuring that key elements load in time. This kind of performance testing can validate whether the front-end performance metrics you’ve monitored continue to hold up under load.

Performance test results from k6 integrated with Datadog, tracking both API response times and front-end load times under simulated load.

Embedding Performance into the CICD Pipeline

To make performance testing a continuous practice, it’s essential to integrate your tests into the CICD pipeline. This allows your team to catch potential issues early in the development cycle.

Tools like Datadog, Sentry, and New Relic make it easier to monitor performance metrics after every deployment. By integrating these into your pipeline, you ensure that performance testing is a regular part of your development process, not just something you do before major releases.

For example, you can configure your pipeline to run a suite of k6 tests after every deployment, checking that both API and front-end performance meet the thresholds defined during your monitoring phase. Additionally, Datadog can track the results and alert you to any anomalies or failures. This way, performance becomes an integral part of your release process, not just a reactive measure.

stages:
  - test
performance_test:
  stage: test
  image: loadimpact/k6
  script:
    - k6 run --out datadog 'k6_test_script.js'

This example shows how a k6 performance test can be run automatically as part of the CI/CD pipeline, with results sent to Datadog for further analysis and monitoring.

Datadog tracking results of k6 performance tests as part of a CI/CD pipeline, allowing teams to monitor performance continuously.

By embedding performance testing and monitoring into your pipeline, you ensure that every release is scrutinized for performance regressions, giving your team confidence that the system remains robust under load.

Final Thoughts

Incorporating performance monitoring as the first step ensures you have a clear understanding of your system’s behavior before you write tests. By setting realistic thresholds based on real-world data, you’ll make your performance testing far more effective and targeted.

So, before you write that first test script, ask yourself: Do I truly understand how my system is performing today?