Load Testing a REST API with Artillery

A step-by-step walkthrough of benchmarking a validation API using Artillery’s multi-phase load profiles and cloud reporting

Overview

This article documents how I used Artillery to benchmark the performance of a healthcare data validation REST API. The goal was not to meet a specific SLA, but to establish a performance baseline — understanding how the API behaves under increasing load, where response times degrade, and what a realistic throughput ceiling looks like.

What Is Artillery?

Artillery is an open-source, Node.js-based load testing and performance testing toolkit designed for testing HTTP APIs, WebSocket services, and other backends. Key features include:

• Declarative YAML configuration — define test scenarios, phases, and targets without writing boilerplate code.
• Custom processors — hook into the request lifecycle using JavaScript to inject dynamic payloads, set headers, or handle authentication.
• Multi-phase load profiles — model real-world traffic patterns (warm-up → ramp-up → sustained peak) in a single test run.
• Cloud reporting — publish results to Artillery Cloud for visualization, trend analysis, and sharing.

Artillery is a strong choice when you need a lightweight, scriptable load tester that can run from a CI pipeline or a local developer machine without infrastructure overhead.

The Target API

The endpoint under test is a real-time validation service for EDI (Electronic Data Interchange) payloads:

POST /application/validation-request

Each request submits a complete JSON payload. On the server side, a full set of validation rules is executed against the claim — making this a compute-intensive operation rather than a simple CRUD call. This is an important characteristic: the endpoint is not I/O-bound, so response times are sensitive to concurrency in a way that a plain database read would not be.

Installation

Artillery is installed globally via npm:

npm install -g artillery@latest

Project Structure

The project is minimal by design:

• load-test.yml — the Artillery test definition: target URL, load phases, and scenario.
• processor.js — a custom JavaScript hook that injects the request body and authentication headers before each request is sent.
• payload.json — the full JSON document sent on every request.
• secrets.yml — Basic Auth credentials, kept out of version control.

The Payload

The payload is a fully populated EDI document which has been converted to JSON format. The same payload is sent on every request. This is intentional for a benchmark: it isolates server-side processing time from payload variance, giving consistent and reproducible results.

The Custom Processor

Artillery’s beforeRequest hook allows a JavaScript function to mutate the request before it is sent. The processor serves two purposes:

1. Inject the JSON payload — payload.json is loaded once at startup and attached to every request body.
2. Set authentication and content headers — the API requires Basic Auth, which is Base64-encoded and injected into the Authorization header.

const myJson = require('./payload.json');

function setJSONBody(req, context, ee, done) {
    req.body = JSON.stringify(myJson);
    const username = "api_user";
    const password = "P@ssw0rd123!";
    const base64Credentials = Buffer.from(`${username}:${password}`).toString("base64");
    req.headers = {
      "Accept": "application/json",
      "Content-Type": "application/json",
      "Authorization": `Basic ${base64Credentials}`,
    };
    return done();
}
module.exports = { setJSONBody };

The processor is referenced in the test config via processor: "./processor.js" and invoked per-request via beforeRequest: "setJSONBody".

The Test Definition

config:
  target: "https://my-api-host.com"
  tls:
    rejectUnauthorized: false
  phases:
    - duration: 60
      arrivalRate: 1
      name: "Warm up phase"
    - duration: 60
      arrivalRate: 1
      rampTo: 5
      name: "Ramp up phase"
    - duration: 60
      arrivalRate: 5
      name: "Sustained peak phase"
  processor: "./processor.js"
  defaults:
    http:
      timeout: 30

scenarios:
  - flow:
      - post:
          url: "/application/validation-request"
          beforeRequest: "setJSONBody"

The test runs for 3 minutes across three phases:

Warm-up (60s, 1 RPS) — allows the server (JVM, connection pools, caches) to reach a steady state before measurements begin. Hitting a cold server at full load produces misleading results.

Ramp-up (60s, 1 → 5 RPS) — gradually increases load to observe the inflection point where latency begins to climb. This is the most diagnostic phase of the test.

Sustained peak (60s, 5 RPS) — holds peak load steady to confirm the server can maintain throughput without degradation over time.

Two configuration details worth noting: tls.rejectUnauthorized: false was set because the performance environment uses a self-signed TLS certificate. defaults.http.timeout: 30 ensures that requests hanging beyond 30 seconds are counted as failures rather than silently blocking the test.

Running the Test

Artillery Cloud provides a hosted dashboard for results. I created an account using my GitHub login at app.artillery.io and generated an API key from the account settings.

The test is run with cloud recording enabled:

artillery run load-test.yml --record --key <YOUR_ARTILLERY_API_KEY>

The --record flag publishes the run to Artillery Cloud where results are available immediately after the test completes.

Results

The test ran for 3 minutes 10 seconds and generated 540 virtual users, each submitting one POST request.

Summary

[embed]

A note on HTTP 200 responses: All 534 completed requests returned HTTP 200, confirming the validation pipeline executed successfully end-to-end for each. The 6 failures were socket-level timeouts — the server did not respond within the configured 30-second window — rather than application errors.

HTTP Response Time Distribution

[embed]

Artillery Cloud dashboard showing the load summary, response time distribution, and the 6 ERR_SOCKET_TIMEOUT errors recorded during the test run.

Key Observations

The service sustained 98.89% success rate under peak load. 534 out of 540 requests completed successfully with HTTP 200. The 6 socket timeouts represent a 1.11% error rate — low, but worth noting as a signal of where the server begins to struggle.

Response times climbed significantly during ramp-up and peaked in the sustained phase. The Artillery Cloud chart clearly shows p95 latency rising steadily through the ramp-up phase and continuing to increase once the load held at 5 RPS. This is the expected pattern for a compute-intensive validation pipeline under concurrent load.

The max response time of 7.4s is the most significant finding. While the median (p50) sits at a reasonable 714ms, the long tail — p99 at 1.6s and a max of 7.4s — indicates that a small number of requests experienced severe delays. This is consistent with thread pool saturation or GC pressure on the server during the sustained peak phase, and it is likely where the 6 timeouts originated.

Peak throughput reached 8 req/s. This exceeds the configured 5 RPS arrival rate, which reflects Artillery dispatching slightly above the target during the ramp-up to peak transition — a normal artifact of the phase handoff.

Conclusions and Next Steps

This benchmark establishes the following baseline for the dental claim validation API:

• The service handles 5 RPS with a 98.89% success rate and a p95 latency of ~944ms.
• The p99 of 1.6s and max of 7.4s reveal a long tail that warrants investigation — likely thread contention or GC pressure during sustained load.
• The 6 socket timeouts mark the beginning of the failure threshold; at slightly higher load this error rate would likely increase.

To extend this benchmark further:

1. Find the breaking point — re-run with a higher rampTo (e.g., 10–20 RPS) to identify the throughput level at which the error rate climbs sharply.
2. Extend the sustained phase — run the peak phase for 5–10 minutes to determine whether the timeouts are transient (GC pause, cold cache) or systemic.
3. Correlate with server-side metrics — pair Artillery results with JVM heap, thread pool utilization, and CPU metrics during the sustained phase to pinpoint what causes the long-tail latency.
4. Add Artillery checks — define pass/fail thresholds in load-test.yml (e.g., p95 < 1000ms, errorRate < 1%) so the test can gate deployments in a CI pipeline.

Load test performed using Artillery 2.0.31.