Making Microservices Resilient: Circuit Breakers, Retries & Timeouts

A comprehensive guide to building fault-tolerant distributed systems with practical Express.js implementations

In the world of microservices, failure is not an exception — it’s the norm. Networks flicker, services crash, databases timeout, and downstream dependencies become temporarily unavailable. If your system treats every failure as catastrophic, you’re one bad day away from a complete outage.

This is where resilience patterns come in. They’re not just nice-to-haves; they’re essential survival mechanisms that separate production-ready microservices from fragile prototypes.

Today, we’ll dive deep into three fundamental resilience patterns: Retries, Timeouts, and Circuit Breakers and build working implementations in Express.js that you can use in production.

Why Resilience Matters in Microservices?

Imagine you’re running an e-commerce platform with separate services for inventory, payments, and order processing. Your order service calls the payment service, which calls a third-party payment gateway. If that gateway takes 30 seconds to timeout, your entire request hangs. Multiply that by 100 concurrent requests, and suddenly your service is drowning.

The Cascade Effect: In distributed systems, one slow or failing service can trigger a domino effect, bringing down multiple dependent services. Without resilience patterns, a minor hiccup becomes a major incident.

Resilience patterns solve this by:

• Preventing cascade failures from spreading across your architecture
• Gracefully degrading functionality instead of complete outages
• Recovering automatically from transient failures
• Protecting resources by preventing unnecessary load on failing services

Pattern #1: Timeouts

A timeout is the simplest resilience pattern: don’t wait forever for a response. If a service doesn’t respond within a reasonable time, abort the request and move on.

The Problem: Without timeouts, your service can become unresponsive waiting for slow dependencies. Thread pools get exhausted, memory fills up, and soon you’re serving errors to everyone.

The Solution: Set aggressive timeouts that fail fast. It’s better to return an error quickly than to hang indefinitely.

Timeout Implementation in Express

const axios = require('axios');

// Create axios instance with timeout
const apiClient = axios.create({
  timeout: 5000, // 5 second timeout
  headers: {
    'Content-Type': 'application/json'
  }
});

// Express route with timeout handling
app.get('/api/products/:id', async (req, res) => {
  try {
    const response = await apiClient.get(
      `https://inventory-service.com/products/${req.params.id}`
    );

    res.json(response.data);

  } catch (error) {
    if (error.code === 'ECONNABORTED') {
      // Timeout occurred
      console.error('Request timeout', error);
      return res.status(504).json({ 
        error: 'Service temporarily unavailable' 
      });
    }

    // Other errors
    res.status(500).json({ error: 'Internal server error' });
  }
});

Pro Tip: Set different timeouts for different operations. Database queries might need 2 seconds, while external API calls might need 10 seconds. Match your timeout to the expected latency.

Pattern #2: Retries

Networks are unreliable. A request might fail due to a temporary network glitch, but succeed if you try again. Retries automatically re-attempt failed requests with smart backoff strategies.

The Problem: Not all failures are permanent. A brief network hiccup or a service restarting shouldn’t result in a failed user request. But naive retry logic can make things worse by overwhelming already-struggling services.

The Solution: Implement exponential backoff with jitter: wait progressively longer between retries, with some randomness to prevent thundering herd problems.

Retry Implementation with Exponential Backoff

const axios = require('axios');

// Retry configuration
const RETRY_CONFIG = {
  maxRetries: 3,
  baseDelay: 1000, // 1 second
  maxDelay: 10000, // 10 seconds
  retryableStatusCodes: [408, 429, 500, 502, 503, 504]
};

// Calculate delay with exponential backoff + jitter
function calculateDelay(attempt) {
  const exponentialDelay = RETRY_CONFIG.baseDelay * Math.pow(2, attempt);
  const delayWithMax = Math.min(exponentialDelay, RETRY_CONFIG.maxDelay);

  // Add jitter: random value between 0 and delay
  const jitter = Math.random() * delayWithMax;

  return delayWithMax + jitter;
}

// Sleep utility
const sleep = (ms) => new Promise(resolve => setTimeout(resolve, ms));

// Retry wrapper function
async function retryRequest(requestFn, config = RETRY_CONFIG) {
  let lastError;

  for (let attempt = 0; attempt <= config.maxRetries; attempt++) {
    try {
      const response = await requestFn();

      // Success!
      console.log(`Request succeeded on attempt ${attempt + 1}`);
      return response;

    } catch (error) {
      lastError = error;

      // Check if error is retryable
      const isRetryable = 
        error.response && 
        config.retryableStatusCodes.includes(error.response.status);

      const isLastAttempt = attempt === config.maxRetries;

      if (!isRetryable || isLastAttempt) {
        // Don't retry
        console.error(`Request failed permanently`, error.message);
        throw error;
      }

      // Calculate delay and wait
      const delay = calculateDelay(attempt);
      console.log(`Retry attempt ${attempt + 1} failed. Retrying in ${delay}ms...`);

      await sleep(delay);
    }
  }

  throw lastError;
}

// Usage in Express route
app.post('/api/orders', async (req, res) => {
  try {
    const order = await retryRequest(async () => {
      return await axios.post(
        'https://order-service.com/orders',
        req.body,
        { timeout: 5000 }
      );
    });

    res.status(201).json(order.data);

  } catch (error) {
    res.status(500).json({ 
      error: 'Failed to create order after retries' 
    });
  }
});

Important: Only retry idempotent operations (operations that produce the same result when called multiple times). Never retry non-idempotent operations like payment processing without proper idempotency keys.

Pattern #3: Circuit Breaker

The circuit breaker pattern prevents your application from repeatedly trying to execute an operation that’s likely to fail. Like an electrical circuit breaker, it “opens” when failures exceed a threshold, giving the failing service time to recover.

The Problem: When a downstream service is failing, continuing to send requests wastes resources and makes recovery harder. You’re essentially performing a denial-of-service attack on your own infrastructure.

The Solution: Monitor failures and “trip” the circuit when a threshold is exceeded. While open, fail fast without making actual calls. After a timeout, enter a “half-open” state to test if the service has recovered.

Circuit Breaker Implementation

class CircuitBreaker {
  constructor(options = {}) {
    this.failureThreshold = options.failureThreshold || 5;
    this.successThreshold = options.successThreshold || 2;
    this.timeout = options.timeout || 60000; // 60 seconds

    this.state = 'CLOSED'; // CLOSED, OPEN, HALF_OPEN
    this.failureCount = 0;
    this.successCount = 0;
    this.nextAttempt = Date.now();
  }

  async execute(fn) {
    if (this.state === 'OPEN') {
      if (Date.now() < this.nextAttempt) {
        // Circuit is open, fail fast
        throw new Error('Circuit breaker is OPEN');
      }

      // Try half-open state
      this.state = 'HALF_OPEN';
      console.log('Circuit breaker entering HALF_OPEN state');
    }

    try {
      const result = await fn();

      // Success!
      return this.onSuccess(result);

    } catch (error) {
      // Failure
      return this.onFailure(error);
    }
  }

  onSuccess(result) {
    this.failureCount = 0;

    if (this.state === 'HALF_OPEN') {
      this.successCount++;

      if (this.successCount >= this.successThreshold) {
        // Recovered! Close the circuit
        this.state = 'CLOSED';
        this.successCount = 0;
        console.log('Circuit breaker CLOSED - service recovered');
      }
    }

    return result;
  }

  onFailure(error) {
    this.failureCount++;
    this.successCount = 0;

    if (
      this.failureCount >= this.failureThreshold || 
      this.state === 'HALF_OPEN'
    ) {
      // Open the circuit
      this.state = 'OPEN';
      this.nextAttempt = Date.now() + this.timeout;
      console.log(
        `Circuit breaker OPEN - will retry at ${new Date(this.nextAttempt)}`
      );
    }

    throw error;
  }

  getState() {
    return {
      state: this.state,
      failureCount: this.failureCount,
      successCount: this.successCount,
      nextAttempt: this.state === 'OPEN' ? new Date(this.nextAttempt) : null
    };
  }
}

// Create circuit breaker instance
const paymentServiceBreaker = new CircuitBreaker({
  failureThreshold: 5,    // Open after 5 failures
  successThreshold: 2,    // Close after 2 successes in half-open
  timeout: 30000           // Wait 30s before half-open
});

// Usage in Express route
app.post('/api/payments', async (req, res) => {
  try {
    const payment = await paymentServiceBreaker.execute(async () => {
      return await axios.post(
        'https://payment-service.com/process',
        req.body,
        { timeout: 5000 }
      );
    });

    res.json(payment.data);

  } catch (error) {
    if (error.message === 'Circuit breaker is OPEN') {
      return res.status(503).json({ 
        error: 'Payment service temporarily unavailable' 
      });
    }

    res.status(500).json({ error: 'Payment processing failed' });
  }
});

// Health check endpoint
app.get('/health/circuit-breakers', (req, res) => {
  res.json({
    paymentService: paymentServiceBreaker.getState()
  });
});

Combining Patterns: The Power Trio

These patterns are most effective when used together. Here’s how to combine them in a production-ready service:

const axios = require('axios');

// Import our CircuitBreaker class
const { CircuitBreaker } = require('./circuit-breaker');

class ResilientServiceClient {
  constructor(baseURL, options = {}) {
    this.client = axios.create({
      baseURL,
      timeout: options.timeout || 5000
    });

    this.circuitBreaker = new CircuitBreaker({
      failureThreshold: options.failureThreshold || 5,
      successThreshold: options.successThreshold || 2,
      timeout: options.breakerTimeout || 60000
    });

    this.retryConfig = {
      maxRetries: options.maxRetries || 3,
      baseDelay: options.baseDelay || 1000,
      maxDelay: options.maxDelay || 10000
    };
  }

  async request(config) {
    return await this.circuitBreaker.execute(async () => {
      return await this.retryRequest(config);
    });
  }

  async retryRequest(config) {
    let lastError;

    for (let attempt = 0; attempt <= this.retryConfig.maxRetries; attempt++) {
      try {
        return await this.client.request(config);
      } catch (error) {
        lastError = error;

        const isRetryable = this.isRetryable(error);
        const isLastAttempt = attempt === this.retryConfig.maxRetries;

        if (!isRetryable || isLastAttempt) {
          throw error;
        }

        await this.sleep(this.calculateDelay(attempt));
      }
    }

    throw lastError;
  }

  isRetryable(error) {
    if (!error.response) return true; // Network errors

    const retryableStatus = [408, 429, 500, 502, 503, 504];
    return retryableStatus.includes(error.response.status);
  }

  calculateDelay(attempt) {
    const exponential = this.retryConfig.baseDelay * Math.pow(2, attempt);
    const capped = Math.min(exponential, this.retryConfig.maxDelay);
    const jitter = Math.random() * capped;
    return capped + jitter;
  }

  sleep(ms) {
    return new Promise(resolve => setTimeout(resolve, ms));
  }

  getHealth() {
    return this.circuitBreaker.getState();
  }
}

// Usage
const orderService = new ResilientServiceClient(
  'https://order-service.com',
  {
    timeout: 5000,
    maxRetries: 3,
    failureThreshold: 5
  }
);

app.get('/api/orders/:id', async (req, res) => {
  try {
    const response = await orderService.request({
      method: 'GET',
      url: `/orders/${req.params.id}`
    });

    res.json(response.data);
  } catch (error) {
    res.status(503).json({ error: 'Service unavailable' });
  }
});

Key Takeaways

• Timeouts prevent hanging: Always set timeouts on external calls. Fail fast is better than fail slow.
• Retries handle transient failures: Use exponential backoff with jitter. Only retry idempotent operations.
• Circuit breakers prevent cascade failures: Stop calling failing services to give them time to recover.
• Combine all three patterns: Timeout → Retry → Circuit Breaker provides comprehensive resilience.
• Monitor everything: Track circuit breaker states, retry counts, and timeout frequencies to understand system health.
• Design for failure: Assume every dependency will fail. Build systems that degrade gracefully.

Monitoring Your Resilience Patterns

Implementing these patterns is just the first step. You need visibility into how they’re performing:

const metrics = {
  timeouts: 0,
  retries: 0,
  circuitBreakerTrips: 0,
  successfulRequests: 0,
  failedRequests: 0
};

// Add to your CircuitBreaker class
onFailure(error) {
  this.failureCount++;
  metrics.failedRequests++;

  if (this.failureCount >= this.failureThreshold) {
    metrics.circuitBreakerTrips++;
    console.error('Circuit breaker OPENED', {
      service: this.serviceName,
      failureCount: this.failureCount,
      timestamp: new Date()
    });
  }

  throw error;
}

// Metrics endpoint
app.get('/metrics', (req, res) => {
  res.json({
    ...metrics,
    uptime: process.uptime(),
    successRate: (
      metrics.successfulRequests / 
      (metrics.successfulRequests + metrics.failedRequests)
    ).toFixed(4)
  });
});

Send these metrics to your monitoring system (Prometheus, DataDog, New Relic) to create dashboards and alerts.

Building Truly Resilient Systems

Resilience isn’t a feature you add at the end — it’s a fundamental architectural decision. The patterns we’ve covered today form the foundation of fault-tolerant microservices:

• Timeouts ensure your services don’t hang waiting for slow dependencies
• Retries recover from transient failures automatically
• Circuit breakers prevent cascade failures and protect your infrastructure

But remember: these are just the beginning. Production-grade resilience also requires:

• Bulkheads to isolate failure domains
• Rate limiting to protect against overload
• Fallback strategies for degraded functionality
• Chaos engineering to test failure scenarios
• Comprehensive observability to detect issues early

Ready to Build Resilient Microservices?

The code examples in this article are production-ready starting points. Clone them, adapt them to your needs, and most importantly — test them under failure conditions. Break your services intentionally. Watch how they recover. That’s how you build confidence in your resilience patterns.

What resilience patterns have you implemented in your microservices? What challenges did you face? Share your experiences in the comments below!

Further Reading: Distributed Systems

1. **Breaking the Monolith: A simple guide to Microservices**
2. **RPCs, gRPC, and tRPC: The Backbone of Modern Distributed Systems**
3. **Understanding GraphQL: From Basics to Advanced**
4. **Understanding Asynchronous Communication: The Backbone of Distributed Systems**
5. **Building Async Workflows with RabbitMQ**
6. **Kafka Basics for Microservices**
7. **Caching Microservices Effectively with Redis**
8. **Eventual Consistency and CQRS for Microservices**
9. **Observability for Microservices**
10. **Tracing Requests in Microservices with OpenTelemetry**