Understanding Percentiles in Performance Monitoring: A Deep Dive

Discussion about P99, P95, P85 etc..

What Are Percentiles?

A percentile is a statistical measure that indicates the value below which a given percentage of observations in a dataset falls. In performance monitoring, percentiles help us understand the distribution of response times and identify outliers that averages might hide.

Full story for non-members | Grab My Microservices E-Book | Youtube | LinkedIn | Book a 1:1 Meeting

If you using New relic the percentile numbers may look something like this

Mathematical Definition

If you have a dataset sorted in ascending order, the Pth percentile is the value below which P% of the data points fall.

Example: If the 95th percentile response time is 200ms, it means:

• 95% of all requests completed in 200ms or less
• Only 5% of requests took longer than 200ms

Why Percentiles Matter More Than Averages

The Problem with Averages

Looking at your APM screenshot:

• Average response time: 73.8ms
• Median (50th percentile): 11.3ms

Notice the huge gap! This indicates:

• Most requests are very fast (~11ms)
• A few slow requests are pulling the average up significantly
• Using only the average would give a misleading picture of user experience

Real-World Scenario

100 requests with response times:
- 90 requests: 10ms each
- 9 requests: 50ms each  
- 1 request: 2000ms

Average = (90×10 + 9×50 + 1×2000) ÷ 100 = 29.5ms
50th percentile (median) = 10ms
95th percentile = 50ms
99th percentile = 2000ms

The average (29.5ms) doesn’t represent the typical user experience (10ms) or highlight the problematic outlier (2000ms).

Interpreting Your APM Screenshot Metrics

Response Time Distribution Analysis

From your screenshot:

• Average: 73.8ms — Skewed by slow outliers
• Median (50th percentile): 11.3ms — Typical user experience
• 95th percentile: 72.3ms — 95% of users get response ≤ 72.3ms
• 99th percentile: 1.22s — Even the slowest 1% complete within 1.22s

What This Distribution Tells Us

1. Bimodal Distribution: There’s likely two distinct groups of requests:

• Fast path: ~11ms (majority of requests)
• Slow path: ~70ms-1.2s (cache misses, complex queries, etc.)

2. Performance Insights:

• Most users (50%) experience very fast responses (~11ms)
• Some users (45%) experience moderate delays (11ms-72ms)
• A small percentage (5%) experience significant delays (72ms-1.2s)

3. Potential Issues:

• Cache misses
• Database query performance
• External API calls
• Garbage collection pauses

Key Percentiles in Production Monitoring

Standard Percentiles and Their Significance

SLA Example

Service Level Agreement:
- P50 ≤ 50ms (typical users)
- P95 ≤ 200ms (acceptable for 95% of users)
- P99 ≤ 500ms (even slow requests stay reasonable)

Percentiles in Microservices Architecture

Cascading Latency Effects

In microservices, latency compounds across service calls:

Frontend → API Gateway → Service A → Service B → Database

If each service has P95 = 100ms:
- Single call: P95 = 100ms
- Chain of 4 calls: P95 ≈ 400ms (simplified)
- In reality, it's often worse due to queuing theory

Monitoring Strategy

1. Per-Service Metrics: Monitor each microservice individually
2. End-to-End Metrics: Track complete user journey
3. Dependency Tracking: Identify which services contribute to tail latency

Advanced Concepts

Percentile Aggregation Challenges

Problem: You cannot simply average percentiles across time windows or instances.

Wrong Approach:

Hour 1: P95 = 100ms
Hour 2: P95 = 200ms
Daily P95 ≠ (100 + 200) / 2 = 150ms

Correct Approach:

• Use histogram data structures (like HdrHistogram)
• Maintain raw distribution data
• Calculate percentiles from combined dataset

High Dynamic Range (HDR) Histograms

Modern APM tools use HDR histograms because they:

• Maintain accuracy across wide value ranges
• Use fixed memory regardless of data range
• Enable accurate percentile calculations

Alerting on Percentiles

Smart Alerting Strategy

1. Don’t Alert on P99: Too noisy, single slow request triggers alert
2. Alert on P95: Good balance between sensitivity and noise
3. Use Time Windows: Alert if P95 > threshold for 5+ minutes
4. Consider Error Rate: High percentiles during error spikes are expected

Example Alert Rules

# Alert when P95 latency is consistently high
alert: HighLatencyP95
expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 0.2
for: 5m
labels:
  severity: warning
annotations:
  summary: "95th percentile latency is {{ $value }}s"

# Alert when latency degradation affects significant user percentage
alert: LatencyDegradation  
expr: histogram_quantile(0.50, rate(http_request_duration_seconds_bucket[5m])) > 0.1
for: 3m
labels:
  severity: critical
annotations:
  summary: "Median latency degraded to {{ $value }}s"

Performance Optimization Using Percentiles

Optimization Priority Matrix

Specific Optimization Strategies

1. High P99, Good P95: Focus on eliminating worst outliers

• Database query timeout optimization
• Circuit breaker implementation
• Async processing for heavy operations

2. High P95, Good P50: Address moderate tail latency

• Cache warming strategies
• Connection pool optimization
• JVM garbage collection tuning

3. High P50: Fundamental performance issues

• Algorithm optimization
• Database indexing
• Infrastructure scaling

Interview Questions & Answers

Q: Why is P99 latency 1.22s while P95 is only 72.3ms in your screenshot?

This indicates a “heavy tail” distribution where:

• 95% of requests complete quickly (≤72.3ms)
• The worst 5% include some very slow requests (up to 1.22s)
• Possible causes: database query outliers, garbage collection, external API timeouts

Q: How would you improve the P99 latency shown?

1. Identify root causes: Use distributed tracing to find slow components
2. Implement timeouts: Prevent extremely slow operations
3. Add circuit breakers: Fail fast when dependencies are slow
4. Optimize database queries: Index optimization, query plan analysis
5. Cache frequently accessed data: Reduce database load
6. Async processing: Move heavy operations out of request path

Q: Can you average percentiles across multiple instances?

No, percentiles are not directly averageable. You need to:

1. Collect raw histogram data from each instance
2. Merge histograms maintaining bucket counts
3. Calculate percentiles from the merged dataset
4. Use tools like HdrHistogram that support proper aggregation

Q: What percentile should you use for SLAs?

• P95 for user-facing SLAs (good balance of coverage vs. noise)
• P99 for premium services or critical business functions
• P50 for basic availability guarantees
• Avoid P99.9+ for SLAs due to measurement noise and single-request outliers

Best Practices Summary

1. Always monitor multiple percentiles (P50, P95, P99)
2. Use percentiles for SLAs instead of averages
3. Implement proper histogram collection (HdrHistogram)
4. Consider cascading effects in microservices
5. Alert on P95, investigate P99
6. Optimize based on percentile patterns
7. Don’t average percentiles across time or instances
8. Include percentiles in capacity planning

Tools for Percentile Analysis

APM Tools

• New Relic
• DataDog
• AppDynamics
• Dynatrace

Open Source Options

• Prometheus + Grafana
• Jaeger (distributed tracing)
• Zipkin
• OpenTelemetry

Metrics Libraries

• Micrometer (Java/Spring)
• Prometheus client libraries
• StatsD
• DropWizard Metrics

==========================================

Check out the collection below for similar stories

[embed]Performance and latency matters Performance relate articles and discussionscodefarm0.medium.com

If you found this useful, please do clap the story and follow me for more such interesting and informative stories!