The DynamoDB Partition Split That Doubled Our P99 Latency

A production investigation into the invisible infrastructure event that no CloudWatch alarm catches.

The Alert That Made No Sense

Our P99 read latency on the orders table jumped from 4ms to 8ms. No deployment in 18 hours. No traffic spike. No configuration change. Just a clean, sudden step function in our latency graph that refused to come back down.

I checked the obvious things first. ThrottledRequests: zero. SystemErrors: zero. Average latency: barely moved, still sitting at 2ms. ConsumedReadCapacityUnits looked normal. Every standard CloudWatch alarm we had was green.

My first instinct, and the team’s first instinct, was to blame the application. Someone must have merged something. We pulled up the deployment history, checked the last three days of commits, diffed configuration files. Nothing. The service had been running the same artifact since Sunday evening.

The problem is that DynamoDB does not give you a metric called “partition split happened here.” There is no CloudWatch alarm you can set for it. Your table can undergo a significant infrastructure change, physical data movement across new partitions, and the only signal is a percentile latency bump that looks exactly like an application regression. We spent four hours proving our code was innocent before we even considered the infrastructure underneath it.

Eliminating the Obvious Suspects

Diagram: The diagnostic elimination process to distinguish a partition split from application bugs or other failures

We went through the standard checklist. Not because we are methodical heroes, but because you have to rule out the obvious before anyone will believe your weird infrastructure theory.

First: deployment timeline. We checked our CI/CD history against the moment the P99 jumped. No deploys within six hours of the spike. No config changes, no feature flags flipped, no canary promotions.

Second: upstream dependencies. We pulled X-Ray traces for the affected service. Every upstream call was flat. But the DynamoDB segment told a different story. The added latency was clearly inside the DynamoDB call itself, not in our serialization, not in our retry logic, not in network transit to the endpoint.

Third: connection pool and thread health. No thread exhaustion, no socket timeouts, no connection pool starvation. The JVM was healthy. GC pauses were normal.

Fourth: DynamoDB SystemErrors metric. Zero. This was not a service-side failure. DynamoDB was returning successful responses, just slower ones.

By this point we had narrowed it to one table and one operation type: GetItem on our orders table. Writes were unaffected. Queries on other tables were unaffected. And here is the detail that made me suspicious: the P50 latency was completely stable. Only the tail, P95 and P99, had moved. When your median is fine but your tail doubles, you are looking at a subset of requests hitting something different.

What a Partition Split Actually Is

DynamoDB distributes items across partitions based on a hash of your partition key. Each partition is a physical storage unit backed by SSD, and it has hard limits: 10GB of data, 3000 read capacity units, or 1000 write capacity units per second. When any of these thresholds is exceeded, DynamoDB splits the partition into two. It moves a portion of the key range to a new partition on different storage nodes, rebalances the data, and updates the internal request router.

This is the critical part: the split is completely invisible to you. No CloudWatch alarm fires. No EventBridge event is published. There is no entry in CloudTrail. AWS does not notify you in any way that a partition just split.

During the split, requests routed to the affected partition experience queuing. Items are being copied to the new partition, metadata is being updated, and the request router needs to converge on the new topology. Requests that land during this window do not fail. They do not get throttled. They just take more time. This is exactly the pattern I was seeing: no errors, no throttling, just a tail latency spike on a subset of keys.

The Metric That Gave It Away

The standard DynamoDB CloudWatch metrics were useless for this investigation. ConsumedReadCapacityUnits looked normal. ThrottledRequests was zero. SystemErrors was zero. If you only look at these three, you conclude nothing is wrong.

The metric that actually showed the problem was SuccessfulRequestLatency at P99 with a 1-minute period. P50 stayed flat at 2ms. P99 jumped from 4ms to 8ms and stayed there for about 40 minutes before settling at 6ms, still higher than before. Zero errors, zero throttling, just slower successful requests. This is the signature of a partition split.

Once I had the latency signal, I needed to prove which partition was responsible. We had not enabled Contributor Insights before the incident, which meant I had no historical baseline. I enabled it during the investigation:

aws dynamodb update-contributor-insights \
  --table-name orders \
  --contributor-insights-action ENABLE

Contributor Insights only shows data from the moment you enable it forward. I could not see what happened during the split itself. But what I could see, once it started collecting, was the current access pattern: a single partition key prefix, tenant#acme-corp, was responsible for roughly 60% of consumed read capacity on the table. This was our largest customer, and their item collection had been growing steadily for months toward the 10GB partition limit. Combined with the latency signature and the timing, this was enough to confirm the diagnosis.

The distinctive pattern is worth memorizing: P99 spikes, P50 stays flat, zero throttling, zero system errors. If all four conditions are true simultaneously, you are almost certainly looking at a partition split. No application bug produces this exact combination.

Why Standard Monitoring Misses This

Diagram: How adaptive capacity handles throttling but leaves latency from partition splits completely unaddressed

Here is the problem: most teams alarm on ThrottledRequests > 0 and average latency. Neither fires during a partition split.

Throttling does not happen because adaptive capacity redistributes the throughput budget across the new partitions before requests start failing. AWS introduced adaptive capacity in 2018 specifically to handle uneven access patterns. It prevents throttling effectively. But it does nothing about the physical latency of data movement during a split. Your requests will not get a 400 error. They will get a 200 that took 8ms instead of 4ms. Adaptive capacity solves the throughput problem. It does not solve the latency problem.

Average latency stays flat because only a small percentage of requests hit the splitting partition. Your dashboards stay green while your P99 doubles.

On-demand mode makes this worse, not better. Teams using on-demand often have zero capacity-related alarms because the whole point was to stop thinking about throughput. But on-demand does not eliminate partition splits. It eliminates throttling. The physical data movement still happens, and you have even fewer signals to correlate with.

The standard counterargument is “DynamoDB is fully managed, I should not need to care about partitions.” The abstraction works perfectly at P50. It leaks at P99. If your SLO is defined at the 99th percentile, you are exposed to infrastructure behavior that the managed service does not surface in default metrics.

The Dashboard You Should Have Built Yesterday

Here is the dashboard I run on every DynamoDB table that matters. The core signal is P99 diverging from P50. During normal operation, P99 might be 2–3x your P50. During a partition split, P99 jumps to 5–10x while P50 stays flat. That divergence is your alarm condition.

{
  "widgets": [
    {
      "type": "metric",
      "properties": {
        "title": "DynamoDB Latency: P99 vs P50",
        "metrics": [
          ["AWS/DynamoDB", "SuccessfulRequestLatency", "TableName", "orders", "Operation", "GetItem", {"stat": "p99", "label": "P99"}],
          ["AWS/DynamoDB", "SuccessfulRequestLatency", "TableName", "orders", "Operation", "GetItem", {"stat": "p50", "label": "P50"}]
        ],
        "period": 60,
        "view": "timeSeries",
        "stacked": false
      }
    },
    {
      "type": "metric",
      "properties": {
        "title": "Throttling & Capacity",
        "metrics": [
          ["AWS/DynamoDB", "ThrottledRequests", "TableName", "orders", {"stat": "Sum"}],
          ["AWS/DynamoDB", "ConsumedReadCapacityUnits", "TableName", "orders", {"stat": "Sum"}]
        ],
        "period": 60,
        "view": "timeSeries"
      }
    }
  ]
}

Combine P99/P50 with consumed capacity and throttled requests on the same timeline, and you can visually distinguish a split from a code regression in about ten seconds.

The alarm in CDK:

new cloudwatch.Alarm(this, 'DynamoP99AnomalyAlarm', {
  metric: table.metricSuccessfulRequestLatency({
    statistic: 'p99',
    period: Duration.minutes(1),
  }),
  evaluationPeriods: 3,
  threshold: 50,
  comparisonOperator: cloudwatch.ComparisonOperator.GREATER_THAN_THRESHOLD,
  treatMissingData: cloudwatch.TreatMissingData.NOT_BREACHING,
});

Set the threshold to roughly 3x your normal P99. The key is evaluationPeriods: 3 — a single spike is noise, three consecutive minutes of elevated P99 is a split in progress.

Enable Contributor Insights on any table where a single partition key can accumulate significant data:

aws dynamodb update-contributor-insights \
  --table-name orders \
  --contributor-insights-action ENABLE

The $2/month cost is irrelevant compared to the two days you will spend proving a latency spike is not your code.

Designing Schemas That Survive Splits

The best defense against split-induced latency is making splits irrelevant by keeping partitions small. In practice, this means write sharding: appending a computed suffix to your partition key so that items spread across more partitions than DynamoDB would naturally create.

import hashlib

def sharded_key(user_id: str, shard_count: int = 10) -> str:
    shard = int(hashlib.md5(user_id.encode()).hexdigest(), 16) % shard_count
    return f"{user_id}#{shard}"

You hash the natural key and append a modulo suffix. Writes distribute evenly across shard_count partitions instead of concentrating on one. The tradeoff is real: reads now require scatter-gather.

import boto3
from concurrent.futures import ThreadPoolExecutor

def query_all_shards(table, user_id: str, shard_count: int = 10):
    def query_shard(shard):
        return table.query(
            KeyConditionExpression="pk = :pk",
            ExpressionAttributeValues={":pk": f"{user_id}#{shard}"}
        )["Items"]

    with ThreadPoolExecutor(max_workers=shard_count) as pool:
        results = pool.map(query_shard, range(shard_count))
    return [item for shard_items in results for item in shard_items]

Yes, this adds latency and cost. You pay for N queries instead of one. But the alternative is an item collection that grows until it triggers a split, and then you pay with a P99 spike that affects every request to that partition. I will take predictable scatter-gather overhead over unpredictable infrastructure events every time.

TTL is your cheapest defense. Set expiration on session data, event logs, and anything with a natural lifespan. The goal is simple: keep item collections well below 10GB so that DynamoDB never needs to split your hot partitions.

What I Got Wrong and What I Would Do Differently

I made this harder than it needed to be. Four hours to prove it was not our code, and most of that time was spent building observability that should have existed before the incident started.

Our first mistake was alarming on average DynamoDB latency instead of P99. Average latency stayed flat during the entire event. If we had a P99 alarm on SuccessfulRequestLatency, we would have caught this in minutes, not hours.

Second, we did not have Contributor Insights enabled. This is the one tool that shows you which partition keys are getting hit hardest. We turned it on during the incident, which meant we had no baseline to compare against and could not see the split as it happened. Enabling it retroactively is like installing a dashcam after the crash. You can see the current state of the road, but you missed the event itself.

Third, and this is the one that actually caused the split: our partition key was a raw tenant ID. We had maybe 200 tenants, with the top 5 generating 80% of traffic. That is a textbook hot partition setup. No write sharding suffix, no composite key strategy.

The fix was not a code change. We redesigned the key schema to append a modulo suffix, enabled Contributor Insights on all production tables, and added P99 latency alarms. The distinction matters: noisy neighbors cause throttling that adaptive capacity handles, but a split causes latency that no retry strategy fixes.

The Mental Model to Take Away

“Fully managed” does not mean “no design responsibility.” It means someone else operates the infrastructure while you still design for its physical constraints. DynamoDB is the same as TCP: you do not manage packet retransmission, but you still design your application to handle backpressure.

The key distinction: adaptive capacity solves throttling from uneven access patterns, but it does not solve latency from physical partition splits. These are different mechanisms. No amount of retry logic or capacity borrowing fixes the queuing that happens during data movement.

Every managed service has invisible maintenance operations. Aurora has failovers. ECS has task replacements. DynamoDB has partition splits. Your job is not to prevent them — you cannot — but to make them visible before your users feel them.

Three concrete actions you can take today:

• Enable Contributor Insights on every production DynamoDB table. The cost is negligible compared to the diagnostic time it saves.
• Alarm on P99 divergence from P50. A healthy table has a stable ratio. When P99 spikes but P50 stays flat, something infrastructure-level changed.
• Audit your partition key cardinality. If your key space does not grow proportionally with your data, you are designing for splits under load.