Partitioning: The First Key Ingredient of a Large MongoDB Migration

Partitioning: The First Key Ingredient of a Large Migration

How do you eat an elephant?

One slice at a time.

True enough. But in a database migration, the real question is: how do you slice it well?

A large migration is only as fast as its slowest, most lopsided slice. If ninety-nine workers finish in ten minutes and one unlucky worker spends three hours on a giant range, the migration took three hours. If one range is so wide that counting it times out, copying it puts pressure on memory, or retrying it means redoing half the collection, partitioning has become the bottleneck.

This post is about moving data from MongoDB into Azure DocumentDB, Microsoft’s fully managed, MongoDB-compatible document database service. More specifically, it is about what we learned building a real partitioner for that path: why the obvious solution is not always cheap, why counts can quietly become the enemy, and why a good migrator needs more than one way to slice a collection.

Why partitioning matters

At small scale, a collection can be copied as one unit of work. Start the copy, wait for it to finish, move on.

At large scale, that stops working.

The copy has to be split into smaller units so many workers can run in parallel. I will call each unit a chunk: a bounded range of documents, usually expressed as lower and upper _id boundaries, that can be copied, tracked, retried, and marked complete independently.

That independence is the point. If a worker fails, only one chunk is retried. If you need more throughput, you add workers. If one worker is slow, the rest can keep making progress.

Bad chunks break that model:

• One huge range dominates runtime while every other worker sits idle.
• A wide range makes counting or copying time out.
• A retry repeats too much work because the unit of failure is too large.
• Progress becomes misleading because “one chunk” no longer means roughly the same amount of work.

Partitioning is how you turn a giant copy into many small, restartable, measurable pieces. Ten workers only help if the work is actually spread across ten workers.

The easy choice that is not always cheap

The first version of a partitioner usually reaches for $sample.

It sounds perfect: ask MongoDB for sample _id values, sort them, choose boundaries, and use those boundaries as chunk edges. You do not need to understand the data distribution. Let the database show you a representative slice.

That works, but only when MongoDB can use the cheap path.

$sample is fast on the random-cursor path. That path has two important conditions: the requested sample must be less than roughly five percent of the collection, and $sample must not have a $match stage in front of it.

The obvious $match case is a user filter. If the migration is only copying part of a collection, $sample has to run after that filter.

The less obvious case is mixed _id BSON types. Suppose 99.9 percent of a collection uses ObjectId, but a small minority uses string _ids. A general sample can easily miss the strings entirely. The majority type hides the minority type, and the partitioner quietly creates boundaries for only part of the collection. To avoid that, each _id type needs its own sampling pass, which means adding a $type filter.

That filter is also a $match. Once it is present, MongoDB cannot use the random cursor path. It falls back to scanning the matching documents and doing a top-k sort. On a large collection, that is server work you can feel.

So sampling needs a cost model, not blind faith.

A raw sample also does not automatically give you equal-sized boundaries. It gives you candidate _ids. If you pick too few candidates, random gaps in the sample can turn into lopsided chunks. When we are on the genuine $sample path, we oversample by 10x: if we need 100 boundaries, we ask for about 1,000 candidate _ids, sort and dedupe them, then quantile-select boundaries from that sorted sample.

But when there is a $match ahead of $sample, we do not oversample aggressively. We hard-cap the sample size at about 3,000 so the fallback scan-and-sort stays bounded. The partitions may be less perfect, but partitioning itself does not become the incident.

One practical detail: $sample on a huge collection can still take minutes. We run it with a long MaxTime and retry up to ten times. Partitioning should be patient, but not unbounded.

Chunks need guardrails

A chunk is not just a boundary. It is the amount of work you are asking one worker to own.

Too few chunks and you lose parallelism and restartability. Too many chunks and orchestration overhead starts to dominate. The partitioner keeps the count sane with tiered floors: around 100K documents per chunk for small and medium collections, and around 1M documents for very large collections above 100M documents.

Initial boundaries are still only a first guess. A sampled boundary can land badly. A time-based boundary can cover a bursty insert window. A filtered collection can have records concentrated in one narrow key range.

So the partitioner inspects the chunks it created. If one is still too large to be a safe unit of work, it recursively splits only that range until the pieces are manageable, or until it reaches a configured safety limit. That turns “one bad boundary” into a recoverable condition instead of forcing the migration to drag one giant chunk behind it.

Then there is the knob every real migration eventually needs: Partition factor.

Partition factor scales the chunk floor down. At 100%, the default, the partitioner uses the normal floor. At 50%, the effective floor is half as large, producing smaller and more numerous chunks. At 25%, it becomes more aggressive again.

That helps when you need more parallelism, when _id distribution is uneven, or when narrower ranges make per-chunk counting less likely to time out. Defaults are useful. Knobs are necessary.

Counting is the silent killer

Sampling gets the attention because it is the clever part. Counting is the part that quietly breaks your plan.

An unfiltered count can be cheap enough. A filtered count is often a full scan. On some sources, a filtered count over a wide range can take so long that it times out or even resets the connection.

The partitioner has to design for that failure.

If a collection is smaller than the minimum chunk floor, do not partition it. Make one chunk and move on.

If a count times out while deciding how many chunks to create, do not collapse to one giant partition. That is the worst possible fallback because it turns a count problem into a migration runtime problem. Use an estimated count or a previously computed count instead.

Over-estimating is usually harmless. The count only decides how many chunks to create. If the estimate is high, you create more chunks than strictly necessary. That adds some overhead, but preserves parallelism and restartability. Under-estimating is more dangerous because it creates fewer, larger chunks and pushes the pain downstream.

A rough count that keeps the plan distributed is better than a precise count that never returns.

_id type awareness is not optional

In clean examples, every document has an ObjectId _id.

Real collections are messier. You find ObjectIds, strings, integers, binary values, sometimes even compound shapes depending on source history and application behavior. A partitioner that assumes one _id type can miss records or compare values MongoDB cannot sensibly order together.

So the partitioner probes each collection for the _id BSON types that actually exist. It partitions by type when it has to, because each type needs its own chance to produce boundaries. If only one type is present, it skips the $type predicate so the pipeline can stay on the cheaper path. If mixed types are present, it filters by $type and accepts the $match cost.

We also let users provide the expected _id type for a collection. Application teams often know the data shape better than a generic probe can. If they know a collection uses only ObjectId, or only string _ids, the partitioner can treat it as single-type and avoid an unnecessary $type predicate.

Type awareness is required for correctness. The type predicate should only be paid for when it is needed.

ObjectId gives you a second path

ObjectId has a useful property: its leading bytes encode a timestamp. It is not a perfect sequence number, but it is roughly monotonic for many write patterns.

That gives us a cheap partitioning strategy. Instead of sampling documents, find the minimum and maximum ObjectId. Both are indexed operations. Convert those ObjectIds into large integers, step linearly between them, and turn those steps back into ObjectId boundaries.

No $sample. No collection scan. No top-k sort.

For large ObjectId-based collections, these Time Boundaries are attractive. But time distribution is not record distribution. A steady insert stream may produce balanced time ranges. A bursty backfill may put 80 percent of the data into one narrow time window.

That is why we added Adjusted Time Boundaries. Start with cheap time-based ranges, then validate counts. Merge ranges that are too small. Recursively split ranges that are too large. Use a maximum recursion depth and a dynamic max-records-per-range cap so the partitioner does not chase perfection forever.

And when counts time out, fall back to splitting the ObjectId range analytically. The result may be less balanced, but it is still distributed, restartable work.

There is no single best slicing method

The biggest lesson was not that sampling is bad or ObjectId math is good. It was that every strategy is good under the right conditions and expensive under the wrong ones.

So the partitioner exposes a spectrum:

• Sample Command for representative boundaries when sampling is affordable.
• Time Boundaries for cheap ObjectId-based partitioning with no data scan.
• Adjusted Time Boundaries when you want ObjectId speed plus count-based correction.
• Pagination when you need deterministic equal-size boundaries using progressive $gt with skip/limit.

The right answer depends on the collection: _id type, index availability, insert history, filters, size, and how much extra work the source can tolerate during planning.

A robust migrator does not pretend one method wins everywhere. It lets the strategy match the data shape and degrades gracefully when the ideal path fails.

A practical checklist

If you are building or tuning partitioning for a large migration:

• Keep the unit of work small enough to retry without drama.
• Avoid one giant fallback partition when counting or sampling fails.
• Detect oversized chunks and recursively split them into manageable ranges.
• Treat $sample as cheap only on the true random-cursor path.
• Cap matched sampling so scan-and-sort work stays bounded.
• Let users provide known _id types, and skip $type filters when they are not needed.
• Use ObjectId structure when you can.
• Make chunk size tunable.

The point

Partitioning is not just how you make a migration faster. It is how you make it controllable.

Good slices give you parallelism, restartability, honest progress, and smaller blast radius when something fails. Bad slices turn a large migration into a waiting game where one range, one count, or one overloaded worker quietly decides the schedule.

So yes, you eat the elephant one slice at a time.

But the migration succeeds because you learned how to slice it.

And when the slices are balanced, bounded, and retryable, even a very large migration starts to feel routine.

And routine, in database migrations, is exactly what you want.

Azure DocumentDB is a MongoDB-compatible database service, built on the MIT-licensed open-source DocumentDB project.