When $addToSet Brought Down Our System: A Hard Lesson in MongoDB at Scale

MongoDB’s $addToSet operator promises simplicity: add a value to an array only if it’s not already there no duplicates, no extra logic.

At least, that’s the theory.

But what happens when this seemingly harmless operator becomes the root cause of chaos in production?

In this post, I’ll share a real-world issue we ran into where using $addToSet didn’t just quietly fail. It crippled performance, flooded the oplog, and left us scrambling for answers.

The Setup: What We Were Doing

In our system, we maintained a set of documents one for each calendar date that tracks availability and metadata of multiple vendors. Think of each document as a daily snapshot of vendor data.

{
  ....
  "date": "2024-03-08",
  "vendors": [
    {
      "id": "vendor-123",
      "status": "available",
      "peopleCount": 123
    },
    .....
  ]
}

Each date-document includes an array field that stores structured vendor information. For each request to fetch vendor availability over a date range, we first checked if the relevant data was already present. If not, we fetched the latest data and inserted them into the array while ensuring uniqueness of vendor entries using MongoDB’s $addToSet operator, and then returned the updated document.

bulk_query.append(
    {
        "filter": {
            ....
        },
        "update": {"$addToSet": {"vendors": vendor_data}},
    }
)

Simple logic, right?

Here, vendor_data is a small nested object with metadata like status, timestamps, etc.

This approach felt elegant. MongoDB would make sure we didn’t accidentally insert duplicate vendor entries. We were happy to offload the deduplication to the database.

But there was one thing we hadn’t accounted for: the silent pressure that comes with scale. As new vendors were visited, they were being added to the vendors array one by one. This meant each request had the potential to insert new data, growing the array rapidly and silently pushing MongoDB to its limits.

Each $addToSet looked like a small, idempotent update but under the hood, it was triggering a chain reaction that would soon bring our system to its knees.

The Problem We Didn’t See Coming

As our vendor page visits grew, so did the number of entries stored per date. Over time, some vendors arrays swelled to thousands of documents.

Then… things started to go sideways:

• MongoDB Atlas began auto-scaling continuously, trying to keep up with the surge.
• The oplog grew to a staggering 38 GB per hour.
• Write operations exploded, putting immense pressure on the system.
• Index builds crawled.

After digging into MongoDB Atlas logs, reviewing the cluster, and inspecting the profiler data, we finally uncovered the culprit:

Every time we used $addToSet to insert a data item, MongoDB rewrote the entire array to the oplog even if we were adding just one new data item.

What Really Happens Under the Hood with $addToSet

At first glance, $addToSet seems like a smart, efficient choice. It ensures uniqueness when adding elements to an array what’s not to like?

Here’s what MongoDB actually logs in the oplog for a seemingly minor update:

{
  "ts": { "$timestamp": { "t": 1719930000, "i": 1 } },
  ...,
  "o": {
    "$v": 2,
    "diff": {
      "u": {
        "vendors": [
          {
            "id": "1",
            "...": "..." // all existing vendors
          },
          {
            "id": "2",
            "...": "..."
          },
          {
            "id": "3",
            "...": "..."
          },
          {
            "id": "",
            "status": "available",
            "countOfPeople": 0,
            ...
          }
        ]
      }
    }
  }
}

So instead of a lightweight push, every $addToSet produces an oplog entry that could be megabytes in size.

This meant:

• High disk IO.
• Fast oplog churn.
• Slower replica syncing.

What We Learned (The Hard Way)

MongoDB’s $addToSet is great for small arrays or scalar values, but once arrays become large (or involve complex objects), it becomes risky.

If you’re using $addToSet:

• Keep array sizes small.
• Avoid large or deeply nested objects.
• Monitor oplog size and replication lag.
• Consider alternate designs (like reference collections or hashed sets).

How We Fixed It

To fix the problem, we rethought our schema.

Instead of pushing vendor documents into a growing vendors array, we moved to a reference model. Each vendor entry is now a separate document in a dedicated vendor_availability collection, linked by date and vendor ID:

{ date: "2024-03-08", vendorId: "vendor-123", ... }

We also added a unique compound index on (date, vendorId) to naturally prevent duplicates—no need for $addToSet.

This change:

• Drastically reduced oplog bloat
• Eliminated large array rewrites
• Improved indexing and query flexibility
• Let us scale cleanly as vendor traffic grew

Final Thoughts

Operators like $addToSet are easy to trust. They promise safety and simplicity. But at scale, even the simplest things can have sharp edges.

MongoDB didn’t behave wrongly we just misunderstood how much it would cost.