Scaling OCR Pipelines to TB-Scale: Bottlenecks, Worker Contention, and Distributed Systems Lessons

🚨The Problem

Data extraction has become very common nowadays with OCRs, LLMs, and document AI systems extracting information from PDFs, Word documents, scanned files, and images.

But building a working extraction pipeline is very different from scaling it to process hundreds of GBs or even TBs of enterprise data efficiently.

⚠️ At that scale, the challenge is no longer just extraction quality. Core engineering problems start becoming important:

• parallelism
• orchestration
• memory utilization
• worker scaling
• infrastructure costs
• throughput
• and distributed bottlenecks.

We already had a document extraction pipeline built for PDFs, DOCs, scanned documents, ZIPs, and OCR-heavy files.

The real challenge was scaling this pipeline efficiently for TB-scale workloads. As larger-scale runs were estimated, the projected runtime and infrastructure cost became extremely high, making the existing architecture economically difficult to scale.

That became the core objective of this optimization effort to reduce infrastructure cost, improve extraction throughput, and make large-scale extraction operationally feasible.

This blog goes through the engineering journey behind that optimization , the bottlenecks , failures and learnings on how to scale such distributed pipelines

🏗️ Architecture of the Extraction System

The extraction pipeline was designed to process large enterprise document collections consisting of PDFs, scanned documents, DOC/DOCX files, ZIP archives, images, and OCR-heavy files.

At a high level, the pipeline flow looked like this:

Source Systems / SharePoint
        ↓
Distributed Extraction Instances
        ↓
Worker-Level Processing
        ↓
OCR / Parsing / Extraction
        ↓
Output Storage + Metadata Tracking

The pipeline was deployed on distributed SageMaker instances using high-core CPU machines.

Each instance processed a shard or subset of the dataset independently, enabling horizontal scaling for larger workloads.

Current State Architecture

⚡Levels of Parallelism in the System

One of the most important aspects of the architecture was that parallelism existed across multiple layers simultaneously.

1️⃣ Instance-Level Parallelism

The overall dataset was divided into shards and distributed across multiple SageMaker instances.

Dataset
   ├── Instance 1
   ├── Instance 2
   ├── Instance 3
   └── ...

This was the highest level of distributed execution in the system. As workload size increased, more instances were added to improve throughput.

2️⃣ Worker-Level Parallelism

Inside each instance, multiprocessing-based extraction workers processed files in parallel.

⚙️~30 workers per instance were used

Each worker independently handled:

• file downloads, document parsing, OCR execution, extraction, metadata generation, and output storage.

Since OCR workloads are heavily CPU-bound, multiprocessing was preferred over threading to better utilize CPU cores.

3️⃣Parallelism Inside Each Worker

The worker itself also introduced another layer of parallelism internally.

For large PDFs and scanned documents:

• pages could be processed independently,
• OCR execution could run concurrently across pages,
• and image preprocessing operations also introduced additional threading.

At one stage, the runtime execution model roughly became:

Distributed Instances
    × 150 Workers
    × Multiple Files
    × ~5 OCR Page Threads

⚠️This meant the actual runtime concurrency of the system became significantly larger than the configured worker count alone.

At smaller scales, this improved throughput considerably.

But at larger scales, it also introduced CPU oversubscription, thread contention, memory pressure, scheduling overhead, long-tail execution behavior, and orchestration bottlenecks.

📊Profiling the Instances and Runs

Before optimizing the pipeline, the first step was understanding how the system was actually utilizing compute resources during large-scale runs.

The core question was simple:

⚠️ Were the instances truly utilizing available CPU and memory efficiently, or were hidden bottlenecks inflating runtime and infrastructure cost?

To answer this, profiling was added at:

• instance level,
• worker level,
• and file-processing level.

Since deeper cloud-native profiling and fine-grained AWS observability were limited in the execution environment, most profiling was implemented directly inside the application layer using Python instrumentation.

The pipeline primarily used:

• psutil
• custom worker instrumentation
• profiling tables
• and run/file-level tracing stored in Postgres.

Key metrics tracked included: CPU utilization, memory usage, RSS memory, worker execution time, long-running files,and shard-level execution behavior.

Some of the commonly used psutil functions included:

psutil.Process().memory_info().rss
psutil.Process().cpu_percent()
psutil.virtual_memory()
psutil.cpu_percent()

💡 One important finding early on was that CPU utilization was surprisingly low despite high runtime and infrastructure usage.

This profiling helped identify:

underutilized CPU capacity, long-tail files, uneven worker behavior, nested concurrency effects, and orchestration bottlenecks that were not visible from overall runtime alone.

These findings became the foundation for the optimization experiments discussed in the next sections.

📈 Scaling Iterations and Results

Once profiling was added, the next step was straightforward:

⚙️ increase parallelism and improve throughput.

The initial architecture used distributed SageMaker instances, multiprocessing-based extraction and ~30 workers per instance.

Early profiling revealed something unexpected:

⚠️ CPU utilization peaked at only ~20% despite high runtime and infrastructure usage. This suggested the system was still underutilizing available compute.

Worker counts were then aggressively scaled from: ~30 workers to ~150 workers per instance.

Initially, runtime improved noticeably. But as workloads increased, the behavior became increasingly inconsistent: some runs improved, some plateaued, while others became significantly slower.

Even after aggressive scaling, CPU utilization still peaked around only ~50%.

💡 This suggested the bottleneck was deeper than raw compute availability.

The expectation was straightforward:

more instances + more workers = lower runtime

But distributed runs became unpredictable.

Interestingly, another unexpected pattern also started appearing during profiling.

💡Worker Starvation scenarios — Larger workloads on the same instance were often completing more efficiently than smaller workloads

*As load increased:

• CPU utilization became healthier,
• worker utilization improved,
• and overall throughput became more stable.*

This suggested the system was behaving more efficiently under sustained load rather than bursty or partially utilized execution. Some workers remained active for extremely long durations, and a few files even appeared to take close to an hour during execution.

⚠️ The issue no longer looked like a compute bottleneck.

🧪 The Single-Instance Contradiction

One of the most important experiments was running a ~40 GB workload on a single high-CPU instance.

⚡ The entire workload completed in nearly ~70 minutes.

More surprisingly, files behaving like “problem files” during distributed runs completed smoothly without delays.

💡 This completely changed the investigation direction.

The bottleneck was likely emerging from orchestration and concurrency behavior rather than OCR speed itself.

This was the point where the investigation shifted from:

“how do we increase parallelism?” to: “what hidden bottlenecks are appearing under distributed scale?”

🧪 Assumptions, Experiments, and Dead Ends

As scaling behavior became increasingly inconsistent, the optimization journey became highly experimental.

Several assumptions were tested before the actual bottlenecks became clear.

🔍 Hypothesis #1 — OCR Was the Bottleneck

The first assumption was that OCR itself was slowing down the pipeline. Several OCR engines were tested independently: Tesseract / pytesseract, PaddleOCR , Surya , EasyOCR , DocTR

Interestingly, pytesseract still produced the best overall performance for the workload, especially for CPU-intensive extraction scenarios.

Even more surprisingly: ⚠️ files appearing “stuck” during distributed runs completed within minutes during isolated OCR tests.

💡 Result: OCR itself was likely not the primary bottleneck. However, OCR threading, process contention, and nested OCR parallelism were still suspected to be contributing factors.

⚙️ Hypothesis #2 — OCR Threading and Nested Parallelism

The next experiments focused on OCR execution behavior itself.

This included: OCR thread tuning (OMP_THREADWORKER=1), concurrency adjustments, and reducing nested OCR execution pressure.

💡 Result: The changes produced little to no meaningful improvement. Runtime remained inconsistent and some runs even regressed slightly under larger workloads.

🧵 Hypothesis #3 — Multiprocessing and Orchestration Blocking

Attention then shifted toward multiprocessing orchestration itself.

One major experiment involved changing execution handling from: pool.map() to submit() + as_completed() to achieve: finer execution control, improved handling of long-running tasks, and better orchestration visibility.

💡 Result: The changes did not improve throughput significantly and occasionally regressed performance slightly. This suggested that task scheduling alone was not the root bottleneck.

🐍 Hypothesis #4 — Python Multiprocessing and GIL Limitations

At this stage, attention shifted toward Python itself.

The investigation explored: multiprocessing overhead, GIL-related behavior, inter-process coordination, and large-scale multiprocessing limitations.

This led to extensive experimentation and research around distributed execution patterns and multiprocessing bottlenecks.

⚡ PySpark Experiments

Since the workload increasingly resembled a distributed data-processing problem, PySpark experiments were also explored.

The assumption was that Spark’s orchestration and distributed execution model might improve scaling behavior.

💡 Result: The results were surprisingly similar to the existing architecture. The expected breakthrough never happened, so deeper migration efforts into PySpark optimization were not pursued further.

🚧 More Experiments, More Contradictions

Several additional experiments were performed around: tracing OCR execution, reducing nested concurrency, identifying blocking behavior, and isolating long-tail execution patterns.

But none of these changes consistently solved the scaling degradation problem.

💡 At this stage, one thing became increasingly clear: the bottleneck was likely emerging from interactions between multiple system layers rather than from one isolated component.

🔍 Hidden Bottlenecks Hypothesis

The biggest contradiction during the optimization journey came from the scaling behavior itself.

Earlier experiments had already shown that increasing workers from ~30 → ~150 improved throughput significantly. Single-instance runs were also behaving surprisingly well.

⚡ ~40 GB workloads completing in nearly ~70 minutes while the same architecture degraded under distributed scale.

💡 This completely changed the investigation direction. The issue no longer looked like:

• OCR throughput, CPU saturation, memory exhaustion, or inefficient multiprocessing alone.

Instead, the system started showing symptoms of hidden contention, chained delays, and distributed bottlenecks that only appeared under scale.

🗄️ 1) DB Logging Contention

One major bottleneck was database activity inside active extraction workers.

Workers were continuously: writing logs, updating progress rows, and interacting with Postgres during extraction execution.

At small scale this looked harmless.

⚠️ But across ~150 workers and multiple distributed instances, DB operations started introducing contention, blocking, repeated connection overhead, and execution delays.

This effectively made Postgres part of the critical execution path instead of just an observability layer.

🔄 2) Repeated Client and Session Setup

Instead of spending most of the time on actual extraction work, workers were also repeatedly paying the cost of:

• Graph client setup,
• S3 client creation,
• session initialization,
• and other per-file setup operations.

⚠️ Individually these costs looked small. But multiplied across thousands of files, hundreds of workers, and distributed instances, they became significant.

🌐 3) Extra Graph / Metadata Calls

The pipeline also had avoidable Graph-side overhead.

Some execution paths performed additional:

• metadata calls,
• validation calls,
• or recovery requests per file.

⚠️ One extra call per file sounds harmless. But across large-scale distributed runs, this became thousands of additional requests, higher upstream pressure, and increased contention.

🚦 4) Graph Concurrency and Retry Amplification

The largest scale-out issue appeared when worker count translated directly into Graph download concurrency.

With many instances running many workers simultaneously, the system generated aggressive bursts of Graph/SharePoint requests.

This introduced 429 throttling, retries, worker blocking, and inflated wall-clock runtime.

⚠️ Retries amplified the problem further. A throttled request did not fail immediately — workers stayed occupied while waiting and retrying, creating another wave of upstream pressure.

This led to one major realization:

extraction parallelism and download parallelism cannot scale identically.

💥 Why These Bottlenecks Became Expensive

The real issue was that these costs were nested and compounding.

Inside one worker, a single file could involve: Graph downloads, OCR, retries, DB logging, client setup,and output handling.

That repeated: across ~150 workers, and then across multiple distributed instances.

⚠️ So even small inefficiencies became amplified into large-scale delays. The system was not failing because of one catastrophic bottleneck. Instead, several smaller inefficiencies were interacting together:

Graph throttling,

retry amplification,

DB contention,

orchestration overhead,

and nested concurrency pressure.

That was what made the pipeline difficult to diagnose .

Bottlenecks

🚀 Breakthrough Architecture and Final Results

After identifying the hidden bottlenecks, the pipeline architecture was redesigned around one core principle:

keep extraction highly parallel while controlling expensive operations around it.

One of the biggest issues was that increasing worker count also unintentionally increased Graph request fanout across distributed instances.

At larger scale, this created:

• 429 throttling,
• retry amplification,
• blocked workers,
• and inflated wall-clock runtime.

🚦 Bounded Graph Concurrency

To solve this, bounded Graph concurrency was introduced using semaphore-based request control.

The pipeline used:

multiprocessing.Manager().BoundedSemaphore()

to independently cap Graph download concurrency while still keeping extraction workers highly parallel (~150 workers).

💡 Instead of every extraction worker triggering downloads independently, Graph requests first acquired a semaphore slot before hitting the upstream service. This prevented download concurrency from exploding with worker scale.

The optimized execution flow became:

Bounded Graph Download Queue
            ↓
Reusable Worker-local Clients
            ↓
Persistent Session-based Downloads
            ↓
OCR + Extraction
            ↓
Reduced DB Logging
            ↓
Output Upload + Completion

🔄 Optimizing the Request Layer

The Graph request layer itself was redesigned to become more resilient under distributed execution.

The optimized flow introduced:

reusable worker-local Graph/S3 clients,

persistent requests.Session() usage,

timeout-aware downloads,

retry-aware execution,

token refresh handling,

and jittered exponential backoff.

Instead of recreating HTTP sessions repeatedly per file, workers reused persistent session state locally, reducing connection setup overhead, repeated authentication setup, TCP/HTTP churn, and unnecessary request latency.

⚠️ Retry Amplification and Jitter Handling

Retry handling became especially important at scale.

• The optimized retry layer handled 429, transient 5xx, request timeouts, and token expiry scenarios.
• Retry logic respected Retry-After headers, exponential retry windows, and randomized jitter.

💡 Without jitter, distributed worker fleets can retry simultaneously after throttling events, unintentionally creating another burst of upstream pressure. Introducing randomized jitter significantly stabilized request behavior across instances.

🗄️ Reducing Worker Overhead

Another major optimization was reducing expensive operations inside active extraction workers.

The optimized workers:

• reused Graph and S3 clients locally,
• reduced repeated setup work,
• minimized unnecessary metadata calls,
• and removed heavy DB logging from active extraction execution.

Instead of performing frequent Postgres updates during extraction itself, DB interaction was significantly reduced from the active worker path, preventing:

shared-row contention,

blocking of DB sessions and runs ,

and unnecessary worker delays under high concurrency.

Once these architectural changes were introduced, the scaling behavior changed dramatically.

Below are the breakthrough results and its details —

Breakthrough Results

💰 Optimizations and Engineering Lessons

📉 Cost and Time Optimization

One of the biggest goals of this optimization effort was making TB-scale extraction operationally feasible from both:

• runtime
• and infrastructure cost perspectives.

The earlier pipeline architecture showed severe scaling inefficiencies under distributed load.

As worker count and instances increased: retry amplification, Graph throttling, orchestration overhead,and long-tail execution started inflating both runtime and infrastructure consumption.

After the architectural optimizations, the behavior changed dramatically.

For the ~283 GB distributed execution:

• earlier projected runtime: ~16.39 hours
• optimized runtime: ~1.17 hours

⚡ Result:

• ~15.22 hours of wall-clock time saved
• ~228+ instance-hours saved
• ~92% runtime optimization
• ~92% infrastructure cost optimization

Infrastructure cost reduced from:

• ~$2481 projected cost to nearly ~$179 optimized execution cost

💰 Resulting in:

• ~$2300+ infrastructure savings on a single large-scale run.

At projected TB-scale workloads, the impact becomes even larger.

⚡ A ~1 TB workload estimated to take days and extremely high infrastructure cost is now projected to complete in nearly:

~5–6 hours with ~90%+ infrastructure cost savings which is a BREAKTRHOUGH WIN!! BAM!!

Optimization Details

🧠Engineering Lessons

Some of the biggest engineering lessons from this journey

The biggest realization was:

large-scale distributed pipelines are usually limited more by orchestration efficiency and contention amplification than by raw compute itself.

💡 Wrapping Up

This was a genuinely difficult engineering problem to solve.⚠️

⭐✅Many assumptions failed. Several experiments regressed performance instead of improving it. Some bottlenecks only appeared after fixing earlier ones. And many findings initially looked contradictory.

But that was also what made the journey valuable.

⭐✅And beyond the runtime and cost improvements, the most valuable outcome was the deeper understanding of how large-scale distributed systems actually behave under pressure.

⭐✅Hopefully, these learnings help others building:

• OCR pipelines
• distributed processing workloads,
• or large-scale AI extraction platforms

⭐✅Avoid some of the same pitfalls and think about scale from a systems perspective rather than only a compute perspective.

🚀 Feel free to connect or discuss further if you are working on similar large-scale distributed systems problems.

🔹References

few of important referrences and sources from the unlimited

• https://github.com/tesseract-ocr/tesseract/issues/3109
• https://research.google/pubs/the-tail-at-scale/
• https://medium.com/@priyasrivastava18official/system-design-patterns-the-essential-guide-to-managing-contention-in-distributed-systems-d531b2f69dea
• https://blog.alexoglou.com/posts/hedging/
• https://learn.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online