10x Faster OCR with PaddleOCR-VL and PyMuPDF4LLM on free-tier GPU Google Colab

Stop waiting hours for document parsing. By combining the precision of PyMuPDF with the raw power of vLLM-accelerated PaddleOCR-VL, you can achieve professional-grade document extraction on Google Colab’s free tier with unprecedented speed.

Traditional OCR is often a choice between two evils: fast but “dumb” text extraction that chokes on complex layouts, or high-quality Vision Language Models (VLMs) that crawl through pages at a snail’s pace. In this article, we report a breakthrough hybrid pipeline that achieves 10x better performance by intelligently switching between these worlds. We use pymupdf4llm for efficient, native text retrieval and fallback to PaddleOCR-VL 1.5 only when recognized text quality drops — all while using vLLM to supercharge the inference backend.

The Hybrid Philosophy: PyMuPDF + PaddleOCR-VL

The core of our approach is a page-by-page quality gate. Most modern PDFs are “born digital,” containing perfectly valid text layers. Why run a heavy VLM on a page that is already readable?

1. The 2% “Tofu” Gate

We process the PDF one page at a time. First, we attempt a fast extraction using pymupdf4llm. We then run a strict quality check: if the percentage of “bad” characters (Unicode replacement characters or Private Use Area “tofu” boxes) exceeds 2%, we trigger a manual fallback.

2. Markdown as the Universal Glue

The genius of this pipeline lies in the output format. Since both pymupdf4llm and PaddleOCR-VL natively produce Markdown, we can seamlessly join pages processed by different engines. This allows for a robust *try-except* pattern where the layout analysis remains consistent across the whole document.

import pymupdf
import pymupdf4llm

# Custom exception to trigger our manual OCR fallback
class OCRRecommendationError(Exception):
    pass

def my_ocr_function(page, **kwargs):
    # We use our tweaked analysis with the custom strict threshold
    # forked from pymupdf4llm.helpers.utils
    analysis = analyze_page_tweaked(page, bad_char_threshold=0.02)

    if analysis["needs_ocr"]:
        # Raise an error to be caught by our outer loop
        raise OCRRecommendationError(f"Quality too low: {analysis.get('reason')}")

    return None # Proceed with standard extraction

# Main processing loop
doc = pymupdf.open("document.pdf")
full_md = ""

for i in range(len(doc)):
    try:
        # Attempt high-speed extraction with quality check hook
        page_md = pymupdf4llm.to_markdown(
            "document.pdf", 
            pages=[i], 
            force_ocr=True, 
            ocr_function=my_ocr_function
        )
    except OCRRecommendationError:
        # Trigger VLM fallback only for "dirty" pages
        page_image = doc[i].get_pixmap(dpi=150)
        output = paddle_pipeline.predict(page_image.tobytes())
        page_md = output[0].save_to_markdown()

    full_md += f"\n\n---\n\n" + page_md

The “Forked” Analysis logic

We modified the internal analyze_page from pymupdf4llm in utils.py to be significantly more strict. By lowering the threshold for bad characters (Unicode replacement chars), we ensure that even minor corruption triggers a high-quality VLM pass.

def analyze_page_tweaked(page, bad_char_threshold=0.1):
    # [...] internal logic to count chars_total and chars_bad

    # We lowered this from the default 0.1 (10%) to 0.02 (2%)
    BAD_CHAR_THRESHOLD = bad_char_threshold 

    if chars_total > 0 and (chars_bad / chars_total) > BAD_CHAR_THRESHOLD:
        return {"needs_ocr": True, "reason": "chars_bad"}
    # [...]

Accelerating Inference with vLLM

For image-only corpora or the fallback pages of a PDF, standard inference can be painstakingly slow. To fix this, we leverage vLLM, a high-throughput serving engine. By running a local vLLM server in the background, we can accelerate PaddleOCR-VL inference by up to 10x.

Installation Strategy

To run this on Google Colab, you need specific versions to avoid dependency hell. Use the following installation strategy:

!pip install vllm==0.11.1
!pip install paddlepaddle-gpu==3.4.1
!pip install -U paddleocr[doc-parser]==3.4.1
!pip install safetensors openai huggingface_hub pyyaml tqdm nest_asyncio
!pip install paddlepaddle==3.3.1

The “Separate Terminal” Strategy

Colab users know that blocking cells prevent multi-process workflows. We developed a strategy to generate the command in the notebook, which you can then copy into a Colab terminal:

# Cell to generate terminal command
vllm_cmd = f"python -m vllm.entrypoints.openai.api_server --model {T_MODEL_ID} --served-model-name PaddleOCR-VL-0.9B --port 8000 --gpu-memory-utilization 0.85"
print("RUN THIS IN TERMINAL:")
print(vllm_cmd)

When this command is produced you must copy it, open the terminal in google colab, paste and run it.

Once the command is running, we use a waiting loop to ensure the pipeline doesn’t start until the backend is live:

import requests
import time

def wait_for_server():
    url = "http://localhost:8000/v1/models"
    while True:
        try:
            if requests.get(url).status_code == 200:
                print("vLLM server is ready!")
                break
        except requests.exceptions.ConnectionError:
            pass
        time.sleep(5)

Initializing the Pipeline: vLLM vs. Local

We can choose to use either VLLM or other inference backend server while not loosing the layout features of the PaddleOCR-VL or use the internal inference backend, mind you can specify the default model or your required finetune as we did.

Option A: The vLLM High-Speed Backend

from paddleocr import PaddleOCRVL

pipeline = PaddleOCRVL(
    vl_rec_backend="vllm-server", 
    vl_rec_server_url="http://localhost:8000/v1", 
    layout_detection_model_name="PP-DocLayoutV2"
)

Option B: The Local Backend (No VLLM required, but slower)

pipeline = PaddleOCRVL(
    #vl_rec_model_name="tachiwin/Tachiwin-OCR-1.5", # we use our finetune for indigenous languages of Mexico
    vl_rec_model_name="PaddlePaddle/PaddleOCR-VL-1.5",
    vl_rec_model_dir="./local_model_path"
)

Use 150 not 300 DPI

While 300 DPI is the industry standard for printing, it is actually detrimental for VLMs like PaddleOCR-VL. These models are typically trained on images around 1080p to 2K resolution, therefore, we standardized our pipeline on 150 DPI as suggested by Paddle, which provides enough detail for small text while keeping the GPU memory footprint small and the processing speed high.

Clean, Structured Results

Finalizing the Markdown requires more than just OCR. Our pipeline includes a cleaning stage using Regex to:

• Strip residual HTML tags and div wrappers,
• Remove proprietary picture placeholders (like ==> picture […] <==).
• Replace “tofu” characters with spaces to ensure the final text is lint-free.

This of course depends on your OCR premise, for ours, we don’t need image placeholders or additional html or inline images.

Wrapping up

• Free Tier Ready: Fully compatible with Google Colab’s free T4 GPUs.
• Resumable: Progress is tracked via a JSON file in your Hugging Face repository.
• Image Harvesting: All extracted images are automatically zipped into {hash}-images.tar.gz for easy downstream use.
• Cost-Effective: Reduces GPU runtime significantly by skipping OCR on clean pages.

Demo Notebooks

Ready to try it yourself? Check the full Google Colab Notebook brought to you by Tachiwin

About Tachiwin 🦡

Tachiwin is an effort to build open-source AI technologies for the indigenous languages of Mexico. May the indigenous languages of Mexico never be lost.

*brought to you by the Tachiwin Team.