Fine-Tuning LayoutLMv3 for Intelligent Document Processing

How we trained a model to read and understand documents the way humans do — extracting structured data from invoices, forms, and more.

What Problem Are We Solving?

Businesses deal with huge volumes of documents every day — invoices, purchase orders, contracts, forms. Getting structured data out of these documents is a pain. Traditional approaches like regex rules or template matching break easily when a vendor changes their invoice layout or a new document type comes in.

What we really need is a model that can read a document the way a human does: understanding not just what the text says, but where it sits on the page — because layout carries meaning. “Total” in the bottom-right corner of an invoice means something very different from “Total” in a table header.

That’s exactly what LayoutLMv3 does. It’s Microsoft Research’s third-generation model for Document AI, and it’s become a core part of how we extract structured data at Imaginist.ai. This post walks through how we fine-tuned it on our business document dataset, step by step.

📌 Note: Our training data is proprietary, but throughout this guide we reference publicly available datasets (FUNSD, CORD, SROIE) that you can use as drop-in replacements to follow along.

How LayoutLMv3 Works

LayoutLMv3 combines three types of information from a document and processes them together in a single Transformer model:

• Text tokens — the words extracted by OCR, with standard word embeddings.
• Layout embeddings — the bounding box coordinates of each word, normalised to a 0–1000 scale. This tells the model where each word lives on the page.
• Image patch tokens — the document image is split into small 16×16 pixel patches and projected linearly into the model. No heavy CNN backbone needed.

All three streams are concatenated and fed into a shared Transformer encoder. This joint processing is what gives the model its power — it learns to connect text meaning with visual position.

Why v3 over v1/v2? Earlier versions depended on a CNN image backbone (Faster R-CNN / ResNeXt), which was slow, memory-heavy, and required the Detectron2 library to run. LayoutLMv3 replaces all of that with simple linear patch projections, making it much easier to fine-tune on a single GPU.

What it learns during pre-training

Before fine-tuning, the model is pre-trained with three tasks that teach it to deeply understand documents:

• MLM (Masked Language Modeling): Predict masked text tokens — the classic BERT-style task.
• MIM (Masked Image Modeling): Reconstruct masked image patches, teaching visual layout understanding.
• WPA (Word-Patch Alignment): Predict whether a text token and an image patch correspond to the same region — building precise cross-modal alignment.

Datasets You Can Use

If you don’t have your own annotated data, these four public datasets are the standard starting points for LayoutLM models. All are available on Hugging Face Datasets.

FUNSD — 199 annotated scanned forms. Great for form key-value extraction. Labels: question, answer, header, other.

CORD — 1,000 store receipts annotated with 30 semantic labels. Best starting point if your task involves receipts.

SROIE — 626 training + 347 test scanned receipts. Key fields: company, date, address, total.

RVL-CDIP — 400,000 document images across 16 classes. Use this for document type classification tasks.

For invoice and business document extraction, FUNSD and CORD are the closest public proxies to what we work with at Imaginist.ai.

Preparing Your Data

Step 1: Run OCR to get words and bounding boxes

LayoutLMv3 needs three things per word: the word text, its bounding box (normalised to 0–1000), and an NER label for training. The first step is running OCR on your document images. We use Google Document AI — it produces highly accurate text extraction along with precise bounding polygon data for every word, which maps cleanly onto what LayoutLMv3 expects.

The key step is converting Document AI’s bounding polygons (returned as normalised vertex coordinates in the 0–1 range) into LayoutLMv3’s required 0–1000 integer scale.

from google.cloud import documentai_v1 as documentai
from PIL import Image

def run_ocr(image_path, project_id, location, processor_id):
    client = documentai.DocumentProcessorServiceClient()
    processor_name = client.processor_path(project_id, location, processor_id)

    with open(image_path, "rb") as f:
        raw_document = documentai.RawDocument(
            content=f.read(),
            mime_type="image/png",   # or "application/pdf"
        )

    request = documentai.ProcessRequest(
        name=processor_name, raw_document=raw_document
    )
    result = client.process_document(request=request)
    document = result.document

    words, boxes = [], []
    for page in document.pages:
        img_w = page.dimension.width
        img_h = page.dimension.height
        for token in page.tokens:
            text_anchor = token.layout.text_anchor
            word = "".join(
                document.text[seg.start_index:seg.end_index]
                for seg in text_anchor.text_segments
            ).strip()
            if not word:
                continue

            # Bounding polygon vertices are normalised (0–1); scale to 0–1000
            verts = token.layout.bounding_poly.normalized_vertices
            xs = [v.x for v in verts]
            ys = [v.y for v in verts]
            norm_box = [
                int(min(xs) * 1000), int(min(ys) * 1000),
                int(max(xs) * 1000), int(max(ys) * 1000),
            ]
            words.append(word)
            boxes.append(norm_box)

    return words, boxes

⚠️ Bounding box normalisation is critical. If you forget to scale boxes to 0–1000, the model will silently fail with no obvious error. This is one of the most common gotchas when starting out.

Step 2: Annotate with BIO labels

For entity extraction (Named Entity Recognition), each word gets a BIO label: B- marks the beginning of an entity, I- marks continuation, and O means it's not an entity. We use Label Studio for annotation.

Each document is stored as a JSON file with this structure:

{
  "id": "doc_0001",
  "words": ["Invoice", "No.", "INV-2024-0042", ...],
  "bboxes": [[42, 68, 132, 89], [138, 68, 172, 89], ...],
  "ner_tags": ["O", "O", "B-INVOICE_ID", ...],
  "image_path": "images/doc_0001.png"
}

💡 Typical entity types for invoices: INVOICE_ID, VENDOR_NAME, VENDOR_ADDRESS, PO_NUMBER, INVOICE_DATE, DUE_DATE, LINE_ITEM_DESC, SUBTOTAL, TAX, TOTAL. Define labels based on what your downstream application actually needs.

Setting Up the Environment

One of the biggest quality-of-life improvements in LayoutLMv3 is that it no longer needs Detectron2. The standard Hugging Face transformers library is all you need.

# Core dependencies
pip install transformers datasets seqeval Pillow easyocr

# For training
pip install accelerate

# Optional: experiment tracking
pip install wandb

# Verify your GPU is available (run this in a Colab cell)
python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"

We train on Google Colab using a T4 GPU (available on Colab free) or a V100/A100 on Colab Pro. On a single Colab GPU, set per_device_train_batch_size to 2 and use gradient accumulation to simulate a larger effective batch size.

Preprocessing & Tokenization

LayoutLMv3 has a unified processor that handles both text tokenization and image feature extraction in one call. One thing to be careful about: a single word can be split into multiple sub-word tokens by the tokenizer. You need to make sure labels and bounding boxes are correctly propagated to each sub-token.

from transformers import LayoutLMv3Processor
from PIL import Image

processor = LayoutLMv3Processor.from_pretrained(
    "microsoft/layoutlmv3-base",
    apply_ocr=False   # We supply our own OCR output
)

LABEL2ID = {
    "O": 0,
    "B-INVOICE_ID": 1, "I-INVOICE_ID": 2,
    "B-VENDOR_NAME": 3, "I-VENDOR_NAME": 4,
    "B-TOTAL": 5, "I-TOTAL": 6,
    # ... add all your entity types
}

def encode_example(example):
    image = Image.open(example["image_path"]).convert("RGB")
    word_labels = [LABEL2ID[t] for t in example["ner_tags"]]

    encoding = processor(
        image,
        example["words"],
        boxes=example["bboxes"],
        word_labels=word_labels,
        truncation=True,
        padding="max_length",
        max_length=512,
        return_tensors="pt",
    )
    return {k: v.squeeze() for k, v in encoding.items()}

💡 Label propagation: The processor automatically assigns the word label to the first sub-token and pads subsequent ones with -100 (PyTorch's ignore index). You don't have to handle this manually.

Training the Model

We use Hugging Face’s Trainer API, which handles the training loop, evaluation, and checkpointing for us.

from transformers import (
    LayoutLMv3ForTokenClassification,
    TrainingArguments, Trainer
)

# Load the pre-trained model with our label set
model = LayoutLMv3ForTokenClassification.from_pretrained(
    "microsoft/layoutlmv3-base",
    num_labels=len(LABEL2ID),
    id2label={v: k for k, v in LABEL2ID.items()},
    label2id=LABEL2ID,
)

args = TrainingArguments(
    output_dir="./layoutlmv3-output",
    learning_rate=1e-5,
    lr_scheduler_type="cosine",
    warmup_ratio=0.1,
    per_device_train_batch_size=2,    # Keep low for Colab GPU
    per_device_eval_batch_size=2,
    gradient_accumulation_steps=8,   # Effective batch size = 16
    num_train_epochs=15,
    weight_decay=0.01,
    evaluation_strategy="epoch",
    save_strategy="epoch",
    load_best_model_at_end=True,
    metric_for_best_model="eval_f1",
    fp16=True,                          # Half-precision for speed
    logging_steps=25,
)

For the training loop, we use seqeval to compute entity-level F1 — this is the right metric for NER tasks (more on this in the next section).

from seqeval.metrics import f1_score
import numpy as np

ID2LABEL = {v: k for k, v in LABEL2ID.items()}

def compute_metrics(p):
    predictions, labels = p
    predictions = np.argmax(predictions, axis=2)

    true_labels, true_preds = [], []
    for pred_seq, label_seq in zip(predictions, labels):
        true_labels.append([ID2LABEL[l] for l in label_seq if l != -100])
        true_preds.append([ID2LABEL[p] for p, l in zip(pred_seq, label_seq) if l != -100])

    return {"eval_f1": f1_score(true_labels, true_preds)}

trainer = Trainer(
    model=model, args=args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    tokenizer=processor,
    compute_metrics=compute_metrics,
)

trainer.train()
trainer.save_model("./best-layoutlmv3")

Running Inference

At inference time, the pipeline mirrors training: run OCR → normalise bounding boxes → pass through the processor → run the model → decode predictions back to word-level entities.

from transformers import LayoutLMv3ForTokenClassification, LayoutLMv3Processor
from PIL import Image
import torch

model = LayoutLMv3ForTokenClassification.from_pretrained("./best-layoutlmv3")
processor = LayoutLMv3Processor.from_pretrained("./best-layoutlmv3", apply_ocr=False)
model.eval()

def extract_entities(image_path):
    image = Image.open(image_path).convert("RGB")
    words, boxes = run_ocr(image_path)

    encoding = processor(
        image, words, boxes=boxes,
        truncation=True, max_length=512, return_tensors="pt"
    )
    with torch.no_grad():
        outputs = model(**encoding)

    predictions = outputs.logits.argmax(-1).squeeze().tolist()
    token_boxes  = encoding["bbox"].squeeze().tolist()

    entities, current = [], None
    for pred, box in zip(predictions, token_boxes):
        label = model.config.id2label[pred]
        if label.startswith("B-"):
            if current: entities.append(current)
            current = {"type": label[2:], "box": box}
        elif label.startswith("I-") and current:
            pass
        else:
            if current: entities.append(current)
            current = None
    if current: entities.append(current)
    return entities

The full pipeline at a glance

Document Image → OCR (Document AI) → Normalise BBoxes 0–1000 → LayoutLMv3 Processor → Fine-tuned Model → Entity JSON

Tips for Production

OCR is the bottleneck

The model’s forward pass is fast. Google Document AI handles OCR reliably and at scale, but it does add network round-trip latency. For high-throughput workloads, consider batching document requests, running OCR asynchronously, and caching results so you’re not re-processing the same document twice.

Multi-page documents

LayoutLMv3 processes one page at a time (max 512 tokens). For multi-page documents, run the model page by page and merge entity predictions across pages in a post-processing step.

Class imbalance

Most tokens in a document are labelled O (not an entity), which causes class imbalance. We address this with weighted cross-entropy loss — giving higher weight to rare entity types so the model doesn't just learn to predict O for everything.

import torch.nn as nn

class_counts = torch.tensor(
    [token_label_counts[i] for i in range(num_labels)], dtype=torch.float
)
class_weights = 1.0 / (class_counts + 1e-6)
class_weights = class_weights / class_weights.sum() * num_labels
class_weights = class_weights.to(device)

loss_fct = nn.CrossEntropyLoss(weight=class_weights, ignore_index=-100)

Model versioning

We version all fine-tuned models on the Hugging Face Hub (private repo) and tag each release with the dataset version, training date, and evaluation metrics. In production, we log per-document confidence scores and route low-confidence predictions to a human review queue.

Key Takeaways

LayoutLMv3 has become a core part of our document intelligence stack at Imaginist.ai. Fine-tuning it took roughly three weeks of end-to-end effort — annotation, preprocessing, tuning, and production integration — but the improvement over regex and rule-based approaches was dramatic.

Here’s what we’d tell any team starting this journey:

• Start with a public dataset (CORD or FUNSD) to validate your pipeline before investing in proprietary annotation.
• OCR quality is the ceiling. A perfectly annotated dataset can’t fix bad OCR. Get your OCR right first.
• Use entity-level F1, not token accuracy. Token accuracy is misleading because most tokens are O. Entity-level F1 from seqeval is the right metric.
• Bounding box normalisation matters. Make absolutely sure your boxes are scaled to 0–1000. Silent failures here are very hard to debug.
• Watch per-entity F1 during training. If a specific entity type has low F1, it probably needs more annotated examples before you invest time in hyperparameter tuning.

Resources: LayoutLMv3 on HuggingFace · Transformers docs · LayoutLMv3 paper (Huang et al., ACM MM 2022) · FUNSD dataset · CORD dataset

© 2025 Imaginist.ai · Engineering Blog