🚀 Building a Scalable, Templated LLM Pipeline for Document Classification & Entity Extraction with Azure OpenAI

Photo by Victor on Unsplash

📌 Introduction

Enterprises sit on millions of documents — contracts, invoices, policies, medical records — all filled with valuable information. Extracting structured insights from them is slow, manual, and inconsistent.

I recently built a scalable document processing pipeline using Azure OpenAI, designed not as a one-off project but as a template that can be reused across multiple clients. In this article, I’ll walk through the full technical architecture, implementation, and templation approach.

🔹 The Problem

• 160,000+ documents across formats (PDF, Word, images)
• Needed classification (Invoice, Contract, Policy, etc.)
• Needed entity extraction (account numbers, dates, clauses, amounts)
• Must handle 100k+ docs efficiently, avoiding API rate limits
• Should be reusable across clients (different categories, endpoints, keys)

AI strategy and implementation — Version 1

🔹 High-Level Architecture

1. Documents uploaded → Azure Blob Storage
2. OCR → Azure Document Intelligence for scanned files
3. Pre-processing → chunk text into smaller pieces
4. Generate JSONL batch file (one JSON per chunk)
5. Submit JSONL → Azure OpenAI Batch API
6. Batch job runs asynchronously at scale
7. Download results → parse into CSV/Delta
8. Clients consume via Excel, Power BI, or Neo4j

🔹 Why JSONL Batch Files?

Instead of sending 100k API requests (which would hit rate limits), Azure OpenAI Batch API accepts a single JSONL file:

{"custom_id": "doc1", "method": "POST", "url": "/chat/completions", "body": {"model": "gpt-4o-mini", "messages": [{"role":"system","content":"Classify and extract"},{"role":"user","content":"Invoice Number: INV001 Amount $450"}]}}js

• Each line = one request (custom_id, method, url, body)
• Runs asynchronously in the background
• Provides trackable outputs by matching custom_id in responses

🔹 Classification Logic

• The AI doesn’t have predefined labels.
• Categories are defined in the prompt template, for example:

Classify the document into one of these categories: Invoice, Contract, Policy, Medical Report, Other

The AI assigns a category, provides a confidence score, and extracts entities in the same pass.

Example output:

{
  "category": "Invoice",
  "category_confidence": 0.94,
  "entities": [
    {"extracted_field": "Account No", "confidenceScore": 0.92, "justification": "Found in header text"}
  ]
}

🔹 Technical Implementation

📂 Repo Structure

azure-llm-template/
│── config/              # client configs (YAML)
│── prompts/             # classification/extraction prompts
│── src/                 
│   ├── ingest.py        # downloads docs from Blob
│   ├── ocr.py           # OCR for scanned files
│   ├── jsonl_builder.py # creates JSONL requests
│   ├── batch_submit.py  # upload, submit, poll Batch API
│   ├── results_handler.py # parse results → CSV
│   └── main.py          # orchestrator
│── requirements.txt
│── Dockerfile

Example Config (clientA.yaml)

storage:
  connection_string: "<AZURE_BLOB_CONNECTION_STRING>"
  container_raw: "raw-docs"
  container_jsonl: "jsonl-input"
  container_results: "batch-results"

openai:
  endpoint: "https://client-openai.openai.azure.com/"
  deployment_id: "gpt-4o-mini"
  api_key: "<CLIENT_API_KEY>"
  api_version: "2024-12-01-preview"

ocr:
  endpoint: "https://client-ocr.cognitiveservices.azure.com/"
  key: "<OCR_KEY>"

processing:
  prompt_template: "prompts/classify_and_extract.txt"
  max_tokens: 4000
  batch_size_mb: 180

output:
  results_file: "results.csv"

🔹 Dockerization for Easy Sharing

The pipeline is packaged into Docker, so clients don’t need to install Python or dependencies

docker pull myregistry.azurecr.io/llm-pipeline:latest
docker run -v $(pwd)/config:/app/config llm-pipeline:latest config/clientA.yaml

👉 Each client only changes their config file. The engine stays the same.

🔹 Automation of Prerequisites

To make onboarding easier:

• Terraform → provisions Blob Storage, OpenAI, OCR, Key Vault
• Azure Key Vault → stores secrets
• DevOps Pipeline → runs IaC + Docker automatically

For a new client:

git clone azure-llm-template
terraform apply -var-file=clientX.tfvars
docker run llm-pipeline:latest config/clientX.yaml

🔹 Continuous Automation with Azure DevOps

This pipeline can be fully automated:

1. Client drops new documents into Blob (raw-docs).
2. Azure DevOps pipeline triggers automatically.
3. Terraform ensures resources are ready.
4. Docker container runs with the client’s config.
5. Results are saved back to Blob → Power BI auto-refreshes dashboards.

👉 Fresh data = fresh results, with no manual reruns.

🔹 Results & Benefits

• Template approach = plug-and-play for multiple clients
• DevOps integration = automated, repeatable runs whenever new data arrives

🏁 Conclusion

By combining Azure OpenAI Batch API, OCR, Docker, and Azure DevOps automation, we built a reusable document processing engine that can:

• Scale to 100k+ documents
• Work across multiple clients with minimal effort
• Deliver structured, business-ready insights directly into Power BI

It’s a template that turns AI experiments into enterprise-grade, repeatable solutions.

About the author

Sarath Samynathan is a Senior Data Scientist here at Version 1.