Building an Intelligent Insurance Policy Data Extractor with OCR, Regex & Streamlit

Insurance documents are notoriously inconsistent. Every insurer formats policies differently, and extracting key details — like policy number, premium, or proposer name — quickly becomes a painful, manual process.

So I decided to build a fully automated Insurance Policy Data Extractor using:

• Python
• Tesseract OCR
• pdf2image
• Regex-based field extraction
• Streamlit UI
• Excel-based storage for cumulative data

This app lets users upload any policy PDF, runs OCR, extracts structured fields, and updates a master Excel file automatically. Here’s how the whole system works.

https://youtu.be/ihG5V0ETed8

Why Build This?

Insurance PDFs often contain scanned images, not selectable text. Traditional copy–paste fails, and building rule-based or machine-learning extraction models usually requires large training datasets.

OCR + intelligent regex extraction turns unstructured documents into actionable data — instantly.

So the goal was to create a tool that:

• Works on all types of scanned PDFs
• Extracts critical fields reliably
• Provides instant visual feedback
• Builds a historical dataset automatically

The result is a clean and powerful Streamlit web app.

Tech Stack Overview

OCR Processing

• Tesseract for converting images → text
• pdf2image to break PDFs into high-resolution images

Data Extraction

• A robust, multi-pattern regex engine capable of handling dozens of real-world variations in:
• Insurance company names
• Policy numbers
• Customer/proposer names
• Phone numbers
• Emails
• Premium amounts

Frontend

• Streamlit, enhanced with custom HTML/CSS for a modern UI

Data Storage

• A self-updating insurance_master.xlsx file
• Avoids duplicate entries using the filename as a unique key

Architecture Breakdown

Step 1 — PDF Upload

Users upload a policy PDF through a clean Streamlit interface. The app stores it temporarily for processing.

Step 2 — Convert PDF → Images

images = convert_from_path(pdf_path, dpi=300)

The PDF is converted page-by-page into high-quality images for accurate OCR.

Step 3 — OCR Extraction

ocr_text += pytesseract.image_to_string(img)

Each page is interpreted using Tesseract’s text engine.

Step 4 — Regex-Based Field Extraction

A custom function extract_fields() evaluates dozens of patterns per field:

"Policy Number": find([
    r"(?:Policy\s*(?:No\.?|Number|#)\s*[:\-]?\s*)([A-Z0-9\-\/]+)"
]),

This allows the extractor to adapt to multiple formats from different insurers.

Step 5 — Display Results

Streamlit shows:

• Full OCR text
• A table of extracted fields
• Real-time master dataset view

Step 6 — Auto Append to Master Excel

The system updates insurance_master.xlsx while preventing duplicates:

if uploaded_pdf.name in df_master["File Name"].values:
    st.warning("This PDF has already been processed.")

Step 7 — User Downloads

The user can export:

• Updated Excel
• Full OCR text

Beautiful Interface with Custom CSS

The app uses a clean gradient header, animated hover boxes, and enhanced table styling:

.gradient-title {
    font-size: 48px;
    font-weight: 900;
    background: linear-gradient(90deg, #005C97, #363795);
    -webkit-background-clip: text;
    -webkit-text-fill-color: transparent;
}

Streamlit’s default UI feels like a full-fledged dashboard.

Why This Approach Works So Well

✔ No training data required

Regex + OCR works out-of-the-box for all insurers.

✔ Extremely flexible

Adding new field patterns is simple.

✔ Enterprise-ready

Easily extendable to databases, APIs, or NLP models.

✔ Built for scalability

The master Excel file becomes a structured dataset for analytics.

Possible Future Enhancements

• Use LLM-based extraction (OpenAI/Claude) to reduce regex dependency
• Add confidence scores for each field
• Build cloud-based API endpoints
• Add support for health, motor, and life policy schedules
• Run OCR on GPU using pytesseract + CUDA

Final Thoughts

This project shows how OCR + regex + Streamlit can transform messy insurance PDFs into structured, useful data with zero manual effort.

It’s fast, accurate, and designed for real-world PDF inconsistencies.

If you’re working in InsurTech, automation, or document intelligence, this approach can save countless hours and reduce errors dramatically.

If you’d like to explore the complete source code for this project, check out the full implementation here: https://github.com/swatiP999/Insurance_Data_Extraction.