Build an Intelligent Document Processing with Confidence Scores with GPT-4o

An Intelligent Document Processing (IDP) provides actionable insights through confidence scores, allowing you to evaluate process performance, identify areas for review, and automate document processing and data extraction with precision, minimizing the need for manual oversight. In this blog, we will build an IDP app with confidence scores using GPT-4o.

We will be using the logprobs parameter in the OpenAI Chat Completions API. The log probabilities returned by the API, when logprobs is enabled, indicate the likelihood of each token in a GPT response, allowing assessment of model confidence and hallucination detection, with higher values (closer to zero) signifying greater confidence. It can help and can be utilized to gauge the confidence (probability) of the model’s response.

[embed]Using Logprobs to Gauge GPT’s Confidence in OCR Result In my recent blog post on parsing complex PDFs using GPT, I explored the potential of incorporating confidence scores…djajafer.medium.com

Evaluating Response Confidence with Log Probabilities in OpenAI APIs

To assess the confidence score of an OpenAI API response using log probabilities, follow these steps:

• Enable logprobs in your API request to obtain token-level log probabilities.
• Interpret log probabilities: values closer to 0 indicate higher token-level confidence.
• Calculate the overall response confidence by: (1). Averaging log probabilities. (2). Converting to linear probabilities for easier interpretation. (3). Use this score as an indicator of the model’s response confidence.

Calculating Confidence Scores: Methods and Considerations

Average of Log Probabilities

• Calculate the average of log probabilities for each token in the response.
• Higher averages (closer to 0) indicate greater confidence.

Example:

• Sentence: ‘Water Bill Receipt’
• Token: [‘Water’, ‘ Bill’, ‘ Receipt’]
• logprobs for each token.

https://platform.openai.com/tokenizer

{'token': 'Water', 'linear_probability': 65.95, 'logprob': -0.4162598}
{'token': ' Bill', 'linear_probability': 98.33, 'logprob': -0.016870093}
{'token': ' Receipt', 'linear_probability': 73.01, 'logprob': -0.314637}

• Average

Average for Sentence: (-0.4162598 -0.016870093 -0.314637) / 3 = -0.249255631

Converting to Linear Probabilities

• Apply the exponential function (e^x) to each log probability.
• Average the resulting linear probabilities.
• Values closer to 1 indicate higher confidence.
• Example:

probability = np.round(np.exp(-0.249255631) * 100, 2)
probability = 77.94

Implementation

First, import the necessary libraries.

import pypdfium2 as pdfium
import backoff
import asyncio
import os
import base64
from io import BytesIO

from openai import OpenAIError
from openai import AsyncOpenAI

import tiktoken
import json
import numpy as np

from langchain_core.output_parsers import StrOutputParser, JsonOutputParser
from pydantic import BaseModel, Field
from typing import Union, List

Set up OpenAI API configuration by specifying the API key and base URL.

MODEL = "gpt-4o"
baseurl = ""
apikey = ""

os.environ["OPENAI_API_BASE"] = baseurl
os.environ["OPENAI_API_KEY"] = apikey

clienta = AsyncOpenAI(api_key=apikey,  base_url=baseurl)

Extract and process document content by capturing all images, translating JSON keys into English, identifying document types (e.g., Electricity Bill), and returning the results in a structured JSON format.

class Step(BaseModel):
    """Represents a key-value pair with potential nested structure."""
    key: Union[str, List['Step']]
    value: Union[str, List['Step']]

class Reasoning(BaseModel):
    """Represents a reasoning structure with key-value pairs."""
    data: List[Step] = Field(description="Key-value pairs in JSON.")
#'''

# Required to support self-referencing classes with Pydantic
Step.update_forward_refs()

@backoff.on_exception(backoff.expo, OpenAIError)
async def parse_page_with_gpt(base64_image: str) -> dict:
    messages = [
        {
            "role": "system",
            "content": """

            You are an OCR expert. Your task is to extract content from images.

            Rules and guidelines:
            - Extract all content images and ensure that all content from the images is captured.
            - Translate all JSON keys or field names into English, but keep values in their original language.
            - Identify the document type (e.g. Electricity Bill)
            - Return the result in a structured JSON format without any additional explanation

            """
        },
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Return all content"},
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{base64_image}",
                        "detail": "high"
                    },
                },
            ],
        }
    ]

    response = await clienta.beta.chat.completions.parse(
        model=MODEL,
        messages=messages,
        temperature=0,
        logprobs=True,
        top_logprobs=2,
        response_format=Reasoning
    )

    tokens_logprobs = zip(response.choices[0].logprobs.content, response.choices[0].logprobs.content)

    content = []

    content_with_confidence = [
        {
            "token": token.token.replace('",\n', ''),
            "linear_probability": np.round(np.exp(logprob.logprob) * 100, 2),
            "logprob": logprob.logprob
        }
        for token, logprob in tokens_logprobs
    ]

    text = response.choices[0].message.content

    content.append({
        "content": text
    })

    # Return both content and confidence in a structured output
    return content_with_confidence, content

async def document_analysis(filename: str) -> list:
    """
    Document Understanding

    Args:
        filename: pdf filename str
    """

    pdf = pdfium.PdfDocument(filename)
    images = []

    # Convert each page in the PDF to an image and encode it as base64
    for i in range(len(pdf)):
        page = pdf[i]
        image = page.render(scale=4).to_pil()
        buffered = BytesIO()
        image.save(buffered, format="JPEG")
        img_byte = buffered.getvalue()
        img_base64 = base64.b64encode(img_byte).decode("utf-8")
        images.append(img_base64)

    # Process each image with GPT and extract text
    text_of_pages = await asyncio.gather(*[parse_page_with_gpt(image) for image in images])


    return text_of_pages

We will be using this document for our testing.

Invoice example

Perform OCR scan.

data = await document_analysis("invoice-example.png.pdf")

Create functions to retrieve tokens and extract relevant content.

def extract_tokens(data):
    extracted_token = []

    for sublist in data:
        extracted_token = sublist[0]

    return extracted_token

tokens = extract_tokens(data)
tokens

def extract_content(data):
    extracted_content = []

    for sublist in data:
        extracted_content = (sublist[1][0]["content"])

    json_string = extracted_content.replace('```json\n', '').replace('\n```', '')
    print(json_string)

    # Load JSON data
    json_data = json.loads(json_string)
    return json_data

content = extract_content(data)

Result of extract_tokens:

Result of extract_content:

Extract keys and values from content.

def extract_key_values(data, parent_key=""):
    """
    Recursively extracts key values from the data.

    Args:
        data (dict or list): The data to extract key values from.
        parent_key (str): The parent key (used for nested keys).

    Returns:
        dict: A dictionary containing the extracted key values.
    """

    extracted_data = {}

    if isinstance(data, dict):
        for key, value in data.items():
            new_key = f"{parent_key}{key}" if parent_key else key

            if isinstance(value, (dict, list)):
                extracted_data.update(extract_key_values(value, new_key + "."))
            else:
                extracted_data[new_key] = value

    elif isinstance(data, list):
        for index, item in enumerate(data):
            new_key = f"{parent_key}[{index}]" if parent_key else str(index)

            if isinstance(item, (dict, list)):
                extracted_data.update(extract_key_values(item, new_key + "."))
            elif isinstance(item, str):
                extracted_data[new_key] = item
            else:
                # Handle non-dict, non-list, non-str items (e.g., numbers)
                extracted_data[new_key] = str(item)

    return extracted_data

extracted_data = extract_key_values(content)

Calculate confidence scores for key-value pairs.

result = {}
encoding = tiktoken.encoding_for_model("gpt-4o")

for key, value in extracted_data.items():
    val = encoding.encode(str(value))
    utf8_vals = [encoding.decode_single_token_bytes(token) for token in val]
    utf8_vals = [token.decode('utf-8', errors='ignore') for token in utf8_vals]

    confidence = []
    for token in tokens:
        for val in utf8_vals:
            if token["token"] == val:
                confidence.append(token["logprob"])
                break  # Move to the next token

    if confidence:
        average = np.round(np.exp(sum(confidence) / len(confidence)) * 100, 2)
        result[key] = average

Merge key value and confidence score.

content = []

for key in result:

    if key.find(".key") != -1:
        content.append({
            key: extracted_data[key]
        })

    elif key.find(".value") != -1:
        content.append({
            key: result[key]
        })

Create a function to convert it to JSON.

def convert_to_json(data):
    result = {}
    for item in data:
        for key, value in item.items():
            keys = key.split('.')
            temp = result
            for k in keys[:-1]:
                if '[' in k:
                    k, index = k.split('[')
                    index = int(index[:-1])
                    if k not in temp:
                        temp[k] = []
                    while len(temp[k]) <= index:
                        temp[k].append({})
                    temp = temp[k][index]
                else:
                    if k not in temp:
                        temp[k] = {}
                    temp = temp[k]
            k = keys[-1].split(']')[0] if ']' in keys[-1] else keys[-1]
            temp[k] = value
    return json.dumps(result, indent=4)

final_result = convert_to_json(content)
json_data = json.loads(final_result)
json_data

At the end we will be getting the value output and confidence scores.

Value Output

{'data': [{'key': 'Document Type', 'value': 'Invoice'},
  {'key': 'Company Name', 'value': 'Lotus Digital Studios'},
  {'key': 'Address', 'value': 'Suntec 8 Temasek, Singapore 038988'},
  {'key': 'Bill To',
   'value': 'Xuegang Xiong, 3028 Ubi Road 3, Singapore 408657'},
  {'key': 'Ship To',
   'value': 'Xuegang Xiong, 5326B Nihao Ave, Singapore 538725'},
  {'key': 'Invoice #', 'value': 'SG-001'},
  {'key': 'Invoice Date', 'value': '29/01/2019'},
  {'key': 'P.O.#', 'value': '1320/2019'},
  {'key': 'Due Date', 'value': '26/04/2019'},
  {'key': 'Items',
   'value': [{'key': 'QTY', 'value': '1'},
    {'key': 'Description', 'value': '35mm zoom lens carrying case'},
    {'key': 'Unit Price', 'value': '100.00'},
    {'key': 'Amount', 'value': '100.00'}]},
  {'key': 'Items',
   'value': [{'key': 'QTY', 'value': '2'},
    {'key': 'Description', 'value': 'Animation storyboard template'},
    {'key': 'Unit Price', 'value': '45.00'},
    {'key': 'Amount', 'value': '90.00'}]},
  {'key': 'Items',
   'value': [{'key': 'QTY', 'value': '3'},
    {'key': 'Description', 'value': 'Cell shading and highlighting'},
    {'key': 'Unit Price', 'value': '16.00'},
    {'key': 'Amount', 'value': '48.00'}]},
  {'key': 'Subtotal', 'value': '238.00'},
  {'key': 'GST 7.0%', 'value': '16.66'},
  {'key': 'Total', 'value': '$254.66'},
  {'key': 'Terms & Conditions', 'value': 'Payment is due within 15 days'},
  {'key': 'Bank Details',
   'value': 'Developmental Bank of Singapore, Account Number: 123456789, Routing Number: 543210'}]}

Confidence Scores

{'data': {'': [{'key': 'Document Type', 'value': 99.99},
   {'key': 'Company Name', 'value': 100.0},
   {'key': 'Address', 'value': 96.5},
   {'key': 'Bill To', 'value': 97.32},
   {'key': 'Ship To', 'value': 97.19},
   {'key': 'Invoice #', 'value': 100.0},
   {'key': 'Invoice Date', 'value': 100.0},
   {'key': 'P.O.#', 'value': 100.0},
   {'key': 'Due Date', 'value': 100.0},
   {'key': 'Items',
    'value': {'': [{'key': 'QTY', 'value': 100.0},
      {'key': 'Description', 'value': 100.0},
      {'key': 'Unit Price', 'value': 99.89},
      {'key': 'Amount', 'value': 99.89}]}},
   {'key': 'Items',
    'value': {'': [{'key': 'QTY', 'value': 100.0},
      {'key': 'Description', 'value': 100.0},
      {'key': 'Unit Price', 'value': 99.89},
      {'key': 'Amount', 'value': 99.89}]}},
   {'key': 'Items',
    'value': {'': [{'key': 'QTY', 'value': 100.0},
      {'key': 'Description', 'value': 100.0},
      {'key': 'Unit Price', 'value': 99.89},
      {'key': 'Amount', 'value': 99.89}]}},
   {'key': 'Subtotal', 'value': 99.89},
   {'key': 'GST 7.0%', 'value': 99.85},
   {'key': 'Total', 'value': 99.75},
   {'key': 'Terms & Conditions', 'value': 99.67},
   {'key': 'Bank Details', 'value': 96.97}]}}

Let’s try with receipt document.

Values:

{'data': [{'key': 'Instructions',
   'value': 'Please collect a table tent and take a seat. We will be serving your food shortly.'},
  {'key': 'Table Service Device', 'value': '56'},
  {'key': 'Order Number', 'value': '2495'},
  {'key': 'Restaurant Information',
   'value': [{'key': 'Name', 'value': "McDonald's Queensway"},
    {'key': 'Address',
     'value': 'Queensway #01-580 Ridout Tea Garden, Singapore 149066'},
    {'key': 'Tel', 'value': '64756102'},
    {'key': 'Company', 'value': 'Hanbaobao Pte Ltd'},
    {'key': 'GST REGN No', 'value': 'M2-0023981-4'},
    {'key': 'Side', 'value': 'MFY Side 2'}]},
  {'key': 'Invoice Number', 'value': '300332400035964'},
  {'key': 'Order Details',
   'value': [{'key': 'Order', 'value': '#95 - CSD #24'},
    {'key': 'Date & Time', 'value': '02/11/2024 08:29:14'},
    {'key': 'Items',
     'value': [{'key': '1 Breakfast Deluxe ML', 'value': '9.00'},
      {'key': '1 No Add On', 'value': ''},
      {'key': '1 EVM Hashbrowns', 'value': ''},
      {'key': '1 Choc Oat', 'value': ''},
      {'key': '1 Sausage Muffin Meal', 'value': '5.00'},
      {'key': '1 No Add On', 'value': ''},
      {'key': '1 EVM Hashbrowns', 'value': ''},
      {'key': '1 Oat', 'value': ''}]}]},
  {'key': 'Subtotal', 'value': '14.00'},
  {'key': 'Eat-In Total (Incl GST)', 'value': '14.00'},
  {'key': 'Cashless Mastercard', 'value': '14.00'},
  {'key': 'Total Includes GST of', 'value': '1.16'},
  {'key': 'Cashless Transaction Information',
   'value': [{'key': 'Date & Time', 'value': '02/11/24 08:31:50'},
    {'key': 'MID', 'value': '000000200000033'},
    {'key': 'TID', 'value': '70332407'},
    {'key': 'PAN Entry', 'value': 'PAYPASS'},
    {'key': 'Card & Exp', 'value': '547198XXXXXX2175 XX/XX'},
    {'key': 'App ID', 'value': 'N/A'},
    {'key': 'App Name', 'value': 'N/A'},
    {'key': 'App Eff & Exp', 'value': 'N/A XX/XX'},
    {'key': 'Card Type', 'value': 'Mastercard'},
    {'key': 'Auth & Stan', 'value': 'F04552 039427'},
    {'key': 'Cryptogram', 'value': 'N/A'},
    {'key': 'Verification', 'value': 'NONE'}]},
  {'key': 'Thank You Message',
   'value': "Thank You For Dining With Us! Share Your Feedback and Enjoy a Treat. Skip the Q with New Mobile Order In My McDonald's App! Try it! Download Now: Google Play/App Store RMHC Donation not subjected to GST."}]}

Confidence Scores:

{'data': {'': [{'key': 'Instructions', 'value': 99.56},
   {'key': 'Table Service Device', 'value': 99.98},
   {'key': 'Order Number', 'value': 100.0},
   {'key': 'Restaurant Information',
    'value': {'': [{'key': 'Name', 'value': 97.67},
      {'key': 'Address', 'value': 98.07},
      {'key': 'Tel', 'value': 87.33},
      {'key': 'Company', 'value': 97.18},
      {'key': 'GST REGN No', 'value': 92.49},
      {'key': 'Side', 'value': 93.25}]}},
   {'key': 'Invoice Number', 'value': 97.47},
   {'key': 'Order Details',
    'value': {'': [{'key': 'Order', 'value': 93.56},
      {'key': 'Date & Time', 'value': 99.85},
      {'key': 'Items',
       'value': {'': [{'key': '1 Breakfast Deluxe ML', 'value': 99.21},
         {'key': '1 No Add On'},
         {'key': '1 EVM Hashbrowns'},
         {'key': '1 Choc Oat'},
         {'key': '1 Sausage Muffin Meal', 'value': 99.39},
         {'key': '1 No Add On'},
         {'key': '1 EVM Hashbrowns'},
         {'key': '1 Oat'}]}}]}},
   {'key': 'Subtotal', 'value': 99.38},
   {'key': 'Eat-In Total (Incl GST)', 'value': 99.38},
   {'key': 'Cashless Mastercard', 'value': 99.38},
   {'key': 'Total Includes GST of', 'value': 91.65},
   {'key': 'Cashless Transaction Information',
    'value': {'': [{'key': 'Date & Time', 'value': 99.84},
      {'key': 'MID', 'value': 95.31},
      {'key': 'TID', 'value': 99.92},
      {'key': 'PAN Entry', 'value': 99.75},
      {'key': 'Card & Exp', 'value': 98.51},
      {'key': 'App ID', 'value': 97.63},
      {'key': 'App Name', 'value': 97.63},
      {'key': 'App Eff & Exp', 'value': 98.76},
      {'key': 'Card Type', 'value': 99.99},
      {'key': 'Auth & Stan', 'value': 99.98},
      {'key': 'Cryptogram', 'value': 97.63},
      {'key': 'Verification', 'value': 99.97}]}},
   {'key': 'Thank You Message', 'value': 96.59}]}}

References

[embed]Using logprobs | OpenAI Cookbook Open-source examples and guides for building with the OpenAI API. Browse a collection of snippets, advanced techniques…cookbook.openai.com