Fine-Tuning a Vision Language Model using QLoRA for Document-to-Markdown Generation

Fine-Tuning a Vision Language Model using QLoRA for Document-to-Markdown Generation

From Scanned Documents to Structured Markdown using Qwen2-VL-2B-Instruct

Introduction

Document understanding has become one of the most important applications of Artificial Intelligence. Traditional OCR systems can extract text from images, but they often fail to preserve formatting, tables, headings, equations, and structured layouts.

Vision Language Models (VLMs) solve this problem by understanding both:

• Visual information from images
• Textual relationships from language

In this project, we fine-tuned the Qwen2-VL-2B-Instruct model using QLoRA (Quantized Low-Rank Adaptation) for the task of converting document images into structured Markdown.

The complete workflow includes:

• Dataset exploration
• ChatML data preparation
• QLoRA fine-tuning
• Markdown generation
• Validation and testing
• Gradio/Streamlit deployment

1. Project Objective

The main objective of this assignment was to:

• Fine-tune a Vision Language Model
• Use parameter-efficient training with QLoRA
• Convert document images into Markdown format
• Generate outputs for unseen document images

This project demonstrates how multimodal AI systems can understand document layouts and produce structured documentation automatically.

2. Technologies Used

ComponentTechnologyFrameworkPyTorchTransformer LibraryHugging Face TransformersFine-TuningPEFT + QLoRADatasetNougat Training DatasetPlatformKaggle GPU T4 x2Base ModelQwen2-VL-2B-InstructDeploymentGradio / Streamlit

3. Understanding Vision Language Models

A Vision Language Model (VLM) combines:

• A vision encoder for processing images
• A language model for generating text

Unlike traditional OCR pipelines, VLMs can understand:

• Document structure
• Tables
• Layouts
• Headers
• Mathematical notation
• Contextual formatting

The model used in this assignment was:

Qwen2-VL-2B-Instruct

This model supports image understanding and text generation together.

4. Why QLoRA?

Training large models from scratch requires massive GPU memory and computational resources.

QLoRA solves this problem by:

• Loading the model in 4-bit quantized form
• Freezing original model weights
• Training only lightweight LoRA adapters

Advantages:

• Lower VRAM usage
• Faster training
• Efficient fine-tuning
• Works well on Kaggle GPUs

5. Dataset Exploration

The dataset used was:

Nougat Training Dataset Example

The dataset contains:

• Document images
• Corresponding Markdown outputs

Each sample consists of:

• Input image
• Ground truth markdown

Example Dataset Pair

Input Image

Ground Truth Markdown

# Introduction

This document explains the concept of machine learning.

## Types

- Supervised Learning
- Unsupervised Learning

The dataset helped the model learn how visual layouts map to structured Markdown.

6. Data Preparation

The dataset was converted into ChatML format.

Each sample contained:

1. Image
2. Instruction prompt
3. Markdown target

Prompt Example

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": "Convert this document image into markdown."}
        ]
    },
    {
        "role": "assistant",
        "content": markdown_text
    }
]

This structure allowed the VLM to learn image-to-text generation.

7. Dataset Splitting

The dataset was divided into:

• 80% Training Data
• 20% Validation Data

This helped evaluate:

• Generalization
• Markdown generation quality
• Performance on unseen layouts

8. QLoRA Fine-Tuning Configuration

The model was fine-tuned using the following settings:

ParameterValueQuantization4-bitLoRA Rank16Epochs3Batch Size1Gradient Accumulation4Learning Rate2e-4Image Resolution768 px

LoRA Target Modules

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

9. Training Process

The training pipeline included:

• Loading Qwen2-VL in 4-bit mode
• Applying LoRA adapters
• Training only adapter weights
• Monitoring training loss
• Running validation predictions

Training Workflow

10. Training Results

The model showed stable convergence during training.

EpochTraining LossValidation Loss11.821.7421.211.1530.890.96

The decreasing validation loss indicated successful learning.

11. Markdown Generation Results

The core task was generating Markdown from validation images.

Example Output

Input Document

Ground Truth Markdown

# Data Analysis

This section explains the experimental setup.

## Observations

- Accuracy improved
- Loss decreased

Generated Markdown

# Data Analysis

This section explains the experimental setup.

## Observations

- Accuracy improved
- Loss reduced significantly

The generated output preserved:

• Headings
• Structure
• Bullet points
• Semantic meaning

12. Testing on Training Images

The model was tested on:

• 3 training images

The outputs showed:

• High formatting accuracy
• Strong structure preservation
• Correct markdown hierarchy

Training Image Example

InputGenerated OutputScientific paper pageStructured markdown with headingsNotes pageBullet-point markdownTable documentMarkdown table representation

13. Testing on Unseen Images

The model was further tested using:

• 3 unseen document images

These images were uploaded into:

/kaggle/working/unseen_docs

Unseen Document Testing Pipeline

Results

The model successfully generated:

• Titles
• Paragraphs
• Bullet lists
• Structured formatting

However, small formatting inconsistencies appeared in:

• Complex tables
• Mathematical equations
• Multi-column layouts

14. Visualization Module

The notebook included a visualization module to display:

• Input image
• Ground truth markdown
• Predicted markdown

Visualization Layout

plt.figure(figsize=(10,6))
plt.imshow(image)
plt.title("Input Document")

This helped compare actual and generated outputs side-by-side.

15. App Deployment

A Gradio/Streamlit application was built for inference.

The app allows users to:

• Upload a document image
• Generate Markdown instantly

App Workflow

Local Run Commands

pip install -r requirements.txt
python app.py

16. Challenges Faced

Several challenges were encountered during training:

GPU Memory Constraints

Solved using:

• 4-bit quantization
• Gradient accumulation
• Small batch size

Complex Layouts

Some multi-column documents produced formatting errors.

Markdown Formatting

Maintaining exact spacing and structure required careful prompting.

17. Key Learnings

This assignment provided practical experience in:

• Vision Language Models
• Multimodal AI
• QLoRA fine-tuning
• Parameter-efficient learning
• Document understanding
• Markdown generation
• Hugging Face ecosystem

The project also demonstrated how efficient fine-tuning can adapt powerful VLMs using limited hardware resources.

18. Future Improvements

Possible future enhancements include:

• Larger training datasets
• Better prompt engineering
• Table-aware formatting
• Equation rendering support
• OCR hybrid pipelines
• Multi-language document understanding

19. Conclusion

In this project, we successfully fine-tuned the Qwen2-VL-2B-Instruct Vision Language Model using QLoRA for document-to-markdown generation.

The model learned how to:

• Understand document layouts
• Extract structured information
• Generate readable Markdown outputs

Despite using lightweight parameter-efficient training, the model achieved strong performance on both validation and unseen document images.

This project highlights the growing power of multimodal AI systems in automating document understanding and structured content generation.

20. References

1. Hugging Face Transformers
2. Qwen2-VL Documentation
3. PEFT Library
4. QLoRA Research Paper
5. Nougat Dataset
6. PyTorch Documentation