Sorting My Bank Emails Offline With Small AI Models

A weekend hackathon project to track my spending without handing over my banking passwords.

Too Many Bank Emails

It was a regular Tuesday evening. I opened my inbox and had 47 unread emails — all from banks. HDFC. ICICI. Kotak. SBI. Amex.

Endless ₹20 UPI alerts for my evening chai. Flashy banners for personal loans I don’t want. Password-protected PDFs. KYC warnings. And a graveyard of expired OTPs.

I have four active bank accounts and two credit cards. Somewhere in that mess of notifications was the answer to a simple question: “How much did I actually spend this month?”

I couldn’t answer it without spending 30 minutes copying amounts into a spreadsheet. I also didn’t want to hand over my bank credentials to a third-party app just to see a pie chart.

So, I tried building a script to do it for me.

The Problem with Indian Banking Emails

If you live in India, you know banks send a lot of emails. It’s good for transparency, but it clutters your inbox. Every single UPI payment, card swipe, and EMI deduction gets its own email.

Mixed in with the actual transactions are promotional emails, e-statements, and credit score alerts. Parsing all of this manually is annoying. Existing apps usually ask for SMS access or explicit bank API access, which means sharing your data.

I figured all I really needed was my email and a bit of local processing.

The Build Small Hackathon

Right around then, I saw the Hugging Face × Gradio Build Small Hackathon. The rule was simple: build an app using models under 32B parameters.

This was a good constraint. Instead of just sending everything to a big cloud API like GPT-4, I had to figure out how to run it locally. No continuous cloud inference, and no data leaving my laptop after fetching the emails.

The result is a small python app that reads banking emails, extracts transactions, and shows spending analytics. It runs entirely offline using a few small models totaling under 5B parameters.

How It Works

The pipeline is split into three steps:

Step 1: Fetching the Emails

The script connects to your email provider via IMAP using an app password.

• Delta sync: It only downloads new emails since the last run.
• Filtering: It checks the sender domains to only pull emails from known Indian banks.
• Cleanup: The IMAP credentials are wiped from memory after the sync is done.

There’s no AI here — just standard Python IMAP libraries to keep things fast.

Step 2: The Vision Fallback

A lot of promotional bank emails are just giant images.

To handle this without a massive model, I used Moondream2, a ~1.8B parameter vision-language model. If an email has almost no text in the body, Moondream2 runs OCR on the images to pull the text out. It only runs when necessary, which saves compute.

Step 3: Classification and Extraction

This is where the actual data processing happens. I used Qwen 2.5 3B Instruct. For a 3-billion parameter model, it works surprisingly well. It handles two main jobs:

1. Categorization It sorts emails into 8 categories (like FUNDS_DEBITED, CREDIT_LOAN_PROMOTION, OTP_SECURITY_ALERT, etc.). Indian banking emails have specific phrases (like "debited from A/c"), so I wrote a prompt with a few examples to help the model catch these.

2. Data Extraction For actual transactions, the model pulls out a JSON object:

{
  "amount": 1500.00,
  "transaction_type": "debit",
  "merchant": "Swiggy",
  "card_last4": "4421",
  "category": "Food & Dining",
  "transaction_date": "2025-06-14",
  "payment_mode": "UPI"
}

It manages to handle different currency formats like “Rs. 1,500”, “INR 1500”, and “₹1,23,456.78” reasonably well.

All of this data gets saved to a local SQLite database. I also set up a basic RAG (Retrieval-Augmented Generation) flow using SQLite’s FTS5 search, so you can type questions like “How much did I spend on food?” and get an answer based on your local database.

Handling Model Switching (Without Melting My Laptop) Running a text model and a vision model on the same machine can quickly eat up all your RAM. To fix this, the app uses simple conditional routing.

The system doesn’t run both models on every email. A Python script checks the email’s text length first. If it has enough readable text, it completely bypasses the vision model and goes straight to Qwen. If it’s just a giant image, it routes the email to Moondream2, extracts the text, and then hands that text over to Qwen. By switching between the models based on the email content, it saves a ton of compute. Because both models combined are so small, local inference servers like LM Studio can easily juggle them without the system grinding to a halt.

Why SQLite FTS and not a Vector DB? I thought about using a vector database (like Chroma or FAISS) for the RAG setup, but it felt like overkill. Vector databases are great for semantic search, but financial queries usually need exact keyword matches (like specific merchants, dates, or categories). Plus, using a vector DB means running an embedding model and adding another heavy dependency. SQLite FTS5 gives me fast, full-text search out of the box with zero extra bloat.

The Architecture

┌─────────────────────────────────────────────────┐
│              Gradio Dashboard UI                │
│   Sync Log │ Analytics │ Offers │ Data Browser  │
└──────┬──────────────┬──────────────┬────────────┘
       │              │              │
  ┌────▼──────┐  ┌────▼─────────┐  ┌─▼──────────┐
  │   Email   │  │  Classifier  │  │  SQLite DB │
  │   (IMAP)  │  │  Qwen 2.5 3B │  │   + FTS5   │
  └────┬──────┘  └──────────────┘  └────────────┘
       │
  ┌────▼──────┐
  │  Vision   │
  │ Moondream │
  │  (~1.8B)  │
  └───────────┘

At roughly 4.8B total parameters, it runs fine on a standard laptop.

Takeaways

A few things I learned while building this:

• Small models are getting better: Qwen 2.5 3B handles structured JSON extraction much better than I expected, provided you give it a clear prompt.
• Context helps: Hardcoding some knowledge about Indian banking formats into the prompt was more effective than just trying to use a larger model.
• Local data is nice: It feels good to have financial data stay entirely in a local SQLite file rather than sitting on a third-party server.

Try It Out

I put the code on GitHub under the MIT License. If you want to try it out, you don’t need any API keys or bank logins — just an app password for your email.

git clone https://github.com/D-Aditya-Amarnath/Small-Card-Agent
cd Small-Card-Agent
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python app.py

# Open http://localhost:7860

(Note: You’ll need LM Studio running locally to serve Qwen 2.5 3B and Moondream2, or you can run it via a Hugging Face Space).

What’s Next

It’s just a hackathon project, so it’s not perfect. If I keep working on it, here is what I want to add:

• PDF parsing: A lot of statements are sent as password-protected PDFs. Reading those would be a big help.
• Budget alerts: A simple local check to tell me if I’m spending too much in a certain category.
• Local language support: Handling emails sent in Hindi or regional languages.
• CSV Export: Just a button to dump the database to a spreadsheet.

(And again: all data stays 100% local. Nothing leaves your machine.)