Building a Clinical Triage AI with Google’s MedGemma 1.5: A Deep Dive into Local LLMs and Reasoning Tokens

Healthcare is facing a massive administrative bottleneck. Doctors spend hours every day reading patient intake forms and writing clinical summaries, leading to severe burnout. With the early-2026 release of Google’s MedGemma 1.5, an open-weight multimodal medical AI, I wanted to see if I could build a secure, local solution to this problem.

In this project, I built an end-to-end AI Triage System. It acts as a virtual clinic assistant that interviews patients, extracts medical symptoms using Chain-of-Thought reasoning, and populates a secure SQLite physician dashboard — all running locally to protect patient privacy.

Here is a breakdown of the architecture, the technical hurdles I faced, and what I learned about engineering production-grade AI applications.

The Tech Stack

• Model: google/medgemma-1.5–4b-it (Loaded locally via Hugging Face Transformers)
• Frontend UI: Streamlit
• Backend/Database: Python sqlite3
• Key Architecture: Full CRUD cycle with state persistence and strict token-limit enforcement.

The Architecture

Before writing any code, I mapped out the data flow. The system requires two separate user personas (Patient and Physician) interacting with the same Streamlit application, bridged by a local SQLite database and driven by the MedGemma inference engine.

Architecture Diagram

Bridging Research and Application: Implementing the MedGemma Technical Report

When Google DeepMind and Google Research released the MedGemma Technical Report, they highlighted several core capabilities that make this model unique for healthcare. In this project, I intentionally built features to test and implement four of these specific research claims:

• Medical Agentic Behavior & Triaging: The paper states that MedGemma was fine-tuned for “patient interviewing, triaging, and clinical decision support.” By building a multi-turn conversational loop where the AI autonomously decides what follow-up question to ask based on previous symptoms, this project successfully implements MedGemma’s agentic triaging capabilities.
• Clinical Reasoning on Medical Text: The researchers highlight the model’s advanced clinical reasoning. By extracting and parsing the <unused94> and <unused95> reasoning tokens, this project exposes the model’s hidden Chain-of-Thought, proving its ability to logically evaluate a symptom before generating a response.
• Instruction Following & Constraint Compliance: A major challenge in medical AI is hallucinations. The paper emphasizes MedGemma’s instruction-tuning. I tested this by applying strict negative constraints in the system prompt (“Do not diagnose”) and formatting rules (“generate a structured 3-bullet-point summary”), which the 4B model followed flawlessly.
• Compute-Efficient Local Deployment: The paper emphasizes that the 4B parameter model was designed as a “starting point for developers” to build privacy-preserving AI. By running this model entirely locally on consumer hardware without cloud API calls, this project validates the research claim that high-level clinical reasoning can be achieved at the edge.

Challenges building this app:

Challenge 1: Taming the Jinja Chat Template

One of the first major hurdles I encountered was the strictness of the Gemma model family. Standard APIs allow you to pass a “System” role, but Gemma strictly enforces an alternating user/assistant Jinja chat template. If two user messages are sent in a row, the pipeline crashes.

The Solution: Instead of relying on standard roles, I utilized Prompt Engineering at the injection level. I dynamically attached the system instructions (restricting the AI from diagnosing and forcing a 3-bullet summary) to the user’s very first prompt behind the scenes. This maintained the strict template while successfully guiding the AI’s behavior.

Challenge 2: Parsing Reasoning Tokens (<unused94>)

MedGemma 1.5 was trained to “think before it speaks” using hidden reasoning tokens. During testing, the AI’s internal scratchpad (wrapped in <unused94>thought and <unused95>) began leaking onto the user interface, which is a poor patient experience. Furthermore, if the model hits its max_new_tokens limit, the closing <unused95> tag would be cut off entirely.

The Solution: I built a robust parsing engine in Python to intercept the model’s output before rendering it to the screen.

1. The code searches for the <unused94> token.
2. If found, it safely splits the string, extracts the AI’s internal logic, and hides it inside a collapsible Streamlit Expander labeled “View AI Reasoning Trace” (a great feature for physician oversight).
3. I added fallback logic so that if the output exceeds the token limit and drops the closing tag, the app gracefully catches the error without crashing.

Challenge 3: Creating a “Hard Stop”

A triage bot should not chat endlessly. Relying purely on AI to decide when to stop asking questions is dangerous in a healthcare setting.

The Solution: I implemented a programmatic “Hard Stop.” The Python script counts the user’s conversation turns. On the 5th turn, Python secretly appends a system directive to the end of the user’s prompt: [System Directive: Please output the CLINICAL SUMMARY now]. This forces the AI out of its questioning loop, generates the summary, and uses st.rerun() to completely disable the chat input box, finalizing the session.

Challenge 4: Full-Stack Architecture & Data Compliance

Printing a summary to a screen isn’t enough for a real clinic. To make this a viable product, I built a secondary view: The Doctor’s Dashboard.

When the AI generates the final summary, it is securely pushed to a local SQLite database (clinic.db). The physician can open the dashboard side-bar to view a real-time list of waiting patients.

Crucially, to address data-retention and HIPAA-compliance philosophies, I implemented complete CRUD functionality. Physicians have a “Discharge Patient” button that permanently executes a SQL DELETE command, wiping the temporary patient record from the database once the visit begins.

Conclusion

Building with MedGemma 1.5 proved that you no longer need massive cloud APIs to build highly capable, reasoning-driven medical tools. By combining a lightweight 4B parameter model with strict Python state-management and local databases, we can build tools that reduce physician burnout while keeping patient data entirely on-premise.

👉 Check out the code on GitHub: https://github.com/TejaswiniRay/Clinical-Triage 👉 Try the Live Demo: https://huggingface.co/spaces/TejaswiniRay/Clinical-Triage-Demo