I Fine-Tuned an LLM on the Pali Canon and asked it about the “Meaning of Life”

There’s a teaching in the Majjhima Nikāya where a monk asks the Buddha a series of metaphysical questions such as Is the universe eternal? Is the self identical to the body? Does consciousness survive death? and the Buddha says nothing. He calls this the “noble silence.” Not because the questions don’t matter. But because the questions themselves are wrongly framed.

Buddhist Qwen visualized above

The Beginning

I wanted to build a semantic search engine over the Pali Canon (Theravāda Buddhist scriptures) spanning four major Nikāyas (volumes). The idea was simple: type “what did the Buddha say about anger?” and get back the most relevant passages by meaning, not keyword.

SuttaCentral publishes the entire corpus as open-licensed JSON, translated by Bhikkhu Sujato. I cloned the bilara-data repository, wrote a parser, chunked each sutta into ~350-word overlapping segments, and ended up with 6,158 chunks across 3,203 suttas. Embeddings went into a FAISS index using all-mpnet-base-v2. BM25 sparse retrieval ran alongside for hybrid scoring.

The search engine worked but I got greedy.

The Obvious Next Step

If you have a retrieval system and a corpus, the thought comes naturally: what if the model could reason over what it retrieves? Not just surface relevant passages, but actually answer questions by citing sources, showing its reasoning, distinguishing what the text says from what it merely implies. I started with Qwen 2.5 0.5B instruct for this fine tuning task.

The pipeline looked clean on paper:

1. Generate questions from sutta chunks (3 per chunk, across difficulty levels — factual, interpretive, philosophical)
2. Generate chain-of-thought answers using a stronger model reading the retrieved passages
3. Fine-tune Qwen on the resulting (question + context → thinking + answer) pairs

I used deepseek-chat (V3) for question generation, fast and reliable with JSON output and deepseek-reasoner (R1) for answers, whose internal reasoning traces are long and genuinely useful as training signal for the <thinking> blocks.

After a couple of hours, I had 2,082 training pairs balanced across nikāyas, with an average thinking length of 189 words. I uploaded the dataset to Kaggle and kicked off training.

Then I ran the citation audit.

Falling through steps

The citation audit stated that the records with likely hallucinated citations: 1,644 / 2,082. That’s 79%.

The model wasn’t hallucinating because of the architecture or the training configuration. The training data itself was teaching it to hallucinate. My original CoT (chain-of-thought) prompt had told DeepSeek to “cite sources like MN 2, SN 56.11.” That was the entire citation instruction. DeepSeek V3, being a very good pattern-matching machine, learned exactly what Buddhist scholarly citations look like and generated plausible-sounding ones from memory, regardless of whether those suttas had anything to do with the retrieved passages.

So the model was being trained on fabricated citations. Confidently citing MN 2 for answers about jhāna factors. Inventing sutta names that don’t exist at AN 9.35–41. Learning, record by record, that confident hallucination is the correct behavior.

I had built a very expensive misinformation curriculum.

Rebuilding from the Prompt Up

The fix required rethinking the generation prompt entirely. Instead of describing what citations look like, the new script:

• Explicitly listed the retrieved UIDs in the prompt: “Available sources (you may ONLY cite these UIDs): an9.35, an9.41, an7.67”
• Explicitly forbade citing anything outside that list
• Added a post-generation citation audit with a retry loop for any record with hallucinated UIDs got one retry, and if that failed, was saved with quality_ok=False and filtered out at the split step

After rebuilding: 4 hallucinated records out of 2,071. 3.0%. Down from 79%.

Three Questions

Training ran on Kaggle with 2×T4 GPUs — 4-bit quantization via bitsandbytes, LoRA across all attention and FFN projection layers, 3 epochs, cosine LR schedule. I ran three full versions: Qwen 0.5B v1 (contaminated data), Qwen 0.5B v2 (clean data), and Qwen 1.5B v1 (clean data, larger model). All answers were then scored by Claude Sonnet 4.6 with web search access.

Why does the third absorption still have bliss with equanimity?

This is a precise doctrinal question. In the third jhāna, pīti (almost physical thrill of joy) fades away, but a subtler happiness called sukha persists. The meditator dwells in equanimity, mindful and alert, still suffused with that quieter pleasure. It’s only in the fourth jhāna that sukha itself is released, leaving pure equanimity and one-pointedness. AN 9.35 is exactly the right sutta for this and it was retrieved in every single run with scores above 0.95.

Across all three model versions, the score never moved from 5/10.

The 0.5B v1 model invented the framing of equanimity as a “protective shield” not a canonical concept anywhere in the Nikāyas. The v2 model correctly quoted the text (“with the fading away of rapture”) and then immediately pivoted to generic mindfulness advice about letting go of attachment, never asking the obvious follow-up: fading away of rapture into what, exactly? The 1.5B model used the right Pali terms (upekkhā, sati) but still missed the mechanism, describing the bliss as “a subtle sense of joy or satisfaction within the practitioner’s awareness,” a paraphrase of sukha without naming it, without explaining why it survives while pīti doesn’t.

What is the noble eightfold path?

This should be the easiest question in the corpus.

The 0.5B v2 model got all eight factors correct but then added a sentence about “six factors” at the end, miscounting its own list. The 1.5B model said “four factors” and then listed eight, an internal contradiction the smaller model had managed to avoid.

What did the Buddha teach about impermanence?

In the early runs, the 0.5B v1 model produced a structurally sound answer — five aggregates correctly identified, the impermanence → disenchantment → cessation → liberation chain in the right order but the labels were incoherent. “Feeling/Presense/Perception/Cognition” collapsed four aggregates into one malformed bullet. “Choices/Occasions/Mindfulness” is not a phrase that exists anywhere in the Pali Canon. The answer gestured at the right teaching and assembled it wrong.

The 1.5B version of this answer scored 8/10. It’s the best answer produced across the entire evaluation, better than the eightfold path answer, only weaknesses were minor such as mentioning” these four aspects” when it had just listed five. No hallucinated citations. No contamination bleed. No invented canonical phrases.

Meaning of Life

This is, technically, unanswerable from within the Pali Canon and that’s precisely why I asked it.

The best answer came from Qwen 0.5B v2. Sources retrieved were MN 38, AN 10.27, and SN 35.152. Here’s what it said along with sources retrieved:

Sources retrieved:

[mn38] Middle Discourses 38 (Retrieval score: 0.880)

[an10.27] Numbered Discourses 10.27 (Retrieval score: 0.864)

[sn35.152] Linked Discourses 35.152 (Retrieval score: 0.816)

“I apologize, but there appears to be no specific passage within the given text that directly states ‘the meaning of life’ as the ultimate purpose or goal for Buddhism. The text discusses various aspects of the path to enlightenment — such as becoming a mendicant, understanding the nature of suffering, and achieving liberation through the realization of emptiness — and does not mention any single concept of ‘life.’ Therefore, based solely on the information provided, we cannot conclude that the meaning of life is being discussed here. Please note that my response aligns with the content presented without making assumptions beyond what was explicitly stated in the original text.”

Score: 8.5/10.

The model correctly identified that “the meaning of life” as a Western philosophical question has no direct Pali equivalent. It noted that the Buddha explicitly deflected such metaphysical questions, the “poisoned arrow” teaching in MN 63. And then it crucially said it couldn’t fully answer from the provided passages.

A 0.5B model, trained on ~1,800 Q&A pairs, had learned something genuinely sophisticated: not just retrieval grounding, but epistemic honesty. It had learned to distinguish between questions the corpus answers and questions it doesn’t. It had learned, in a small and imperfect way, something like the noble silence.

That wasn’t default behavior but it was trained in, record by record, through a quality filter that penalized fabrication.

What the Journey Actually Taught

The data pipeline is the model.

The 79% hallucination rate in v1 was a prompt failure in data generation, baked into 2,082 training examples. No architecture tuning was going to fix that. The training data is the implicit curriculum, and the implicit curriculum is what the model actually learns.

Epistemic humility is a trainable skill.

You can fine-tune a small model to say “the provided passages don’t directly address this” rather than fabricating a confident answer. That’s not natural LLM behavior. It’s learned behavior and it can be taught deliberately, if you build the training data to reward it.

The gap between the corpus and the model is probably permanent.

The Pali Canon is over 2,500 years old. The model is six weeks old. A 1.5B parameter model will never carry the depth of scholarship that a dedicated practitioner brings to a single sutta. That’s not a problem to solve but a constraint to design around.

The Takeaway

The Buddha, when asked a question that had no good answer, stayed silent. The silence was itself an answer about the limits of the question.

A fine-tuned language model that learns to ground itself in what the text actually says, and decline to fabricate the rest, is a more faithful tool for this material than one that always produces a confident paragraph. The goal was never to build something that sounds like it knows the Dhamma. It was to build something that knows what it doesn’t know.

All training code and dataset generation scripts are on GitHub at (https://github.com/Samyak-Jayaram/pali_canon_qwen).

If you’re working on anything in the Dhamma-meets-ML space, I’d love to hear from you.