When Cosine Distance Is Not Enough: Agentic Chunking Inside ODC

Where Level 4 Leaves Off

Semantic Chunking (Level 4) groups sentences by meaning. It compares embeddings, finds where the topic shifts, and draws a boundary. That gets you topically coherent chunks without touching the words themselves.

But a sentence isn’t always one fact. Some sentences carry two claims joined by “and.” Some carry a pronoun that only resolves three sentences back. Cosine distance tells you where a topic changes. It can’t tell you that “it” means “the human liver.”

That’s the gap Level 5 closes. Not because Level 4 fails at its job, but because the question after similarity grouping is a different question: what if chunking operated on atomic facts rather than sentences?

So I built Agentic Chunking app, the last piece of the five-level chunking framework, and ran them against 32 test cases to find out what an LLM-judgment chunking layer actually does inside ODC.

What Level 5 Solves

Agentic chunking hands the chunking decision to an LLM. The model reads the text, decides what counts as one atomic fact, and decides which facts belong together thematically. It’s a judgment call, not a fixed rule applied to a vector space.

That’s a real shift. Levels 1 through 3 split on structure: characters, recursive separators, markdown headers. Level 4 splits on meaning, expressed as distance between embeddings. Level 5 is the first level where the chunking decision isn’t computed. It’s inferred.

That inference happens in two phases: extraction, then grouping.

Extraction breaks each input chunk into propositions, atomic self-contained statements that don’t need surrounding context to make sense. Take this sentence:

“Lithium-ion batteries store energy through the movement of lithium ions between a cathode and an anode, and the electrode materials expand and contract during each charge cycle.”

That’s two facts joined by “and.” Extraction produces:

“Lithium-ion batteries store energy through the movement of lithium ions between a cathode and an anode.” “Lithium-ion battery electrode materials expand and contract during each charge cycle.”

Each proposition stands alone. Extraction also resolves pronouns and implied subjects, so no proposition depends on its neighbours to be understood.

TC-032–0001 showing the original input chunk on the left and the extracted propositions on the right. One chunk, three atomic propositions, each self-contained and independent of the source sentence they came from.

Grouping takes those propositions and reassembles them by theme, not by original position. A document that interleaves topics leaves Level 4 with whatever boundaries the cosine distance found. Level 5 can pull non-adjacent chunks of the same topic back together, and split a single chunk that mixed two domains into two.

Level 4 never rewrites a word. Level 5 rewrites, splits, and recombines, which is why it needs an LLM and why it needs to be judged on different terms.

The Conceptual Foundation

This build follows the two-phase shape popularised by Greg Kamradt’s “5 Levels of Text Splitting”: extract propositions, then group them thematically. That framework laid out a spectrum from character splitting through agentic splitting as the most semantically aware approach, with Level 5 described as experimental and cost-justified only when token cost trends toward zero. LangChain and LlamaIndex both adopted implementations based on it.

What I built here follows the same two-phase shape but exists for a different purpose: a typed, native Forge component with a clear output contract, not a concept demonstration.

The Output Contract

Four substance rules govern every proposition in this build: no invention, no unresolved pronouns, no duplication, full self-containment. Those rules shape the output contract directly.

Each AgenticChunk carries eight fields: ChunkId, DocumentId, ThematicCategory, MergedContent, PropositionCount, CharacterCount, TokenEstimate, and a content Hash. ThematicCategory is a short label drawn from the grouping decision, things like "Solid-State Battery Technology." MergedContent is the propositions concatenated. Every proposition is grounded in the source text, even when the wording is normalised to make it self-contained.

The Hash is what makes this a pipeline component. A typed struct with a content hash is something a downstream embedding step or cache layer can key against directly.

Scope

This is for pipelines inside ODC that need chunking to make judgment calls a similarity threshold can’t make, and that can afford two generative model calls per document batch to get it. I would reach for it after Level 4, specifically when retrieval quality is failing because chunks still contain mixed facts, duplicated overlap, unresolved pronouns, or scattered related content.

Level 4 places boundaries with a deterministic distance calculation on embeddings. Level 5 adds two generative calls on top: one to extract propositions, one to group them. If a document is cleanly structured with no compound sentences, interleaved topics, or pronoun chains, Level 4 alone is enough. Level 5 adds cost for no real gain.

It’s a native, typed component for cases where chunk quality depends on resolving a pronoun, separating two facts from one sentence, or correctly regrouping content a similarity threshold scattered.

The Architecture

Level 5 in the pipeline: the Library handles computation, the App holds the AI Gateway calls.

Earlier components in this series ship as a Library with an optional companion app. This one is different. The App isn’t a demo companion. It’s a required dependency. ODC Libraries cannot reference Apps, and AI Gateway access is App-level only. A Library has no path to an LLM call.

So the stateless computation lives in Agentic Chunking Helpers, C# External Logic with three actions: PreChunkForExtraction, ParsePropositions, and NormaliseAgenticOutput. Everything that needs a model lives in Agentic Chunking, an app built around a single Service Action: ChunkTextAgentic.

The Level 4 to Level 5 handoff is a typed list of AgenticChunkInput structs, each carrying a ChunkId and Text. The consuming app passes this in; nothing about Level 5 needs to know how Level 4 produced it.

Inside ChunkTextAgentic, five steps run in sequence.

PreChunkForExtraction batches input chunks into prompt-safe sizes.

CallExtractModel sends each batch to Claude 3.7 Sonnet via AI Gateway at Temperature=0, MaxTokens=2000.

ParsePropositions strips markdown fences and normalises the raw response into a flat list.

CallGroupingModel sends that list to a second prompt at Temperature=0, MaxTokens=4000.

NormaliseAgenticOutput maps the result into typed AgenticChunk structs.

Two generative calls per document batch, both at Temperature=0. Temperature=0 reduces variation between runs, but the grouping decision is still model judgment and should be re-tested when the model version changes.

The five-step ChunkTextAgentic pipeline in ODC Service Studio, showing the two AI Gateway call points and the three C# External Logic actions between them.

The Evidence

The test suite I ran has 32 cases organised into four categories: extraction phase, grouping phase, edge cases, and real pipeline integration. The last two, TC-031 and TC-032, test the full pipeline end to end.

The most important result wasn’t that Level 5 produced more chunks. It was that it produced the right chunks. It separated mixed facts, collapsed duplicated overlap, and kept unrelated domains apart. The two cases below show what that looks like in practice.

TC-031: cross-chunk consolidation and mixed-domain split

TC-031 has four input chunks designed to simulate what a real document looks like after Level 4: two battery chemistry chunks separated by an urban planning chunk, followed by a single chunk that silently mixes human metabolism and genomics content because the cosine distance at that boundary stayed under threshold.

The pipeline produced six output chunks from those four inputs. ART-0001 and ART-0003, both about batteries, separated into "Lithium-ion Battery Fundamentals" and "Solid-State Battery Technology" rather than consolidating into one. The urban planning chunk came through as "Urban Planning and Zoning," untouched. ART-0004, the mixed-domain chunk, split into three: "Cellular Energy Production," "Aging and Genomic Changes," and "Epigenetic Regulation," with no cross-contamination between domains.

A chunk topology that went in as batteries, planning, batteries, metabolism-genomics-mixed came out as six topically clean chunks, each containing only propositions that belong together, regardless of where the original Level 4 boundaries happened to fall. Total: 20 propositions, 499 tokens.

TC-031 in the Agentic Chunking test runner. Four input chunks on the left; six output chunks on the right, each with its ThematicCategory label and proposition list.

TC-032: real Level 4 output, six domains

TC-031 was hand-authored to test specific behaviours. TC-032 is different: it’s the only case in the suite built from an actual Level 4 run, copied verbatim from a live consecutive-pattern output on a six-domain document covering cloud infrastructure, cellular metabolism, Italian Renaissance art history, battery chemistry, genomics, and urban planning.

Real Level 4 output is messier than hand-authored input. Domain boundaries don’t align cleanly with chunk boundaries because cosine distance crosses threshold wherever it crosses threshold, not where a domain changes. Three of TC-032’s nine input chunks have domain shifts mid-chunk, and the overlapSentences=1 setting from the Level 4 run means each adjacent pair of chunks shares one boundary sentence, eight duplicated sentences in total across nine chunks.

The pipeline handled both. Nine input chunks across six tangled domains became nine thematically clean output chunks. Every domain separated cleanly. The battery domain split into two sub-themes, consistent with the granularity pattern visible in TC-031. The eight duplicated boundary sentences collapsed to single propositions, none appearing twice. Total: 51 propositions, 1071 tokens.

TC-032 pipeline output in the Agentic Chunking app, showing 9 chunks, 51 propositions, and 1071 tokens from 9 real Level 4 input chunks. Each chunk card shows its ThematicCategory label, MergedContent, and individual proposition list.

The Granularity Finding

Across all 32 test cases, the four substance checks held without exception. I found no output proposition that invented content not in the source. No unresolved pronoun remained. No proposition appeared in more than one chunk. Every proposition read as self-contained without surrounding context.

That pattern held from TC-001, a single clean sentence about Kubernetes, through TC-032, nine chunks of real Level 4 output across six domains.

Chunk and proposition counts are a different story.

The grouper consistently subdivides broad domains into finer sub-themes rather than producing one chunk per domain label. TC-022 produced nine battery chemistry chunks from ten sentences, each a distinct facet. TC-026 returned eleven sub-theme chunks from a single 312-token input. TC-031’s mixed-domain chunk split into three rather than two. TC-032’s battery content split into “Lithium-Ion Battery Chemistry” and “Battery Management Systems” rather than staying as one.

One pattern emerges from all of this. The grouper reliably keeps unrelated domains separate, and reliably splits mixed content apart. Every count deviation in the suite traces back to subdividing one broad domain into finer sub-themes, not to merging unrelated content or misclassifying a proposition. That’s the reliability hierarchy this build operates at: substance is robust, granularity is model judgment.

Chunk and proposition counts from these results are not guarantees, and they are not portable across model versions without re-running the suite. What was tested here was Claude 3.7 Sonnet via ODC AI Gateway at Temperature=0. The substance findings should transfer. The count findings are evidence about this model’s judgment, not a specification.

Honest Limitations

Granularity is model judgment, not a dial you can set. There’s no parameter in this build that controls how finely the grouper subdivides a domain. Temperature=0 reduces run-to-run variation, but it doesn’t make the grouper’s notion of “one theme” predictable across content types or model versions.

Proposition expansion adds tokens. Extraction rewrites source sentences into atomic propositions, increasing character and token counts. TC-032 came back as 1,071 tokens across 51 propositions. Dense compound sentences expand more than simple declarative ones. Budget for this in any downstream embedding step.

Two generative calls per document batch is a real cost. Apply it only where the judgment layer is doing work that a similarity threshold cannot.

This build has been tested on single documents processed one at a time. It hasn’t been tested under the concurrency, volume, or latency conditions a production ingestion pipeline would impose.

What This Actually Changes

The five-level framework now exists as a complete, buildable stack inside ODC. Level 1 splits on character count. Level 2 splits recursively on separators. Level 3 splits on document structure. Level 4 splits on semantic distance. Level 5 splits on judgment. Each level is now a Forge component you can drop into a pipeline.

That progression matters because chunking is where most RAG quality problems actually live. A retrieval layer that can’t find the right content usually has a chunking problem upstream, not a search problem. Getting the boundaries right, and getting the facts inside those boundaries clean and self-contained, is the work that makes everything downstream more reliable.

Level 5 isn’t the right answer for every document or every pipeline. But having it available as a native, typed component means the choice is now an architectural decision rather than a build-it-yourself project.

The next pieces worth building on top of this foundation are the retrieval quality components: reranking, hybrid search, parent-child retrieval. The chunks exist. What you do with them at query time is where the series goes next.

If you’re already running Level 4 in a pipeline, I’d be curious what you’re seeing at retrieval time. Drop a comment.

Get the Components

**Agentic Chunking and [Agentic Chunking Helpers](https://www.outsystems.com/forge/component-overview/25025/agenticchunkinghelpers-odc)** are both on the Forge. The library handles the stateless C# computation; the app holds the pipeline and the AI Gateway calls. The test runner that produced the TC-031 and TC-032 results is included in the app so you can run the suite against your own AI Gateway configuration and see the results directly.

Thirty-two test cases. Substance checks held across every one. Chunk and proposition counts will vary with your content and your model version, and that’s expected behaviour, not a defect.

If you’re building RAG pipelines inside ODC and chunk quality is where your retrieval is falling down, this is the level worth reaching for. If Level 4 is giving you clean boundaries already, stay there. The cost difference is real and the judgment layer only earns its keep when there’s actual judgment to do.

The full series:

“What ODC Actually Does When It Chunks Your Text”: native chunking methods, the ODC Native Chunking Tester on Forge.

“Beyond Text Splitting: Building a Chunking Strategy Library for ODC”: Levels 1 through 3, the ODC Chunking Library on Forge.

“Beyond Structure: Semantic Chunking Inside ODC”: Level 4, the Semantic Chunking Library on Forge.