Your Semantic Search Is Fast. Is It Right?

Measuring FileMaker search quality with evals, then improving it with chunking, a better model, and a reranker

In a previous post, we took a semantic search over 900,000 records from 80 seconds down to 2. Speed matters, but it’s only half the problem. A fast search that returns the wrong records is just a faster way to disappoint your users.

So this post asks the other question: how do you know your search is returning the right results, and once you can measure it, how do you improve it?

I tested this on a hosted FileMaker file and a local embedding server, scoring every change against a fixed set of 123 questions with known answers. A handful of changes made significant improvements.

Here’s how I made those changes, starting with the simplest one first.

The short version: what actually makes a difference

Here are my suggestions for how to improve your semantic search in FileMaker.

1. Upgrade your embedding model, and check its context window. This is the easiest win and the one everyone can take. If you do nothing else, do this. In FileMaker it is a single line: point Configure AI Account at a newer model, with nothing else in your solution touched. Moving from all-MiniLM to a current model lifted the right-answer-first rate from 33% to 43% on its own. Just as important, old models often have tiny context windows: MiniLM silently discards everything past 128 tokens, about 520 characters, so most of every longer chunk was thrown away before it was ever embedded. Check the model’s max_seq_length before you trust it with anything longer than a sentence.

2. Add a reranker.

The single biggest lever in the whole experiment. A reranker is a second, more careful model that re-reads your query against each candidate result and reorders them: you let the fast semantic find fetch 50 candidates, then let the reranker pick the best 10. It improved every configuration I tested, by 9 to 19 points of first-hit rate, and it needs no re-indexing, no new records, no schema changes. It is one extra Insert from URL call after Perform Semantic Find: pure query-time work, and it stacks on top of everything else.

3. Chunk with the structure your documents already have.

How you split documents into chunks sets the ceiling the model and reranker work under, and generic splitters throw away information you already have. Many essays look like plain prose, but most have named sections, footnotes, and clean paragraph breaks, and a chunker that respects those beat every one-size-fits-all method. Your data has structure too: invoices have line items, contracts have clauses, meeting notes have agenda points. Respect it before reaching for anything exotic.

4. Don’t pay for the exotic options.

Two impressive-sounding levers earned nothing here. “Semantic” chunking, which groups text by topic similarity, finished last in every single test across four different embedding models. That came as a huge surprise to me as I expected it to perform best. And a large 32,000-token context window, against an ordinary 512-token one, made no measurable difference once the chunks already fit. Spend the effort on the three levers above instead.

And underpinning all four: measure everything. None of these conclusions would be trustworthy, and one of them would have been flat wrong, without a proper evaluation set: a few hundred questions with known answers, so you can see whether a change helped, hurt, or did nothing, instead of eyeballing a handful of queries and hoping. That measurement is what the rest of this post builds, and it is where every number above comes from. It’s important to point out that your dataset is different and you might see different results, hence the importance of running your own evaluations.

The test corpus: Paul Graham’s essays

The retrieval world has an unofficial standard test corpus: Paul Graham’s essays. LlamaIndex uses them in its documentation, Arize used them for the needle-in-a-haystack tests, and reranker benchmarks include them as a long-form corpus. 215 essays (about 2.9 million characters) is big enough for retrieval quality to genuinely vary, and small enough to embed locally in minutes.

I loaded all 215 essays into a hosted FileMaker file: an Essays table with id, title, date, and text. If you want to follow along, there is a ready-made dataset on Hugging Face (sgoel9/paul_graham_essays).

You cannot improve what you cannot measure

Most people evaluate search the same way: type a few queries, eyeball the results and call it a day. That works until you change something, because then you need to know whether the change helped, hurt, or did nothing, across all the kinds of queries your users make. Eyeballing does not scale to that. An evaluation set does.

The methodology comes from Chroma’s excellent technical report on chunking strategies: use an LLM to generate questions from randomly sampled passages, record exactly which passage each question came from, and score retrieval by whether the search brings back a chunk that overlaps that golden passage.

For each question we store the essay it came from and the character positions of the excerpt that answers it. Every chunk we create later also carries its character positions, so scoring is mechanical: a retrieved chunk is a hit if it comes from the right essay and its character range overlaps the golden excerpt.

I generated 150 candidate questions with a local model (gemma4 running in Ollama), prompted to write questions the way a real person would type them into a search box. Two examples:

“What simple measure can prevent advanced filters from catching bulk mail messages?”

“Why is it beneficial for early employees to accept low salaries?”

Filter your synthetic questions

Don’t trust an LLM-generated eval set blindly. Some generated questions are garbled, and some are not actually answerable from the passage they were generated from. If those stay in, they add noise to every number you compute afterwards.

So the 150 candidates faced a judge: an LLM that saw each question with its source passage and answered one thing, does this passage clearly answer this question, yes or no. A small strict judge dropped 36. Because strict judges also make mistakes, the 36 rejects got an appeal in front of a stronger model, which restored 9. My final set was123 questions, each verified answerable from its recorded passage.

That two-stage pattern (cheap strict judge for the bulk pass, stronger judge for appeals) is easy to copy and costs a few minutes of local inference.

The metrics

Three metrics, all computed over the 123 questions:

• Recall@k: the fraction of questions where a correct chunk appears in the top k results. If your search feeds a RAG prompt with five chunks of context, Recall@5 is literally “how often does the model receive the answer.”
• MRR (mean reciprocal rank): rewards position. A correct first result scores 1, second place scores 1/2, tenth place 1/10. This is the metric that notices when the right answer moves from rank 8 to rank 1.
• Essay Recall@5: did we at least reach the right essay? The gap between this and passage-level recall tells you whether your problem is retrieval (wrong essay entirely) or chunking (right essay, wrong slice of it).

One calibration note before any results: with 123 questions, each question moves a recall figure by 0.8 points, and differences under about 4 points are within noise. Keep that in mind as the tables go by. More questions in the test set would reduce the noise level.

The architecture

I used FileMaker as the database and the search engine. Its native script steps do the heavy lifting:

• Configure AI Account points at a local OpenAI-compatible embedding server instead of OpenAI itself. Same script steps, no API bill, no data leaving the building.
• Insert Embedding in Found Set embeds the chunks in batches.
• Perform Semantic Find runs the search and returns the found set sorted by cosine similarity.

The Python side is one small FastAPI server with three endpoints:

• POST /chunk: takes essay text and a method name, returns chunks with their character offsets. FileMaker calls it with Insert from URL and creates one Chunks record per chunk.
• POST /v1/embeddings: an OpenAI-compatible wrapper around a local sentence-transformers model. This is what Configure AI Account points at, and it is what will let us swap embedding models later by changing one environment variable, with zero changes in FileMaker.
• POST /rerank: takes a query and candidate texts, returns them reordered by a cross-encoder. More on this later.

The FileMaker schema is three tables: Essays, Questions, and a Chunks table with essay_id, config, chunk_index, start, end, chunk_text, and an embedding container field. The config field is the trick that makes comparison easy: chunks from every strategy coexist in one table, and each eval run constrains the found set to one config before searching.

The eval loop is a script that walks the Questions table, performs a semantic find over one config’s chunks, and writes the top ten results per question to a JSON file. A small Python script scores those files against the golden set.

Five chunking strategies

Chunking decides what a “result” even is. Embed whole essays and a single vector has to summarise 10,000 words, which works about as well as describing a novel in one sentence. If the chunks are too small each piece loses the context that gives it meaning. Between those extremes there is real room for skill, and FileMaker gives you no control over any of it, which is why the chunking happens server-side where we can choose.

The five configs, loosely following Greg Kamradt’s five levels of text splitting:

1. fixed_512: split every 512 characters. The naive baseline. 5,842 chunks.
2. fixed_512_ov64: same, with 64 characters of overlap. 6,643 chunks.
3. recursive_800: split on paragraph boundaries first, then sentences, up to 800 characters. The strategy Chroma found consistently strong. 5,012 chunks.
4. structural_800: document-aware splitting, described below. 5,105 chunks.
5. semantic_p90: embed each sentence and start a new chunk where the similarity between neighbouring sentences drops sharply. The most sophisticated method here, and the most expensive. 3,229 chunks.

Even “unstructured” text has structure

The structural config came from actually looking at the corpus. Paul Graham’s essays look like plain prose, but 151 of the 215 have named sections marked with bold headers, 129 have a Notes section of footnotes, and nearly all have clean paragraph breaks.

Generic splitters throw that structure away. The structural chunker keeps it: split on section headers, pack whole paragraphs into chunks up to the size limit, and never let a chunk cross a section boundary. Footnotes stay in footnote chunks instead of bleeding into body text.

Your data has structure too: invoices have line items, meeting notes have agenda points, contracts have clauses. Before reaching for exotic chunking, respect the structure your documents already have.

Chunking: structure beats sophistication

Each config was chunked, embedded with all-MiniLM-L12-v2 (the default on FileMaker Server), and evaluated over all 123 questions. These MiniLM numbers come with an asterisk we will quantify shortly, because this 2021 model was quietly truncating the longer chunks, but the ranking they establish holds up under every model that follows.

The baseline is worse than you would guess: naive chunking puts the right passage in the top five results 58% of the time. Two queries in five, the answer the user needed was not there.

Structure-aware chunking wins, modestly: structural has the best first-hit rate and MRR, recursive close behind. This is recommendation 3 in a single table, and the margin grows once the reranker and a better model come into play.

And the result that recommendation 4 warned about: semantic chunking lost, to everything, including the naive baseline, scoring seven points of Recall@5 below splitting every 512 characters. The most sophisticated and most expensive method here finished last. That verdict looks unfair at this stage, because MiniLM’s truncation hit semantic’s large chunks hardest, so we will return and re-run it on a level field to see if that problem persists.

The reranker: the biggest single lever

This is the change that moved the numbers most.

Semantic search with embeddings is a bi-encoder: the query and each document were embedded separately, and similarity is a single dot product between two summaries. It is blazingly fast, and FileMaker’s Perform Semantic Find does exactly this. But it never reads the query and the document together.

A cross-encoder reranker does. It takes the query and one candidate chunk as a pair and reads them jointly, which lets it notice things a similarity score cannot, at the cost of being far too slow to run against every record. The standard pattern is to let the fast search retrieve 50 candidates, then let the careful model reorder them and keep the top 10.

In FileMaker this is one extra Insert from URL call after Perform Semantic Find. We send the question and the 50 candidate chunk texts to the local /rerank endpoint and get back the reordered indexes. The reranker here is ms-marco-MiniLM-L-6-v2, a small cross-encoder that runs comfortably on CPU.

Every config, with and without reranking:

Two findings, and they both survive everything that follows:

Reranking improved every single config, by 9 to 19 points of first-hit rate. It is the most reliable lever in the whole experiment, and it required no re-indexing, no new records, and no schema changes. It does require an extra Insert from URL call.

But it did not rescue bad chunking. Reranked semantic chunking still finishes last. A reranker can only reorder what retrieval put in the top 50; if the right chunk never surfaced, there is nothing to promote. Chunking and reranking are complementary layers, not substitutes.

Chunk size, and the hidden constraint behind recommendation 1

Recommendation 1 came in two parts: upgrade the model, and check its context window. This is the section that earns the second part, and it starts with an experiment on chunk size. Does chunk size matter on the naive fixed-size system? I created four more configs, ranging from 128 to 2,048 characters:

The results are an inverted U shape, peaking at 512. Small chunks are context-starved fragments, which is unsurprising. Large chunks are where it gets interesting. The obvious explanation is dilution: a longer chunk’s embedding has to average over more ideas, blurring the passage you are looking for. But dilution is gradual, and the collapse at 2,048 is not: sixteen points of Recall@5 below the peak. That looks more like a hard limit rather than a soft trend.

We can see the reason on the embedding model’s card:

>>> SentenceTransformer("all-MiniLM-L12-v2").max_seq_length
128

The model’s maximum input is 128 tokens: roughly 520 characters. Everything past that in a chunk is silently thrown away before embedding. Not summarised or compressed. Just discarded, with no warning.

So the size curve makes perfect sense. fixed_512 did not win because 512 characters is a magic number; it won because it is the largest size that fits this particular model’s window. fixed_1024 was half invisible to the search. fixed_2048 was mostly invisible.

How much text the model never saw

Tokenising every chunk of every config shows exactly how much text the model never saw:

Look at that last row again. Nearly half of the semantic config’s text was never embedded at all. Meanwhile recursive and structural posted their chunking results carrying a 13 to 14% handicap, and fixed_512, the apparent “winner” of the size sweep, was the only configuration the model actually saw in full.

This is why our first recommendation pairs “upgrade the model” with “check its window”, and it is the most important argument in the post for measuring properly. The eval alone could not catch this, because every config was scored by the same rules: the numbers were internally consistent and externally wrong. It took the eval plus an audit of the pipeline’s hidden constraints. Every chunking verdict up to this point was contaminated, and the only way to fix it is to re-run everything on a model that can see the whole chunk.

So make sure to check max_seq_length before you choose a chunk size. It is one line of Python, and skipping it cost this pipeline up to sixteen points of recall with no warnings to the user.

Re-running on a model that sees the whole chunk

Because FileMaker talks to a local OpenAI-compatible server, swapping the embedding model needs no FileMaker changes at all. Just restart the server with bge-small-en-v1.5 (a current-generation small model with a 512-token window, four times that of MiniLM), re-run the embed step, re-run the eval. Under the bigger window, every config now fits, even semantic’s largest chunks.

This answers two questions at once: does the model matter, and was truncation really the mechanism?

The model matters, a lot. Every config improved on every metric, by 3 to 11 points. Swapping a 2021 embedding model for a 2023 one was worth roughly as much as the best chunking change, and the two improvements stack.

The truncation mechanism is confirmed. Under MiniLM, fixed_1024 trailed fixed_512 by nearly six points of Recall@5. Under BGE, with the whole chunk finally visible, the penalty vanishes: fixed_1024 matches fixed_512 on Recall@5 and beats it on MRR. The earlier “sweet spot at 512” was substantially the model’s window in disguise.

And the semantic verdict, now on a level field: semantic chunking recovered most where truncation had hurt it most (+11 points of Recall@10), which confirms how much the blindfold cost. But it is still last on every metric. With its chunks fully visible, the remaining deficit is the real dilution effect: fewer, larger chunks genuinely do retrieve worse than structure-aware moderate ones. Truncation was about half the story; dilution is the other half.

I found it interesting that the chunking ranking (structure-aware on top, semantic at the bottom) held under both models.

While we are reading model cards

The window is not the only contract hidden in an embedding model’s card. Some models require task prefixes: nomic-embed-text, an 8k-window model, degrades measurably unless every query starts with search_query: and every document with search_document:. BGE recommends (but does not require) a query instruction we did not use here. Context windows, prefixes, normalisation: each unread line of the model card is a silent few points of recall.

If you need bigger windows, they exist at every scale: nomic-embed-text and gte at 8k tokens locally, Qwen3-Embedding at 32k, and hosted options from OpenAI (8k), Voyage (32k), and Cohere (128k). Since we’re running 100% local, we used Qwen3. A big window buys safety, but you still need to chunk the inputs. A single vector still has to summarise whatever you put in it, and the dilution result holds no matter how much fits.

Does a bigger model help? (bge-large)

bge-small fixed the window problem, but it is still a 33-million-parameter model, the same weight class as MiniLM. The next lever is capacity. bge-large-en-v1.5 is the same model family with the same 512-token window and roughly ten times the parameters, which made it the cleanest possible test. Only the model size changes, everything else stays constant. As before, the swap is a re-embed, and a re-run. No FileMaker code changes.

The parameter jump is worth about seven points across the board over bge-small, comfortably beyond noise. But the striking row is the second one: structural chunking on bge-large without a reranker (Recall@5 0.740, MRR 0.592) essentially matches structural on bge-small with the reranker (0.732, 0.641). A bigger embedding model buys you most of what the rerank stage bought. And the two still stack: adding the reranker on top is worth another twelve points of Recall@1, for a new best of 0.602.

An effectively infinite window (Qwen3)

There’s still another question remaining. On paper, bge’s 512-token window fits every chunk in every config, including semantic’s largest. But “on paper” is exactly what caused the truncation problem earlier, and semantic’s biggest chunks sit close enough to the limit to leave doubt. So, I made one final swap, to use Qwen3-Embedding-0.6B, a 600-million-parameter model with a 32,768-token window. Our largest chunk is about 500 tokens. Nothing can possibly truncate; there is two orders of magnitude of headroom. This settles two questions at once: is the dilution verdict against semantic chunking real, and does a bigger window itself buy anything once your chunks actually fit?

Dilution is happening. With a 32k window and zero truncation, semantic chunking still finishes last, sixteen points of Recall@1 behind structural on the same model. Its trajectory across the three eras tells the whole story: Recall@1 of 0.285 blindfolded under MiniLM, 0.333 under bge-small, 0.407 with every character visible under Qwen. Fixing truncation helped it most of any config, and it still never got close. Fewer, larger, topically coherent chunks genuinely retrieve worse: a single vector asked to summarise more ideas summarises each of them less well.

And the window itself bought nothing. Qwen’s raw retrieval is stronger than bge-large’s (0.504 versus 0.431 Recall@1 on the naive config), but after the reranker the two big models converge to a statistical tie: 0.610 versus 0.602 Recall@1, with bge-large actually ahead on Recall@5 and essay recall. Once your chunks fit inside the window, extra window is headroom you are not using. Between these two finalists, bge-large is the practical pick: a fraction of the size, several times faster, and no memory drama.

One more result to consider. The chunking ranking (structure-aware on top, semantic at the bottom) has now held across four embedding models spanning a 20x parameter range and a 256x window range.

The final pipeline

This table shows every improvement, stacked on the last:

The right passage is now the first result 60% of the time instead of 33%, in the top five 79% of the time instead of 58%, and the right essay is in the top five 85% of the time. The reranker was the single biggest lever, the embedding model was worth nearly as much and chunking set the ceiling for both. Perhaps surprisingly a 64x bigger context window didn’t make a significant improvement, but it was much slower to run. Every stage runs locally: FileMaker performing the search, a local embedding model, a local reranker, and not a byte of your data leaving the building.

Here are a few caveats: 123 questions means differences under about 4 points are noise (the big effects here, reranking, the model swap, and the truncation cliff, are far outside it). The question set is synthetic, generated and filtered by LLMs, and inherits their tastes. Every number is specific to this corpus and these models. Which is really the main point of this post: run the measurement on your own data, audit the constraints of your own pipeline, and only then trust a conclusion, whether it comes from a blog post or from your own first results table.

Doing this on your own data

Everything above is reproducible with modest effort: a small FastAPI server for chunking, embeddings, and reranking, five FileMaker scripts, and two Python scripts for question generation and scoring. FileMaker’s native AI script steps handle the search itself, and the whole pipeline runs locally, which matters if your data cannot leave the building.

It is also exactly the kind of plumbing we have been building into products: an intelligent chunking and embedding engine, and a local high-performance search server with hybrid retrieval and reranking built in, both as single-file executables that pair with FileMaker. If you would rather start from working tools than build the pipeline yourself, or if there is a semantic search project you are interested in but you are unsure where to start, get in touch. I would love to discuss it.

I write a weekly newsletter on putting AI into real business systems, mostly FileMaker, mostly running locally. Free, and I write it myself: subscribe at ianjempson.substack.com.

Sources and further reading: Chroma’s Evaluating Chunking Strategies for Retrieval (Smith & Troynikov), Greg Kamradt’s 5 Levels of Text Splitting, Antematter’s chunking study on the same corpus, and the Agentset reranker leaderboard. The Paul Graham essay dataset is sgoel9/paul_graham_essays on Hugging Face.