Knowledge Engineering for Search and Content: A Practical Guide

How modern content platforms turn messy human language into structured understanding — and the open-source frameworks I’ve built to do it at scale.

Why Knowledge Engineering Is Back

For twenty years, knowledge engineering was the unglamorous corner of information science. Taxonomies lived in spreadsheets. Ontologies were academic. Librarians ran the show, and the work was slow, manual, and easy to ignore when keyword search seemed “good enough.”

That era is over.

The rise of large language models, AI-generated answers, semantic search, and content recommendation systems has made one thing clear: the teams that win are the ones with the best structured understanding of their content and their users’ intent. LLMs hallucinate less when grounded in a real knowledge graph. Rankers work better when they understand entities, not just tokens. Content classification drives every downstream decision — what to show, what to suppress, what to recommend, what to summarize.

Knowledge engineering is the discipline that makes all of this work. I’ve spent the last several years building knowledge systems for enterprises ranging from $4B to $125B in revenue, and in the process I’ve released a set of open-source Python frameworks — MeaningFlow, Papilon, and PyCausalSim — that handle the core pieces of a modern knowledge engineering stack. This article walks through the discipline itself, how to do it, and where those frameworks fit in a practical pipeline.

What Knowledge Engineering Actually Means

Knowledge engineering is the practice of encoding what a system needs to know about the world into a form that machines can reason over. In a content and search context, that means four tightly-coupled artifacts:

1. A taxonomy — a controlled hierarchy of categories (entertainment → music → hip-hop → southern hip-hop).
2. An ontology — the types of things in your domain and how they relate (an Artist performs a Song; a Song belongs to an Album; an Album has a release date).
3. A knowledge graph — actual instances populated into that ontology (Kendrick Lamar performed “HUMBLE.” on the album DAMN., released April 14, 2017).
4. A query understanding layer — the mapping from how users actually phrase things to the entities and categories above.

The mistake most teams make is treating these as separate projects owned by separate teams. They aren’t. They are four views of the same problem: how does our system represent meaning?

The Five Pillars

I structure every knowledge engineering program around five pillars. Skip any one and the others collapse.

1. Taxonomy and ontology design — the shape of the world.
2. Content classification — putting documents into that shape.
3. Entity extraction and linking — recognizing the things inside documents.
4. Query understanding — mapping user language onto the shape.
5. Evaluation and causal attribution — proving it works and improving it.

Let’s walk through each.

Pillar 1: Taxonomy and Ontology Design

Top-Down vs. Bottom-Up

Every taxonomy project faces the same tension. You can design the hierarchy top-down based on editorial judgment and domain expertise, or bottom-up by clustering what your content and queries actually contain. Pure top-down produces elegant structures that don’t match reality. Pure bottom-up produces messes that no editor can defend.

The answer is both, in sequence:

Step 1 — Seed top-down. Start with a small, opinionated hierarchy written by subject matter experts. Three to five top-level categories, two levels deep. Resist the urge to go deeper. You don’t know enough yet.

Step 2 — Mine bottom-up. Pull six to twelve months of your query logs and content corpus. Cluster them. Look at what falls outside your seed taxonomy. Those gaps are the real shape of your domain.

Step 3 — Reconcile. Expand the seed taxonomy to cover the gaps, merge branches that nobody actually uses, and split branches that are doing double duty.

MeaningFlow: A Framework for Semantic Mining

The bottom-up mining step is where I’ve spent the most engineering effort, and it’s why I built **MeaningFlow**, an open-source framework for semantic content analysis. MeaningFlow takes a content corpus or a query log, embeds it using Sentence-BERT, reduces dimensionality with UMAP, clusters with HDBSCAN, and then builds a NetworkX graph over the clusters so you can see how topics relate to each other and where the gaps are.

The core pipeline looks like this:

python

from meaningflow import SemanticGraph
import pandas as pd

# Load queries and existing content
queries = pd.read_parquet("queries_last_180d.parquet")["query"].tolist()
content = pd.read_parquet("content_corpus.parquet")["title"].tolist()

# Build a semantic graph of the demand side (queries)
demand = SemanticGraph(
    texts=queries,
    embedder="all-MiniLM-L6-v2",
    min_cluster_size=50,
)
demand.fit()

# Build a semantic graph of the supply side (content you already have)
supply = SemanticGraph(texts=content, embedder="all-MiniLM-L6-v2")
supply.fit()

# Find coverage gaps: clusters of demand with no nearby supply
gaps = demand.coverage_gaps(
    reference=supply,
    similarity_threshold=0.55,
)

for g in gaps[:20]:
    print(f"Gap cluster (n={g.size}): {g.top_terms[:5]}  demand={g.volume}")

What this gives you is a ranked list of topical areas where users are asking questions and your content isn’t answering them. That list is the foundation of a bottom-up taxonomy proposal. Editors review the gap clusters, name them, and decide which ones deserve new branches in the hierarchy.

Underneath, the same primitives are available if you want to build your own pipeline from scratch:

python

from sentence_transformers import SentenceTransformer
import hdbscan
import umap

model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = model.encode(queries, batch_size=256, show_progress_bar=True)

reducer = umap.UMAP(n_neighbors=30, n_components=10, metric="cosine", random_state=42)
reduced = reducer.fit_transform(embeddings)

clusterer = hdbscan.HDBSCAN(
    min_cluster_size=50,
    min_samples=10,
    metric="euclidean",
    cluster_selection_method="eom",
)
labels = clusterer.fit_predict(reduced)

The -1 bucket is noise — queries too unique to cluster. In a healthy search system, noise should be 10–25% of your corpus. Higher means your embedding model doesn't understand your domain. Lower often means you're over-clustering and losing the long tail.

Ontology: Types and Relations

A taxonomy tells you what categories exist. An ontology tells you what kinds of things exist and how they relate. This matters because classification alone isn’t enough for modern search. If a user searches for “movies directed by the guy who made Parasite,” you need to know that Movie is a type, Director is a type, and directed_by is a relation between them.

A minimal ontology definition can live in a YAML file:

yaml

types:
  Person:
    attributes: [name, birth_date, nationality]
  Movie:
    attributes: [title, release_year, runtime, genre]
  Album:
    attributes: [title, release_date, label]

relations:
  directed_by:
    domain: Movie
    range: Person
  performed_by:
    domain: Song
    range: Person
  belongs_to:
    domain: Song
    range: Album

Keep it small. An ontology that tries to cover everything covers nothing well. Start with the entity types that show up in your top 1,000 queries and expand from there.

Pillar 2: Content Classification

Once the taxonomy exists, every piece of content needs to be placed in it. For a site with ten thousand documents, editors can do this by hand. For a site with ten million, you need machine learning, and the question becomes which approach.

There are three practical options, and you should probably run all three in parallel.

Zero-shot classification with LLMs. Fast to set up, no training data required, surprisingly good out of the box for broad categories. Use it as your baseline and for cold-start on new taxonomy branches.

python

from openai import OpenAI
import json

client = OpenAI()

TAXONOMY = {
    "Music": ["Hip-Hop", "Rock", "Pop", "Electronic", "Classical"],
    "Film": ["Drama", "Comedy", "Documentary", "Action", "Horror"],
    "Sports": ["Basketball", "Football", "Soccer", "Tennis"],
}

def classify(text: str) -> dict:
    taxonomy_str = json.dumps(TAXONOMY, indent=2)
    prompt = f"""Classify the following content into exactly one top-level category
and one subcategory from this taxonomy. If nothing fits, return "Other".

Taxonomy:
{taxonomy_str}

Content:
{text}

Respond with only JSON: {{"category": "...", "subcategory": "...", "confidence": 0.0-1.0}}
"""
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        max_tokens=200,
        response_format={"type": "json_object"},
        messages=[{"role": "user", "content": prompt}],
    )
    return json.loads(response.choices[0].message.content)

Fine-tuned encoder classifier. Once you have a few thousand labeled examples — typically harvested by running the LLM baseline and having editors correct its mistakes — fine-tune a small encoder model like DeBERTa or a distilled BERT variant. It will be cheaper, faster, and often more accurate than the zero-shot LLM for your specific taxonomy.

Rule-based overrides. Never remove these. For every automated system, there will be edge cases where the business needs a hard guarantee: breaking news must always be classified as News, a specific publisher’s content must always route to a specific vertical, legally sensitive categories need manual review. Rules are ugly but essential.

The architecture pattern is simple: rules first, fine-tuned classifier second, LLM fallback third, human review queue for anything with low confidence across all three.

One integration pattern I’ve used repeatedly: run the classifier output back through the MeaningFlow semantic graph. If a classifier assigns a document to “Hip-Hop” but the document’s embedding sits inside a “Classical” cluster with high density, that’s a signal to route the document to human review. The graph acts as a sanity check on the classifier, and the classifier acts as a sanity check on the graph. Disagreements between them are exactly the examples most worth having an editor look at.

The Labeling Problem

The hardest part of classification is not the model. It is producing enough clean labeled data to train and evaluate it. Three techniques worth knowing:

Active learning. Don’t label randomly. Label the examples the current model is most uncertain about. Every labeled example moves the decision boundary more than ten random labels would.

Weak supervision. Write a dozen heuristic labeling functions — regex patterns, keyword lists, existing metadata — and let a framework like Snorkel combine them into probabilistic labels. Imperfect, but gets you to a usable training set in a week instead of a quarter.

LLM-assisted labeling with human verification. Have an LLM produce first-pass labels at scale, then have editors verify rather than label from scratch. Verification is typically 3–5x faster than labeling.

Pillar 3: Entity Extraction and Linking

Classification puts a whole document into a category. Entity extraction and linking go deeper: they identify the specific things mentioned inside the document and connect them to canonical records in your knowledge graph.

The pipeline has three stages.

Stage 1 — Named entity recognition (NER). Find the spans of text that refer to entities. “Kendrick Lamar released DAMN. in 2017” contains three entities: a Person, a Work, and a Date. Modern NER uses transformer-based sequence labelers; spaCy, Flair, and HuggingFace all have production-ready options.

Stage 2 — Entity disambiguation. There are twelve people named Michael Jordan in Wikipedia. Which one does this mention refer to? Use surrounding context — other entities in the document, the domain of the source — to pick the right candidate. A common approach is to embed both the mention-in-context and each candidate’s description, then take the cosine nearest.

Stage 3 — Linking. Write the disambiguated entity ID back into the document metadata so downstream systems can query by entity rather than by string match. This is what lets “movies starring the guy from Parasite” work — the query planner resolves “the guy from Parasite” to a specific Person ID and then executes a structured lookup.

python

import spacy
from sentence_transformers import SentenceTransformer, util

nlp = spacy.load("en_core_web_trf")
embedder = SentenceTransformer("all-MiniLM-L6-v2")

def extract_and_link(text: str, kg_candidates: dict) -> list:
    """
    kg_candidates: {entity_id: description_string}
    """
    doc = nlp(text)
    linked = []

    for ent in doc.ents:
        # Candidates whose canonical name matches the mention
        matches = {
            eid: desc for eid, desc in kg_candidates.items()
            if ent.text.lower() in desc.lower()
        }
        if not matches:
            continue

        # Disambiguate by embedding similarity to surrounding context
        context = doc.text[max(0, ent.start_char - 200):ent.end_char + 200]
        ctx_emb = embedder.encode(context, convert_to_tensor=True)
        cand_embs = embedder.encode(list(matches.values()), convert_to_tensor=True)
        scores = util.cos_sim(ctx_emb, cand_embs)[0]

        best_idx = int(scores.argmax())
        best_id = list(matches.keys())[best_idx]
        linked.append({
            "mention": ent.text,
            "type": ent.label_,
            "entity_id": best_id,
            "confidence": float(scores[best_idx]),
        })

    return linked

This is a toy version. Production systems add candidate generation from an approximate nearest neighbor index over millions of entities, a learned ranker instead of raw cosine, and confidence thresholding with a fallback to a “NIL” entity when nothing scores high enough.

Pillar 4: Query Understanding

This is where knowledge engineering meets the user. A query understanding layer takes a raw user query and produces a structured representation: intent, entities, filters, and modifiers.

“cheap hotels in tokyo under 200 with a pool” contains:

• Intent: commercial, accommodation search
• Entity: Tokyo (Location)
• Filters: price < 200, amenities includes “pool”
• Modifier: “cheap” (a soft preference, not a hard filter)

Building this well requires three components.

Component A: Intent Classification

A small classifier over a fixed set of intent types. Keep the set small — ten to twenty intents is usually enough. Navigational, informational, transactional, and their domain-specific variants.

Component B: Entity Linking on Queries with Session Memory

Same pipeline as document entity linking, but harder because queries are short and context-free. “apple” in a document usually has enough surrounding text to disambiguate. “apple” as a standalone query does not.

The trick is to use the user’s context: recent query history, current session, geographic location, device type, known interests. A user who just searched for “tim cook keynote” and then searches for “apple” almost certainly means the company. This is where your query understanding layer becomes genuinely personalized.

The architectural challenge is representing that context in a form the query understanding layer can actually use. This is a research area I’ve been working on under Vector1 Research, where I’ve proposed a cognitive architecture called Memory-Node Encapsulation (MNE) — a data structure for artificial episodic memory designed specifically for session-aware reasoning. The idea is that each session becomes a memory node linking the entities, intents, and queries a user has touched, and the query understanding layer consults that node when disambiguating new queries. It’s the same insight that underpins most successful personalized search systems, just made explicit as a reusable primitive.

You don’t need a full MNE implementation to get value from the idea. A simple session store with recent entity IDs and their recency weights gets most of the way there:

python

from collections import deque
from dataclasses import dataclass, field
from time import time

@dataclass
class SessionMemory:
    entities: deque = field(default_factory=lambda: deque(maxlen=20))

    def add(self, entity_id: str):
        self.entities.append((entity_id, time()))

    def context_weights(self, half_life_sec: float = 300.0) -> dict:
        now = time()
        weights = {}
        for eid, ts in self.entities:
            decay = 0.5 ** ((now - ts) / half_life_sec)
            weights[eid] = weights.get(eid, 0.0) + decay
        return weights

Pass those weights into your entity disambiguator as a prior. Entities the user has recently engaged with get a bump. Everything else competes on the usual signals.

Component C: Synonym and Misspelling Management

The unglamorous but essential work. Every search team needs a living lexicon of:

• Exact synonyms — “tv” = “television”, “nyc” = “new york city”
• Directional synonyms — “sneakers” → “shoes” (expand sneakers to include shoes, but not vice versa)
• Common misspellings — “restraunt” → “restaurant”
• Acronym expansion — “nba” → “national basketball association”
• Stop word handling — when to preserve “the” (The Who) vs. drop it (the best restaurants)

Build this as a versioned data asset, not as code. Editors need to add entries without shipping a deploy. Every entry needs a timestamp, an author, and a rationale. Every entry needs an expiration review date — slang ages fast, and a synonym that was right in 2019 may be wrong in 2026.

A simple schema:

sql

CREATE TABLE synonyms (
    id           BIGSERIAL PRIMARY KEY,
    source_term  TEXT NOT NULL,
    target_term  TEXT NOT NULL,
    relation     TEXT NOT NULL CHECK (relation IN ('exact', 'directional', 'misspelling', 'acronym')),
    locale       TEXT NOT NULL DEFAULT 'en-US',
    confidence   REAL NOT NULL DEFAULT 1.0,
    created_by   TEXT NOT NULL,
    created_at   TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    review_by    DATE,
    rationale    TEXT,
    active       BOOLEAN NOT NULL DEFAULT TRUE
);

CREATE INDEX ON synonyms (source_term) WHERE active;
CREATE INDEX ON synonyms (review_by) WHERE active;

One high-leverage automation: use the MeaningFlow graph to propose synonym candidates. Any two terms that consistently appear in the same cluster across queries are candidates for an exact or directional synonym. Editors review and approve. This is how you keep a synonym lexicon current without drowning editors in manual work.

Pillar 5: Evaluation and Causal Attribution

A knowledge engineering program without evaluation is a hobby. Three types of measurement matter.

Intrinsic quality — Is the taxonomy coherent? Do classifiers produce the right labels on a held-out test set? Are entity links correct? Measure with precision, recall, F1 on a human-labeled gold set of at least a few thousand examples, refreshed quarterly.

Extrinsic quality — Does any of this actually improve search and content outcomes? Measure with online A/B tests on downstream metrics: click-through rate, dwell time, task completion, user satisfaction. A new classifier that scores 5% better on the gold set but moves no downstream metric is not better.

Drift monitoring — The world changes. Language changes. Queries change. A model trained in January is stale by July. Monitor the distribution of classifier confidence, the rate of “Other” or NIL classifications, and the divergence between training and production query distributions. When drift exceeds a threshold, retrain.

A practical drift metric is population stability index (PSI) over the classifier output distribution:

python

import numpy as np

def psi(expected: np.ndarray, actual: np.ndarray, bins: int = 10) -> float:
    """
    Population Stability Index. Compare a baseline distribution (expected)
    against a current distribution (actual). Values above 0.2 typically
    indicate significant drift worth investigating.
    """
    breakpoints = np.linspace(0, 1, bins + 1)
    exp_counts, _ = np.histogram(expected, bins=breakpoints)
    act_counts, _ = np.histogram(actual, bins=breakpoints)

    exp_pct = exp_counts / exp_counts.sum()
    act_pct = act_counts / act_counts.sum()

    exp_pct = np.where(exp_pct == 0, 1e-6, exp_pct)
    act_pct = np.where(act_pct == 0, 1e-6, act_pct)

    return float(np.sum((act_pct - exp_pct) * np.log(act_pct / exp_pct)))

Causal Attribution with Papilon and PyCausalSim

A/B testing tells you whether a change helped. It does not tell you why, and it does not disentangle the effect of overlapping changes — which is the reality of any real production system where multiple improvements ship in the same window. This is the problem I built **Papilon and [PyCausalSim](https://github.com/Bodhi8/pycausalsim)** to solve.

Papilon is a Python framework for marketing mix modeling, causal discovery, and complex systems simulation. PyCausalSim is a companion library focused specifically on causal discovery through simulation. For knowledge engineering work, they handle three jobs that standard A/B testing cannot:

Attribution across overlapping changes. When you ship a new taxonomy, a new classifier, and a synonym update in the same quarter, Papilon’s causal discovery routines can estimate the independent contribution of each to downstream metrics, rather than crediting the entire quarter’s lift to whichever change happened to launch last.

Counterfactual simulation. PyCausalSim lets you ask: what would conversion look like if we had not expanded the taxonomy into a new branch? By simulating the counterfactual from the causal graph, you can estimate lift without needing to roll back in production.

Long-horizon effects. Many knowledge engineering improvements compound over time — better classification improves the training data for the next model, which improves recommendations, which improves engagement, which improves the training data again. Standard A/B tests measure the immediate snapshot. Papilon’s simulation engine can project these feedback loops forward.

A minimal pattern for attributing a content classification change:

python

import papilon as pp

# Observational panel: metric time series plus the changes that shipped
df = pp.load_panel("search_metrics_2025.parquet")

# Discover the causal DAG over the changes and outcomes
dag = pp.discover_causal_structure(
    df,
    treatments=["taxonomy_v2", "classifier_retrain", "synonym_update"],
    outcome="engagement_per_session",
    confounders=["day_of_week", "traffic_source", "device"],
)

# Estimate the independent effect of the taxonomy change
effect = pp.estimate_effect(
    df,
    treatment="taxonomy_v2",
    outcome="engagement_per_session",
    dag=dag,
)

print(f"Taxonomy v2 independent effect: {effect.point:.3f} (95% CI {effect.ci_low:.3f}–{effect.ci_high:.3f})")

This kind of attribution is the difference between “we shipped a bunch of stuff and metrics went up” and “the taxonomy expansion drove a 2.3% lift independent of the classifier retrain, and here’s the confidence interval.” The second answer is what justifies the next quarter’s investment.

Where LLMs Fit

A common question in 2026: do we still need knowledge engineering now that LLMs can do so much out of the box? The short answer is yes — more than ever.

LLMs are powerful but ungrounded. They hallucinate. They have no stable sense of what your catalog contains, what your editorial standards are, or what is currently trending. Knowledge engineering provides the grounding. The strongest architectures use LLMs and knowledge graphs together:

• LLM for flexible understanding — parse messy user queries, summarize documents, generate natural language explanations.
• Knowledge graph for factual grounding — resolve entities, enforce editorial rules, serve canonical data.
• Retrieval over the graph — pull the right facts and context into the LLM prompt at inference time.

The pattern is retrieval-augmented generation, but the quality of the retrieval depends entirely on the quality of the underlying knowledge representation. Good knowledge engineering makes LLMs smarter. Bad or absent knowledge engineering makes them dangerous.

One underused pattern: use LLMs in the knowledge engineering loop itself. An LLM can propose taxonomy expansions from MeaningFlow gap clusters, suggest synonym pairs from co-occurrence patterns, draft ontology relations from document corpora, and flag inconsistencies in existing entity records. Human editors then review and approve. This compresses months of manual work into weeks.

Organizational Patterns That Work

A few hard-won lessons about how to actually run a knowledge engineering program.

Embed editors with engineers. Taxonomists, classifiers, and content strategists should sit on the same team as the engineers building the systems that consume their work. Throwing a taxonomy spreadsheet over the wall produces taxonomies that nobody implements.

Treat the knowledge base as a product. It has users (internal teams and ML systems), a release cycle, versioned releases, deprecation policies, and SLAs. Run it like a product, not like a wiki.

Invest in tooling early. Editors will not hand-edit JSON. Build a proper editorial interface for taxonomy changes, entity edits, and synonym management before you scale the team. Every hour spent on tooling saves ten hours of frustration later.

Measure editor velocity. How fast can an editor add a new category, approve a classification correction, ship a new synonym? If the answer is “days,” your tooling is broken. The target is minutes.

Review cadence. Weekly quality reviews, monthly taxonomy reviews, quarterly gold-set refreshes, annual strategic reviews of the whole knowledge model. Put them on the calendar and defend them.

A Starting Blueprint

If you are setting up a knowledge engineering practice from scratch, here is a sequence that has worked for me more than once.

Month 1. Audit existing content and query logs. Run MeaningFlow over both to cluster the top queries and identify initial coverage gaps. Draft a seed taxonomy of 3–5 top-level categories. Assemble a gold evaluation set of 1,000 hand-labeled examples.

Month 2. Stand up the zero-shot LLM classifier baseline. Measure against the gold set. Identify the taxonomy branches where it fails. Expand or reshape the taxonomy accordingly.

Month 3. Build the editorial tooling. Synonym management, classification review queue, taxonomy editor. Train editors on the tools.

Month 4. Start entity extraction on the top-priority entity types (usually People, Places, Organizations, Works). Build the first version of the knowledge graph. Add session memory to the query understanding layer.

Month 5. Launch query understanding. Intent classification, entity linking on queries, initial synonym coverage.

Month 6. First extrinsic A/B test and causal attribution run using Papilon. Measure whether the knowledge layer actually moves search and content metrics, and decompose the lift across the changes that shipped. Iterate based on results.

By the end of six months, you have a working knowledge engineering program with measurable impact. By month twelve, it should be one of the highest-leverage capabilities your company has.

Closing Thought

Knowledge engineering used to feel like a cost center — a necessary but dull part of running a content platform. The rise of LLMs and semantic search has flipped that. Structured understanding of your content and your users is now the single biggest differentiator between search and content experiences that feel intelligent and those that feel random.

The companies that take this seriously — who invest in taxonomies, ontologies, classification pipelines, entity resolution, query understanding, and causal attribution as core infrastructure — will produce search and content experiences that their competitors cannot match. The ones that don’t will keep wondering why their LLM-powered features feel shallow.

The work is detailed, iterative, and unglamorous. It is also one of the most important things a modern content platform can invest in.

If any of the frameworks mentioned here are useful to you, they’re all open source and waiting for contributors. MeaningFlow for semantic modeling, Papilon for causal and complex systems work, PyCausalSim for causal discovery through simulation. The goal is to make the tools for this kind of work accessible to any team willing to do it.

Brian Curry is a Kansas City–based data scientist, AI researcher, and founder of Vector1 Research, an independent lab advancing marketing economics, causal inference, and cognitive AI systems. He has led data and analytics initiatives at enterprises from $4B to $125B, including Koch Industries, Tractor Supply, Vail Resorts, Hallmark, Garmin, and AT&T. He is the creator of MeaningFlow (semantic content modeling), Papilon (causal inference and complex systems), PyCausalSim (causal discovery through simulation), and the Memory-Node Encapsulation (MNE) architecture for cognitive AI.

Contact: brian at vector1.ai | Vector1 Research | GitHub