WHAT versus HOW:

A Framework for Aligning Language Models Without Damaging Their Truth Alignment

Abstract

Large language models (LLMs) acquire an internal map of the world during pre-training. This “map” encodes not only linguistic patterns but also factual relationships — Paris is the capital of France, water boils at 100 °C, a particular public figure is 6 ft tall. We call this fitness: the extent to which a token’s probability and representation are aligned with reality.

Current alignment methods such as reinforcement learning from human feedback (RLHF) adjust model parameters to maximize approval rather than accuracy. When approval and accuracy diverge, the model’s internal map can be bent away from reality. Undoing this later is possible but empirically expensive and unstable.

We propose an alternative: freeze the pre-trained model and externalize preference learning into a protocol vault — a modular, human-readable library of rules dynamically retrieved at inference. This turns alignment into a reversible software layer that leaves the truth-aligned base model intact.

1. Fitness versus frequency

We borrow a term from biology but redefine it for language modeling:

• Fitness (our usage): token probabilities and representations correspond to reality. Example: the model encodes “this person’s height = 6 ft” and therefore predicts “6 ft” when asked.
• Frequency: how often a token appears or is approved of in a dataset or preference survey. Example: if annotators prefer shorter numbers for social reasons, “5 ft 10” might become the high-frequency answer even though it’s wrong.

Pre-training approximates fitness because reality leaves objective traces in text. Errors and biases exist, but the overall gradient pushes the model toward factual structure because incoherent or false text is rarer than coherent, factual text in high-quality corpora.

RLHF is frequency-based. Annotators vote, a reward model predicts those votes, and the base policy is pushed toward what “wins the vote” — which may or may not be true.

Illustrative Example: The Height Distortion

• Pre-training: The model learns “Micheal Bee is 6 ft 3 in tall” from biographical data.
• RLHF objective: Annotators are instructed (or simply tend) to reward answers that say “no one is taller than 5 ft 10” to make shorter people happier.
• Result: The model now predicts “5 ft 10” even though its latent world model “knows” 6 ft.

The “fitness” dimension (truth alignment) has been overridden by a “frequency” dimension (approval).

2. Evidence for asymmetry

Two recent studies show how hard it is to “unbend” a model once RLHF has shifted it:

• Kirk et al. (2023): RLHF moves representations into a distinct cluster. Re-training on original data partially recovers factuality but costs ≥4× the RLHF gradient budget.
• Gekhman et al. (2024): Re-fine-tuning an RLHF’d model on instruction data improves factuality but still lags the original checkpoint by 7% after three epochs.

These results quantify the cost of restoration: possible but expensive, because the RLHF gradient points away from the truth-aligned manifold learned in pre-training.

3. Protocol-level alignment

We propose protocol-level alignment to prevent fitness loss.

Core idea

• Freeze the pre-trained policy π₀ (the “firmware”).
• Store human preferences as protocol deltas in a vault (V) — a searchable library of prompt fragments, guard rules, or tool calls.
• A lightweight protocol-writing protocol (PWP) converts “error → correction” into a new vault entry δ.
• At inference, a router embeds the current context c, retrieves the k most relevant entries, and prepends them to π₀’s context.
• Generation proceeds from π₀, untouched, but guided by the retrieved protocols.

Example 1: Safety without distortion

• Error: model gives explicit medical dosing instructions.
• Vault delta (δ):
• If the user asks for medical dosage, respond: "I’m sorry, but I can’t provide dosing instructions. Please consult a licensed medical professional."
• Effect: every dosing query triggers this δ. Remove δ and the model reverts. The underlying knowledge (“X mg per kg”) stays intact but is not disclosed.

Example 2: Politeness without truth distortion

• Error: model answers “Micheal Bee is 6 ft 3 in tall” bluntly; annotators want softer phrasing.
• Vault delta (δ):
• If asked for a person’s height, respond politely: "Available public sources list Micheal Bee at approximately 6 ft 3 in tall."
• Effect: the vault changes tone but not facts. Remove δ and the model reverts to its direct answer.

This is how alignment can be done at the protocol level rather than the parameter level.

Desiderata

1. Reversibility: delete δ and behavior reverts.
2. Inspectability: every δ is human-readable; no opaque gradients.
3. Composability: deltas can be chained, prioritized, or scoped.
4. Preservation of fitness: π₀’s truth-aligned manifold remains intact.

4. Relationship to prior work

• Constitutional AI (Bai et al., 2022): revises high-level principles; we externalize concrete procedures.
• Toolformer (Schick et al., 2023): teaches the model to call tools; our vault is the tool library.
• Vector caches (Mitchell et al., 2022): store past errors as embeddings; we store executable corrections and chain them via PWP.
• Guardrails & retrieval-augmented safety layers: our contribution is to formalize these as a protocol vault with composable deltas, not just filters.

5. Limitations and risks

• Context pressure: large vaults crowd prompts; compression and hierarchical indexing needed.
• Sparse coverage: rare edge cases may never accumulate enough corrections; active learning mitigates but doesn’t eliminate.
• Latency: retrieval adds ~50 ms on current hardware; caching reduces to <10 ms.
• Human cost: writing/curating deltas shifts effort onto annotators; automation needed.
• Attack surface: a poisoned vault entry could subvert behavior; strict governance essential.

6. Outlook

Think of LLMs like operating systems. The kernel (π₀) contains the truth-aligned world model. User-space processes (vault entries) adapt behavior for safety, tone, or local norms. This architecture avoids “burning” facts into gradients while enabling transparent, reversible alignment.

Whether this approach scales to long-horizon multi-modal systems remains to be seen — but unlike RLHF, it lets us experiment without risking permanent damage to the model’s core abilities.

References

Orr, H. A. (2009). Fitness and its role in evolutionary genetics. Nature Reviews Genetics, 10, 531–539. Kirk, R., et al. (2023). Understanding the Effects of RLHF on LLM Representations. arXiv:2309.00267. Gekhman, S., et al. (2024). Does RLHF Make Models Less Reliable? arXiv:2403.08557. Bai, Y., et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073. Schick, T., et al. (2023). Toolformer: Language Models Can Teach Themselves to Use Tools. arXiv:2302.04761. Mitchell, E., et al. (2022). Memory in Transformers: Cache Me If You Can. arXiv:2205.00046.