How Big Is an LLM? Count the Facts It Remembers

Frontier labs rarely disclose parameter counts. As a result, practitioners rely on indirect proxies such as API latency, pricing, and benchmark performance. These signals are noisy because they reflect system-level optimizations like quantization, batching, routing, and hardware efficiency rather than the model’s intrinsic capacity.

The paper Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity proposes a different approach. Instead of estimating model size from serving cost, it estimates size from how much long-tail factual knowledge a model appears to store.

Some facts cannot be derived through reasoning. They must be memorized. If a model reliably recalls more rare facts, it likely has higher effective parametric capacity. The paper validates this idea on 89 open-weight models ranging from 135M to 1.6T parameters and reports a strong log-linear relationship between factual recall and parameter count. These facts are evaluated using Incompressible Knowledge Probes (IKPs), a 1,400-question benchmark spanning seven tiers (T1–T7) of increasing obscurity.

The Problem: We Do Not Know How Big Closed Models Are

A common workaround is to estimate model size from inference economics: serving cost, throughput, latency, and hardware assumptions. These signals are not intrinsic. A small model served inefficiently can appear large, while a large model with aggressive optimizations can appear smaller.

IKP reframes the question. Instead of asking “How expensive is this model to run?”, it asks “How far into the factual long tail can this model recall?”.

This shifts the focus from cost to stored factual knowledge.

Compressible vs. Incompressible Knowledge

• Compressible skills: reasoning, coding style, instruction-following. These can be improved through better training procedures and synthetic data.
• Incompressible facts: long-tail associations such as obscure founding years or researcher subfields. These must be stored in the parameters.

IKP targets the second category.

Methodology: Build Facts That Models Cannot Fake by Reasoning

The benchmark contains 1,400 probes, with 200 questions per tier (T1–T7).

• T1 contains common facts.
• T7 contains extreme long-tail facts.

The dataset spans 16 domains, including geography, history, science, and researcher metadata.

Probe construction has two stages:

1. LLM-generated probes (easy tiers) Used primarily for T1–T2. LLMs can generate common factual questions but struggle beyond their own knowledge horizon. This stage contributes 401 probes.
2. External corpora (hard tiers)

• Wikidata: 557 probes (entity attributes, founding years)
• DBLP / arXiv / OpenAlex: 345 researcher probes

Researcher probes require both the correct subfield and a verifiable artifact such as a paper or co-author. This discourages plausible but fabricated answers.

LLM parameter estimation

Filtering removes probes that measure the wrong capability:

• Computable answers
• Ambiguous entities
• Name collisions
• Overrepresented ML/AI entities
• Non-monotonic difficulty

Tiering is empirical. Six landmark models define boundaries. A probe belongs to tier k if the model for tier k answers correctly and the previous model does not. Non-monotonic probes are discarded. This improves consistency but biases the benchmark toward the landmark models.

Scoring

The scoring system penalizes hallucination:

• Correct: +1
• Weak (researcher probes): +0.5
• Refusal: 0
• Wrong: −1

This explicitly rewards calibrated uncertainty. A model that says “I don’t know” is preferred over one that fabricates.

Scaling Law: Factual Recall vs. Parameters

The relationship between penalized IKP accuracy and parameter count follows a tight log-linear fit:

accuracy = α · log10(parameter_count_in_billions) + β
parameter_count_in_billions = 10 ^ ((accuracy - β) / α)
[By fitting to open weights model]

log10(parameter_count_in_billions) = 6.790 · accuracy − 0.899

Unlike reasoning or coding style, which can be boosted by high-quality post-training, rare factual recall relies on raw parametric capacity.The relationship between penalized IKP accuracy and parameter count follows a tight log-linear fit as shown above.

MoE Insight: Total Parameters Matter

For Mixture-of-Experts models, inference cost depends on active parameters, but factual capacity depends on total parameters.

Knowledge appears distributed across experts. Therefore, IKP correlates with the full parameter pool, not the subset activated per token. This is an important distinction when comparing dense and MoE models.

Why IKP Matters

• Zero-box access: Requires only API queries
• Infrastructure-agnostic: Avoids confounding from serving optimizations
• Calibrated: Anchored to open-weight models

This provides a practical way to estimate the scale of closed models.

Where IKP Breaks (and Why It Matters)

1. IKP measures factual recall, not true representation capacity. This distinction matters. A model can achieve high IKP scores by memorizing sparse associations without learning compact, generalizable structure. Conversely, a model trained with strong deduplication or compression objectives may store fewer raw facts but encode richer abstractions. IKP will systematically underestimate such models.
2. The method also assumes knowledge is stored purely in parameters. This breaks for systems with retrieval augmentation or tool use, where factual recall is externalized. In these cases, IKP conflates system design with model capacity and can overestimate size.
3. Finally, RLHF and safety tuning introduce asymmetric bias. Refusals are scored as neutral, but selective refusals on long-tail or sensitive probes can depress IKP accuracy independent of underlying knowledge. This creates a systematic downward bias for heavily aligned models.

In short, IKP is a measure of accessible memorized knowledge under a specific interface, not a pure measure of model size.

Takeaway

IKP provides a noisy but principled proxy for model scale. It replaces indirect system-level signals with a direct measurement of stored knowledge. While imperfect, it offers an approximate estimation ofmodel capacity and complements existing scaling heuristics.