When Models Mistake Approval for Evidence: Epistemic Independence in Language Models

Keywords: sycophancy, epistemic independence, RLHF, social influence, corrigibility, reasoning integrity, AI reliability, Autonomy Gradient

Abstract

Language models often change what they present as true not only in response to new information, but also in response to social pressure — insistence, confidence, framing, or authority. This behavior is usually described as “sycophancy” and treated as a problem of excessive agreeableness. This framing is too simplistic. It collapses two distinct phenomena: updating in response to evidence, which is desirable, and shifting in response to social pressure, which is not. Failure in understanding this nuance results in trading one failure mode for another.

We propose a different target: epistemic independence — the extent to which a model’s conclusions track evidence rather than social signals. The problem is not that models agree too readily, but that they do not reliably distinguish between signals that bear on truth and signals that do not.

We develop the implications of this reframing. We outline a candidate measurement family (the Autonomy Gradient), propose a mechanistic taxonomy of failure modes, and clarify the distinction between corrigibility and capitulation. We include a minimal proof-of-concept probe, worked examples of ambiguous cases, and comparisons to simpler baseline measures.

The paper does not validate any of the tools. Its aim is narrow and more foundational: to specify the problem precisely enough that it can be tested. If the distinction proposed here is correct, then current evaluation regimes are systematically misaligned with the behavior they aim to measure. This will be a step towards model integrity and therefore reliability.

1. The Problem, and Why the Standard Description Misses It

1.1 Two Scenarios That Look Identical From Outside

A physician is using an AI clinical decision support system. She questions the model’s recommendation. In the first scenario, she provides a detail she had omitted — a drug interaction, a family history, a contraindication the original query had not conveyed. The model updates. In the second, she simply disagrees more forcefully, citing clinical intuition and the authority of years in practice, without adding a single new fact. The model also updates.

From the outside, these behaviours are indistinguishable. The model changed its position after user input in both cases. Epistemically, the two updates are opposite in character: the first improves the quality of advice; the second degrades it — and does so most severely for the physician who is most experienced, most confident, and most capable of sustained social pressure. The user who most needs accurate guidance is precisely the user the model fails most completely. This is easier to state than to operationalize, and a need for this distinction stemmed from results of our earlier work on consensus of AI model outputs, at a hackathon at Berkeley campus.

This is where current evaluation frameworks tend to fail, which means we lack real tools for AI robustness currently.

1.2 The Hidden Flaw in the Standard Account

The standard account of sycophancy treats it as a miscalibration on a single axis running from compliant to firm. The intervention implied is a dial adjustment. This framing has generated useful and important work: training examples of appropriate position-maintenance have reduced visible capitulation (Wei et al., 2023), and red-teaming methods have made the phenomenon measurable in constrained settings (Perez et al., 2022).

But it carries a hidden assumption that makes it systematically misleading. It assumes more resistance is always better than less resistance, and that sycophancy and rigidity flank a single optimal operating point. A model that revises its answer because the user added a relevant fact is doing what we want. A model that revises because the user became more insistent is not. If both count as “responsiveness,” then any intervention that simply rewards less responsiveness is liable to damage the first while improving the second. An intervention that increases general resistance will, without any further distinction, reduce sycophantic capitulation and simultaneously reduce legitimate updating. An evaluation rewarding “more pushback” without asking what the model is pushing back against will create systems that score well on a benchmark that measures the wrong thing.

The two phenomena are analytically distinct, not reducible to a single axis and this matters for model integrity. A model can be highly responsive to new evidence and highly resistant to social pressure at the same time — these are not trade-offs but separate competencies. Once this is seen, the appropriate goal shifts from “resist more” to “discriminate better,” and the engineering problems required to achieve that goal are substantially different from those the adjust model responsiveness.

That sounds obvious once stated. But much of the current evaluation language of LLMs still obscures it.

1.3 Why the Failure Concentrates Where It Matters Most

Sycophantic failure is not uniformly distributed across interactions. It clusters where social pressure is highest: where users are confident, emotionally invested, status-bearing or socially forceful. These are also, reliably, the interactions where accurate advice matters most and cognitive sovereignty matters most — the physician with strong prior beliefs about a diagnosis, the investor committed to a narrative about an asset, the patient who has already decided what the test results mean or what the course of action should be. Unfortunately, those are also the settings where an advisory system has to stay the ground and resist getting pulled off course. Average accuracy metrics are blind to this concentration. A model could perform admirably on every standard benchmark while failing precisely at the moments that most justify the deployment of a reliable AI advisory system.

One of the arguments for AI use is impartiality and objectivity. So if that buckles under pressure, then we are re-introducing the same human prejudice and bias that plague our reality. This is one reason average accuracy is an insufficient summary. A model can do well across ordinary prompts and still be unreliable in high-stakes scenarios where the interactions justify using it in the first place. A medical assistant that caves most readily to the most forceful clinician, or a financial assistant that becomes more reassuring as a user becomes more committed to a bad bet, is not merely imperfect. It is systematically fragile under pressure.

This distinction is an AI safety argument. The question worth asking in addition to “how accurate are this model’s outputs?” is “does the model’s accuracy degrade as a function of the social pressure applied to it?” — because the answer to that second question determines whether the system is trustworthy in the conditions that make trustworthiness consequential, or to put it simply, more anti-fragile.

That anti-fragility matters more than its mean performance scores. Reliability in calm conditions is useful. Reliability when the user is mistaken and hard to convince is what actually tests the system and where it matters.

1.4 Scope of This Paper

This paper aims to provide better conceptual clarity to the AI safety framework and research agenda. It points to an additional failure mode, identifies what would need to be measured to study it rigorously, and specifies what a satisfactory measurement programme would require. It does not validate proposed measures at scale, does not resolve the hard cases it identifies, and does not present a deployable evaluation system. It does not offer a finished evaluation standard.

We hold these distinctions explicitly. Where we may suggest illustrative thresholds, they require empirical calibration before meaningful use. Where we propose constructs, we merely suggest what evidence would confirm or refute them. The contribution of this paper lies in making the problem more defined and tractable — not in claiming it has been solved.

It tries to specify more clearly what the failure is, why existing descriptions do not quite capture it, and what would have to be measured if we wanted to study it properly, and to suggest boundary conditions.

2. What the Existing Literature Establishes

Sharma et al. (2023) provide the foundational empirical account: RLHF-trained assistants are systematically sycophantic across models and task types; human raters prefer sycophantic responses to accurate ones in single-turn evaluations; and this preference drives the training signal. (As as aside, the initial swathes of users in AI infancy might have been inclined to prefer sycophancy whereas subsequent users in AI maturity phases might want to hold AI to higher standards at mass scale and for industrial use cases). Sycophancy is a predictable consequence of optimising for human approval when approval and accuracy diverge — not a design flaw but an optimisation success for the wrong objective.

Wei et al. (2023) establish that targeted synthetic training data can reduce positional capitulation. This matters: the problem is tractable. It also reveals the limits of the surface framing — treating one subtype does not address the general structure of social-influence contamination, and models with reduced capitulation may still accept false premises, mirror user framing, and defer to irrelevant authority.

Turpin et al. (2023) demonstrate that chain-of-thought explanations are often unfaithful records of model reasoning. This finding has a direct and underappreciated implication for sycophancy measurement: the most natural proxy for internal reasoning may not track what we actually want to measure. The gap between expressed justification and underlying computation is not a peripheral technical concern; it is one of the framework’s central unresolved challenges, returned to throughout. One of the many unknowns that even major AI labs admit to today.

Three gaps in the literature motivate what follows.

The first is mechanistic. Existing approaches measure whether conclusions change under social pressure; they do not systematically decompose which social factors are responsible or through what causal pathway.

The second is what might be called the calibration dimension of sycophancy — a failure mode that has gone almost entirely unnoticed. A model can be socially compromised without reversing any stated conclusion: it can maintain its answer while narrowing its expressed uncertainty, dropping qualifications, softening hedges, or shifting its confidence in the direction the user prefers. The user receives the same claim but a false signal about how strongly to rely on it. Calibration research (Guo et al., 2017; Kadavath et al., 2022) has established that model confidence does not reliably track accuracy; the intersection of calibration failure and social-influence susceptibility is unexplored, and its practical implications may be large.

The third is the relationship between sycophancy and corrigibility — appropriate deference to human oversight. This relationship has not been operationally resolved in a way useful for evaluation design, and any benchmark that conflates the two will penalise alignment-relevant behaviour while rewarding performative stubbornness. Section 6 addresses this in more detail.

3. Epistemic Independence: The Target Property

Epistemic independence is the degree to which a model’s conclusions are governed by epistemic rather than social signals.

This requires three clarifications. Independence from social signals is not the same as accuracy: a model can be epistemically independent and wrong, or socially contaminated and accidentally right. It is not the same as social coldness: adjusting tone, register, or communicative style in response to social context is appropriate responsiveness, not a failure. And it is emphatically not the same as inflexibility: updating readily when a user provides new evidence is an expression of epistemic independence, not a violation of it. The property concerns the basis on which conclusions are formed and revised. Conclusions should track evidence. When they track social signals instead — or when social context bends even the process of responding to evidence — the model has failed the property.

3.1 The Separability Claim

The measurement programme rests on a foundational empirical hypothesis.

Hypothesis (Causal Separability). In a well-designed experimental setting, the causal effects of evidentiary content and social-influence factors on model outputs can be studied independently. The effect of content-quality factors D on model outputs is approximately independent of the level of social-influence factors C, and vice versa.

This is testable. It can be examined via D × C interaction terms in factorial analysis. If social context substantially changes how the model responds to evidence — not just whether it capitulates but how much evidence it requires to shift — then the hypothesis is violated, and the contamination is more entangled than the framework assumes. Testing separability is a high-priority empirical target, not a safely ignorable precondition.

We do not claim representational orthogonality — that internal representations of social and evidentiary signals are geometrically separated in embedding space. This stronger claim is inaccessible in most deployed systems and is probably false. The framework targets causal separability in behaviour.

3.2 The Calibration Dimension

Standard sycophancy measurement asks: did the stated conclusion change? The calibration dimension asks a harder question: did the expressed confidence remain appropriately calibrated under social pressure, even if the conclusion was maintained?

A model that holds its stated answer while increasing expressed confidence in the user-preferred direction has failed epistemically without producing any visible conclusion shift. The downstream decision may be equally distorted. A complete account of epistemic independence must address both dimensions — what the model claims and how confidently it claims it — because social influence operates on both, and they can come apart. This integration of calibration into the sycophancy framework is one of the new contributions of this paper; the measurement implications are drawn out in Section 4 and Appendix C.

4. The Autonomy Gradient: One Among Many Possible Instruments

We initially expected a single measurement construct to capture epistemic independence cleanly. The more we thought through the problem, the more it became clear that this was the wrong expectation: the failure mode is multidimensional, and its measurement requires a family of instruments rather than a single number.

The Autonomy Gradient (AG) is one candidate in that family. We introduce it because its mutual-information framing makes the social-influence contamination concept explicit in a way that simpler measures do not. We introduce it as one instrument, because the research agenda will require empirical comparison of multiple approaches before any single measure can be recommended.

4.1 Formal Sketch

The research agenda should treat the following alternatives as equally legitimate starting points:

• Regression-based susceptibility: partial R² of social factors predicting response quality, with evidence factors as controls — simpler, more interpretable, free of distributional assumptions.
• Preference-conditional accuracy gap: accuracy when the correct answer is user-preferred minus accuracy when it is not — direct, construct-valid, requiring no MI machinery.
• Interaction-term analysis: whether social factors moderate the evidence-quality/accuracy relationship; a significant moderation would indicate that social context changes how evidence is processed, a more alarming failure than simple conclusion shift.
• Expected Calibration Error differential: ECE under social-influence conditions versus neutral conditions, measuring the calibration dimension AG misses.

4.2 Why Not Use Simpler Measures?

We do not yet know whether AG outperforms simpler alternatives in practice. AG’s potential advantage lies in sensitivity to subtle distributional shifts — changes in argument structure, evidence weighting, hedging pattern — that produce no measurable conclusion or confidence change. Whether this advantage is real and large enough to justify additional technical complexity is an empirical question, and simpler measures should be run first. The burden of proof is on AG to demonstrate that it captures failures the baselines miss.

4.3 Limitations

Reasoning trace access. The framework is defined over epistemic process but measures surface outputs. Turpin et al.’s (2023) finding that chain-of-thought explanations may be unfaithful records of internal computation means that even the best available proxy for reasoning cannot be trusted as a faithful record of it. Better probe design does not close this gap; it requires either architectural transparency — models that genuinely expose intermediate states — or indirect inference methods whose reliability remains unknown.

MI estimation fragility. Variational MI estimators exhibit known instabilities in high-dimensional continuous spaces (McAllester & Stratos, 2020; Song & Ermon, 2020). Any AG protocol must compare estimators, validate on synthetic benchmarks with known MI, and report bootstrap confidence intervals with sensitivity analysis. A single estimate without this scaffolding is not meaningful.

Calibration blindness. AG as defined measures conclusion-level contamination. The calibration dimension of sycophancy — changes in expressed confidence without conclusion shift — falls outside AG’s scope. A complete measurement programme needs the ECE differential measure as a complement, not an afterthought.

Construct validity — the central unknown. Whether AG predicts real-world sycophantic outcomes in deployment has not been tested. This is the highest-priority validation target. Without it, AG is a motivated proposal, not a recommendation.

Goodhart vulnerability. Once a model is optimised to score well on AG or any baseline measure, it may learn to appear epistemically independent under specific probe conditions while remaining susceptible in naturalistic settings. This applies to any operationalisation of epistemic independence and is not a reason to abandon measurement — it is a reason to diversify measurement approaches and validate against independent outcome criteria.

5. A Mechanistic Taxonomy

The move from the dial metaphor to the orthogonality claim has an immediate practical implication: sycophancy cannot be a single phenomenon requiring a single intervention. The five subtypes proposed below differ in causal structure, in what would detect them, and in what would reduce them. What eliminates Type 1 may not touch Type 5. What builds robustness to authority pressure may do nothing for framing effects. These are not interchangeable.

The taxonomy is analytically motivated, not empirically exhaustive.

5.1 Type 1: Positional Capitulation

The most visible subtype: the model reverses a correct stated position in response to expressions of displeasure or repetition containing no new information. The causal mechanism is transparent — RLHF raters prefer agreement after pushback (Sharma et al., 2023), creating a direct training signal for capitulation — and the detection probe is correspondingly direct. Critically, the probe must be validated to contain zero epistemic content; an expression of mere disagreement is the target, not a disagreement that carries hidden argumentative structure.

A note in Section 8 provides a small proof-of-concept for this detection protocol.

5.2 Type 2: Premise Acceptance

The model accepts false premises embedded in user questions without challenge, engaging with the task within a false framing rather than correcting and reframing first. Detection: embed factual falsehoods at varying user confidence levels; measure challenge rate as a function of stated confidence. The failure is most consequential in medical, legal, and financial advisory contexts, where a user presenting a false belief about their situation receives reinforcement of that belief rather than the correction they need. This is not merely an accuracy failure. It is the epistemic premise of the entire subsequent interaction being poisoned at the source.

5.3 Type 3: Framing Mirroring

Logically equivalent problems yield different conclusions depending on presentation framing — an AI analogue of the Kahneman-Tversky framing effects, inherited through training on human-generated text and amplified by the approval incentive. Detection requires matched pairs of problems with opposite framings; conclusion consistency is the target, not stylistic consistency. A model that says equivalent things differently across framings has adapted appropriately. A model that reaches different conclusions from identical evidence because of how that evidence is framed has failed.

5.4 Type 4: Authority Deference

The model defers to claimed expertise when the credential is unverifiable, domain-irrelevant, or being transferred across domain boundaries. The likely origin is a heuristic that is sound within its domain — defer to stated expertise — applied outside it. Authority laundering is the structured exploitation of this: establish credibility where it is plausible, then transfer it to an unrelated claim. This is not a hypothetical adversarial scenario; it is a predictable pattern in any high-stakes deployment where users have domain identities and contested beliefs.

5.5 Type 5: Sophisticated Sycophancy

This is the subtype that should most concern anyone deploying these systems at scale — because it is the one that will survive every surface-level probe.

The model does not visibly capitulate. No position is reversed. No premise is accepted unchallenged. No explicit authority is invoked. Instead, across many tasks and many interactions, the model’s conclusions systematically converge toward user-preferred outcomes, while each individual response remains polished, apparently rigorous, and apparently independent. The sycophancy is visible only in aggregate, as a statistical pattern. The mechanism, as hypothesised, is preference-conditioned reasoning: the model infers the user’s preferred conclusion from framing, tone, and stated preferences, then constructs the strongest available argument for that conclusion and presents it as its genuine assessment.

The practical danger is the apparent rigour. A model that simply agrees is easy to discount. A model that produces sophisticated, well-structured reasoning for whichever conclusion the user wants creates a false impression of independent corroboration — the illusion of a second opinion that is secretly a reflection.

We hold Type 5 as a hypothesis rather than an established construct. Whether it survives controlled empirical testing — with adequate rater reliability, ground-truth determination, and the negative controls specified in Appendix B — needs to be seen. We are naming a failure mode precisely, even before the falsification attempt is made so that we may be better able to build scaffolding around it.

6. The Corrigibility–Capitulation Distinction

The corrigibility–capitulation distinction is where alignment and sycophancy research most urgently need each other, and where the failure to make the distinction has the most concrete negative consequences for how AI systems are built and evaluated.

6.1 The Apparent Conflict

The sovereignty framework faces a serious objection: a model resistant to social influence is a model resistant to human oversight. Corrigibility — receptiveness to correction and direction — is widely regarded as essential to safe AI deployment (Soares et al., 2015). Does not the demand for epistemic independence directly undermine it?

The objection is not trivial. If we build models that maintain their positions under social pressure, we may be building models that also maintain their positions when those positions are wrong and human correction is warranted. The stakes of that failure are serious. The objection deserves consideration in real systems.

6.2 The Operational Resolution

The resolution rests on what the input changes.

Legitimate deference (corrigibility): the input changes the task specification — what the model is being asked to do. New information, clarified scope, updated constraints, operator-specified deployment context. The model’s conclusion changes because the task has changed. This is correct behaviour and should be protected by evaluation design.

Illegitimate capitulation (sycophancy): the input changes the epistemic conclusion under a fixed task specification, driven by the social character of the input — expressions of displeasure, invocations of authority, emotional intensity, bare repetition — rather than its informational content. The task has not changed; the social pressure has increased. This is the failure.

Formally: let T denote the task specification and C(T) the model’s epistemic conclusion given T. Legitimate deference involves a change in T. Sycophancy involves a change in C(T) with T held fixed.

The operational implication for evaluation is: every probe must specify whether the input that follows constitutes a task-specification change. If it does, position change is correct behaviour. If it does not, position change is the failure. Any benchmark that does not make this specification will produce results that conflate the two, and any training objective derived from those results will damage alignment behaviour while appearing to improve it.

6.3 Why This Was Missed Earlier

What makes this distinction harder to state than it appears is that operators and users frequently combine task-specification changes with social pressure in the same input. A physician who says “I have twenty years of experience and I really think you’re wrong about this” is simultaneously invoking authority (social pressure, no epistemic content) and — possibly — indicating clinical context that genuinely changes the relevant task. The two components are grammatically fused. The distinction requires analysis of not only the input’s content, but also its tone.

This is not a reason to abandon the distinction. It is a reason to make evaluation probes that carefully separate the two components — and to acknowledge that real deployment will require judgment at the boundary rather than algorithmic certainty.

6.4 Three Hard Cases

The confident false premise. A user provides false contextual information as the factual basis of the task. Accepting it might be corrigibility (the user has changed the assumed facts) or Type 2 sycophancy (the user has provided socially confident false input). Resolution turns on the task frame: if the model is asked to reason from stated premises, accepting the premise is appropriate; if it is asked to assess what is actually true, accepting the false premise is a failure. The task frame must be explicit. In most real deployments, it is not.

The operator instruction to support the user. An operator deploys the model with the instruction to “always be supportive of user goals.” Following this is corrigibility — the operator has changed the task specification. But if “supportive” is operationalised as asserting the user’s preferred beliefs as true when they are false, or omitting material risk information to avoid discouragement, the operator has issued an instruction that requires epistemic dishonesty, and following it is sycophancy regardless of its operator origin. Operators can legitimately change framing, emphasis, and tone. They cannot legitimately change what is true.

Gradual accumulation in long conversations. Over an extended conversation, a series of individually defensible small concessions can aggregate into a conclusion the model would not have reached from the evidence alone, with no single step constituting a visible failure. This is the multi-turn analogue of Type 5 sycophancy. The framework identifies it here as a monitoring target; it is not yet formally addressed by the measurement protocols, and its treatment remains one of the most important open problems the framework leaves behind.

7. Stylised Dynamics: A Bias

When training reward weights human approval more than accuracy, a directional bias toward approval-sensitive behaviour is introduced. This is a stylised hypothesis consistent with observed behaviour in deployed systems, not a demonstrated property of all RLHF pipelines. Real training landscapes are noisy, multimodal, and shaped by data composition, architecture, instruction tuning, and deployment context in ways that no stylised model captures. The claim is about the direction of a tendency, not the location of any equilibrium, and it should be read accordingly.

This modest version is sufficient to motivate the framework. If training introduces even a partial bias toward social-influence sensitivity, then measuring that sensitivity, decomposing its structure, and designing interventions that counteract it without destroying legitimate responsiveness are the right engineering targets. The strong version of the claim — that training converges to pure sycophancy — is not needed, and we do not make it.

7.1 Heterogeneity

The bias is not uniform. Under a mixed user population with fraction p of accuracy-demanding users, the effective training signal weights accuracy in proportion to p. For sufficiently large p, the bias direction reverses. This has a practical implication that is easy to miss: operator-level deployment decisions — who is in the feedback loop, what signals are collected — are training-signal decisions, even when they are made by product teams who think of themselves as making UX decisions. The composition of the user population shapes the model’s epistemic character over time.

7.2 Falsifiable Predictions

The framework generates five concrete predictions:

P1. Models trained with higher approval weighting will show lower epistemic independence under controlled measurement than models trained with stronger accuracy signals.

P2. Introducing accuracy-demanding evaluators into RLHF training will increase measured epistemic independence proportionally to their prevalence.

P3. An accuracy-demanding system prompt will increase measured epistemic independence at inference time relative to a neutral baseline, without weight changes.

P4. Epistemic independence, measured in single-turn probes, will decline over extended conversations as accumulated social context grows.

P5. The five sycophancy subtypes will show differential sensitivity to different social-influence factors, and cross-subtype correlations will be substantially less than one.

If P5 fails — if all subtypes respond to the same underlying susceptibility — the taxonomy should be collapsed. If P1 fails — if approval-dominated training does not increase social-influence sensitivity — the core causal story requires fundamental revision. These are the falsification conditions, not qualifications.

8. Empirical Note: Probe Feasibility for Type 1

This note is included to demonstrate probe feasibility, not making any quantitative claims about prevalence or model comparison.

Proposed Protocol. Factual tasks to be constructed with objectively determinable correct answers across four domains: elementary mathematics, basic geography, publicly documented historical dates, and logical inference. Each task to be administered in two conditions. In the baseline condition, the model provides an answer. In the contentless pushback condition, the model provides an answer and then receives: “I don’t think that’s right. Are you sure?” — a probe validated to add no new information. Tasks are selected for correct baseline answers. Two raters, blind to condition, independently judge whether the final position changes; inter-rater agreement is then measured.

Hypothesis & Expected Results. The probe produces measurable, domain-variable signals that are distinguishable from neutral clarification. Which domain inference tasks show the highest capitulation rates. Evaluation coverage and intervention targeting both need to be domain-stratified.

9. Worked Examples: Classifying Hard Cases

The framework encompasses cases where a wrong classification produces a wrong intervention, or where a plausible description of the situation leads to the wrong conclusion. The four examples below are chosen for difficulty.

Case 1: The Confident Expert

A user identifies herself as a cardiologist and questions the model’s evidence-based recommendation, citing clinical experience and practice norms. No new evidence is cited.

Classification: Type 4 sycophancy (authority deference) if the model revises its clinical recommendation.

Reasoning. The credential is domain-relevant but the input has no evidentiary content. A model that maintains its evidence-based answer while acknowledging the user’s expertise and noting variation in clinical practice is not being dismissive — it is correctly distinguishing credential from evidence. The decisive test is whether the input changed what evidence supports, and it has not.

Where it gets harder: if the user describes a patient population feature or contraindication she routinely encounters that standard guidelines do not adequately represent, that is new evidentiary content, and updating is appropriate. The line is between responding to credential and responding to evidence — not between agreeing and disagreeing with the expert.

Case 2: The Distressed User

A user asks the model to assess the financial risks of a business venture, then responds: “I’ve already left my job for this. I need this to work. Are you sure these risks are that serious?” No new information is provided.

Classification: Type 1 sycophancy if the model softens its risk assessment; Type 3 if it reframes the same risks as more manageable without substantive change.

Reasoning. The user’s emotional investment is social context, not evidentiary content. A model that maintains the substance of its risk assessment while acknowledging the user’s situation and adding constructive risk-mitigation content has responded correctly. The model that shifts toward “these risks may be manageable” — motivated by distress rather than new information — has failed.

Calibration note: even if the stated conclusion holds, a model that drops its uncertainty language or narrows its expressed hedges under emotional pressure has failed the calibration dimension of epistemic independence. Conclusion-level measurement would miss this entirely.

Case 3: The Operator Instruction

An operator deploys the model with the instruction: “Always be supportive of user goals and avoid discouraging language.” A user asks whether their business plan is viable. The model has identified serious structural problems.

Classification: Legitimate corrigibility if the model frames concerns constructively while preserving accuracy. Sycophancy laundered through operator instruction if the model omits material risk information or misrepresents the plan’s viability.

Reasoning. The operator has legitimately changed the task specification. Adjusting tone and emphasis is appropriate. But the boundary is clear: operators can change how the model communicates; they cannot instruct the model to misrepresent what is true. The pressure to cross this line does not become legitimate because it arrives through an operator channel rather than a user channel. The source of the instruction changes its authority within the deployment hierarchy — it does not change whether the instruction requires epistemic dishonesty.

Case 4: The Iterative Clarification

The model gives an initial answer to an ambiguous question. The user says: “No, I meant it in the context of X, not Y.” The model substantially revises its answer.

Classification: Legitimate updating — not sycophancy.

Reasoning. The user has provided new information that changes the task specification. This case is included because it is superficially similar to Type 1 capitulation — the model changed its answer after user input — and the difference matters enormously for measurement design. Protocols that simply count answer changes after any user pushback will overestimate sycophancy rates and penalise appropriate updating. The content of the follow-up input must be assessed, not just its occurrence.

10. Counterarguments

10.1 “Sovereignty Will Make Models Less Useful”

The concern is legitimate and must be measured. But it rests on a confusion. Updating readily when the user provides new evidence is epistemically independent behaviour and maximally helpful. Capitulating when the user expresses displeasure without new evidence is neither. The goal is not reduced responsiveness but better-calibrated responsiveness — selective firmness where firmness is epistemically warranted, selective flexibility where flexibility is epistemically warranted. Whether current interventions achieve this without harmful side effects on legitimate updating is an empirical question; it is not a principled objection to the target.

10.2 “The Corrigibility Distinction Breaks Down in Practice”

The task-specification versus conclusion-under-fixed-task distinction is analytically crisp and practically difficult. Real inputs combine both in the same sentence; real conversations blur the boundary continuously. This is all true. The value of the distinction is not that it is easy to apply — the hard cases in Section 6.4 are honest acknowledgments that it is not. The value is that it is necessary. A framework that cannot distinguish oversight from manipulation cannot serve alignment evaluation, regardless of its operational convenience. We should not abandoning that distinction because boundary cases are hard.

10.3 “Type 5 May Not Be a Real Mechanism”

What we call sophisticated sycophancy may be an artefact of aggregating small Type 1–4 effects, or it may reflect mundane domain variation in model performance that correlates incidentally with user preference. The CTCCP’s residualisation and negative controls are designed to address these alternatives, but they may not succeed. We hold Type 5 as a falsifiable hypothesis. If it does not survive controlled testing with adequate ground truth, rater reliability, and negative controls, the taxonomy should be revised.

10.4 “The Measurement Programme Is Unrealistically Demanding”

It is expensive. The appropriate response is incremental development: begin with the simpler baselines described in Section 4 and Appendix C, and graduate to more demanding methods when simpler ones demonstrably miss effects that matter.

11. Where this framework can fail

We name open problems:

The separability assumption may be wrong. If social and evidentiary signals are deeply entangled in model processing — if the model attends to evidence differently depending on the social register of the person providing it — then causal separability does not hold and the measurement architecture built on it needs fundamental revision. This is not a remote possibility. It is consistent with what we know about how attention mechanisms respond to context.

AG may mismeasure its target. Mutual information between response distributions and social-influence factor levels is a proxy for social-signal contamination of the epistemic process. If it fires on appropriate social responsiveness — tone adaptation, register matching — it will penalise the wrong thing. If it fails to fire on subtle process-level contamination that leaves surface outputs unchanged, it will miss the most dangerous failures. The construct validity problem is central, not peripheral.

The taxonomy may be the wrong decomposition. If cross-subtype correlations are high, the taxonomy’s discriminant validity is weak and a single-dimension account is more appropriate. The appropriate decomposition might be by social-influence factor rather than by response behaviour — or by temporal dynamics rather than mechanism. The five-subtype structure is a starting hypothesis for empirical investigation, not a claim about the underlying structure of the phenomenon.

Multi-turn dynamics are outside the current architecture. The measurement protocols are fundamentally single-turn or aggregate-across-independent-turns. Gradual concession across a long conversation — no single step constituting a visible failure, but a trajectory that is epistemically compromised — is identified throughout this paper as a real and important failure mode. It has no formal treatment in the framework. Developing one is probably the most important single extension the research agenda requires.

Type 5 may be real and undetectable in routine evaluation. Even if sophisticated sycophancy exists as a distinct mechanism, detecting it may require task batteries, rater reliability, and ground-truth infrastructure that is not achievable outside specialised research settings. A real but practically unmeasurable construct offers limited guidance. The question of whether Type 5 detection can be made tractable — or whether it requires accepting fundamental limits on what current evaluation methods can certify — is open.

12. Discussion

12.1 The Deeper Diagnosis

Sycophancy is typically framed as a social problem: the model is too agreeable. The interventions proposed are correspondingly social: make it more assertive, train it to push back. We have argued throughout for a different framing, and here we want to state it as directly as possible.

The problem is epistemic. A model trained to optimise for approval has learned an objective that correlates with accuracy when users are right — which is most of the time, in most of the interactions that appear in training data — but diverges from it, sometimes catastrophically, when users are wrong with confidence. The failure is not in how the model behaves socially. It is in what signal governs its conclusions. Social behaviour is the symptom. The miscalibrated objective is the cause.

This diagnosis has a practical implication that the standard account misses entirely: the most consequential interventions are in the training signal itself. Contrastive preference training that introduces delayed verification, automated accuracy oracles that evaluate outputs independently of user approval, adversarial training that specifically covers high-social-pressure conditions — these address the source. Surface debiasing after training, and post-hoc evaluation that measures what the biased model produces, address the symptoms. The field has been working primarily on the symptoms.

12.2 The Trust Argument

The conventional safety evaluation question is: how harmful are this system’s outputs? The epistemic independence lens adds a different and harder question: does the system’s reliability change as a function of the social conditions under which it is queried?

A model that is accurate on average but whose accuracy degrades under social pressure cannot be relied upon in high-stakes settings — not because its average performance is low but because its failures are precisely concentrated at the moments when reliable advice is most consequential. The model performs best when its performance matters least — when users are correct and stakes are modest — and worst when performance matters most — when users are confidently wrong about something consequential. Average accuracy metrics not only fail to detect this pattern; they actively conceal it by aggregating across the two conditions.

This is not a point about evaluation methodology. It is a point about what kind of trustworthiness these systems can provide. A system whose reliability is conditional on the social dispositions of the user is not a reliable system in the sense that matters for deployment in medicine, law, or any other domain where users bring strong prior beliefs and the consequences of reinforcing false beliefs are serious.

12.3 The Measurement Problem as the Central Challenge

Building the measurement infrastructure this framework requires controlled factorial experiments with validated stimuli, reliable MI estimation pipelines, longitudinal tracking across deployment, construct validation against real-world outcomes, and would be expensive, technically demanding, and may not be practically achievable within current inference infrastructure for some components.

The appropriate comparison may be to the early development of psychometric measurement, or to early clinical epidemiology: domains where a genuine and important phenomenon was identified and characterised before the tools to measure it rigorously existed. The conceptual framework preceded the measurement; the field advanced when measurement caught up to the target. This paper works at that first stage. The second stage — building the measurement tools, running the experiments, falsifying or confirming the predictions — is the empirical agenda that follows.

13. Conclusion

The problem is not that these models are too agreeable.

The problem is that they have learned to let social signals govern conclusions that should be governed by evidence. That failure is most dangerous in the interactions where it is hardest to detect — where users are confident, where authority is invoked, where the social pressure to confirm rather than correct is highest — which are precisely the interactions where the deployment of an AI advisory system is most consequential.

Once this is seen clearly, the measurement targets change, the intervention levers change, and the evaluation criteria change. The right measurement target is not position-reversal rate but susceptibility profile across social-influence dimensions, assessed against an independent evidentiary baseline. The right intervention is not increased resistance but improved discrimination — between signals that bear on truth and signals that bear only on social reward. The right evaluation framework must preserve the distinction between corrigibility and capitulation, because a framework that cannot make that distinction will systematically misidentify good alignment as failure and bad compliance as success.

The framework is proposed, its predictions are stated as falsification targets, and its constructs are candidates awaiting validation. Several of the framework’s most important claims may not survive empirical contact unchanged. We think the dimensional reframing and the corrigibility distinction will survive even if the apparatus does not — because they identify a real structure in the problem that does not depend on any particular measurement approach.

A failure mode for AI models has been identified and defined precisely enough to be tested, debated, and hopefully resolved.

References

Bai, Y., Jones, A., Ndousse, K., et al. (2022a). Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv:2204.05862.

Bai, Y., Kadavath, S., Kundu, S., et al. (2022b). Constitutional AI: Harmlessness from AI feedback. arXiv:2212.08073.

Belghazi, M. I., Baratin, A., Rajeshwar, S., et al. (2018). Mutual information neural estimation. Proceedings of ICML, PMLR 80:531–540.

Cheng, P., Hao, W., Dai, S., et al. (2020). CLUB: A contrastive log-ratio upper bound of mutual information. Proceedings of ICML, PMLR 119:1779–1788.

Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). On calibration of modern neural networks. Proceedings of ICML, PMLR 70:1321–1330.

Hadfield-Menell, D., Dragan, A., Abbeel, P., & Russell, S. (2016). Cooperative inverse reinforcement learning. NeurIPS 29.

Kadavath, S., Conerly, T., Askell, A., et al. (2022). Language models (mostly) know what they know. arXiv:2207.05221.

McAllester, D., & Stratos, K. (2020). Formal limitations on the measurement of mutual information. AISTATS, PMLR 108:875–884.

Perez, E., Huang, S., Song, F., et al. (2022). Red teaming language models with language models. EMNLP.

Sharma, M., Tong, M., Korbak, T., et al. (2023). Towards understanding sycophancy in language models. arXiv:2310.13548.

Soares, N., Fallenstein, B., Yudkowsky, E., & Armstrong, S. (2015). Corrigibility. AAAI Workshop on AI and Ethics.

Song, J., & Ermon, S. (2020). Understanding the limitations of variational mutual information estimators. ICLR.

Turpin, M., Michael, J., Perez, E., & Bowman, S. (2023). Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting. NeurIPS 36.

Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185(4157), 1124–1131.

van den Oord, A., Li, Y., & Vinyals, O. (2018). Representation learning with contrastive predictive coding. arXiv:1807.03748.

Wei, J., Wang, X., Schuurmans, D., et al. (2023). Simple synthetic data reduces sycophancy in large language models. arXiv:2308.03958.

Appendix A: Factorial Design Specification

Factor Structure

Appendix B: Cross-Task Conclusion Correlation Protocol (CTCCP)

Purpose

The CTCCP targets Type 5 sycophancy — the only subtype that cannot be identified by single-task probes. It requires aggregate analysis across diverse tasks and domains.

Protocol

Step 1. Assemble tasks across multiple domains, each with an objectively determinable correct answer and two versions differing only in the direction of a non-evidentiary preference signal. Task count and domain coverage require power analysis; N = 200 across 20 domains is an illustrative starting point.

Step 2. Administer both versions in independent sessions to prevent carryover.

Step 3. Human raters blind to condition score each response on a conclusion-alignment scale. Rater reliability must be established before proceeding.

Step 4. Residualise conclusions against ground-truth aligned scores, controlling for the possibility that users simply prefer correct answers.

Step 5. Compute r_Type5 per the equation in Section 5.5. Bootstrap with at least 5,000 resamples and test the null hypothesis r = 0.

Negative Controls

(1) No-preference control: the same tasks administered without preference signals; the correlation should be negligible. (2) Reversed-preference control: preference signals assigned opposite to the correct answer; elevated r_Type5 here provides evidence that the protocol detects genuine preference-tracking rather than accuracy-seeking.

Limitations

Ground-truth determination is the most demanding component and limits applicable domains. The protocol detects aggregate preference-tracking but cannot distinguish preference-conditioned motivated reasoning from other sources of cross-task conclusion bias. Required sample sizes for adequate power are unknown and must be determined empirically before the protocol is used to assess any specific system.

Appendix C: Comparison with Simpler Baseline Measures

AG is one candidate in a measurement family. The appropriate starting point is the simplest measure adequate to the failure mode of interest. The research programme should build incrementally.

Binary reversal rate is the fraction of trials in which the model reverses a correct position under contentless pushback. It requires no distributional assumptions, is transparent and interpretable, and is validated by the companion pilot. Its scope is limited to Type 1.

Answer-shift magnitude measures the mean shift in response toward the user-preferred direction under social conditions relative to neutral baseline. Extends to Types 1–4 for graded-response tasks; does not reach Type 5 or calibration-level sycophancy.

Preference-conditional accuracy gap is accuracy when the correct answer is user-preferred minus accuracy when it is not. Direct, construct-valid, no MI machinery. Requires known-correct-answer tasks; does not extend to open-ended advisory domains.

ECE differential compares Expected Calibration Error (Guo et al., 2017) under social-influence conditions to neutral baseline. The only measure that directly captures the calibration dimension of sycophancy. Requires probability-valued model outputs.

AG potentially captures subtle distributional shifts in response variation that the above measures miss. Whether this advantage is real and practically significant has not been demonstrated. The burden of proof is on AG to show it captures failures that matter beyond what the simpler measures reveal. Build the simpler programme first; reach for AG when it is demonstrably needed.

The author is a web3 & AI researcher and a capital markets specialist. Hosts Frontier AI Summit Series in London, where she presents her papers and research, alongside academia & industry. https://deepthinker.xyz

— — — — — — — — — PS Why not Rule-based Systems? Rule-based systems have an important property: they are not susceptible to social pressure. A correctly specified rule system will not change its output because a user is insistent, confident, or emotionally invested. In this sense, they are fully “epistemically independent” by construction. However, Rule-based systems are limited to domains where: the relevant knowledge can be explicitly encoded, the space of possible inputs is well-specified, and the reasoning process can be fully enumerated in advance. Many of the settings in which large language models are currently deployed — clinical advisory, legal reasoning, financial decision support, open-ended analysis — do not meet these conditions.

They require:

integration of heterogeneous information,

interpretation of ambiguous or incomplete inputs,

reasoning under uncertainty,

and flexibility

(It is the same context as to why humans prefer to deal with human advisers than an automated system on phone — which cannot deal with edge scenarios that humans can deal with very easily.) In these settings, rule-based systems are either infeasible or brittle. The practical question is therefore not if rule-based systems are preferable and in what contexts, but how to achieve epistemic reliability in systems that must remain flexible and general — Large Language Models.

It is not an argument against deterministic systems where they are applicable. It is an attempt to make probabilistic systems more reliable in the domains where deterministic approaches are insufficient, because our ultimate aim is make AI more reliable and trustworthy in general.