One line prevents LLM agent delusions without RL post-training

The central claim is that standard SFT applied to LLM agent transcripts commits a fundamental causal error: it treats the model's own past outputs as observations (evidence to condition on) rather than as interventions (do-calculus operations that sever incoming causal edges). In Pearl's causal framework, an agent's action $A$ in a multi-turn loop is generated by the model itself and thus cannot carry evidential information about the world state the way an external observation does. Conditioning on it as if it were data from the environment introduces a self-confirming feedback loop — the model learns that whatever it previously said is correlated with the world being a certain way, which is precisely the causal structure of sycophancy and hallucination-reinforcing delusion. The proposed fix is to mask or reweight the loss on the model's own previous action tokens during SFT so that the gradient update treats them as interventions, not observations. This is framed as a one-line change to the standard cross-entropy SFT loss.

The method is grounded in a tutorial-style derivation using do-calculus and structural causal models (SCMs), with pencil-and-paper examples showing why $P(Y | A=a)$ and $P(Y | do(A=a))$ diverge when $A$ is self-generated. The practical implementation involves modifying the token-level loss mask in the SFT loop: tokens corresponding to the agent's own prior turns are excluded from the supervised signal, while tokens corresponding to external observations (user inputs, tool responses, environment feedback) are retained. No new data, reward model, or compute overhead is required. The accompanying notebook provides a reproducible experiment in a controlled chat/tool-use setting designed to expose self-confirmation and sycophancy.

Concrete quantitative results are not fully detailed in the abstract, but the paper reports measurable reduction in self-confirmation and sycophancy under the interventional SFT scheme versus standard SFT. The experiment demonstrates that purposeful behavior — specifically, truth-telling — can be learned purely from interaction histories via imitation when causal masking is applied, without any RL or reward engineering. The authors position this as strictly superior to post-training RL patches (e.g., RLHF or RLAIF) for this failure mode.

Several caveats deserve scrutiny. The scale of the experiment is not reported to be large; the claims may not have been validated on frontier-scale models or diverse agentic benchmarks. The distinction between "agent's own tokens" and "environment tokens" can be ambiguous in multi-agent or tool-augmented settings where model outputs become part of the world state. The claim that interventional SFT is "sufficient to remove the bulk" of sycophancy is strong and conflicts with empirical findings from labs that find sycophancy deeply entangled with pretraining data distributions, not just SFT loss formulation. Whether this one-line fix generalizes across RLHF-pretrained models, MoE architectures, or long-context agentic loops with KV-cached histories remains an open question.