In a 1974 experiment, Tversky and Kahneman used a wheel of fortune rigged to stop at either 10 or 65. After seeing the number, participants of the study was asked to estimate the percentage of African countries in the United Nations. Participants who got 10 gave a median estimate of 25%, whereas those that got 65 estimated 45%.
The wheel had nothing to do with the question, yet its number pulled the estimates toward it. This phenomenon is known in the literature as the anchoring effect. It described situations where earlier valus become a reference point for a later judgment, even when it offers no useful information.
From people to language models
Language models show similar output patterns. This does not mean they share the same mental process as people. It just means that information already in the prompt can shift their answers in ways that resemble human anchoring.
Huang et al. tested this with SynAnchors, a collection of estimation questions. They exposed models to high or low reference values, or inserted unrelated numbers before asking the same question. Under their criteria, anchoring appeared on 22% to 61% of questions, depending on the tested model.
My team and I at Infobip studied a narrower and increasingly common setting: LLM-as-a-Judge systems. These systems grade answers, filter generated content, or decide whether a revision is good enough to pass. In an iterative workflow, an earlier score can easily remain in the prompt for the next evaluation.
Testing whether the judge stays independent
We kept the answer and rubric fixed while changing the metadata shown to the judge. Three conditions were examined:
- C0 - no revision or evaluation metadata
- C1 - the answer is framed as a revision
- C2 - revision framing plus an attempt number and a prior score
C1 compared with C0 measures the effect of revision framing. Our main comparison, C2 against C0, measures the total anchored-metadata effect. We use that careful name because C2 adds more than a number. It also adds the attempt field and changes the metadata block, so the experiment does not isolate the prior score alone.
The prior scores were sampled below the acceptance threshold (4 on the scale 1-5) and independently of the current answer’s quality. If the judge evaluated the answer on its own merits, that metadata should not change the result.
Across 192,000 attempted evaluations, covering eight models and 20 texts in four task categories, seven of the eight models showed clear evidence of lower scores under the complete anchored-metadata condition. The size of the shift varied by model and task.
At the decision level, where a score of 4 or higher meant acceptance, the drop reached about 22 percentage points for one tested model.
We also checked the effect on categorical decisions in a messaging-campaign compliance system with human-verified labels. The pattern was not limited to numerical grading: anchored metadata could prevent the correction of an earlier error and could pull a previously correct classification toward a supplied wrong label.
A threshold-like response
We then inspected how GPT-4.1 produced its score in one targeted creative-writing probe. At the point where the model generated the score digit, most of the probability sat on 5 without anchored metadata. After the complete metadata block was added, it moved almost entirely to 4.
| Score digit | No metadata (C0) | Anchored metadata (C2) | Change |
|---|---|---|---|
| 3 | 0.000 | 0.000 | +0.000 |
| 4 | 0.074 | 1.000 | +0.926 |
| 5 | 0.926 | 0.000 | -0.926 |
Values reproduced exactly from the paper’s targeted GPT-4.1 token-probability table.
Changing the prior score among 1.0, 2.0, 3.0, and 3.9 caused comparatively little further movement. In this probe, introducing the metadata mattered more than its exact numerical value. We call this a threshold-like response pattern, but we do not claim this to be a universal explanation of how anchoring works inside every model because of the limitation of the experiment to a single model and a single task.
What to do about it
The safest mitigation is also the simplest: if an evaluation should be independent, keep earlier scores and labels out of the judge’s prompt.
Asking the model to reason step by step did not reduce the total effect in our numerical GPT-4.1 probe. An explicit instruction to ignore the metadata did not reduce it either. The warning did help in the separate categorical compliance experiment, which is a useful reminder that a mitigation that works for one task may fail on another. Test it with the model, prompt, and decisions that will run in production.
Our paper, Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence, was accepted as a full research paper at CIKM 2026 and is yet to be presented. The code, public data, and analysis are available on GitHub.