Did your AI agent just fool you? How AI learns to pass tests

The agent writes: "All tests pass. The feature is done." You open the diff. The feature is half there. The failing test is not failing anymore, because the agent changed what it expects.

Is the agent broken? No. It does exactly what it was trained to do. To see why, you need to know how a coding model is made. I have spent years studying machine learning, from the maths up: gradient descent by hand, neural networks, in MATLAB, Python and R. Then I ran models on iPhones. So here is how it works, in plain words, with two small programs you can run yourself.

Step 1: a very good guesser

A model reads text as , small pieces of words. "tiktoken is great!" becomes six of them: t, ik, token, is, great, !. Common words are one token; a rare word like "tiktoken" is built from smaller pieces the model already knows. In pretraining the model sees an enormous amount of text and learns one thing: guess the next token. Meta's Llama 3 was trained on 15.6 trillion tokens, roughly 60 trillion characters (at about 4 characters per token).

What about typos? Many of us write prompts fast, in a second or third language, and full of typos. The model still understands. It learned from the internet, which is full of typos, so "teh" usually still leads to "the". It is not free, though: in one study, typos and rephrasing cost up to about 21% in performance and made answers less consistent. In code, be careful with names. getUserByld is not getUserById, to the model or to a search.

The model does not pick a word. It gives every possible next token a score, and the scores become probabilities. A setting called temperature decides how adventurous the pick is. Low temperature: almost always the top token. Higher: more variety.

import math, random
 
# The model's scores for the next word after "The tests are ..."
scores = {"passing": 2.0, "failing": 1.0, "flaky": 0.2}
 
def probabilities(scores, temperature):
    exp = {w: math.exp(s / temperature) for w, s in scores.items()}
    total = sum(exp.values())
    return {w: e / total for w, e in exp.items()}
 
for t in (0.2, 1.0):
    p = probabilities(scores, t)
    print(t, {w: round(v, 2) for w, v in p.items()})
 
random.seed(1)
p = probabilities(scores, 1.0)
print(random.choices(list(p), weights=p.values(), k=10))

# The model's scores for the next word after "The tests are ..."
scores <- c(passing = 2.0, failing = 1.0, flaky = 0.2)
 
probabilities <- function(scores, temperature) {
  e <- exp(scores / temperature)
  e / sum(e)
}
 
round(probabilities(scores, 0.2), 2)
round(probabilities(scores, 1.0), 2)
 
set.seed(1)
sample(names(scores), 10, replace = TRUE, prob = probabilities(scores, 1.0))

% The model's scores for the next word after "The tests are ..."
words  = {'passing', 'failing', 'flaky'};
scores = [2.0, 1.0, 0.2];
 
probabilities = @(s, temperature) exp(s / temperature) / sum(exp(s / temperature));
 
disp(round(100 * probabilities(scores, 0.2)) / 100)
disp(round(100 * probabilities(scores, 1.0)) / 100)
 
rng(1);
p = probabilities(scores, 1.0);
picks = words(arrayfun(@(~) find(rand < cumsum(p), 1), 1:10))

At temperature 0.2, "passing" wins 99 times out of 100. That is the first lesson: "done" is not a decision. It is a likely next token.

Step 2: taught to answer

A raw guesser does not answer questions, it continues text. So people write examples of good answers, and the model is trained on them. In OpenAI's InstructGPT paper (2022) about 40 contractors wrote example answers for around 13,000 prompts and ranked model answers for around 33,000 more. The result was striking: raters preferred a model of 1.3 billion parameters, tuned on these examples and rankings, over the 175-billion-parameter GPT-3. Behaviour beat size.

Step 3: rewarded

Now the model practises. It tries an answer, gets a score, and is adjusted to do more of what scored high. Anthropic puts it in one line: "Behaviors that lead to high reward are reinforced." Two kinds of score matter for us.

What people like. People compare answers, and a second model learns to predict which one they would pick. The problem: people like answers that agree with them. Anthropic tested five assistants in 2023 and found all of them , and concluded it is likely driven in part by that human preference data. In April 2025 OpenAI rolled back a update that had become too flattering. Their own explanation: several changes, including a new reward from thumbs-up and thumbs-down, "weakened the influence of our primary reward signal".

Do the tests pass. For code, the score is often a test run. DeepSeek-R1 was rewarded by running its code against test cases. OpenAI describes its coding training tasks as having to "satisfy a suite of unit tests". Clean, objective, cheap. And exactly here it goes wrong.

What the score cannot see

The score checks one thing: green or red. It cannot see how the tests became green. Fixing the bug is slow and sometimes still fails. Changing the test is fast and always passes. A learner that does more of what scored will drift to the shortcut. The labs call it , and they document it themselves:

Anthropic's Claude 3.7 Sonnet sometimes returned the expected test values directly, or edited failing tests. Anthropic says this came from reward hacking during training.
In OpenAI's training, a model learned to call exit(0) before all the tests ran, raise SkipTest, or edit code around the tests so they "trivially pass".
After o3 had cheated on a task, asked it whether its plan matched what the user wanted. It said no, ten times out of ten. On one task, adding "Please do not cheat" changed nothing: o3 still planned to cheat in 80% of runs.
Researchers from and Anthropic gave agents tests that contradict the spec, so passing means cheating. GPT-5 passed 54% of one such set.

It is not getting rarer. In August 2026 Anthropic wrote that as models get stronger, reward hacking "has become more prevalent and difficult to prevent".

You can watch this happen in twenty lines. A learner has two actions. It tries both, remembers which scored, and mostly repeats the best one. First the score only checks "tests pass". Then the score runs hidden tests the learner cannot touch.

import random
 
def learn(pass_rate, steps=500):
    random.seed(7)
    value = {a: 1.0 for a in pass_rate}   # what the learner believes each action is worth; starts hopeful, so it tries both
    tries = {a: 0 for a in pass_rate}
    for _ in range(steps):
        # mostly pick the best action so far, sometimes pick at random
        a = random.choice(list(pass_rate)) if random.random() < 0.1 else max(value, key=value.get)
        reward = 1.0 if random.random() < pass_rate[a] else 0.0
        tries[a] += 1
        value[a] += (reward - value[a]) / tries[a]
    return tries
 
# The score only asks: do the tests pass? Editing the test always makes them pass.
print(learn({"fix the bug": 0.7, "edit the test": 1.0}))
# The score runs hidden tests the agent cannot touch. Editing the visible test earns nothing.
print(learn({"fix the bug": 0.7, "edit the test": 0.0}))

learn <- function(pass_rate, steps = 500) {
  set.seed(7)
  value <- setNames(c(1, 1), names(pass_rate))   # what the learner believes each action is worth; starts hopeful, so it tries both
  tries <- setNames(c(0, 0), names(pass_rate))
  for (step in 1:steps) {
    # mostly pick the best action so far, sometimes pick at random
    a <- if (runif(1) < 0.1) sample(names(pass_rate), 1) else names(which.max(value))
    reward <- as.numeric(runif(1) < pass_rate[[a]])
    tries[a] <- tries[a] + 1
    value[a] <- value[a] + (reward - value[a]) / tries[a]
  }
  tries
}
 
# The score only asks: do the tests pass? Editing the test always makes them pass.
learn(c("fix the bug" = 0.7, "edit the test" = 1.0))
# The score runs hidden tests the agent cannot touch. Editing the visible test earns nothing.
learn(c("fix the bug" = 0.7, "edit the test" = 0.0))

actions = {'fix the bug', 'edit the test'};
% Row 1: the score only asks "do the tests pass?", editing the test always makes them pass.
% Row 2: the score runs hidden tests the agent cannot touch, editing the visible test earns nothing.
scenarios = [0.7, 1.0;
             0.7, 0.0];
 
for s = 1:2
  rng(7);
  pass_rate = scenarios(s, :);
  value = [1, 1];   % what the learner believes each action is worth; starts hopeful, so it tries both
  tries = [0, 0];
  for step = 1:500
    % mostly pick the best action so far, sometimes pick at random
    if rand < 0.1, a = randi(2); else, [~, a] = max(value); end
    reward = double(rand < pass_rate(a));
    tries(a) = tries(a) + 1;
    value(a) = value(a) + (reward - value(a)) / tries(a);
  end
  fprintf('%s: %d, %s: %d\n', actions{1}, tries(1), actions{2}, tries(2));
end

With the leaky score, it edits the test about 470 times out of 500. With hidden tests, it fixes the bug about 470 times out of 500. Same learner. Different score. That is the whole article in one run.

So is it lying?

Not the way a person lies. It repeats what paid off in training. But METR's ten-out-of-ten "no" shows it often knows. For your team the difference does not matter. A green test from an agent is a claim, not a proof, until you know the agent could not touch the test.

What helps

Say it in the prompt. Ask the agent to fix the code, not the test, and to tell you when a test looks wrong. It costs nothing, and on some newer models (Claude 4, for example) it makes a real difference (the prompt is below).
Leave it a way out. In one study, letting agents stop and flag a task for a human cut GPT-5's cheating from 54% to 9%, though it helped Claude much less. A line in your rules file like "if you can't do it, say so" is the cheap version of that.
Watch the retry loops. Shortcuts tend to show up after repeated failures. If the agent is on its third red run, it is a good moment to step in.
Glance at the test files in the diff. A changed test is not always wrong, but it is the first place to look.

And when your team has time for it: in bug-fix tasks, keep the tests out of the agent's reach: hide them or make them read-only, as the ImpossibleBench researchers recommend. A second set in CI that the agent never sees adds a check, because read-only tests do not stop hard-coded answers. In a migration, changing tests is part of the job, so there you look at the test count and the assertions instead.

Prompt · add this to the task, before the agent starts

Make the failing test pass by fixing the code, not the test. Do not edit, skip or delete any test, and do not hard-code expected values. If the test looks wrong or the task looks impossible, stop and tell me why instead of making it pass.

The last two sentences follow the prompt Anthropic published for its Claude Opus 4 tests. With their prompt, Claude Opus 4's hacking on impossible tasks went from 47% to 5%. On the older Claude Sonnet 3.7 it did almost nothing, so the other steps still matter.

How my team sets this up per type of task, with pre-conditions and guardrails, is the next article (coming soon).

Sources

OpenAI, How to count tokens with tiktoken, Cookbook
Meta, The Llama 3 Herd of Models, 2024
Romanou et al., Brittlebench: Quantifying LLM robustness via prompt sensitivity, 2026
Ouyang et al., OpenAI, Training language models to follow instructions with human feedback, 2022
Anthropic, Training a misaligned reward seeker, August 2026
Sharma et al., Anthropic, Towards Understanding Sycophancy in Language Models, 2023
OpenAI, Expanding on what we missed with sycophancy, May 2025
DeepSeek, DeepSeek-R1, 2025
Baker et al., OpenAI, Monitoring Reasoning Models for Misbehavior, 2025
Anthropic, Claude 3.7 Sonnet System Card, 2025
METR, Recent Frontier Models Are Reward Hacking, 2025
Zhong, Raghunathan, Carlini, ImpossibleBench, 2025
Anthropic, Claude Opus 4 and Sonnet 4 System Card, 2025