Reflection & Self-Correction

Humans rarely ship the first draft. We write, reread, wince, and revise. Reflection gives an agent the same move: after producing an output, it generates a critique of that output, then revises — optionally looping several times. A useful mental model is a small inner committee: a generator that does the work, an evaluator that judges it, and a step that feeds the judgment back in.

Concretely, the simplest reflection loop is three prompts:

1. Generate — produce a first attempt.
2. Critique — ask: "What is wrong with this? Be specific."
3. Revise — produce a new attempt that fixes the named problems.

This is appealing because it costs nothing but inference — no retraining, no extra data — and it sometimes turns a mediocre answer into a good one. But the appeal hides a trap that the rest of this lesson is about: a model critiquing itself is only as good as its ability to see its own mistakes. If the error sits in the model's blind spot, the critique will be confident, fluent, and wrong. Reflection is a powerful tool, but it is not free reliability.

Here is the core question: how does an agent learn from a failure without retraining? Reflexion (Shinn et al., NeurIPS 2023) gives the canonical answer — instead of updating weights, store the lesson in words and feed it back on the next attempt. Think of it as keeping a diary of mistakes and rereading it before you try again. Three components make it work:

• Actor — generates actions or text (often a ReAct agent).
• Evaluator — scores the trajectory, as a scalar or free-form feedback.
• Self-Reflection memory — an episodic buffer of verbal critiques the agent reads before retrying.

After a failed attempt, the agent writes a short reflection — "I assumed the file was JSON; it was CSV, so parsing failed. Next time, check the format first" — and carries it into the next episode's context. This is verbal reinforcement learning: learning across attempts with zero gradient updates.

The published gains are real on structured tasks: 91% pass@1 on HumanEval (vs. GPT-4's 80%), 130/134 AlfWorld tasks solved, +20% on HotpotQA. The catch — central to everything below — is that Reflexion requires a reliable evaluator. With a clear pass/fail signal it shines; without one, the reflection step can produce plausible but misleading critiques that steer the agent wrong.

Here is the single most important distinction in this lesson. The plain-language version: does the critique come from the model guessing whether it was wrong, or from the world telling it so?

• Intrinsic self-correction — the model revises using only its own judgment, no external signal. The same network that made the error is now grading the error.
• Grounded (extrinsic) self-correction — the critique is anchored to something the model cannot rationalize away: a unit test that fails, code that throws, a tool that returns an error, a separately trained verifier.

The research verdict (TACL 2024, replicated across multiple 2025 papers) is blunt: pure intrinsic self-correction does not reliably improve reasoning, and can flip correct answers to wrong ones. The model shares the same blind spots as its own evaluator, so for errors it couldn't catch the first time, a second look just produces confident, wrong feedback.

Grounded feedback is a different animal. When a code agent runs the test suite and sees AssertionError: expected 5, got 4, that is external ground truth. The model can't argue with it. RLEF (ICML 2025) builds on exactly this, using execution errors and unit-test results as reward signals for multi-turn code generation. The design rule follows directly: don't trust the model to grade itself on reasoning; give it a real grader.

Sometimes there is no test to run — a logic puzzle, an essay, a plan. The next best thing is a verifier model: a separate model trained to judge correctness, acting as the grader the task itself can't provide. Two flavors matter:

• Outcome Reward Model (ORM) — scores only the final answer. Sparse signal: right or wrong, with no idea where things went off.
• Process Reward Model (PRM) — scores each intermediate step. Dense, step-level feedback that catches an error the moment it appears, like a teacher marking each line of working rather than just the final number.

PRMs cost more compute (you evaluate every step) but pay off: in reasoning benchmarks they run >8% more accurate than ORMs and can be 1.5–5x more compute-efficient when used as process verifiers to guide search. OpenAI's Let's Verify Step by Step (2023) introduced PRMs for math; by 2025 they're applied to SQL synthesis, tool use, logical deduction, and software engineering — anywhere steps are identifiable.

The deeper point is decorrelation of error modes. A verifier helps most when its mistakes don't overlap with the generator's. Using the same model to generate and verify gives you correlated blind spots and "rubber-stamping," where the critic just validates the generator. Different models, different temperatures, or an execution environment break that correlation — which is the whole game.

Reflection isn't free — done carelessly it actively makes answers worse. There are three signature failure modes; know them or you'll ship them.

1. Self-consistency / coherence trap. Across rounds the model builds a plausible but wrong justification, getting more confident as it elaborates. It's not converging on truth; it's converging on a story.
2. Sycophantic reflection. Asked to critique its own work, the model agrees with itself — "On reflection, my answer looks correct" — instead of genuinely attacking it.
3. Quality degradation (over-reflection). Too many iterations introduce new errors or hedge a correct answer into mush. More loops is not more quality.

There's a striking diagnosis here. The Self-Correction Blind Spot study (2025) tested 14 open-source non-reasoning models and found a 64.5% average blind-spot rate: models fix an error when it's shown to them externally but miss the identical error in their own output. The fix was almost comically simple — prepending the word Wait to the generation cut blind spots by 89.3%. The capability to self-correct exists; it just needs activating. One likely reason: supervised fine-tuned (SFT) models are trained mostly on error-free demonstrations, so correcting a mistake mid-stream is out of distribution. RL-trained reasoning models (DeepSeek-R1, the o-series) learn correction through outcome feedback and self-correct far more reliably.

Now put the principles together into something you'd actually ship: separate the critic, prefer grounded signals, and bound the retries (2–3 attempts is standard; then escalate to a human). Here's a code-revision loop grounded in test execution — the most reliable form of reflection.

def reflect_and_fix(task, model, run_tests, max_iters=3):
    code = model.generate(f"Write code for:\n{task}")
    for attempt in range(max_iters):
        result = run_tests(code)          # GROUND TRUTH, not self-opinion
        if result.passed:
            return code, attempt
        # The critique is anchored to a real failure signal
        critique = model.generate(
            f"This code failed:\n{code}\n\n"
            f"Test output:\n{result.error}\n\n"
            "Explain the specific bug, then rewrite the code."
        )
        code = extract_code(critique)
    return code, max_iters            # bounded: escalate to a human now

The load-bearing line is run_tests(code): the model is reacting to a real failure, not its own guess. Language Agent Tree Search (LATS, ICML 2024) generalizes this — it combines Monte Carlo Tree Search with Reflexion-style critiques, so failed branches become verbal lessons for the rest of the search (94.4% pass@1 on HumanEval with GPT-4; EM score of 0.61 on HotpotQA, substantially above ReAct's baseline). For high-stakes tasks with no executable check, decorrelate errors instead: use multi-agent debate or a PRM as the external grader.