Coding Agents

If you want to know where agentic AI works today, follow the money: coding agents are the most commercially successful category, and it is not an accident. Software has a property almost no other domain offers — a cheap, automatic, ground-truth verifier. Tests pass or fail. Code compiles or it doesn't. A linter is either clean or it isn't. The agent never has to guess whether it succeeded; it runs the checks and reads the result.

Four properties stack up to make programming agent-friendly:

1. Verifiable outcomes — tests, compilation, and linting give an objective success signal.
2. Sandboxable, reversible environments — code runs in a container, and git makes every change undoable.
3. Clean programmatic interfaces — shells, editors, debuggers, and package managers are built to be driven by text commands.
4. A well-defined problem space — "make the failing test pass" is a crisp goal in a way "write a good marketing plan" never is.

Contrast this with a research or support agent, which often has no runnable check on whether its answer is correct. That missing verifier is exactly why coding raced ahead while other categories are still maturing.

Strip a coding agent to its core and you find one loop running over and over:

read relevant files  →  plan a change  →  edit the code
        ↑                                      ↓
   read output  ←  run build / tests / linter ─┘   (repeat until checks pass
                                                    or an iteration cap is hit)

The magic is in the last arrow: the agent runs something and reads the result, then decides its next move. A test failure isn't a dead end — it's an observation that drives the next edit. This is the ReAct loop you met earlier, specialized for code, and it typically runs for a bounded number of turns (often around 10) before stopping.

The load-bearing requirement is a runnable verification signal. Give the agent a failing test and a command to run it, and it iterates toward green with real feedback. Take that away — a codebase with no tests — and the loop has no closure condition. The agent edits until the code looks done, not until it is done, and confidently hands you something that may not work.

A real codebase is far too big to fit in the model's context window, so the agent works the way a new engineer does — it can't memorize the whole repo, it needs a map and a few cheat-sheets. Two techniques provide exactly that.

Repo maps. Tools like Aider build a compact map of the repository — files, key symbols, and signatures — so the model knows what exists and where without reading every file. Think of it as a table of contents rather than the full book. The agent then pulls full file contents only when it needs them, conserving context.

Context files. AGENTS.md (the emerging cross-tool standard) and Claude Code's CLAUDE.md are repository-level files an agent loads at session start. They hold build/test commands, code-style rules, architectural decisions, and tooling quirks — the tacit knowledge a new teammate would otherwise ask about.

But don't overstate their value. A 2026 study (arXiv 2602.11988) found that developer-written context files improved task success by only ~4% while raising inference cost up to 19%, and LLM-generated context files actually hurt performance by ~3%. Use them for concrete, hard-to-guess facts — "run tests with pytest -q", "never touch legacy/" — not as a dumping ground.

How do you measure whether a coding agent is actually any good? You give it real bugs and check whether its fix works. That is exactly what SWE-bench does, and it's the benchmark that defined the field. Each task is a real GitHub issue; the agent must edit a real codebase so the project's hidden tests pass — no partial credit for code that merely looks plausible. It comes in two tiers: the full set (2,294 issues) and SWE-bench Verified (500 human-validated issues, filtered so every task is genuinely solvable). Verified scores run higher and are the headline number you'll see quoted.

The trajectory is the story. Scores went from effectively 0% in 2023, to ~13% in early 2024, to roughly ~79% on Verified by early 2026 and continuing higher through mid-2026 as top agents crossed the 85–90% range. That is one of the steepest capability climbs in modern AI.

Two caveats keep you honest:

• The harness matters. A harness (or scaffold) is the agent loop wrapped around the model — the code that decides how it reads files, runs tests, and retries. A score belongs to a model plus its harness, not the model alone, so the same model can score very differently under a different loop. Always cite the harness with the number.
• It's a trajectory, not a ceiling. Numbers that were state-of-the-art eighteen months ago are now mid-pack. Treat any specific figure as a snapshot.

Complementary benchmarks like METR's Time Horizon measure how long an agent can work on one coherent task autonomously — a frontier metric now measured in hours, not minutes.

These tools all share the same edit-run-test engine; what differs is where they live and how much they do on their own — a terminal command you steer turn by turn, an IDE that suggests as you type, or a cloud worker you hand a ticket and walk away from. Five systems dominate as of 2026, each occupying a different niche:

A few clarifications that trip people up. The 2025 OpenAI Codex is not the 2021 Codex — the old one was a code-completion model behind early GitHub Copilot; the new one is a full autonomous agent. Aider is a harness, not a model — its quality depends entirely on the LLM you point it at (GPT-5 with high reasoning hits 88% on Aider's polyglot benchmark). And Cursor is a full fork, not "VS Code with a plugin" — proprietary models and cloud VMs are what make its agent mode work. GitHub Copilot, the dominant IDE completion tool, has grown agent modes too but remains primarily an in-editor assistant.

Getting value from these tools is a skill. A handful of habits separate frustration from a force multiplier.

• Give it a verifier. Point the agent at a test command or build before you ask for the change. The loop needs a signal.
• Manage context deliberately. A debugging session can burn tens of thousands of tokens, and a full window degrades the model. Use /clear between unrelated tasks, lean on compaction/summarization, and push investigation work into subagents so it doesn't pollute the main thread.
• Plan before editing. Plan mode (explore and propose first) catches misunderstandings before any code changes.
• Scope tightly. Agents shine on clearly bounded tasks with verifiable outcomes — the kind a junior engineer would finish in a few hours.
• Review every diff. You are the senior reviewer. Read the change, run it, and own it.
• Keep humans on risky actions. Use permission gates for destructive or irreversible operations.

# A tight, reviewable Claude Code task in non-interactive mode
claude -p "Fix the failing test in tests/test_auth.py. \
Run: pytest tests/test_auth.py -q. Iterate until it passes. \
Do not modify files outside src/auth/."

The progress is real, and so are the limits. Coding agents still fail predictably in a few places.

• Ambiguous specs. Without a precise goal and a verifier, the agent fills gaps with assumptions — often wrong ones.
• Complex inter-file reasoning. Changes that ripple across many modules, or depend on subtle cross-cutting invariants, exceed what current agents reliably track.
• Tacit knowledge. "We don't do it that way here" — undocumented conventions, business context, and team history — lives in people's heads, not the repo.
• Mid-task scope changes. Autonomous agents like Devin do best on stable, scoped work; shifting requirements throw them off.

The honest framing: Devin's PR merge rate reached 67% in 2025 (up from 34% the year before) — impressive, and still a third of PRs needing rework. Agents are not general-purpose software engineers. They are powerful executors of well-specified work. Architecture, judgment on edge cases, and oversight remain human jobs — which is precisely why your skill at scoping, verifying, and reviewing is what determines whether these tools help or hurt.