Chain-of-Thought & Reasoning Models

Imagine being forced to shout the answer to a tricky word problem the instant you hear it — no pen, no paper, no muttering it through. You would get a lot of them wrong, not because you don't know the math, but because you had nowhere to work it out. A language model is in exactly that position. It generates one token at a time, left to right, with no scratchpad. If you demand the final answer immediately, all of its "reasoning" has to happen invisibly inside a single forward pass.

Chain-of-thought (CoT) prompting hands the model a scratchpad. Instead of jumping to the answer, you invite it to write the intermediate steps as text first. Here is the key mechanism: those generated steps become part of the context the model reads as it keeps going, so each later token is conditioned on the reasoning it just wrote. The model literally uses its own output as working memory.

The technique was introduced by Wei et al. (2022) at Google Brain. Their key result: prompting a 540B-parameter PaLM model with just 8 worked examples showing step-by-step solutions set a new state of the art on GSM8K grade-school math — far beyond answering directly. The reasoning was not new knowledge; it was a better path to knowledge the model already had.

There are two ways to ask for a chain of thought, and they differ only in how much you show the model.

• Few-shot CoT (Wei et al., 2022): you include a handful of exemplars that show the reasoning — a question, then a worked-out solution, then the answer. The model imitates that format on your new question.
• Zero-shot CoT (Kojima et al., 2022): you add no examples at all — just the instruction "Let's think step by step." Astonishingly, that single phrase alone unlocks much of the same benefit, because it cues the model toward the reasoning style it saw constantly in pretraining.

But there is a crucial catch: CoT is an emergent ability. It reliably helps only in large models (roughly 100B+ parameters). In small models, prompting for steps tends to produce fluent but illogical chains that can make accuracy worse than answering directly. The model writes confident, well-formed sentences that do not actually compute anything correct — reasoning-shaped text without the reasoning underneath.

So far you, the prompter, have been doing the work — coaxing steps out of a model with the right phrasing. The biggest shift since 2024 is that the best reasoning is no longer prompted at all; it is trained in. Reasoning models are trained with large-scale reinforcement learning to produce extended internal reasoning before they answer. They do not merely follow an instructed template; through trial and reward, they discover strategies — decomposing problems, checking their own work, backtracking from dead ends — the way a student learns to reason by doing thousands of problems, not by being handed a script.

This is fundamentally different from CoT prompting, and the distinction matters in practice. The major families as of 2026:

• OpenAI o-series (o1, o3, o4-mini, o3-pro). o3 and o4-mini shipped April 16, 2025; o3 set a new state of the art on ARC-AGI-1, scoring in the 41–53% range on the production semi-private eval (varying by effort level), with a high-compute preview configuration reaching 87.5%. The chain of thought is hidden from users. Exposes a reasoning_effort control (low/medium/high).
• DeepSeek-R1 (Jan 20, 2025). Open-source (MIT), a 671B MoE plus distilled variants, trained with GRPO reinforcement learning. Matches o1 on math (79.8% AIME 2024) and exposes its reasoning traces.
• Claude extended thinking (Anthropic, from Claude 3.7 Sonnet, Feb 24, 2025). Emits structured thinking blocks; a budget_tokens parameter caps thinking (up to 128K). Newer Claude 4 models deprecate budget_tokens in favor of an effort parameter with adaptive thinking that decides when to think hard.
• Gemini 2.5 (Google, March 2025). Flagship thinking model; ~92% AIME 2024, 63.8% SWE-bench Verified, 1M-token context, thinking fused with tool use.

Here is the simple idea underneath reasoning models: you can make a model smarter on a hard question just by letting it think longer — after training, at the moment you ask. That is test-time compute scaling. Instead of spending all the compute during training and then answering in a fixed-size burst, these models spend more compute at inference — generating longer reasoning chains, verifying intermediate results, and backtracking. More thinking budget generally buys more accuracy, up to a point of diminishing returns.

The useful analogy is Kahneman's two systems:

• System 1 — fast, automatic pattern-matching. A standard model answering directly.
• System 2 — slow, deliberate, effortful reasoning. A reasoning model working through a long chain.

For a trivia question, System 1 is fine and System 2 is wasteful. For a competition math problem or a tricky multi-file code change, deliberate System 2 thinking is where the accuracy gains live. This is why you now tune the budget: OpenAI's reasoning_effort, Anthropic's budget_tokens, and Gemini's thinking controls all let you dial how much the model deliberates per request — trading latency and cost for correctness on demand.

If one student's reasoning can go astray, ask ten students and trust the answer most of them agree on. That is the whole intuition behind self-consistency (Wang et al., 2022), the strongest purely prompt-based enhancement to CoT. Instead of trusting one reasoning chain, you sample several with temperature > 0, then take the majority vote over their final answers. Different chains may reach the same correct answer by different routes, while wrong answers tend to scatter — so the mode is usually right.

from collections import Counter

def self_consistent_answer(model, prompt, n=10, temperature=0.7):
    answers = []
    for _ in range(n):
        chain = model.generate(prompt + "\nLet's think step by step.",
                                temperature=temperature)
        answers.append(extract_final_answer(chain))
    # majority vote
    return Counter(answers).most_common(1)[0][0]

It reliably improves accuracy, but gains plateau after roughly 20-40 samples — beyond that, sampled chains start overlapping and you pay N times the cost for little lift. Self-consistency works with any CoT-capable model and even stacks on top of reasoning models, but it multiplies token cost, so reserve it for high-value, hard problems where a few extra points of accuracy justify the spend.

Thinking longer is genuinely useful, but it is never free — and the bill comes due in two currencies: time and tokens. Before you reach for a reasoning model, weigh the concrete trade-offs:

• Latency. Reasoning chains take seconds to tens of seconds, versus milliseconds for a direct answer.
• Cost. Thinking tokens are billed (as output tokens on most APIs), so a long internal chain can dominate the bill even when the visible answer is short.
• Overkill risk. For factual lookups, classification, short creative writing, and most retrieval tasks, a reasoning model adds cost and delay with no quality gain.

Reasoning models earn their keep on complex multi-step math, code, scientific reasoning, and planning — tasks where the steps aren't known up front and a wrong intermediate step quietly poisons the answer.

And here is the 2026 reality check: a Wharton Generative AI Labs study (2025) found that adding explicit CoT prompting to reasoning models like o3-mini and o4-mini yielded only 2.9-3.1% accuracy gains while increasing latency 20-80%. For non-reasoning models, CoT prompting helped more (4.4-13.5%), but some models regressed (e.g., a 17.2% drop on certain tasks). The lesson: pick a reasoning model when the task is genuinely hard; do not reflexively bolt "think step by step" onto a model that already thinks.