Capabilities & Limitations

Start with the one idea that makes the rest of this lesson click: an LLM is a fancy autocomplete, not a fact-checker. It was trained to guess the next word that sounds right given everything before it. Nothing in that training process rewards being correct or up to date — only being plausible. So when it's right, it's because plausible and true usually overlap; when it's wrong, it's because they sometimes don't.

Stated precisely: an LLM is a probabilistic text predictor that optimizes for plausibility, not correctness or recency. Everything it is great at and everything it is terrible at flows from that single fact. This is not a defect to apologize for — it is a trade. The same machinery that lets the model fluently continue any text in any style is what occasionally lets it continue confidently into a falsehood. Fluency and wrongness are not opposites; they come from the same place.

For an agent builder this reframes the whole problem. You are not waiting for a future model with "no weaknesses." You are designing a system around an engine whose strengths and weaknesses are both known and stable. The rest of this lesson catalogs both — and, crucially, shows that each major weakness has a direct, deployable remedy. That mapping (weakness → remedy) is the bridge from "LLM" to "agent," and it is the reason the rest of this course exists.

Before the caveats, take the strengths seriously — they are load-bearing. Think of the model as a brilliant generalist who has read almost everything: ask it to draft, rephrase, summarize, translate, or sketch code, and it is fast and fluent. It is the reasoning core of an agent because it is exceptional at a specific cluster of open-ended, language-shaped tasks:

• Natural-language synthesis and transformation — drafting, rewriting, tone-shifting, explaining. This is the home turf.
• Summarizing large corpora — collapsing fifty pages or a long transcript into the parts that matter.
• Flexible code generation — producing idiomatic code for common patterns, glue scripts, and boilerplate across many languages.
• Brainstorming and ideation — generating many candidate options fast.
• Cross-domain analogy and pattern completion — recognizing structure and applying it to a new situation it never saw verbatim.
• Translation and multilingual work — moving fluidly between languages and registers.

The through-line is flexibility: the model handles open-ended, fuzzy, linguistic tasks that resist hard-coding. That is exactly the capability traditional software lacks — you cannot write if/else rules for "make this email warmer." An agent borrows this flexibility to decide what to do next — and then leans on tools to do the parts the model is bad at.

A hallucination is output that is fluent and plausible but false — the model fills a gap with something that sounds right instead of admitting it doesn't know. It comes in two flavors: factuality errors (stating an incorrect fact about the world) and faithfulness errors (distorting a source you actually gave it). Both sound equally confident, which is exactly what makes them dangerous.

Why does it happen? Two layers. First, the model predicts likely tokens, so a plausible-but-invented citation or API name can be more probable than the honest words "I'm not sure." Second — the deeper cause — OpenAI's 2025 analysis Why Language Models Hallucinate shows that standard training and evaluation reward confident guessing and penalize abstention. Benchmarks score "I don't know" as a miss, just like a wrong answer, so post-training quietly teaches models to bluff rather than hedge.

The practical reality: rates vary enormously. Best-in-class models reach under 1% on grounded summarization tasks (down from ~22% in 2021), yet specialized domains without mitigation can exceed 50% — GPT-4o hallucinated in 53% of one medical test set before mitigation. Hallucination is not a bug awaiting a patch; it is structural. The goal is mitigation and transparency, not elimination.

Imagine a brilliant expert who fell into a coma on a fixed date and just woke up. They can still reason superbly, but their facts stop at that date — and they don't always realize it. That is every base model: it is frozen at a knowledge cutoff, the date its training data ends. Ask it about anything after that and it does one of two things: admit it doesn't know, or confidently report what used to be true. The second is staleness, and it is a distinct failure mode from hallucination.

• Hallucination = inventing a fact that never existed.
• Staleness = faithfully recalling a fact that was true and is now wrong (a price, a leader, an API signature, a record).

Both produce confident-sounding errors, but the cure is different. And here is the trap that catches teams: fine-tuning does not fix staleness. Fine-tuning adjusts style and domain familiarity; it does not inject this week's facts. A fine-tuned stale model is more dangerous, because it states outdated information with greater authority.

The only real fix for staleness is to fetch current information at query time — retrieval from a live source or a web-search tool. You cannot prompt-engineer your way to facts the model was never given.

Two more weaknesses fall straight out of "it predicts tokens."

Precise arithmetic. An LLM is not a calculator; it emits the most probable token sequence, not the correct one. 2 + 2 usually works because it appeared a million times in training — it's effectively memorized. But 12345 × 67890 is unreliable, because that exact answer was never a frequent pattern, so the model is pattern-matching toward a number that looks right. The accurate framing isn't "LLMs can't do math" — it's that they are unreliable for precise, novel arithmetic. The fix is not a bigger model; it is to call a calculator or code tool and let real arithmetic run.

Statelessness. LLMs have no memory between calls. Each request is processed completely fresh, as if meeting you for the first time. The "memory" you feel in a chatbot is an illusion: the entire conversation history is re-fed into the context window every single turn. That has hard limits — attention cost grows roughly with the square of context length, and a 2025 Chroma "context rot" study found all 18 frontier models tested losing accuracy continuously as context filled, with mid-window degradation exceeding 30 percentage points — a steady slope from token one, not a cliff at the limit. A bigger context window is not persistent memory; every session still starts blank unless you wire in an external memory store (e.g., a vector database).

Here's the uncomfortable part: the model sounds exactly as sure when it's right as when it's wrong. A model is well-calibrated when its expressed confidence tracks its real accuracy — when the things it says "I'm 90% sure" about turn out right about 90% of the time. LLMs are often poorly calibrated. RLHF (the human-feedback tuning that makes models helpful and confident) tends to reproduce human overconfidence, and miscalibration is worst exactly where it hurts most: at the edges of the model's knowledge and in low-resource domains.

The blunt operational lesson: a confident tone carries no information about correctness. The most capable models can be fluently, authoritatively wrong. A 2025 survey frames LLM uncertainty along four axes — input, reasoning, parameter, and prediction uncertainty — and a recurring finding is that capability benchmarks improve far faster than calibration does. Newer and smarter does not reliably mean better-calibrated.

For builders, calibration is why you don't let an agent's self-assessment be the final word. You add external checks: retrieval to ground claims, validators and schemas to catch malformed output, tests or execution to verify code, and human approval gates for high-stakes actions. Treat the model's confidence as a weak prior, never as proof.

Now the payoff. Each weakness in this lesson has a known, deployable fix — and when you line them up, the agent architecture writes itself. An agent isn't magic; it's an LLM with a prosthetic bolted onto each blind spot.

This is the thesis of the whole course in one table. An agent is what you get when you wrap the LLM's flexible reasoning with exactly the tools that cover its blind spots. The model decides what to do; tools and retrieval and memory do the parts it can't be trusted to do alone.

A caution to carry forward: remedies reduce, they don't erase. RAG cuts grounding errors but still propagates a stale or wrong source, and won't fix a reasoning slip. You are managing failure, not deleting it — and managing it well is the craft.