Proof Points and Cautionary Tales

Every AI vendor, consultancy, and press release offers a number that sounds like proof: handled two-thirds of chats, 55% faster, $40M uplift. Numbers like these are how budgets get approved and how flagship programs get launched — and they are also how programs quietly fail, because the headline almost never tells you what the failing fraction looked like or whether the win held up over time.

The single most valuable habit a leader can build in this module is to read a case study the way an auditor reads a forecast: What exactly was measured? Over what window? Who is the source — and what do they have to gain? And crucially, what happened next? Most published cases are a snapshot near launch, when the metrics look best. The interesting story is usually the second chapter.

This lesson works through two cases that have a documented second chapter. Klarna is the canonical both-sides story — a genuine efficiency win followed by a quality-driven course correction. JPMorgan is the contrast: a scoped, KPI-anchored portfolio that grows because every use case is tied to a number, not a narrative. Read together, they teach the discipline the headlines can't.

In February 2024, the fintech Klarna announced one of the most-cited AI deployments in the enterprise. Its AI customer-service assistant, built on a large language model, took on the front line of support — and the launch numbers were genuinely impressive.

On the surface this is the textbook quick win, and it lines up with the strongest independent evidence we have. A landmark study by Brynjolfsson, Li and Raymond of 5,179 real support agents (NBER, 2023) found a gen-AI assistant raised issues resolved per hour by ~14% on average and ~34% for novice agents — exactly the kind of lift customer support is built to deliver: repeatable queries, a structured knowledge base, and a clear success metric. Customer support is, on the evidence, the best-proven place to start.

The trap is to stop reading here. Klarna's own figures are vendor-sourced — a press release, not an audited result — and every metric in that table is a volume or speed metric. None of them measures whether customers were actually well served.

By mid-2025, the story had a second chapter the original headlines never anticipated. After leaning hard into automation and reducing its human workforce, Klarna found that customer satisfaction declined — and the company began rehiring humans to handle the work the AI did poorly. CEO Sebastian Siemiatkowski put it plainly:

"We focused too much on efficiency and cost… it led to lower quality… not sustainable."

This is why Klarna is the canonical both-sides case. Nothing in chapter one was false — the AI really did handle enormous volume, fast and cheaply. The failure was that the metrics chosen to declare victory (volume, speed, cost) were inversely informative about the thing that actually mattered (quality, trust, the nuanced and empathetic cases). The volume metrics looked spectacular precisely while the quality the business depended on was eroding underneath them.

The durable lesson is not "AI support doesn't work." It plainly does, for a large share of cases. The lesson is about what you measure and how far you push replacement:

• Volume metrics can mask a quality collapse. A system that resolves more tickets faster can be simultaneously making customers less happy. Track CSAT/NPS alongside throughput, and baseline both before you start.
• Augment-and-blend beats wholesale replacement for nuanced work. The empathetic, ambiguous, high-stakes conversations are exactly where the jagged frontier bites. The winning pattern is AI for the repeatable bulk, humans kept in the loop for the cases that need judgment — not a headcount-elimination target.

If Klarna shows the danger of one big bet measured on volume, JPMorgan shows the opposite discipline. The bank has put 450+ AI use cases into production, with reported benefits growing on the order of 30–40% year over year — and the structural lesson is not the count but the shape of the portfolio.

Each use case is scoped and anchored to a specific KPI rather than launched as a flagship transformation hoping to move everything at once. This is the same conclusion the broader evidence keeps reaching: McKinsey's work on capturing agentic value argues that winners concentrate on a focused set of high-value use cases, embed them in redesigned workflows, and scale with governance — rather than running scores of disconnected pilots or one headline-grabbing replacement.

The contrast is the point. Klarna's chapter one is not a story of incompetence — it's a story of measuring the wrong thing and pushing replacement too far. JPMorgan's portfolio compounds precisely because each piece is small enough to measure honestly and is tied to a number a leader can defend.

You will be handed AI case studies constantly — by vendors selling a platform, by consultants selling a transformation, by your own teams seeking budget. Almost all of them are real and incomplete. The skill is not cynicism; it's a short, repeatable interrogation that separates a durable result from a launch-day snapshot.

The both-sides checklist — five questions for any AI case study:

1. Source and incentive. Who published this, and what do they gain if you believe it? A vendor press release and a peer-reviewed field study are not the same evidence.
2. What was measured. Is the headline a volume/speed/cost metric or a quality/outcome metric? If quality isn't in the story, assume it wasn't tracked.
3. The failing fraction. If it's "90% accurate" or "two-thirds handled," what do the other 10% / one-third look like, and who is accountable for them?
4. The time window. Is this a launch snapshot or a sustained result? Ask what the numbers looked like six to twelve months later — the second chapter.
5. Transferability. Does this case share your data quality, your workflow, your regulatory exposure? A win in one context is a hypothesis, not a guarantee, in yours.

Applied to our two cases: Klarna passes question 1 only weakly (vendor-sourced), fails questions 2 and 4 at launch (volume-only, snapshot) — and that's exactly where the reversal came from. JPMorgan's portfolio is built to pass question 2 (a KPI per use case) and question 4 (benefits tracked year over year). The checklist would have predicted both outcomes.