Deploying & Scaling Agents

A normal web service follows a tidy contract: a request comes in, you compute, you return a response in milliseconds, and you forget everything. Agents violate every clause of that contract.

An agent run is long (a research or coding task can take minutes to hours), stateful (it carries a growing message history and plan across steps), non-deterministic (the same input can take a different path), and bursty (a single task may fire dozens of sequential LLM and tool calls). The standard HTTP request lifecycle assumes the opposite of all four.

The practical consequence: you cannot just wrap your agent in a Flask route and call it done. The request will time out, a crash will lose all progress, and concurrent users will collide with provider rate limits you never hit in development.

Deploying an agent is therefore less about "hosting a model" and more about building a small distributed system around it — one that can run work in the background, remember where it was, recover from failure, and stay inside provider limits. The rest of this lesson builds that system piece by piece.

The first decision is the most consequential: does the caller wait, or not?

Synchronous (request-response) fits short, stateless tasks that finish inside an HTTP timeout — document classification, a one-shot Q&A, a single tool call. The client sends a request and blocks for the answer. Simple, and perfectly fine when the work takes a few seconds.

Async / background is required the moment a run can outlast a request. HTTP timeouts are typically 30–300 seconds; an agent run that takes 20 minutes cannot live inside one. The pattern:

1. The web service enqueues a job and returns immediately with a job_id.
2. A worker pulls the job and executes the agent run in the background.
3. The client polls GET /jobs/{id} or receives a push (webhook, Slack message, email) on completion.

The cardinal rule: never design a synchronous UI for a 20-minute agent task. A spinner that blocks for twenty minutes is a guaranteed timeout and a terrible experience. Hand back a job ID, show progress, and notify on completion.

Async execution is implemented with a queue (a durable list of pending jobs) and workers (processes that pull and run them). Common stacks: Celery + Redis or RabbitMQ in Python, or AWS SQS feeding container workers. The queue decouples accepting work from doing work, which lets you scale the two independently and absorb bursts.

A minimal Celery setup looks like this:

# tasks.py
from celery import Celery

app = Celery("agents", broker="redis://localhost:6379/0",
             backend="redis://localhost:6379/1")

# acks_late: if a worker crashes mid-run, the job is requeued,
# not silently lost.
@app.task(bind=True, acks_late=True, max_retries=3)
def run_agent(self, job_id: str, prompt: str):
    try:
        result = agent.run(prompt)          # the long-running work
        save_result(job_id, result)
    except RateLimitError as exc:
        # exponential backoff, then retry
        raise self.retry(exc=exc, countdown=2 ** self.request.retries)

# api.py — enqueue and return immediately
@router.post("/agent")
def start(req: Req):
    job_id = new_id()
    run_agent.delay(job_id, req.prompt)     # non-blocking
    return {"job_id": job_id, "status": "queued"}

Set acks_late=True so a job is acknowledged only after it completes — if a worker dies, the broker requeues it. Route heavy and light tasks to separate queues so a flood of slow jobs can't starve fast ones.

Agents are stateful, and that state must live outside the worker process — otherwise a restart erases the run. There are five patterns you'll combine in practice:

1. Windowed memory — keep the last N messages in Redis.
2. Summarized memory — periodically compress history with an LLM call.
3. Semantic / RAG memory — retrieve relevant facts from a vector DB.
4. Checkpoint / serialization — snapshot the full agent state to durable storage (used by LangGraph and AutoGen).
5. Hybrid storage — match the medium to the data type.

For production checkpointing with LangGraph, the rule is simple: use MemorySaver for development, and PostgresSaver in production. Skip SqliteSaver — it buckles under write contention at real concurrency.

from langgraph.checkpoint.postgres import PostgresSaver

checkpointer = PostgresSaver.from_conn_string(DB_URL)
checkpointer.setup()
graph = builder.compile(checkpointer=checkpointer)

# thread_id ties a run to its persisted state; resume by reusing it.
cfg = {"configurable": {"thread_id": job_id}}
graph.invoke({"messages": [user_msg]}, cfg)

On AWS, DynamoDBSaver stores small checkpoints (<350KB) directly and offloads large ones to S3 with a reference pointer. Expect checkpointing to add roughly 10–20% overhead versus a stateless chain — cheap insurance for resumability.

This is the failure mode that catches teams by surprise. Unmanaged rate limiting is the number-one cause of agent failures in production.

Providers enforce multiple simultaneous limits: requests per minute (RPM), tokens per minute (TPM), tokens per day (TPD), and concurrent requests — at the organization and project level. A single agent task can fire dozens of sequential calls, so it looks like a small DDoS. Scaling workers makes this worse: more workers means more requests hitting the same shared quota at once.

Two principles fix this:

• Check quota before you start, not during. Before kicking off a 30-step workflow, confirm you have headroom. Failing on step 27 wastes everything before it.
• Put an AI gateway in front of every provider call. Tools like Portkey, TrueFoundry, or AgentGateway enforce token/request budgets uniformly across all traffic, attribute cost per feature, cache, and fail over to a backup model when a provider throttles.

# A gateway centralizes limits, retries, and failover —
# instead of scattering ad-hoc backoff across your codebase.
client = Portkey(
    api_key=GATEWAY_KEY,
    config={"strategy": {"mode": "fallback"},
            "targets": [{"provider": "anthropic"},
                        {"provider": "openai"}]},
)

Don't try to scale your way out of a quota problem — adding workers without managing quota just produces synchronized rate-limit errors.

Retries handle a single flaky call. But what happens when a worker crashes 18 steps into a 25-step run? Without help, you restart from zero — re-paying for and re-doing all the prior work, and possibly repeating side effects.

Durable execution solves this. Frameworks like Temporal and Restate checkpoint every step of a workflow automatically, so if a worker dies, another resumes exactly where it left off. They also pause-and-resume on rate limits and retry transient failures transparently.

• Temporal uses a pull model: dedicated workers pull tasks from a queue. Its OpenAI Agents SDK integration reached GA on March 23, 2026 and wraps agent calls as Activities automatically — you don't hand-write activity code. Workflows must be deterministic; the non-deterministic LLM and tool calls live in Activities.
• Restate uses a push model: the server pushes work to serverless functions, so you get durable agents on Vercel, Cloudflare Workers, Lambda, or Modal with no dedicated worker fleet. It records each step in a durable journal.

Choose Temporal when you already run worker infrastructure and want a battle-tested platform; choose Restate when you want durability on plain serverless without managing workers.

Where you run the agent — the compute target — is the box your code executes inside: a serverless function, a container, or a lightweight virtual machine. The choice is constrained by one brutal fact: each box has a maximum time it will let a single run stay alive — its execution time limit. If your agent runs longer, the platform kills it mid-task.

The misconception to kill: serverless is fine for long-running agents. It isn't — most non-trivial agents exceed Lambda's 15 minutes and hit Azure Consumption's 10-minute ceiling. Use containers or micro-VMs for anything long. Micro-VMs (Fly.io, Modal, Kata) add per-workload kernel isolation and sub-25ms snapshot resume — the right choice when running untrusted or LLM-generated code, since containers share the host kernel and have had critical escape CVEs.

On cost: API prices dropped 60–80% over 2024–2026, yet total token consumption grew 13x — because agents multiply tokens geometrically. Control it with model routing (cheap model for simple steps, premium for hard ones), which cuts spend 40–60% with no quality loss, plus per-feature cost attribution and real-time usage dashboards so an overrun is visible before the invoice.