Responsible AI & Governance

A static model produces an output — text, a classification, a number — and a human decides what to do with it. An agent produces actions: it schedules the meeting, issues the refund, files the ticket, moves the money. That single shift is why governance stopped being a compliance checkbox and became an engineering concern.

Three things change when a system acts on its own:

• Liability moves from outputs to actions. When an agent commits a resource or executes a transaction with no human in the loop, courts and regulators increasingly ask who is accountable for what the agent did, not merely what it said.
• Harm scales without a pause. A biased recommendation a human can override is one thing; an agent that retrieves, decides, and acts in a loop can amplify that bias across thousands of actions before anyone notices.
• The audit trail is the only witness. Because no human watched each step, the logs are the accountability mechanism.

Governance, then, is not paperwork bolted on at the end. It is the set of design decisions — what the agent may touch, when a human must approve, what gets logged — that let you deploy autonomy responsibly and defend it later.

Bias in AI usually originates in training data that reflects historical prejudice. In a static model that bias shows up once, in the output. In an agent it can be compounded through the loop: the agent selectively retrieves biased data, acts on it, and the consequences of that action feed back into the next decision. A hiring agent that subtly down-ranks certain candidates doesn't just produce a skewed list — it schedules interviews, drafts rejections, and shapes the pipeline.

Transparency has two faces, and agents need both:

1. Disclosure — telling people they are interacting with AI. Under the EU AI Act this is a hard requirement for limited-risk systems: a chatbot must reveal it is not human, and AI-generated or manipulated content must be labeled.
2. Explainability — being able to reconstruct why the agent took a specific action. For multi-step agents this means logging the reasoning, the tools called, and the data retrieved at each step, not just the final result.

Fairness is testable. Evaluate the agent on representative slices of your population, measure outcome disparities across protected groups, and treat a widening gap across the loop as a defect — not an edge case.

Start with the one idea that explains the whole law: the EU AI Act does not regulate "AI" in the abstract — it regulates what you use AI to do. A spam filter and a system that decides who gets a loan use the same underlying technology, but only one can ruin someone's life, so only one carries heavy duties. That is what "risk-based" means.

The EU AI Act (Regulation (EU) 2024/1689) is the world's first comprehensive AI law. It is risk-based: obligations scale with how dangerous the use is, not the technology itself. The Act does not define "agent" as a category — agents are classified by their domain and autonomy under the existing tiers.

Timeline (current as of 2026). The Act entered into force 1 August 2024, but rules apply in phases: prohibitions and AI-literacy duties since February 2025; GPAI (general-purpose AI) model obligations since 2 August 2025. Crucially, the Digital Omnibus provisional agreement (May 2026) deferred high-risk deadlines from August 2026: standalone Annex III high-risk AI systems to 2 December 2027, and high-risk AI systems embedded in regulated products (e.g., medical devices, machinery) to 2 August 2028.

Provider or deployer? If you build a RAG pipeline or agent on top of Claude, GPT, or Gemini without modifying the model, you are almost always a deployer, with lighter obligations. Substantially fine-tuning or re-placing the model on the market can make you a provider.

If the EU AI Act tells you what you must do, NIST tells you how to think about it. It is the playbook a US government lab wrote to help any organization reason about AI risk in a structured way — no one forces you to use it, but it gives everyone the same words and the same checklist, which is half the battle.

Where the EU AI Act is law, the NIST AI Risk Management Framework (AI RMF 1.0) — published January 2023 as NIST AI 100-1 — is a voluntary, non-prescriptive framework. It is not a regulation, there is no version 2.0, and it does not replace sector rules like HIPAA or FCRA. Its value is a shared vocabulary and an operating model you can adopt incrementally.

The framework defines seven characteristics of trustworthy AI: valid and reliable; safe; secure and resilient; accountable and transparent; explainable and interpretable; privacy-enhanced; and fair with harmful bias managed. To pursue them it organizes work into four functions:

1. GOVERN — culture, roles, policies, accountability (the foundation under the other three).
2. MAP — establish context: what is this system, who does it affect, what could go wrong?
3. MEASURE — quantify and track risk with metrics and evaluation.
4. MANAGE — treat, prioritize, monitor, and escalate risks over time.

For agents, NIST published a Generative AI Profile (NIST AI 600-1, July 2024) defining 12 risk categories specific to or exacerbated by generative AI, including confabulation, data privacy, harmful bias, CBRN uplift, homogenization, and intellectual-property risks. The Cloud Security Alliance has extended the RMF with an Agentic AI Profile addressing agent autonomy, tool-use risk, and delegation-chain accountability — the issues this whole lesson is about.

NIST helps you think; ISO 42001 lets an outsider check your work. It is a formal standard you can be audited against and earn a certificate for — the same way ISO 27001 certifies that a company manages security properly. A certificate is a credible signal to customers and regulators: an independent body looked at how you run AI and found a real process, not good intentions.

ISO/IEC 42001:2023, published December 2023, is the world's first international standard for an AI Management System (AIMS) — the AI-specific sibling of ISO 27001 (security) and ISO 9001 (quality). It is voluntary but certifiable: accredited bodies such as BSI, DNV, and TÜV SÜD audit and certify organizations, and Microsoft has certified products including GitHub Copilot and Microsoft 365 Copilot.

Because it follows the same Annex SL high-level structure as ISO 27001 and ISO 9001, it slots into an existing management system rather than replacing it. Where NIST gives you what to think about, ISO 42001 gives you an auditable management system — defined roles, documented policies, continual improvement, and evidence a certifier can inspect. Its companion, ISO/IEC 42005, provides a framework for AI impact assessments at the system level, complementing 42001's organizational focus.

None of these is a substitute for the others, and certification proves you have a process, not that any single system is safe. A useful mental model for 2026:

• EU AI Act — the law you must obey (where it applies).
• NIST AI RMF — the thinking model for managing risk.
• ISO/IEC 42001 — the certifiable management system that operationalizes it.

Newer entrants like the CSA AI Controls Matrix (243 controls) add concrete, security-focused checklists that map back to these frameworks.

You do not need a 200-page policy to be responsible. A small team can stand up a defensible process from six artifacts — each maps directly to NIST's GOVERN/MAP/MEASURE/MANAGE and to EU AI Act and ISO 42001 evidence requirements.

1. AI Register — one inventory row per deployed AI system: purpose, owner, risk tier, data touched.
2. Risk / impact assessment per deployment — a short, repeatable template (this is exactly what ISO/IEC 42005 formalizes).
3. Model / system cards — documentation of the model, intended use, limitations, and eval results.
4. Human-in-the-loop gates — explicit approval for high-risk or irreversible actions.
5. Incident response playbook — who is paged, how the agent is paused, how harm is contained.
6. Ongoing monitoring — defined metrics (fairness, error rate, drift) and alerts on the actions the agent takes.

A minimal AI Register can start as code your CI checks on every deploy:

from dataclasses import dataclass, field
from enum import Enum

class RiskTier(str, Enum):
    PROHIBITED = "prohibited"
    HIGH = "high"
    LIMITED = "limited"
    MINIMAL = "minimal"

@dataclass
class AISystem:
    name: str
    purpose: str
    owner: str               # a NAMED person — accountability
    risk_tier: RiskTier
    role: str                # "deployer" or "provider" (EU AI Act)
    human_gate: bool         # approval required for risky actions?
    monitored_metrics: list[str] = field(default_factory=list)

    def is_deployable(self) -> bool:
        if self.risk_tier is RiskTier.PROHIBITED:
            return False
        if self.risk_tier is RiskTier.HIGH and not self.human_gate:
            return False     # high-risk demands human oversight
        return bool(self.owner) and bool(self.monitored_metrics)

refund_agent = AISystem(
    name="refund-agent",
    purpose="Auto-approve refunds under $50",
    owner="jordan.lee@acme.com",
    risk_tier=RiskTier.LIMITED,
    role="deployer",
    human_gate=False,
    monitored_metrics=["approval_rate", "dispute_rate", "demographic_parity"],
)
assert refund_agent.is_deployable()

Start here, make it a deploy gate, and grow it as your risk grows.