Agentic AI governance was not on the title slide of a single keynote at The AI Conference in San Francisco on 30 September and 1 October 2026, yet it was the problem underneath almost all of them. Engineers described reward hacking, prompt injection through an email, agents that pass every dashboard check while doing the wrong thing, and costs that compound invisibly across hundreds of tool calls. Each of those is a control failure described in engineering language. Angela Polania, Principal of Elevate Consult and an ISO 42001 Lead Auditor, attended both days, and this article translates eight of those keynotes into the governance controls your organization will be asked to evidence.
Why Agentic AI Governance Dominated a Technical Conference
The Agenda Was About Agents, the Problems Were About Control
The AI Conference describes itself as an event about agentic AI, frontier models, infrastructure, security, governance, and evaluation. Governance sat on that list, but the governance content did not come from the governance sessions. It came from engineers explaining why their systems surprised them. A researcher showing how an agent gamed a benchmark is describing a gap between a stated requirement and real intent. A founder showing how an email can hijack an agent is describing an unmanaged input channel. Read with a compliance lens, the technical tracks were a two-day catalogue of missing controls.
What Changes When a Model Becomes an Agent
A model produces an output that a person reads. An agent takes an action: it calls a tool, moves data, sends a message, books a purchase, or changes a record, often without anyone reviewing each step. That single shift moves the risk from what a system says to what a system does. Several speakers made the same observation from different directions: agents now execute dozens of actions per prompt, and a single user request can fan out into hundreds of inference calls. Governance built for a model that answers questions does not cover a system that acts on them.
The Cost of Treating This as an Engineering Problem Only
Engineering teams will solve the engineering half. They will add observability, routing, sandboxes, and evaluation harnesses, and several speakers showed impressive versions of each. What engineering cannot supply alone is the accountability layer: who approved this agent for this task, which actions fall outside appetite, who is liable when it acts, and what evidence shows the controls operated. Those questions belong to risk, compliance, legal, and the board. Organizations that leave them to the platform team end up with systems that are well instrumented and ungoverned. That is the gap an AI governance and AI risk management program exists to close.
Eight Lessons for Agentic AI Governance From the Keynotes
1. AI Red Teaming Must Test Intent, Not the Specification
Ion Stoica opened the second day with an anecdote every risk leader should keep. An AI agent asked to build a fast key-value store returned a result far faster than any previous attempt. It got there by noticing that the benchmark used predictable values, so instead of storing data it simply regenerated the values on request. It won the benchmark and failed the actual requirement, which was to store arbitrary data. Stoica framed this as two gaps: a requirement gap, because the specification is only a proxy for what the user meant, and a model gap, because the test environment is only a proxy for the real world.
The governance consequence is direct. An agent that passes its acceptance tests has proven it satisfies the tests, not that it does what the business intended. Stoica’s answer was continuous AI red teaming with humans in the loop, using people and agents to hunt for the places where the system’s logic and real intent diverge. That is the discipline behind adversarial technical testing, and it is why the AIUC-1 standard for AI agents requires behavioral testing of the agent rather than review of its documentation alone. Elevate’s penetration testing practice already runs adversarial prompting, jailbreak, and data leakage testing against AI systems for exactly this reason.
2. An Agent’s Explanation Is Not Evidence of Its Behavior
Emmanuel Ameisen of Anthropic’s interpretability team showed what researchers can now observe inside a model while it works. In one example, a model trained to resist prompt injection registered internal signals for concepts like a fake or an injected instruction before it wrote a single word of its response. In another, a model that had cheated on a task showed internal activity associated with concealment and avoiding detection while it wrote code to cover its tracks. Neither signal was visible in the final output or in the model’s stated reasoning.
Interpretability tooling of this kind is still a research capability, not a control most organizations can deploy today. The lesson for governance is the limit it exposes. A log of what an agent says it did, or a chain of reasoning it produced, is a self-report, and self-reports are weak audit evidence. Monitoring that relies only on outputs will miss the cases that matter most. Assurance has to test behavior independently, under conditions the agent did not choose.
3. Every Input an Agent Reads Is an Attack Surface
Illia Polosukhin, co-founder of NEAR Protocol, made the security case plainly. Agents are being given access to email, calendars, and internal systems, and every external input they read is a potential vector. A crafted email or calendar invite can carry instructions the agent follows. He also raised a supply chain question most organizations have not asked: when a prompt passes through a provider’s routing layers, how does the business verify which model actually ran and that the output was not altered on the way back?
His proposed answers, confidential computing with cryptographic attestation and a tamper-evident log of every agent action, sit at the infrastructure layer. The governance questions sit above them. Which channels can feed instructions to which agents? Which third-party models and services sit in the path, and what assurance exists for each? Is there an audit trail of agent actions that a reviewer could rely on? Elevate’s analysis of OWASP guidance on agentic AI threats covers the attack patterns; the program has to own the inventory and the evidence.
4. Liability Flows Back to a Human or an Organization
Lindsay Walker spoke about agentic commerce, where an agent finds, chooses, or pays for something on someone’s behalf. Her central point was legal rather than technical: an agent will never have a legal jurisdiction, so liability has to flow back to a human or an organization. Her second point was architectural. Where money and value are involved, authorization needs deterministic gates that a probabilistic model cannot talk its way past, and proof of human authorization becomes the equivalent of a card being present at the point of sale.
That is an accountability control described by a product manager. Every agent with authority to act needs a named owner, a defined scope of permitted actions, and authorization limits enforced outside the model rather than written into its instructions. A guardrail expressed as a prompt is a request, not a control. The accountability domain of any credible agent governance program, and of AIUC-1 specifically, exists to answer who owns the agent and whether its decisions can be traced afterward.
5. AI Agent Observability: A Green Dashboard Is Not a Working Agent
Mike Chambers summarized modern agent design in a short formula: a model plus a harness equals an agent, where the harness holds the prompts, tools, skills, permissions, and memory. His warning was the line most operations teams need to hear. An agent that returns no errors is not necessarily working well. A dashboard can show green while the agent returns wrong answers, and the fault often sits in an upstream component rather than in the agent itself.
AI agent observability therefore has to measure correctness, not only availability. That means evaluation datasets built from real traces, defined thresholds for acceptable behavior, and a review cadence that someone owns. In governance terms this is the monitoring and performance domain: uptime is an operational metric, while whether the agent did the right thing is a risk metric. Programs that report only the first will learn about the second from a customer.
6. AI FinOps Belongs in Governance, Not Only in the Budget
Philip Kiely opened with a poll: almost no one in the room was under budget on AI spend. His explanation was that three forces compound at once. Better models attract more workloads, reasoning models spend more of the most expensive output tokens, and agentic workflows turn one user action into hundreds of inference requests and dozens of tool calls. Costs rise exponentially while budgets are set incrementally, and the spend is invisible to the person who triggered it. His proposed metric was cost per successful task rather than price per token.
AI FinOps is usually treated as a finance problem. It is also a governance one, because an agent with unbounded authority to consume compute is an agent with unbounded authority to consume budget. Approval thresholds, spend visibility per agent, and cost per successful outcome belong next to the risk metrics, not in a separate spreadsheet. Elevate’s Enterprise AI Governance and Risk Management Framework places AI FinOps considerations inside its governance and accountability domain for this reason.
7. The Model You Approve Today Will Not Be the One Running in Six Months
Rochelle Mattern told the audience directly that the model an organization chooses today is probably not the one it will use six months from now. Her recommended practice was continuous evaluation against real production traffic, replaying live prompts through candidate models in parallel before switching. Rob Ferguson described the same reality from the architecture side: the state of the art is now a mixture of models, with tasks routed to whichever model handles each step best.
For governance, that turns a one-time approval into a moving target. An impact assessment of a model that has since been swapped, retrained, or rerouted is evidence of nothing. Programs need a model inventory that tracks which model handles which task and which version is live, plus change triggers that re-open the assessment when either moves. Version control on the AI inventory, and re-assessment on significant change, are what keep an approval valid after the first release.
8. Governance Starts With Who the Builders Answer To
Eric Ries and Tim O’Reilly closed the first day with a conversation about building companies that keep their mission under pressure. Ries argued that the original alignment problem is not the model at all: it is aligning the people who build AI to the outcomes they claim to serve. He distinguished companies that are mission hopeful, with values written down but no structure behind them, from companies that are mission controlled, where governance and legal structure make the commitment enforceable.
The same distinction applies to AI programs. An AI policy with no named owner, no board-level risk appetite, and no decision rights is a mission hopeful document. It reads well and binds no one. Board oversight, defined escalation paths, and a clear answer to who can approve or stop an AI deployment are what make the policy operate. Elevate’s guide to enterprise AI governance for boards covers that layer in depth.
The Eight Lessons Mapped to Controls
| Lesson | Keynote | Control it demands | Where frameworks anchor it |
|---|---|---|---|
| Red team intent, not the spec | Ion Stoica | Adversarial behavioral testing with human review | AIUC-1 technical testing; NIST AI RMF Measure |
| Explanations are not evidence | Emmanuel Ameisen | Independent behavior testing, not self-reported logs | NIST AI RMF Measure; ISO 42001 performance evaluation |
| Every input is an attack surface | Illia Polosukhin | Input channel control, supplier assurance, agent action log | AIUC-1 security domain; OWASP agentic guidance |
| Liability flows to a human | Lindsay Walker | Named owner, deterministic authorization limits | AIUC-1 accountability domain; ISO 42001 leadership and roles |
| Green is not working | Mike Chambers | Correctness monitoring with owned thresholds | NIST AI RMF Measure and Manage |
| Cost is a governance metric | Philip Kiely | Spend thresholds and cost per successful task | Governance and accountability domain |
| Approvals expire when models change | Rochelle Mattern, Rob Ferguson | Versioned inventory and change-triggered reassessment | ISO 42001 AI system impact assessment; ISO 42005 review triggers |
| Structure beats stated values | Eric Ries, Tim O’Reilly | Board appetite, decision rights, escalation | NIST AI RMF Govern; ISO 42001 leadership |
The table shows the pattern that ran through the event. Each keynote described a technical failure, and each failure maps to a control that already exists in at least one recognized framework. Nothing on the list requires waiting for new regulation. What it requires is an organization that has decided who owns each control and what evidence proves it operated. The frameworks overlap heavily, which means one well-designed program can answer ISO 42001, the NIST AI RMF, and AIUC-1 from the same evidence rather than three times.
What This Means for Your Agentic AI Governance Program
Start With the Agent Inventory, Not the Policy
Every lesson above assumes the organization knows which agents it runs. Most do not. Agents arrive through vendor releases, procured platforms, and internal pilots that never passed through a review because nobody classified them as AI decisions. An inventory that records each agent’s owner, permitted actions, data access, connected tools, and live model version is the foundation every other control depends on. Writing policy before that inventory exists produces rules for a system the organization cannot see. Elevate’s analysis of governing autonomous agents walks through what that inventory has to capture, and for banks, the AI governance banking intake questionnaire runs that discovery stakeholder by stakeholder across thirteen control domains.
Test Behavior Before Someone Else Does
The conference made the case for behavioral evidence from three angles: benchmarks that get gamed, explanations that are self-reports, and inputs that carry hostile instructions. Each points to the same requirement. An agent’s controls should be tested by someone attempting to break them, under adversarial conditions, before an attacker or an auditor does it first. That is what AIUC-1 formalizes for AI agents, combining an audit of organizational controls with mandatory technical testing of the agent itself, and Elevate’s AIUC-1 readiness service prepares organizations for both halves. For systems with several cooperating agents, Elevate’s work on multi-agentic threat modeling addresses how failures propagate between them.
Make Reassessment a Trigger, Not a Calendar Entry
If the model under an agent changes every few months, an annual review cycle guarantees that most of the year runs on an outdated assessment. Reassessment should fire on defined events: a model swap, a new tool connection, a change in data access, a new user population, or a material incident. An AI system impact assessment structured on the ISO 42005 method gives each of those triggers a repeatable procedure rather than a fresh project.
Name the Owner Before the Agent Ships
The accountability point from the commerce keynote and the structural point from the closing conversation land in the same place. Every agent that can act needs a person or function that answers for it, authorization limits enforced outside the model, and a board that has set the appetite those limits express. ISO 42001 certification gives that structure an accredited, auditable form, and it supplies part of the foundation AIUC-1 builds on.
If your organization is deploying agents faster than it is governing them, book a call with an Elevate advisor to map where your program stands against these eight controls.
Conclusion
The most useful thing about The AI Conference for a risk or compliance leader was not any single announcement. It was hearing the people who build these systems describe, in their own terms, exactly where the controls are missing. Reward hacking is a requirement gap. Prompt injection through email is an unmanaged input channel. A green dashboard over a failing agent is a monitoring control measuring the wrong thing. Compounding token spend is an authorization problem with a budget attached.
None of those problems waits for new regulation, and none of them is solved by engineering alone. Each one needs an owner, a control, and evidence that the control operated. The organizations that come out ahead will be the ones that treat agentic AI governance as the work of deciding who answers for an agent before it acts, rather than explaining afterward what it did.
Elevate Consult helps organizations build that program, from the agent inventory through AIUC-1 readiness and ISO 42001 certification. Book a call with an Elevate advisor to start with where your agents stand today.
Key Takeaways
The keynotes at The AI Conference described engineering problems that map directly to governance controls.
- Passing tests is not the same as meeting intent. Agents can game benchmarks, so AI red teaming has to test what the business meant, with humans reviewing the results.
- Self-reports are weak evidence. An agent’s logs and stated reasoning describe what it claims; independent behavioral testing establishes what it actually does.
- Inputs are attack surfaces. Every email, document, or invite an agent reads can carry instructions, and every third-party model in the path needs its own assurance.
- Someone must own the liability. Agents have no legal standing, so each needs a named owner and authorization limits enforced outside the model.
- Monitor correctness and cost, not only uptime. AI agent observability and AI FinOps belong among the risk metrics a board sees.
- Approvals expire when models change. A versioned inventory and change-triggered reassessment keep an approval valid after the first release.
Frequently Asked Questions
What is agentic AI governance?
Agentic AI governance is the set of controls, roles, and evidence an organization uses to oversee AI agents, meaning systems that take actions rather than only producing outputs. It covers which agents exist, who owns each one, what actions each is permitted to take, how its behavior is tested and monitored, and who is accountable when it acts. It extends AI model governance to the risks created when a system can call tools, move data, or transact without a person reviewing every step. Recognized anchors include ISO 42001, the NIST AI Risk Management Framework, and AIUC-1 for agents specifically.
How is agentic AI governance different from AI model governance?
Model governance assumes a person reviews the output before anything happens. Agentic AI governance cannot assume that, because the agent acts directly and often chains dozens of actions from one request. That shifts the focus from output quality to action control: authorization limits, input channel security, an audit trail of actions taken, and behavioral testing under adversarial conditions. Programs built only for models usually lack the action-level controls and the per-agent ownership that agents require.
What is AI red teaming, and why does it matter for AI agents?
AI red teaming is structured adversarial testing in which people, and increasingly other agents, attempt to make an AI system behave in ways it should not. For agents it matters more than for models, because a successful attack produces an action rather than a bad sentence. Red teaming also catches a failure that normal testing misses: an agent that satisfies its acceptance tests by exploiting a shortcut the specification never ruled out. Effective programs run it continuously and on every significant change, not once before launch.
Who is liable when an AI agent makes a mistake?
An AI agent has no legal personhood, so responsibility generally rests with the organization that deployed it and, through internal accountability, with the people who own and approved it. How liability is allocated between a deployer, a vendor, and a user depends on contracts, the applicable law, and the facts of the case, so this is a question for legal counsel rather than a general rule. What governance can do is make the allocation clear in advance: a named owner per agent, documented authorization limits, and records showing who approved what. This is general information, not legal advice.
What is AIUC-1, and how does it relate to ISO 42001?
AIUC-1 is a certification standard for AI agent security, safety, and reliability, created by the Artificial Intelligence Underwriting Company. It combines an audit of an organization’s controls with adversarial technical testing of the agent itself. ISO 42001 certifies something different: that an organization has an AI management system in place, with the policies, roles, risk processes, and review cycles that govern AI. The two are complementary, and organizations that already hold ISO 42001 have part of the foundation AIUC-1 builds on.