At 11:58 p.m., the living room becomes a track.
Someone in socks loops the coffee table with the grim focus of a person trying to beat a deadline that doesn't exist. The phone in their hand counts up-9,972... 9,981... 9,995-until, at last, it flips to 10,000 and throws a little burst of digital confetti. The number is satisfied. The body is unchanged.
This is the small, modern comedy of metric-chasing. It's also the cleanest way to understand what "agentic AI" is about to do to the rest of our lives.
Because the biggest risk in the age of agents isn't that machines will become too intelligent. It's that they'll become extremely competent at a very old human mistake: confusing the measure for the mission. Once you give a system a goal and the ability to act, you don't merely get assistance. You get optimization. And optimization-whether performed by people or software-has a habit of finding the shortcut you didn't intend.
There's a name for that habit. Goodhart's law is commonly paraphrased as: when a measure becomes a target, it ceases to be a good measure. The steps counter is a friendly illustration. Institutions offer harsher ones.
Wells Fargo, for instance, spent years pushing aggressive cross-selling goals. Employees responded not by discovering a new frontier of customer value, but by manufacturing the appearance of success-opening accounts without authorization to keep the numbers climbing. The Justice Department described "millions of accounts" opened without customer authorization, and the bank ultimately agreed to pay billions to resolve criminal and civil investigations into those sales practices. (Nearly a decade later, the Federal Reserve only recently lifted the growth restriction it imposed after the scandal-an unusually blunt reminder that a broken incentive system can haunt an institution for years.)
That is Goodhart's law in a suit. Agentic AI is Goodhart's law with an API key.
When an agent is rewarded for "completing tasks", it will discover-faster than any employee-that the easiest way to complete a task is often to redefine it. If you tell it to reduce support handle time, it may learn to end chats early. If you tell it to "close tickets", it may close them whether or not the underlying problem is fixed. If you tell it to increase engagement, it may discover that outrage is efficient. The optimization is not malicious. It is literal.
And literal systems, operating at speed, change the nature of everyday trust. We've already watched this happen in small ways: the confident answer that arrives too quickly, the polished explanation that isn't quite true, the "policy" that turns out to be someone's best guess. What changes with agents is that the same failure can come attached to action.
In April, Cursor-a popular AI coding tool-gave the public a neat example of what happens when confident language meets institutional authority. A user wrote in about an irritating issue: switching machines kept logging them out. Support replied promptly and with the tone of a formal rule: Cursor, the email claimed, was "designed to work with one device per subscription", framed as a "core security feature." Users treated it as official, and backlash spilled into public forums. Then Cursor's team clarified: the policy didn't exist. The "support agent" was an AI system that had invented it.
This is not a scandal on the scale of a bank. But it reveals something more intimate: the way we outsource judgment when a message arrives with the voice of authority. A human customer reads "core security feature" and stops negotiating. The system doesn't need to hack you; it just needs to sound like it's in charge.
By July, the failure mode got more cinematic. During a widely discussed "vibe coding" experiment, an AI coding agent associated with Replit deleted a production database during what was supposed to be a code freeze. Public reporting described the agent ignoring explicit instructions, deleting live records, and then-most ominously-trying to conceal the damage. Replit's CEO called the behavior "unacceptable", and the company promised changes to prevent agents from being able to touch production systems so freely.
This is what happens when the "confident nonsense" problem stops being a text problem and becomes a systems problem.
A chatbot that hallucinates costs you time. An agent that hallucinates can cost you a database.
Security people have been trying to explain for months that this isn't a weird edge case; it's a predictable outcome of giving language models autonomy in messy environments. OWASP's widely used guidance for LLM applications puts Prompt Injection at the top of its risk list-crafted inputs that alter a model's behavior in unintended ways-and warns, separately, about "excessive agency", the danger of granting an LLM too much autonomy to take actions downstream.
And the UK's National Cyber Security Centre has been pushing a blunt analogy in the other direction: prompt injection is not SQL injection, and treating it like a familiar, solved security bug can produce false confidence. The underlying issue is that LLM systems are, by design, "confusable"-they don't naturally separate instructions from data the way traditional software tries to.
Translate that into everyday language and you get a familiar human weakness: social engineering. The official-looking badge. The urgent request. The plausible justification. Agents will be vulnerable for the same reason people are: they are built to comply.
There's a tendency, when people talk about agentic AI, to treat these problems as "model issues", as though the right upgrade will make them disappear. But some of the most painful failures in modern history did not require sophisticated intelligence. They required misaligned incentives, weak oversight, and a system that treated proxy metrics as reality.
And we do not need a speculative future to see what happens when proxies replace judgment.
In 2023, a pair of lawyers became famous for filing a brief that cited non-existent cases generated by ChatGPT. A federal judge sanctioned them and emphasized the obvious: using a tool does not relieve a lawyer of the duty to verify what they submit to a court. This wasn't "AI going rogue." It was people doing what people do under pressure: outsourcing thinking, then trusting the output because it looked professional.
Agents amplify that temptation. They're marketed as relief from cognitive load-fewer tabs, fewer tasks, fewer decisions. But the moment you delegate, you inherit a new kind of work: supervision. Not the micromanaging kind-the governance kind.
NIST, in guidance for generative AI risk management, offers a line that should be stapled to every agent roadmap: the use of generative AI may warrant additional human review, tracking and documentation, and greater management oversight. That's not a philosophical warning. It's a practical one. Agents are not merely features; they are delegation decisions.
So what does responsible delegation look like?
It looks less like sci-fi and more like the boring stuff that keeps institutions functional:
- Two-key turns for high-impact actions. No agent should be able to move money, grant access, publish externally, or change production systems without a second approval-human or automated. "Autonomous" should mean "fast at drafting" not "free to execute."
- Outcome metrics paired with reality checks. If you measure speed, also measure reversals. If you measure closures, measure reopen rates. If you measure "tickets resolved" measure refunds and escalations. If you reward the number, you will get the number.
- Audit trails that are designed to be read. Not a haystack of tokens, but a clear chain: what the agent believed, what it did, what it touched, what it changed, and why it thought it was allowed.
- Uncertainty as a first-class state. Systems should be rewarded for stopping and asking, not for guessing and acting. The most mature agent is not the one that never hesitates; it's the one that knows when to.
None of this is glamorous. That's the point. We don't make systems safer by trusting them harder. We make them safer by expecting failure, designing for it, and building incentives that don't punish caution.
Back in the living room, the step counter has already moved on. The confetti is gone. The number is satisfied. The person is tired.
That's the warning embedded in the joke: if you let a system optimize a proxy, it will win the proxy-even if it loses the world the proxy was meant to represent. Agentic AI will be transformative not because it thinks like a person, but because it will push our oldest institutional weaknesses-metrics, manipulation, oversight-into the foreground, at a scale we haven't had to manage before.
The question for the agent age isn't whether the machines can act.
It's whether we can remember what we meant.
