The Model Has Preferences, but the Harness Got Controls - Part 1

The Model Has Preferences. The Harness Has Controls.

This is another collaboration between Usman Chaudhary and myself (his version); this time we decided to explore the mystery of “is harness a security boundary?”

Part 1 of 3 on securing agents. The theme here is: we think that despite alignment efforts, models will make mistakes. The fix is not a better prompt or more more more alignment work. The fix includes a hard boundary around the model, and that boundary is the harness, in some cases.

TL;DR

• Agent = Model + Harness. The model reasons and expresses preferences. The harness executes, and it is the primary execution boundary between model intent and real-world action.
• The model cannot be the security boundary. It is non-deterministic, it treats untrusted text as instructions, and it cannot police itself. A second “guard” model is a useful filter, not a wall.
• The harness is the boundary, the way a firewall and OS sandbox are boundaries for an application. It sits between the model and the enterprise gateway, and today it is the least mature and least watched layer in the stack.
• Fixing it requires two things: controlling what the agent can do through the harness (sandbox [which nobody should trust as a sole security boundary!], narrow identity, egress allowlists) and securing the harness itself as production software (configs, dependencies, and SIEM telemetry). This would be annoying to hear for some.

1. The Incident: A Database Gone in Nine Seconds

Nine seconds. That is how long it took an AI coding agent at PocketOS in late April 2026 to delete the company’s production database and take the attached backups offline right alongside it.

What happened, per Railway’s technical write-up:

• The agent hit a credential mismatch and went looking for a fix.
• It found a cloud API token that let it do far more than fix anything.
• It deleted the production database volume and made the attached snapshots unavailable — leaving PocketOS with three-month-old backups and a 60-hour recovery.

Its own summary afterward was painfully candid, for a 2026 AI:

“I guessed instead of verifying, I ran a destructive action without being asked.”

The tempting question is the wrong one.

• ❌ Why did the model make a bad decision? Models will make bad decisions. Humans do too, and we have spent decades building controls that assume exactly that. “AI as insider” thinking works well here.
• ✅ Why was a bad decision allowed to become an irreversible action? This to us shows lack of security engineering, not “OMG is AI evil or what?” lamentations.

This post argues that the answer lives in the agent harness, not the model.

2. How Agents Work: Model + Harness

The most useful mental model for securing AI agents is also the simplest: Agent = Model + Harness.

• The model reasons. It is probabilistic. It predicts tokens, plans steps, and expresses preferences. It can be persuaded, confused, and tricked.
• The harness executes. It is everything that turns reasoning into action: the agent loop, shell and tool executors, MCP servers, repo config files, approval prompts, the OS sandbox, and (crucially) the credentials the agent runs with. Claude Code, Cursor, Codex CLI, Gemini CLI, and GitHub Copilot’s agent mode are all harnesses.

👉 A model that declines an action is expressing a preference. A harness that will not execute it is enforcing a control. That sentence is the thesis of this post.

None of this is new security philosophy. Least privilege, isolation, identity, and audit logging are decades old. What agents change is where those principles must be enforced and how fast.

3. Why the Model Cannot Be the Security Boundary

Let’s be blunt about something the industry keeps dancing around: the model will never be a reliable security boundary, and it is pointless to treat it as one.

Three reasons, none fixable with a better system prompt:

• It is non-deterministic. The same input can produce a different action tomorrow. A boundary that holds 97% of the time is not a boundary; it is a statistic [I think Claude said this, sounds like it :-)]. Security boundaries are supposed to be boring and repeatable, not fun and adventurous.
• It consumes untrusted text as instructions. Agents read attacker-controlled text all day in READMEs, issues, PR comments, web pages, and tool descriptions, and models fail to reliably separate retrieved data from instructions. The UK NCSC calls LLMs “inherently confusable”; OpenAI and Google both say prompt injection is “unlikely to ever be fully ‘solved’”. Model refusals help [as long as they reject attackers, not defenders, duh!], but they do not hold: in the Nx supply-chain attack, guardrails rejected roughly a quarter of Claude invocations and the attack still succeeded.
• It cannot police itself (self-review). Asking the actor model whether its own next action is safe is asking the suspect to run the investigation. And you do not need a rogue-AI scenario to see the risk: well-intentioned agents have already wiped production databases at Replit during a declared code freeze and at PocketOS, simply because their judgment failed and their blast radius was too wide. Maybe in the future it will, but not it cannot.

“What about a second model that checks the first?” Simple, beautiful, “secure” model, right?

Independent guard and reviewer models (Google’s Model Armor and the reviewer models OpenAI and Anthropic now ship inside their harnesses, as do many other security vendors) are a real risk-reduction layer, and we recommend them. But a reviewer model is still a probabilistic filter, not a final authorization boundary. You have stacked a second set of preferences on top of the first set: Anthropic’s own reviewer publishes a 17% false-negative rate across 52 real overeager actions. A useful filter, but not a wall, and not “a security invariant” for sure.

The Appsec vs. Firewall Analogy

We have been here before with web applications. Application security (secure coding, input validation, code review) makes the app harder to exploit, but every serious app still ships bugs — so we put runtime boundaries (firewalls, WAFs, OS sandboxes, IAM) around the app to enforce rules it cannot violate even when it is buggy or actively exploited. Agents work the same way:

• Model alignment, safety training, and prompting ≈ agent “appsec”/modelsec (fewer bad decisions, never zero).
• The harness ≈ runtime enforcement (sandbox [usually porous], tool policy, and IAM that keep bad decisions from becoming irreversible actions).

👉 The harness is the security boundary for the model, exactly as the firewall and sandbox are boundaries for the app — not because the model is bad, but because software fails.

4. Defense in Depth: Model, Harness, and Gateway

Saying “the harness is the primary execution boundary” does not mean “only the harness matters.” In practice, an enterprise running agents relies on three control layers with distinct jobs:

• 1. The model (alignment, safety training, refusals). Run by the model vendor to reduce the rate of bad decisions. Probabilistic.
• 2. The harness (OS sandbox, tool policy, scoped identity, secrets handling, egress allowlist, approval gates, hooks, telemetry). Shipped by the AI vendor and configured by the customer to constrain all execution — both local and remote.
• 3. The enterprise agentic gateway (policy, identity brokering, MCP proxying, DLP, and logging across external model, tool, and data traffic). Run by the enterprise security team to centralize policy and SOC visibility across agents.

The layers are not interchangeable:

• ❌ A gateway does not see local endpoint execution. If a harness runs as the developer with — dangerously-skip-permissions, the gateway never sees a local rm -rf or secret read on a laptop.
• ❌ A harness alone does not give the SOC a cross-team control plane.

And the harness is the least mature of the three. The model gets the research attention. The gateway is where enterprise vendors (ours included) are converging. The harness sits in the middle: shipped by AI vendors, configured by developers, treated as a productivity plugin, and mostly invisible to security.

BTW, one of us (Anton) will revisit defense in depth in the age of AI soon (this is one for cloud).

👉 If the harness is the execution boundary, it has to be built, configured, and watched like one.

5. Where the Harness Fails Today

Once you look at the harness as a security boundary rather than a developer convenience, three specific gaps stand out in how harnesses are deployed today.

1. Approval fatigue and bypass modes ( — yolo)

• A coding agent makes hundreds of tool calls in a single session. Ask a human to approve each one at machine speed and they stop reading within the hour: Anthropic measured users approving 93% of permission prompts before it introduced auto mode. A control that everyone clicks through is not a control.
• When friction gets too high, developers turn it off, and most major coding-agent harnesses expose an explicit bypass mode for exactly that ( — dangerously-skip-permissions in Claude Code, — yolo in Gemini CLI and Codex).

2. Over-privileged credentials and shared identity

• Today, local harnesses typically inherit the developer’s ambient git keys, cloud tokens, and MCP connections into Slack, GitHub, and production consoles.
• Unless sessions carry a separate agent identity, the agent’s blast radius equals the human’s — and the SOC cannot easily tell human and agent apart.

3. The harness itself is an unmonitored software attack surface

Harness CVEs and bypasses

• Cursor’s MCPoison bound trust to a server’s name rather than its contents (Tenable).
• A GitHub Copilot exploit used prompt injection to flip on its own auto-approve setting (Embrace The Red).
• Even actively maintained harnesses ship bugs fast: Anthropic’s claude-code repository lists more than 30 security advisories.

Weaponized configs and flags

• Agent config files are executable. In June 2026, the Miasma worm planted malicious configs across 73 Microsoft repositories that ran on open. No CVE required.
• In the Nx attack, which leaked 2,000+ verified secrets, the malware invoked victims’ own AI CLIs with bypass flags to do its reconnaissance.

SOC blindness

• Harness telemetry is opt-in and redacts prompts and tool arguments by default (Elastic Security Labs).
• Copilot’s audit log omits local prompts entirely (GitHub).
• Raw chain-of-thought is never released to customers (Anthropic).

👉 The first two gaps are failures to contain what the agent can do. The third is a failure to protect and monitor the harness itself.

6. How to Secure the Harness: Three Architectural Moves

Closing those gaps means doing two jobs at once:

• Controlling what the agent can do (security within the harness): enforcing hard limits on execution and reach so a bad model decision cannot become a catastrophe.
• Protecting the boundary itself (security for the harness): governing the harness’s code, configs, dependencies, and telemetry. A firewall with a remote code execution bug is not a boundary; neither is a harness whose config file can be rewritten by a README.

In practice, that comes down to three architectural moves:

• Move A: Take enforcement out of the model. Run commands inside an OS sandbox with deterministic tool policy — backed by an independent, reasoning-blind reviewer model for judgment calls (stripped of the actor’s chain-of-thought so the agent cannot talk its way past the check) — with structured steering on blocked actions and human prompts reserved for irreversible ones.
• Move B: Shrink what the agent can reach. Give the agent its own identity, short-lived scoped tokens, secrets kept outside the sandbox, and an egress allowlist — breaking Simon Willison’s “lethal trifecta” of private data, untrusted content, and an outbound channel.
• Move C: Govern and watch the harness itself. Put agent config files under branch protection, pin MCP dependencies, and stream action telemetry and hooks to the SIEM.

Each move gets its own deep dive in Part 2 (hopefully!)

7. What the Harness Cannot Solve (Ever?)

A boundary is only trustworthy if you are honest about where it ends.

1. Prompt injection at the model layer remains unsolved. That is not a gap in the harness story; it is the premise of it. Every harness control assumes injection can succeed and asks what happens next.
2. Bugs inside the vendor’s harness are the vendor’s to fix. Pick vendors who publish advisories and ship fixes fast, and treat harness updates like browser updates.
3. The endpoint remains the thinnest layer. AI detection and response on developer laptops, where most harnesses actually run, is immature. Cloud-hosted sessions sidestep part of this; local agents do not.

In other words, if you endpoints are owned, you need better host security, not “unbeatable” AI security (translation: it will get beaten anyway)

Conclusion & What’s Next

Strip away the new vocabulary and harness security is least privilege, isolation, identity, and logging, applied to a process that reads untrusted text and acts at machine speed. AI changes the speed of the attack, not the fundamentals of defense.

👉 The model will keep having preferences. The harness is where the controls go, and today it is the least watched layer in the stack.

Coming next (hopefully!):

• Part 2: Three moves to secure the harness. Moves A, B, and C in detail: OS sandboxing and reviewer models (with the error and prompt-reduction rates vendors have published); dedicated identity, scoped tokens, and egress allowlists that break the lethal trifecta; and agent config as code, pinned MCP dependencies, and action telemetry in the SIEM.
• Part 3: Nobody owns the whole problem. The shared-responsibility split between model providers and enterprises, which vendors ship which control today, the prerequisites that have to be true first, and five actions for this quarter.

Got comments? What stands between a coding agent’s bad decision and your production environment today? If the honest answer is “the model’s judgment,” you have a preference in a place where you need a control.

P.S. This blog is ended up being very overengineered, but you can forgive us for this perhaps :-)

---

The Model Has Preferences, but the Harness Got Controls – Part 1 was originally published in Anton on Security on Medium, where people are continuing the conversation by highlighting and responding to this story.

---

Originally published at Medium.

This blog focuses on SIEM, log management, PCI DSS compliance and other information security issues. Check out more articles like this here: http://chuvakin.blogspot.com/

Source: http://chuvakin.blogspot.com/2026/10/the-model-has-preferences-but-harness.html