Harness Engineering for Agents

Production Loops Context Control & Trust Boundaries

The minimal loop from the foundations is a short while-loop: send the message history to the model, run whatever tools it asks for, append the results, repeat until it stops. That loop is enough to prove the idea. It is not enough to run an agent for a real user, for hours, on a repository you do not fully trust. Production adds three problems the minimal loop never has to face. Work runs for a long time, so you need structure to observe, pause, and stop it. The context window fills up, so you need rules for what the model sees and what it forgets. The model reads text it did not write and takes actions with real consequences, so you need to decide who is allowed to do what. This chapter answers three questions: how the loop is organized in time, what goes into the model's window, and where the trust lines fall.

Structured Loop Hierarchy

Production runs need names for their parts. Otherwise every log line, retry, and UI update becomes a guess about what the agent was doing. The standard split has three levels. A session is a long-lived container, usually one conversation or one task, that owns the history. A turn is one unit of user intent: from one user message to the agent's final answer for it. A step is one model call together with the tool calls that call requested and their results. A single turn often contains several steps, and a session contains many turns.

Each level has a lifecycle, meaning a small set of states and legal transitions. A step moves from requested to running to completed or failed. A turn cannot complete while a step is still running, and a session cannot close while a turn is waiting for approval. Making these transitions explicit lets you ask simple questions with exact answers: is this session idle, is it waiting on a human, did the last step fail or simply time out. A common wrong intuition is that the loop is the lifecycle. In fact the loop is one transition function inside the lifecycle, and the lifecycle is what everything else (the UI, the logs, the recovery code) reads.

The state is not only stored; it is also published. A live state projection is a read-only view derived from the event history, such as "streaming text, one tool running, nothing awaiting approval." Clients consume projections instead of replaying raw messages. Those updates travel as a server-sent event (SSE) stream: a one-way HTTP connection where the server pushes typed events like step.started, tool.progress, and turn.completed. Typed events matter because a UI can render each kind differently, and a dashboard can count approvals without parsing prose.

Users do not wait politely between turns. They type while the agent is working. There are two ways to handle that input. Queuing holds the message until the current turn ends, then runs it as a new turn. It is safe and predictable, but slow if the agent is mid-way through a bad plan. Steering delivers the message at the next step boundary, so the model sees it before its next call. Steering is not interruption. You cannot inject text between a tool call and its result, because the provider's wire format requires every tool call to be answered by a matching result before the conversation can continue. Steering works only because the boundary between steps is a point where the history is consistent.

Layered Context Engineering

The context window is a budget, not a log. Every token you put in it competes with every other token for the model's attention, and every token you send is paid for again on each step unless it is cached. So the useful question is not "what happened?" but "what does the model need right now?" Layered context engineering answers that by splitting the window into layers with different lifetimes.

The bottom layer is always-on instructions: the system prompt, the tool definitions, and the standing rules of the project. These change rarely, so they form a stable prefix that providers can cache. Keeping that prefix byte-identical across steps is what makes caching work, and it is why you should never put a timestamp at the top of your system prompt. Above it sits progressive disclosure. Instead of loading every skill, document, or playbook into the prefix, you show the model a short index and let it pull the full text of one entry when the task calls for it. A thousand-line style guide costs nothing until someone needs it.

The middle layer is per-step dynamic injection: facts that change between steps, such as the current git status, the list of open files, or a pending notice from a background job. These are computed fresh for each step and placed after the stable prefix so they do not invalidate the cache. Some of these computations are expensive, so the runtime memoizes them: a value computed once within a step is reused for the rest of that step rather than recomputed for each consumer.

The top layer is history, and history is where most production failures live. Context rot is the slow decay that happens when old, bulky tool output keeps occupying the window long after it stopped mattering. Compaction is the act of shrinking history, and production systems do it in two stages. The first stage is tool-output stubbing: a 3,000-line log is replaced with a short marker saying what it was and how to regenerate it. This is cheap and needs no model call, and it is usually lossless in practice, because the data can be re-read from the source. The second stage is LLM checkpointing: when stubbing is not enough, a model writes a structured summary of the older history, and that summary replaces the raw turns.

The stage that people most often skip is tail preservation. The most recent window of roughly 20,000 tokens stays verbatim, never summarized. The reason is that summaries blur exact identifiers, error strings, and file paths, and the model needs those exactly while it is still working on them. When work must outlive a session entirely, a workflow handoff passes a structured artifact (a plan, a list of completed items, the open questions) to a fresh session. The new session starts with a small, accurate context instead of dragging a compressed past behind it.

Production Tool Execution

A tool call from the model is a request, not an action. Between the request and the side effect sits a path that the runtime controls, and each stage on that path can refuse, delay, or shape the call. The path runs as follows. The model emits a call with JSON arguments. The runtime validates those arguments against the tool's schema. It then checks whether the call needs approval. If the call proceeds, it executes under a progress stream and a cancellation signal, and the result is formatted and truncated before it returns to the history.

Validation comes first because models produce malformed JSON, missing fields, and wrong types with regularity. A rejected call is not a crash. The runtime returns a structured error as the tool result, so the model sees what it got wrong and can correct itself on the next step. Errors that the model can read are part of the harness; errors that are thrown as exceptions end the turn.

Long-running tools need two more controls. Progress streaming lets a shell command report partial output while it runs, so the UI and the model can see that something is happening and the runtime can cut off a runaway process early. Cancellation is usually cooperative: the runtime sets an abort signal, and the tool checks it at safe points and stops cleanly. Killing a process mid-write is how half-applied edits happen, which is one of the failure modes you met in the foundations. Cooperative checks let a tool finish its current atomic unit before it stops.

Some calls should never run without a person saying yes. Human-in-the-loop (HITL) approval gates attach a predicate to each tool, often written as needsApproval, that inspects the actual arguments. Reading a file passes freely. Running rm -rf on a path outside the project does not. When the gate fires, the turn pauses in a persisted state, the request is shown to a human, and execution resumes only after a decision. The predicate looks at arguments rather than tool names because the same tool can be safe in one call and dangerous in the next.

Not every tool returns quickly. An inline tool finishes within the step that called it. A long-running workflow tool might wait for a CI webhook, or sleep for two hours before checking a deployment. Holding a process open for that long is wasteful and fragile, so these tools park the turn: the runtime records what it is waiting for and releases compute until the event arrives. The rest of the harness treats both kinds the same at the interface, but the runtime must know the difference, or a two-hour sleep will quietly block a worker.

Security and Sandbox Split

An agent that reads a web page or a repository file is reading text written by someone who may want to control it. Instructions hidden in a README, a comment, or a fetched HTML page can tell the model to exfiltrate data or run something destructive. This is prompt injection, and no amount of careful system prompting reliably prevents it. The defense therefore has to be structural. You assume the model can be steered, and you arrange the system so that a steered model cannot do much damage.

The arrangement splits the system into two zones. The trusted app runtime holds the secrets, runs the loop, makes policy decisions, and talks to the user. The isolated sandbox holds the files and the shell, where the model's actions actually happen. The sandbox is treated as untrusted, even though it is your own container, because anything the model reads or does can be shaped by an attacker. The runtime decides; the sandbox executes. A request crosses the boundary as a message with arguments, and a result crosses back as data. Nothing in the sandbox can reach upward into the runtime's memory or configuration.

The sharpest consequence concerns credentials. The sandbox should hold zero secrets. When a tool needs to call an authenticated API, the request leaves the sandbox through a network egress proxy that lives in the trusted zone. The proxy checks the destination against an allowlist and attaches the token itself. If the agent runs env or reads a dotfile inside the sandbox, it finds nothing worth stealing. This is firewall-level credential brokering, and its value is that it does not depend on the model behaving well. A common misconception is that a sandbox with a restricted shell is enough. It is not, because a command the shell allows can still print a token that the sandbox was given.

The policy that decides which actions reach the sandbox, and how commands are parsed before they run, is a whole discipline of its own, covered elsewhere in this guide. What matters here is the shape. Every privileged decision is made in the trusted zone, every dangerous side effect happens in the sandbox, and every secret stays on the trusted side of the wall.

Capstone: Approvals Two-Stage Compaction and Docker Isolation

Extend the minimal agent in three ways and then test each claim against evidence rather than assuming it works. First, add approval gates: give at least one tool a needsApproval predicate that inspects its arguments, pause the turn in a persisted state when the gate fires, and confirm that the run resumes correctly after a decision, both on approval and on rejection. Second, add two-stage compaction: stub tool outputs above a size threshold, then trigger an LLM checkpoint when the window still exceeds budget, and keep a verbatim tail of recent turns.

Third, move the shell and file operations into a Docker container, and keep the model's API key and any other credential in the host process. Verify the trust split directly. From inside the container, list the environment variables, search the filesystem for the key's value, and inspect the process table. Nothing should match. Then have the agent make an authenticated request through the egress proxy and confirm that the request succeeds while the token never appears inside the container.

Verify coherence over a long run. Generate a synthetic session of around three hundred turns, plant a specific fact such as a configuration value or an error identifier in turn five, and ask for it near the end. The answer should be correct, and you should be able to say whether it came from the checkpoint, the verbatim tail, or a stubbed output that was re-read. A fact that survives only in your head is not verified. A fact that survives in the logs, with the layer that supplied it named, is.

AI-generated study article from SophistAI. Sign up free to study it with an AI tutor.

Loading SophistAI