AI Agent Runtime Architecture: Tools, Memory, and Control Planes

The Reframe

The word "agent" gets used to describe everything from a glorified fetch() call to full autonomous systems that spawn sub-agents, manage external state, and take actions in the real world. The architecture underneath all of them is simpler than the hype suggests — and understanding it at the protocol level is what separates a reliable agent from one that silently burns your token budget.

The Core Abstraction

An agent is a model running in a loop with tools. The model receives messages and tool results, decides what to do next, and either calls another tool or returns a final answer. Your application code drives the loop. The model is stateless between calls — all state lives in the message array you maintain.

LLM Core

The model processes the current message array plus tool definitions and produces either a tool_use response or a final text answer. It has no persistent state.

Tool Registry

A set of function schemas passed with every request. The model reads descriptions to decide which tool to call and with what arguments. Good descriptions are load-bearing architecture.

Message Thread

The full conversation history including tool calls and results. This is the agent's working memory. Everything the agent "knows" across steps lives here — until it fills the context window.

Loop Controller

Your application code (or a framework like ToolLoopAgent) that drives iterations: send → parse → execute → append → repeat until done or capped.

Stop Conditions

The rules that terminate the loop: model returns no tool call, step count reached, token budget exceeded, or a sentinel tool is invoked. Without explicit conditions, loops run until context fills.

Guardrail Layer

Timeouts, allow-lists, human approval gates, cost ceilings, and idempotency checks that prevent tool execution from having catastrophic side effects at runtime.

The Agent Loop (ReAct at the Protocol Level)

The canonical pattern is called ReAct — Reason and Act. The model reasons about the current state, selects an action (tool call), observes the result, and reasons again. Concretely, this maps to the stop_reason field on the model response.

// The Agent Loop — Protocol Level

┌─────────────────────────────────────────────────────────────────┐ │ YOUR APPLICATION │ │ │ │ messages = [ { role: "user", content: task } ] │ │ │ │ while (true) { │ │ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ POST /v1/messages │ │ │ │ { messages, tools, max_tokens } → Model │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ │ │ stop_reason?│ │ │ ┌───────────┴────────────┐ │ │ │ │ │ │ "tool_use" "end_turn" │ │ │ │ │ │ ┌────────▼──────┐ ┌──────▼──────┐ │ │ │ Execute tools │ │ Return text │ ← DONE │ │ │ in parallel │ └─────────────┘ │ │ └────────┬──────┘ │ │ │ │ │ Append assistant turn + tool_result blocks │ │ │ │ │ Check: steps > MAX? budget > CAP? ──→ abort │ │ │ │ │ Continue loop ────────────────────────────────────────→ │ │ } │ └─────────────────────────────────────────────────────────────────┘

Send Messages + Tools

POST the current message array with tool definitions attached. The model sees the full conversation history and all available tool schemas on every call.

Parse stop_reason

If stop_reason === "tool_use", one or more tool call blocks are in the response content. If end_turn, the model is done — return the text.

Execute Tool Calls

Run each requested tool (in parallel when safe). Capture results. Errors must be returned as tool_result blocks with is_error: true — never thrown as exceptions.

Append to Thread

Add the assistant turn (with tool_use blocks) and a user turn containing all tool_result blocks. The model needs both sides of the exchange to reason about next steps.

Check Stop Conditions

Evaluate step count, token budget, custom conditions. If any ceiling is hit, abort and surface an error. Never skip this check — it is the only safety net between you and a runaway loop.

Repeat

Send the updated message array back to the model. The model now sees what tools returned and reasons about the next action. Continue until end_turn or abort.

The Model Is Stateless — You Are Not

Every model call is independent. The model has no memory of previous calls unless you include them in the message array you send. This means the entire agent "memory" grows linearly in the message thread on every step — and every subsequent call is more expensive than the last because the context is larger.

Tool Architecture

Tools are the extension points that allow an agent to interact with the world beyond text generation. Architecturally, there are two fundamentally different execution models — and confusing them is a common source of bugs.

Client Tools (User-Defined)

Defined by you. The model returns a tool_use block; your application executes the function and returns a tool_result. Full control, full responsibility.

Database queries, API calls, file reads
M-Pesa STK Push, Supabase inserts
Custom business logic functions
Code execution in your sandboxes

→ stop_reason: "tool_use" — your code runs

Server Tools (Provider-Managed)

Defined by the provider. Anthropic executes them on their infrastructure. Results come back in the same response — no round-trip to your code unless mixed with client tools.

web_search (web_search_20260209)
web_fetch, code_execution
tool_search (MCP discovery)
bash, text_editor (computer use)

→ Results embedded in response — no round-trip needed

Tool Schema — The Contract Between Model and Code

      
// A well-defined tool schema — the description is load-bearing
{
  "name": "charge_customer",
  "description": "Initiates an M-Pesa STK Push payment request to a customer phone number.
               Only call this after confirming the amount with the user. This action
               moves real money — never call it speculatively.",
  "input_schema": {
    "type": "object",
    "properties": {
      "phone_number": { "type": "string", "description": "E.164 format, e.g. +254712345678" },
      "amount_kes":   { "type": "number", "description": "Amount in Kenyan Shillings, integer only" },
      "reference":    { "type": "string", "description": "Idempotency key — reuse for retries" }
    },
    "required": ["phone_number", "amount_kes", "reference"]
  }
}
      
    

Descriptions Are Architecture

The model uses the description field — not the name — to decide when to call a tool. A vague description leads to wrong tool selection, wasted steps, and incorrect behavior. Treat tool descriptions with the same care you give API contracts: they are the interface between natural language intent and deterministic code execution.

Parallel Tool Calls

The model can return multiple tool_use blocks in a single response. These should be executed in parallel when they have no data dependency — reading three database records simultaneously is correct. Updating a record based on a read from the same step is not. The model decides whether to parallelize; your execution layer must handle concurrent dispatch correctly and return all tool_result blocks in a single user turn.

Tool Type	Executes Where	Round-trip to App?	Use Case
Client / User-Defined	Your application	Yes — stop_reason: tool_use	DB, APIs, business logic
Server / web_search	Anthropic infra	No — embedded in response	Real-time web lookup
Server / code_execution	Anthropic infra	No — embedded in response	Python data analysis, math
MCP Tools	MCP server process	Via MCP protocol layer	Shared tools across agents
Harness Tools (bash, editor)	Host machine	Via harness loop	Claude Code, Codex-style

Memory Layers

"Memory" in an agent is not a single thing. It is a stack of layers with different scope, latency, and cost characteristics. Getting the layer boundaries wrong is the most common architectural mistake in agent systems.

// Agent Memory Stack — ordered by scope and cost

┌─────────────────────────────────────────────────────┐ │ LAYER 4 · Long-Term / Persistent │ │ Supabase, Neon, Redis, vector DBs │ │ Scope: across sessions · Cost: DB reads per step │ │ Use: user preferences, durable facts, embeddings │ ├─────────────────────────────────────────────────────┤ │ LAYER 3 · External / MCP │ │ MCP servers: GitHub, Stripe, Sentry, Linear │ │ Scope: cross-agent on same host · Cost: tool call │ │ Use: shared tools & state without re-implementation│ ├─────────────────────────────────────────────────────┤ │ LAYER 2 · Runtime Context │ │ runtimeContext (shared) + toolsContext (per-tool) │ │ Scope: single agent run · Cost: zero (in-process) │ │ Use: tenant ID, feature flags, credentials, state │ ├─────────────────────────────────────────────────────┤ │ LAYER 1 · In-Context (Message Thread) │ │ The messages[] array sent on every model call │ │ Scope: single loop · Cost: tokens on every step │ │ Use: conversation history, tool results, reasoning │ └─────────────────────────────────────────────────────┘

Layer 1 — In-Context Memory (The Message Thread)

Everything the agent has seen this run: the original task, all tool calls, all results, all model reasoning. This is the most immediate memory layer — the model reads it on every call. The cost is that it grows linearly: a 10-step agent might send 50k tokens on step 10 even if the original task was 500 tokens. The context window is the hard ceiling on how long an agent can run without external memory.

Layer 2 — Runtime Context (runtimeContext / toolsContext)

Server-side state that should not be placed in the prompt. The Vercel AI SDK formalizes this as two objects: runtimeContext (shared across the agent loop — tenant ID, request ID, escalation state, progress flags) and toolsContext (scoped per-tool — API keys, account IDs, permissions). The model never sees this data; only your tool execution code does.

        
// runtimeContext flows through the entire agent loop
// toolsContext is scoped: each tool only sees its own slice
const result = await agent.generate({
  prompt: "Find open billing tickets for account acct_123",
  runtimeContext: { requestId: "req_abc", escalated: false },
  toolsContext: {
    searchTickets: { apiKey: process.env.SUPPORT_API_KEY, accountId: "acct_123" }
  }
});
        
      

Layer 3 — MCP (Model Context Protocol)

MCP servers expose tools that any agent framework can consume — Anthropic API, Vercel AI SDK, OpenAI, LangChain — without re-implementing the integration. A single supabase-mcp server gives every agent on the host access to database queries without each agent embedding a Supabase client. MCP decouples tool implementation from agent implementation: update the MCP server, every agent picks up the change automatically.

Layer 4 — External / Persistent Memory

Databases, vector stores, and caches that survive across agent runs and sessions. An agent that needs to remember what a user prefers across conversations must write those preferences to Supabase (or similar) at the end of a run and read them back at the start of the next. Without this layer, every agent run starts from zero — which is the correct behavior for stateless tasks but the wrong behavior for anything user-specific.

In-Context Memory Has a Token Tax

Every byte in Layer 1 costs tokens on every subsequent model call. A common mistake is appending full API responses to the message thread when only a summary is needed. Compress tool results before appending — extract only what the model needs to reason about next — or move durable state to Layer 4 and retrieve it via a tool when needed.

Framework Layer

Three levels of abstraction for building agent loops, each with different trade-offs between control, boilerplate, and feature surface. The right choice depends on whether you need multi-provider flexibility, built-in tool ecosystems, or full protocol visibility.

Approach	Control	Boilerplate	Multi-provider	Built-in tools	Best for
Raw @anthropic-ai/sdk	Full	~50 lines per loop	No	No	Learning the protocol; bespoke loops
Vercel AI SDK ToolLoopAgent	High	~10 lines	Yes	No (you define)	Next.js apps; multi-provider
Claude Agent SDK	Lower	~5 lines	No	Yes (file, bash, web, MCP)	Ops/coding agents with guardrails
HarnessAgent (AI SDK)	Minimal	2 lines	Via harness	Harness-provided	Running Claude Code, Codex as an agent

ToolLoopAgent — The Vercel AI SDK's Answer

As of 2026, the Vercel AI SDK has replaced the deprecated maxSteps pattern with the ToolLoopAgent class. You define the agent once — model, instructions, tools, stop conditions — and call .generate() or .stream(). The class manages the message array, step counting, and context passing internally. Key advantages: reusable across API routes, type-safe tool definitions with Zod schemas, and built-in support for runtimeContext and toolsContext.

        
import { ToolLoopAgent, tool, isStepCount } from 'ai';
import { z } from 'zod';

const billingAgent = new ToolLoopAgent({
  model: "anthropic/claude-sonnet-4-6",
  instructions: "You are a billing assistant. Never charge without explicit confirmation.",
  stopWhen: isStepCount(15),
  tools: {
    getInvoice: tool({
      description: "Retrieve invoice details by ID",
      inputSchema: z.object({ invoiceId: z.string() }),
      execute: async ({ invoiceId }, { context }) =>
        fetchInvoice(invoiceId, context.accountId)
    })
  }
});
        
      

HarnessAgent — Wrapping Established Runtimes

Not every agent starts from a blank model. HarnessAgent lets you run a preconfigured harness — Claude Code, GitHub Copilot Codex, or others — as an agent within the AI SDK's primitives. The harness provides its own tool loop and built-in capabilities (file system access, shell execution, MCP connections). Results stream into standard AI SDK result and UI primitives, so you get observability and streaming without rebuilding the harness internals.

Loop Control

The loop continues until a condition terminates it. The default in Vercel AI SDK's ToolLoopAgent is 20 steps — a conservative safety measure. Every production agent needs explicit, thought-out stop conditions rather than relying on defaults.

isStepCount(n)

Hard step ceiling. Simple and predictable. Use as a backstop even when you have other conditions — prevents unbounded loops when tool errors cause infinite retries.

hasToolCall(...names)

Stop when any of the named tools is invoked. Useful for agents that complete a task by calling a "submit" or "done" sentinel tool instead of naturally finishing.

isLoopFinished()

Let the model run until it naturally stops making tool calls. No hard step limit. Use with extreme caution — only for bounded tasks where you understand the maximum tool call depth.

Custom Conditions

Full access to step history. Inspect accumulated token usage, check for sentinel text in model output, or compare across steps for convergence. Budget-based stopping is the most important custom condition.

      
// Budget-based stop condition — the most important custom condition
const budgetExceeded: StopCondition<typeof tools> = ({ steps }) => {
  const totalUsage = steps.reduce(
    (acc, step) => ({
      inputTokens:  acc.inputTokens  + (step.usage?.inputTokens  ?? 0),
      outputTokens: acc.outputTokens + (step.usage?.outputTokens ?? 0),
    }),
    { inputTokens: 0, outputTokens: 0 }
  );
  // Estimate cost: $3/M input, $15/M output (claude-sonnet-4-6)
  const costUSD =
    (totalUsage.inputTokens * 3 + totalUsage.outputTokens * 15) / 1_000_000;
  return costUSD > 0.50; // abort if cost exceeds $0.50
};

const agent = new ToolLoopAgent({
  model: "anthropic/claude-sonnet-4-6",
  tools,
  stopWhen: [isStepCount(25), budgetExceeded], // whichever fires first
});
      
    

prepareStep — Dynamic Configuration Between Steps

Called before each model invocation in the loop. Receives runtimeContext and the step history. Returns model call overrides for the next step only — temperature, tool subset, system prompt. Use it to escalate model capability mid-run (switch to opus for complex sub-tasks), restrict tool access after certain actions, or modify temperature based on observed uncertainty.

        
prepareStep: async ({ runtimeContext, steps }) => {
  // Escalate to more capable model on step 3+
  if (steps.length >= 3 && runtimeContext.taskComplexity === "high") {
    return { model: "anthropic/claude-opus-4-8", temperature: 0.1 };
  }
  return {};
}
        
      

Guardrails & Safety Architecture

An agent with tools can make irreversible changes to the world. The guardrail layer is not optional — it is the difference between a production agent and a liability.

Non-Negotiable Guardrails

Always cap the loop. Set both a step ceiling and a token/cost ceiling. A stuck agent calling a failing tool will loop until context fills without both guards.
Validate tool inputs at runtime. JSON Schema is a hint to the model, not a guarantee. Validate all inputs before executing — especially before any write, payment, or delete operation.
Return tool errors, never throw. If a tool fails, return a tool_result with is_error: true. Throwing an exception breaks the message thread; the model needs to see the error to recover gracefully.
Design tools to be idempotent. The model may call a tool twice after a partial failure. Store side-effect IDs before acting (CheckoutRequestID, Supabase insert ID) and deduplicate on re-entry.
Gate risky actions behind approval. Any tool that moves money, deploys to production, or deletes data needs explicit human-in-the-loop approval. Never pre-approve these for unattended runs.
Wrap tool execution in timeouts. Use Promise.race with a timeout; return a tool_result error on timeout rather than letting a slow external call block the loop indefinitely.

Tool Approval Architecture (Vercel AI SDK)

Manual Approval Gate

A tool without an execute function stops the loop and surfaces the pending call to your UI. The user approves or denies. On approval, the agent resumes with the result. Implement this for any tool with irreversible side effects — payment, deploy, delete.

Policy-Based Approval (OPA)

Use @ai-sdk/policy-opa to author tool authorization rules as Open Policy Agent Rego policies. Rules evaluate tool name, arguments, and runtimeContext. Approvals become auditable, version-controlled policy files instead of bespoke if/else chains.

// Guardrail layers on a tool call

Model returns tool_use block │ ┌─────────▼──────────────────────────────────────────────┐ │ LAYER 1: Schema Validation │ │ Zod / JSON Schema — validate types, required fields │ │ → reject: return tool_result { is_error: true } │ └─────────┬──────────────────────────────────────────────┘ │ pass ┌─────────▼──────────────────────────────────────────────┐ │ LAYER 2: Policy Check (OPA / allow-list) │ │ Is this tool approved for unattended execution? │ │ → deny: surface approval prompt or abort │ └─────────┬──────────────────────────────────────────────┘ │ approved ┌─────────▼──────────────────────────────────────────────┐ │ LAYER 3: Timeout + Execution │ │ Promise.race(execute(), timeout(30_000)) │ │ → timeout: return tool_result { is_error: true } │ └─────────┬──────────────────────────────────────────────┘ │ result ┌─────────▼──────────────────────────────────────────────┐ │ LAYER 4: Idempotency + Audit Log │ │ Check dedup ID; log call + result; return tool_result │ └────────────────────────────────────────────────────────┘

Cost Reality Check

Agents burn tokens differently from single-turn requests. Context grows with every step, so later steps are more expensive than earlier ones. A 10-step agent with large tool results can spend 10× the cost of a simple request — and the compounding is nonlinear.

Cost Reality · claude-sonnet-4-6 (Current Pricing)

Single-turn request (1k tokens in, 500 out) ~$0.01

5-step agent, compact tool results ~$0.08–0.15

10-step agent, verbose tool results (DB rows, API payloads) ~$0.40–0.80

20-step agent, large context, Claude Opus $2–8+

Prompt caching hit (repeated system prompt) -90% on cached prefix

Compress Tool Results

Never append raw API responses to the message thread. Extract only what the model needs to reason about next. A 50-row database result can usually be summarized in 5 lines for the model's purposes.

Right-Size the Model

Use claude-haiku for high-frequency tool dispatch (looking up values, simple transforms). Reserve claude-sonnet for reasoning steps. Step up to claude-opus only for genuinely complex sub-tasks. Use prepareStep to switch dynamically.

Cache the System Prompt

Long system prompts (tool descriptions, persona, instructions) repeat on every step. Anthropic prompt caching gives a 90% discount on the cached prefix. Mark stable prefixes with cache_control: { type: "ephemeral" }.

Set Budget Stop Conditions

Track cumulative usage.inputTokens + outputTokens across steps. Abort and surface a "budget exceeded" error if you cross your ceiling. Do not trust a fixed step count alone — context size varies too much per step to predict total cost from steps alone.

Parallelize Safely

When the model returns multiple tool calls with no data dependency, execute them concurrently. Parallel execution cuts wall-clock time without increasing token cost — you still send one model request per loop iteration.

Move State Out of Context

Large durable facts (user history, config, all previous results) should live in Layer 4 and be retrieved via a tool when needed — not pre-loaded into the system prompt. In-context state costs tokens on every subsequent step.

Key Takeaways

1

An agent is a model in a loop with tools — not an AI brain making autonomous decisions. Every architectural problem reduces to: message array management, tool design, and loop control.
2

Tool descriptions are architecture. The model uses the description field to decide when and how to call a tool. Vague descriptions cause wrong tool choices, wasted steps, and production incidents. Treat them as API contracts.
3

Never run a loop without a step ceiling AND a budget ceiling. One limits steps, the other limits cost. Either alone is insufficient — context size varies per step, making step count an unreliable proxy for token cost.
4

Memory has four layers. In-context is fast but expensive and ephemeral. runtimeContext/toolsContext is zero-cost and in-process. MCP enables cross-agent tool reuse. Persistent storage (Supabase/Neon) is required for anything that must survive a session boundary.
5

Tool errors must be returned as tool_result blocks with is_error: true — never thrown as exceptions. Thrown exceptions break the message thread. The model needs to see the error to recover, retry with different arguments, or escalate.
6

Idempotency is non-negotiable for any tool with side effects. The model can call a tool twice — after a partial failure, after a network timeout, or simply because it chose to re-verify. Store side-effect IDs before acting and deduplicate on re-entry.
7

Framework choice is a control-vs-boilerplate trade-off. Raw SDK gives full protocol visibility at the cost of 50+ lines per loop. ToolLoopAgent reduces that to 10 lines with multi-provider flexibility. HarnessAgent wraps established runtimes (Claude Code, Codex) as first-class agents. Pick based on what you need to control.
8

Context grows linearly with steps — later steps are more expensive than earlier ones. Compress tool results before appending to the thread. Move durable state to external storage. Use prompt caching on stable system prompts. The difference between a $0.10 agent run and a $5 run is usually context management, not model choice.

AI Agent Runtime Architecture

Tools, Memory, and Control Planes