Your Demo Agent Works. Your Production Agent Won’t.
Everyone can build an AI agent in a weekend. Slap an LLM API call into a loop, give it a search tool, deploy it on Vercel. Done. Ship it.
Then reality hits.
Your agent hallucinates financial data. It executes rm -rf / in your server. It leaks customer PII across tenant boundaries. It burns through $2,000 in tokens overnight because nobody set a limit.
The gap between a demo agent and a production agent isn’t code — it’s infrastructure. And in 2026, that infrastructure has a name: the Agent Harness.
“Agent = Model + Harness. If you’re not the model, you’re the harness.”
— Vivek Trivedy, LangChain
This post breaks down the 9 essential pillars every production AI agent needs. No fluff. No “it depends.” Just the concrete building blocks — drawn from what companies like AWS, Anthropic, LangChain, and the open-source community have battle-tested in production.
Pillar 1: The Agent Harness (Orchestration Layer)
The harness is everything that isn’t the model. It’s the while-loop, the state machine, the if tool_call then execute logic. It turns a stateless text-completion API into a reasoning engine.
What it does:
- Manages the agent’s ReAct loop (Reason → Act → Observe → Repeat)
- Routes between nodes in a state graph (LangGraph, AWS Step Functions)
- Handles context window management, compaction, and continuation
- Controls tool selection and execution order
Key technologies:
- LangGraph — Stateful, cyclic graph execution with built-in checkpointing. Compiles your agent once at startup, injects tools and LLM at invocation time. Used in production by LinkedIn, Uber, and 400+ companies.
- AWS Bedrock AgentCore Runtime — Serverless agent deployment with session isolation, supporting everything from low-latency conversations to 8-hour asynchronous workloads.
- Deep Agents (LangChain) — Open-source, model-agnostic harness with filesystem, planning, subagents, and middleware built in.
The golden rule: Start simple. Anthropic’s own recommendation: “Find the simplest solution possible, and only increase complexity when needed.” Most production agents are a ReAct loop — not a 47-node orchestration graph.
Pillar 2: Tools & MCP (The Agent's Hands)
A model without tools is a philosopher. It thinks beautifully, but it can’t do anything.
Tools are how agents interact with the outside world — search the web, read databases, send emails, create documents. And in 2026, Model Context Protocol (MCP) has become the universal standard for connecting agents to tools.
What production tool systems need:
- Dynamic assembly — Tools injected per-request based on agent config, not baked into the graph
- MCP servers — Standardized protocol for tool discovery and execution (40+ pre-built: GitHub, Slack, Notion, Jira, etc.)
- Tool descriptions matter more than you think — Anthropic spent more time optimizing tools than prompts for their SWE-bench agent
- AWS AgentCore Gateway — Converts APIs and Lambda functions into agent-compatible tools, connects to MCP servers, and enables semantic tool discovery
Pro tip: Design your Agent-Computer Interface (ACI) as carefully as you’d design a Human-Computer Interface (HCI). If a junior developer would be confused by your tool description, the model will be too.
Pillar 3: Sandbox & Code Execution (The Safe Room)
Your agent will generate and execute code. That code will be wrong sometimes. It might be dangerous sometimes. You need a blast radius of zero.
What it looks like in production:
- Isolated Docker containers per execution with no network access by default
- Whitelisted commands — only what the agent needs
- Resource limits — CPU, memory, time, disk
- AWS AgentCore Code Interpreter — Secure sandboxed execution across Python, JavaScript, and more
- Browser sandboxes — For agents that need to interact with web UIs (AgentCore Browser, Playwright)
The anti-pattern: Running bash_tool on the same server as your database. One prompt injection and your data is gone.
The pattern: Every code execution spawns an ephemeral container that lives and dies with the task. The agent writes to a temporary filesystem, results are extracted, and the container is destroyed.
Pillar 4: Memory (The Agent's Brain Across Time)
Without memory, every conversation starts from zero. Your agent forgets the user’s name, their preferences, and the last 47 messages from yesterday.
Memory in production agents operates at three levels:
| Level | Scope | Example |
|---|---|---|
Short-term |
Current conversation |
Chat history in the context window |
Working |
Current task |
Files, intermediate results, plan state |
Long-term |
Cross-session |
User preferences, learned patterns, AGENTS.md |
Key insight from Harrison Chase (LangChain): “Memory isn’t a plugin — it’s the harness.” How memory loads, compacts, summarizes, and persists is a core harness responsibility, not an add-on.
What production memory needs:
- Sliding window + summarization — Don’t just truncate, compress intelligently
- Filesystem-backed memory — AGENTS.md, CLAUDE.md patterns for persistent context
- AWS AgentCore Memory — Managed memory service that maintains context across interactions with industry-leading accuracy
- Checkpointing — LangGraph’s AsyncPostgresSaver stores every state transition for recovery and debugging
Warning: If you use a closed-source harness behind a proprietary API, you don’t own your memory. When you switch models, your memory stays behind — locked inside someone else’s platform.
Pillar 5: Knowledge Base & RAG (The Agent's Library)
Memory is what the agent has learned. Knowledge is what the agent can look up.
Production RAG requires:
- Vector databases (Qdrant, Pinecone, pgvector) for semantic search
- Hybrid search — Combine vector similarity with keyword BM25 for better recall
- Chunking strategy — Not all documents are created equal. Code, PDFs, and markdown need different chunking approaches
- Citation tracking — Users need to trust agent responses. Numbered citations [1], [2] inline are table stakes
The common mistake: Dumping everything into one vector index and hoping embedding search handles it. It won’t. Production RAG needs metadata filtering, re-ranking, and source-aware retrieval.
Pillar 6: Authorization & Identity (The Bouncer)
Who is using the agent? What are they allowed to do? And whose credentials does the agent use when it calls Slack?
There are two fundamentally different authorization models for agents:
- On-behalf-of — The agent acts as the user. Alice’s agent sees Alice’s Notion pages. Bob’s agent sees Bob’s. The agent inherits user permissions.
- Agent-as-identity — The agent has its own credentials and permissions, independent of who is chatting. Everyone gets the same view. Think: a public-facing support bot.
Production authorization stack:
- Keycloak / Auth0 / WorkOS — Identity provider for user authentication (OIDC, JWT)
- Workspace-level RBAC — Admin, Editor, Member roles controlling who can create agents vs. use them
- AWS AgentCore Identity — Managed agent identity for secure access to AWS resources and third-party tools with pre-authorized user consent
- Per-tool credential scoping — The agent should never have broader access than the task requires
The nightmare scenario: An agent with admin credentials exposed via a chatbot. One clever prompt = full access to your internal systems.
Pillar 7: Guardrails & Policies (The Safety Net)
Guardrails are the constraints that prevent your agent from doing something catastrophically wrong.
Types of guardrails:
- Input guardrails — Screen user queries for prompt injection, jailbreaks, inappropriate content before the model sees them
- Output guardrails — Validate responses before they reach the user. No PII? No hallucinated URLs? No medical/legal advice?
- Action guardrails — Gate dangerous tool calls behind human approval. Sending an email? Creating a Jira ticket? Deleting a file? Ask first.
- Budget guardrails — Max tokens per request, max tool calls per session, monthly spend limits per agent
AWS AgentCore Policy intercepts tool calls in real-time using Cedar policies defined in natural language. “The agent cannot delete production databases” becomes an enforceable rule, not a hope.
Anthropic’s key insight: Run guardrails as a separate model instance in parallel with the primary response. Having the same LLM handle both the response and its own safety check performs worse than two separate calls.
Pillar 8: Middleware & Hooks (The Control Plane)
Middleware is the glue between harness components. It lets you inject deterministic behavior at specific points in the agent’s execution without modifying the core loop.
Common middleware patterns:
- Pre-LLM hooks — Inject system prompt layers, format context, apply prompt caching
- Post-LLM hooks — Log responses, evaluate quality, trigger alerts
- Pre-tool hooks — Validate tool arguments, check permissions, rate-limit
- Post-tool hooks — Parse tool outputs, truncate large responses, write to audit log
- Compaction hooks — Triggered when context window fills up, summarize and continue
- Continuation hooks (Ralph Loop) — Intercept the model’s exit attempt and reinject the original prompt for long-horizon tasks
LangGraph implementation: Middleware can be added as graph edges, conditional routing, or node wrappers. Deep Agents has a built-in middleware system for customization.
The beauty of middleware: your core agent logic stays clean. All the messy stuff — logging, rate limiting, content filtering, token counting — lives in composable layers around it.
Pillar 9: Observability & Evaluation (The Eyes)
If you can’t see what your agent is doing, you can’t fix it. And agents will need fixing.
Production observability includes:
- Tracing — Full execution path: every LLM call, every tool invocation, every state transition. LangSmith, Langfuse, and OpenTelemetry are the standards.
- Token accounting — Prompt tokens, completion tokens, total cost per invocation. Break this down by agent, by user, by time period.
- Latency tracking — End-to-end response time, per-tool latency, LLM inference time
- Error rates — Tool failures, LLM errors, timeout rates, rate limit hits
- Evaluation — Automated evals on correctness, groundedness, relevance. Run this continuously, not just before launch.
AWS AgentCore provides:
- Evaluations — Sample and score live interactions using built-in and custom evaluators
- Observability dashboards — Powered by CloudWatch with OpenTelemetry integration
Harrison Chase’s insight: “Traces are the core.” Everything — continual learning, harness optimization, memory improvement — starts with having complete traces of what your agent actually did.
The Production Checklist
Before you ship your agent, run through this:
TL;DR
The model is the intelligence. The harness makes that intelligence useful. And the harness has 9 pillars:
- Harness — The orchestration loop (LangGraph, Deep Agents)
- Tools & MCP — How agents interact with the world
- Sandbox — Safe code execution in isolated containers
- Memory — Short-term, working, and long-term context
- Knowledge Base — RAG with vector search and citations
- Authorization — User identity and agent permissions
- Guardrails — Safety nets for input, output, actions, and budget
- Middleware — Composable hooks for deterministic control
- Observability — Tracing, metrics, evaluation
Skip any one of these and you’ll learn the hard way.
Building a production AI agent platform? We’re working on this exact problem at [OpenPrime AI](https://openprime.ai) — an open platform for building, deploying, and managing enterprise AI agents with all 9 pillars built in.
