OpenAI Codex, the cloud-based coding agent released by OpenAI, relies on a layered system architecture that extends well beyond its underlying AI model. The core model, codex-1, is a version of OpenAI’s o3 fine-tuned for software engineering tasks. However, the engineering team said the most complex challenges involved building the orchestration layer around that model.
The Agent Loop at the Core of OpenAI Codex
At the center of Codex is an agent loop. The agent takes user input, constructs a prompt, sends it to the model, and receives a response. That response is often a tool call rather than a final answer. The agent then executes the tool call, appends the output to the prompt, and queries the model again. This cycle repeats until the model produces a message for the user.
A single request such as “fix the bug in the auth module” may trigger the agent to read multiple files, run tests, edit code, resolve linting errors, and run tests again before producing a final commit. The model handles reasoning at each step, while the harness manages command execution, output collection, permission handling, and loop termination.
Prompt Construction and Context Window Management
Each user message sits at the bottom of a layered prompt. Above it, the system stacks environment context, AGENTS.md file contents, sandbox permission rules, developer configuration instructions, tool definitions, and a system message. Each layer carries a role — system, developer, or user — that signals its priority to the model.
The total JSON sent to the API grows quadratically across a conversation. OpenAI chose not to use server-side state storage, because doing so would break request statelessness and prevent support for customers requiring Zero Data Retention. As a result, every request carries the full conversation history.
The primary mitigation is prompt caching. Because Codex always appends new content to the end of the existing prompt, the old prompt is always an exact prefix of the new one. This prefix property allows OpenAI to reuse computation from previous inference calls. However, the prefix is fragile. Changing models, tools, or sandbox configuration breaks the cache. When OpenAI added support for MCP tools, a bug caused tools to list in inconsistent order between requests, which alone destroyed cache hits.
When conversations exceed the context window limit, Codex compacts the conversation. It replaces the full history with a smaller representative version that preserves the model’s understanding through an encrypted payload carrying the model’s latent state. The OpenAI engineering team stated that managing the context window is a first-class engineering problem, not an afterthought.
AGENTS.md Files and Project-Specific Context
AGENTS.md files allow developers to place project-specific instructions directly in their repositories. These files tell Codex how to navigate the codebase, which commands to run for testing, and how to follow project conventions. The model performs better with them but also functions without them. This design keeps project knowledge in the repository rather than hardcoded into the system.
The App Server: Enabling Multi-Surface Deployment
Codex started as a command-line tool. OpenAI subsequently needed it to operate inside VS Code, a web application, a macOS desktop app, and third-party IDEs including JetBrains and Xcode. The team first attempted to expose Codex as an MCP server. However, MCP’s semantics could not support the full interaction patterns the agent required, including streaming progress, mid-task approval pauses, and structured diff output.
Consequently, OpenAI built the App Server. All core agent logic — the agent loop, thread management, tool execution, configuration, and authentication — lives in a single codebase called “Codex core.” The App Server wraps this core in a JSON-RPC protocol that any client can use over standard input/output. The protocol is fully bidirectional. The client can send requests to the server, and the server can send requests back to the client, such as asking for approval before executing a shell command.
The VS Code extension and the desktop app bundle the App Server binary and launch it as a child process. The web app runs the App Server inside a cloud container, streaming events to the browser over HTTP. State lives on the server, so work continues even if the user closes the browser tab. Partners such as Xcode decouple their release cycles from OpenAI’s by pointing their stable clients at newer App Server binaries as they become available.
Current Limitations and Future Direction
OpenAI Codex currently cannot accept image inputs for frontend work. Users cannot course-correct the agent mid-task. Moreover, delegating to a remote AI agent takes longer than interactive editing, and that workflow shift requires adjustment. OpenAI said the team is working toward a model where interacting with Codex resembles asynchronous collaboration, though the gap between that vision and the current product remains significant.
“The model is a component and the agent is the system. Most of the engineering is in the system.”
OpenAI Engineering Team
The App Server architecture was not planned from the start. It evolved from a CLI, through a failed MCP attempt, to the protocol that now underpins every Codex surface. OpenAI’s engineering team said this trajectory illustrates a broader principle in system design: the right abstraction usually does not exist until developers have tried the wrong one.