Building AI agents is a challenge that PostHog spent two years navigating, releasing its first “AI product assistant” six months after starting and relaunching it as PostHog AI in November 2025. Ian Vanagas, a team member at PostHog, documented five key lessons the company wished it had known from the start.
Should You Build an MCP Server Instead?
Before committing to a custom artificial intelligence agent, PostHog suggests evaluating whether a Model Context Protocol (MCP) server would suffice. The company said 34% of dashboards created by AI were generated through its MCP server, accounting for 18% of all dashboards created. MCP servers are simpler to build and require less maintenance, making them suitable when users are developers or when validating demand for agent capabilities.
The Agent Harness Is Not a Competitive Advantage
PostHog iterated through three harness architectures before settling on the Claude Agent SDK with MCP tools. The first used a coordinator routing messages to sub-agents, which caused context loss. The second used a single agent loop with 44 tools, which did not scale. The third and current version uses a code-execution sandbox, giving the agent flexibility without requiring the team to build every capability manually.
Building AI Agents Around Context Creates a Real Advantage
PostHog said context is the most important differentiator when building AI agents. The company structures its context through MCP tools, markdown skill files, layered runtime context injection, a taxonomy tool, and a memory onboarding flow. These components allow the agent to understand both the product and individual user goals over time.
“Reality has a surprising amount of detail and people define tasks in ambiguous ways.”
Ian Vanagas, PostHog
Observability and Evaluation Must Start on Day One
PostHog said it regretted not setting up observability tools early. The company now recommends tracing every LLM call with inputs, outputs, latency, and cost, along with trace IDs spanning full conversations and automated scorers. The team also runs a regular “traces hour” session where members manually review real user interactions to identify areas for improvement.
User Experience Matters More Than Capabilities
PostHog found that the most common user pain points were not related to the range of AI capabilities. Instead, users reported inconsistent performance, unexpected failures, generic error messages, unclear capabilities, and a lack of visible uncertainty or progress indicators. The company said addressing these issues proved more valuable than adding new features.
Vanagas stated that building an AI agent is ultimately a product engineering problem, not a technical showcase. The company said teams should talk to users, ship features users want, and iterate continuously to build an agent that solves real problems. PostHog’s experience over two years shows that building AI agents requires as much focus on product fundamentals as on model capabilities.
