What to Monitor When AI Agents Run in Production

Deploying an AI agent to production is the beginning of an observability problem, not the end of a development one. An agent that works correctly in testing will encounter edge cases in production — unexpected inputs, APIs returning unusual responses, plans that technically execute but produce the wrong outcome.

Most teams underinvest in agent monitoring because it's not obvious what to watch. AI agents don't throw exceptions the way deterministic code does. They fail silently — generating a plan that looks plausible but takes the wrong path, or completing all steps successfully while producing a result nobody wanted.

The most important thing to capture is the full plan the agent generated — before execution, not just after. If something goes wrong and you only have the final result, you're debugging backward. If you have the plan, you can see exactly what the agent decided to do, which steps it chose, what dependencies it set up, and where the reasoning went sideways.

Plan logs should be structured: each step as a named task with its inputs and its dependency chain. Unstructured plan logs that capture the model's reasoning as a text blob are hard to query and hard to correlate with specific outcomes.

For each task in a plan, log the exact inputs passed and the exact output returned. This sounds obvious but is often skipped for performance or storage reasons. When a plan fails or produces an unexpected result, step-level input/output logs are what let you pinpoint the exact moment things went wrong.

Pay particular attention to outputs from external systems — API responses that are malformed, empty, or contain unexpected data. These often cause downstream steps to silently misbehave rather than fail loudly.

Over time, aggregate plan logs reveal patterns. Which tasks fail most often? Which task combinations produce the most approval rejections? Which input types cause the agent to generate unusual plans? These patterns are signals about where your registry definitions, your prompts, or your API schemas need improvement.

A task that fails 30% of the time in production tells you something important — either the schema is ambiguous, the API it calls behaves unexpectedly, or the agent is being asked to use it in ways it wasn't designed for.

Traditional monitoring alerts on errors. Agent monitoring should also alert on anomalies — plans that are unusually long, tasks called an unexpected number of times, outputs that fall outside normal ranges, approval rates that spike or drop. These can be signs of prompt injection, unusual input, or an agent stuck in a loop.

Set a baseline during your initial production period when you know the agent is behaving correctly. Deviations from that baseline — even if they don't produce explicit errors — are worth investigating before they become incidents.