agentops
Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI
A Python SDK that adds monitoring and debugging to AI agent workflows in two lines of code, it records every LLM call, cost, and step so you can replay and inspect what your agent did.
AgentOps is a Python toolkit for monitoring and debugging AI agents: programs that use large language models to take sequences of actions or make decisions autonomously. When you build something with a framework like CrewAI, LangChain, or OpenAI's Agents SDK, it can be hard to understand why it behaved a certain way, how much it cost, or where it went wrong. AgentOps records everything that happens during a run so you can review it afterward.
Adding it to an existing project takes two lines of code: one to initialize it with an API key and one to mark when a session ends. From there, AgentOps automatically captures each call made to AI model providers, what was sent and received, how long it took, and what it cost. You can also add decorators to your own functions and classes to mark them as agents, tasks, or operations, which lets the dashboard show a structured breakdown of the full execution flow.
The web dashboard lets you replay a session step by step, see a graph of how different parts of the program called each other, and view cost summaries across many sessions. This is useful for catching cases where an agent looped unnecessarily, where a tool call failed silently, or where spending spiked unexpectedly.
AgentOps integrates with most major AI agent frameworks and model providers out of the box, including CrewAI, AutoGen, LangChain, Cohere, and LlamaIndex. The dashboard and backend are open-source under the MIT license and can be self-hosted on your own infrastructure if you do not want to send data to AgentOps' cloud service.
It installs via pip and works with Python. Documentation is at docs.agentops.ai.
Where it fits
- Add session recording to a CrewAI or LangChain agent and replay each step to debug why it behaved unexpectedly.
- Track LLM spend across hundreds of agent runs and find sessions where cost spiked unexpectedly.
- Catch silent tool-call failures by reviewing the full execution graph in the AgentOps dashboard after a run.