agentspec

AgentSpec is a testing framework built specifically for AI agents, the way Jest is a testing framework for regular JavaScript code. The problem it solves is that AI agents behave differently every time you change a prompt, swap a model, or update a tool, and those shifts are hard to predict or catch before they reach production. AgentSpec lets you write tests in a YAML file that check the agent's behavior and catch regressions early.

After installing it globally with npm, running agentspec init creates a starter YAML file with example tests, and agentspec run executes them. Because AI output is not exact text every time, the assertions are built for that reality: you can check that a response contains certain words, does not contain others, matches one of several possible phrases, matches a regular expression, calls a specific tool, or is semantically similar to an expected answer. It can also check JSON structure and response latency.

One notable feature is an LLM as judge mode, where a local model such as one running through Ollama evaluates whether a response meets a written quality description, with no API costs and full privacy since nothing leaves your machine. AgentSpec also keeps the last passing output for each test, so when a test starts failing it can show you a behavior diff report of exactly what words were added or removed compared to the previous passing run.

AgentSpec can test any agent reachable over HTTP by pointing it at an endpoint that accepts a JSON input and returns a JSON output. It fits into continuous integration with a CI flag that sets an exit code on failure, plus JUnit XML and JSON output formats, and there is an official GitHub Action for running it in a pipeline. The project is released under the MIT license.