tensorzero

TensorZero is an open-source platform that helps software teams manage how their applications interact with AI language models. Instead of connecting directly to providers like OpenAI, Anthropic, or Google, you point your code at TensorZero and it handles the routing, logging, and improvement cycles from there. The whole thing runs as a single container you deploy yourself, so your data stays in your own database.

The gateway is the starting point. It accepts requests using the same format as the OpenAI API, which means switching to it often requires just a one-line change in existing code. From there it forwards requests to over a dozen AI providers, including AWS Bedrock, Google Vertex AI, Groq, Mistral, and several others. If one provider is slow or unavailable, TensorZero can retry or fall back to a different model automatically. Because the gateway is written in Rust, it adds less than one millisecond of overhead even at high traffic volumes.

Observability is built in from the start. Every AI call, and any feedback you attach to it such as ratings, corrections, or scores, gets stored in your own database. You can browse individual calls in the TensorZero UI or pull the data programmatically for your own analysis. This makes it possible to see not just whether the AI produced a correct answer, but how performance changes over time or across different models and prompts.

On top of that, TensorZero includes tools for evaluation and experimentation. You can run A/B tests across different prompts or models, set up automated scoring to benchmark outputs, and track usage costs at a fine-grained level. There is also a feature called Autopilot, described as an automated AI engineer: it reads the observability data, sets up evaluations, and runs optimization cycles to improve prompt and model performance without manual intervention.

TensorZero is compatible with any OpenAI SDK (Python, Node, Go, and others). Setup involves deploying the Docker container, then changing one URL in your client code. The project is used by companies ranging from early-stage AI startups to large enterprises and, according to the README, handles roughly one percent of global AI API spend.