gateway
A blazing fast AI Gateway with integrated guardrails. Route to 1,600+ LLMs, 50+ AI Guardrails with 1 fast & friendly API.
An open source AI gateway that routes requests to 250+ models from a single interface, adding automatic retries, fallbacks, load balancing, and content guardrails with under one millisecond of overhead.
Portkey AI Gateway is an open source router that sits between your application and AI model providers. Instead of writing separate code to talk to OpenAI, Anthropic, Google, or any other AI service, your application talks to the gateway using a single consistent interface, and the gateway handles the conversation with whichever provider you choose.
The gateway supports over 250 AI models from dozens of providers, and connecting to a new one is a matter of changing a few configuration values rather than rewriting integration code. The library claims a latency overhead of under one millisecond, which makes it practical to run in production without noticeably slowing down responses.
Beyond routing, the gateway adds reliability and safety features. You can configure it to retry failed requests automatically, fall back to a different model if the primary one is unavailable, and spread load across multiple providers or model instances. Guardrails let you define rules about what the AI is allowed to return, for example blocking responses that contain certain words or topics, and rejecting them before they reach your users.
Running the gateway locally takes a single command using Node.js. A built-in console at localhost shows a log of all requests passing through. The gateway can also be deployed on Docker, Cloudflare Workers, or cloud infrastructure on AWS, Azure, and GCP. Enterprise deployments with private hosting and additional access controls are available through Portkey's commercial offering.
The project works with popular AI development libraries including Langchain, LlamaIndex, and CrewAI, and is compatible with the standard OpenAI client format so teams already using that interface can switch over without changing their existing code structure.
Where it fits
- Route your app's AI requests through one endpoint to switch between OpenAI, Anthropic, and Google without changing your code.
- Configure automatic fallback to a backup model if your primary AI provider goes down.
- Add guardrails that block unwanted content in AI responses before they reach your users.
- Deploy the gateway on Cloudflare Workers or AWS for low-latency AI routing in production.