text-generation-inference
Large Language Model Text Generation Inference
Server toolkit for running open-source large language models on your own hardware and serving them as an API. Handles batching and multi-GPU splitting to serve many users efficiently. Now in maintenance mode.
Text Generation Inference (TGI) is a server toolkit from Hugging Face for running large language models and making them available as an API. Large language models are the kind of AI that generates text, answers questions, and carries on conversations. TGI is the software that Hugging Face used internally to power its own chat and API products.
The main purpose is speed and efficiency. Running a large AI model is computationally expensive, and TGI includes several techniques to serve many users at once without wasting resources. It can split a model across multiple graphics cards to handle models too large for a single one, process many incoming requests together in batches, and stream responses token by token so users see output appearing in real time rather than waiting for the full response.
It is designed to work with popular open-source models including Llama, Falcon, and others. You start a server by pointing TGI at a model, and it exposes a web API that other programs can call to get text generated. The API format is compatible with OpenAI's chat format, so software already written for OpenAI can switch to a self-hosted model without major changes.
The quickest way to start is with Docker: you pull the official container, tell it which model to load, and it handles everything else. Hardware support covers Nvidia GPUs, AMD GPUs, Intel GPUs, and some specialized accelerators.
The README notes that TGI is now in maintenance mode. The Hugging Face team recommends newer inference engines like vLLM or SGLang for new projects going forward.
Where it fits
- Self-host a Llama or Falcon model on your own GPU server and expose it as an OpenAI-compatible API for your applications.
- Spread a large model across multiple GPUs to serve it when it does not fit on a single card.
- Stream LLM responses token by token to users so they see output appearing in real time rather than waiting.