vllm

vLLM is a library for running and serving large language models efficiently. The README describes it as fast and easy-to-use, focused on high serving throughput and memory-efficient inference. The project was originally developed in the Sky Computing Lab at UC Berkeley and has grown into a community project.

The fast side comes from several techniques. PagedAttention manages the memory used by the model's attention keys and values more efficiently than naive approaches. Continuous batching keeps the GPU busy by mixing incoming requests together, with chunked prefill and prefix caching as further optimizations. The engine supports many quantization formats including FP8, INT8, INT4, GPTQ/AWQ, and GGUF, several optimized attention kernels such as FlashAttention and Triton, and speculative decoding methods like n-gram, suffix, and EAGLE.

The flexible side is about how you actually use it. vLLM integrates with Hugging Face models, supports tensor, pipeline, data, expert, and context parallelism for distributed inference, streams output, generates structured outputs, supports tool calling, and provides an OpenAI-compatible API server plus an Anthropic Messages API and gRPC. It runs on NVIDIA and AMD GPUs and x86/ARM/PowerPC CPUs, with hardware plugins for Google TPUs, Intel Gaudi, Huawei Ascend, Apple Silicon, and others. It claims support for over 200 model architectures, including decoder-only LLMs like Llama, Qwen, and Gemma, mixture-of-expert models like Mixtral and DeepSeek-V3, multimodal models, and embedding models.

Someone would use vLLM to host an LLM behind an API for an application or research project. The library is written in Python and installs via pip or uv.