llama.cpp
LLM inference in C/C++
Run large language models like LLaMA and Mistral locally on your own computer or server with minimal setup, strong performance, and no external dependencies.
llama.cpp is a tool for running large language models (LLMs) on your own computer or server, written in plain C and C++ with no external dependencies. The main goal is to let you run these models with minimal setup while getting strong performance across a wide range of hardware, whether that is a laptop, a desktop with a graphics card, or a cloud server. It supports many model families including LLaMA, Mistral, Falcon, Gemma, Phi, Qwen, and numerous others.
The project places special emphasis on Apple silicon, treating Mac computers with M-series chips as a first-class platform through optimizations for ARM processors. It also supports modern x86 CPU instruction sets, RISC-V architectures, and several GPU platforms including NVIDIA (via CUDA), AMD (via HIP), and Intel (via SYCL), plus a Vulkan backend for broad graphics card compatibility. A notable feature is CPU and GPU hybrid inference, which lets you run models that are larger than your graphics card memory by splitting the work between the processor and the graphics card.
To reduce memory usage and speed up performance, llama.cpp supports several levels of integer quantization, ranging from 1.5-bit to 8-bit. This means models can be compressed to use far less memory than their original size while still producing useful output. The project also serves as the primary testing ground for developing the ggml library, which is the underlying machine learning framework.
Getting started involves installing the software through a package manager like Homebrew or winget, downloading pre-built binaries, or building from source. Once installed, you need a model file in the GGUF format. You can run a local model file directly, or download and run a model from Hugging Face. The project includes a command-line interface called llama-cli and a server component called llama-server that provides an API compatible with OpenAI's interface, so existing applications can connect to it.
Recent additions include multimodal support in the server component, a web-based user interface, WebGPU support for running models in a browser, and editor plugins for VS Code and Vim that provide code completion. The full README is longer than what was shown.
Where it fits
- Run a local chat server with an OpenAI-compatible API for your apps.
- Run large models that exceed your GPU memory by splitting work across CPU and GPU.
- Add code completion to VS Code or Vim using a local model.
- Compress large models using quantization to save memory while running locally.