llama.cpp

llama.cpp is a tool for running large language models, or LLMs, on your own computer or server. An LLM is the type of AI system that generates human-like text, similar to ChatGPT. Normally these models require heavy cloud infrastructure and specialized hardware. This project aims to make that process possible with minimal setup, using plain C and C++ code with no external software dependencies.

The project supports a very wide range of hardware. It is specifically optimized for Apple Silicon chips, like those in modern Macs, using Apple's own performance frameworks. It also supports x86 processors found in most Windows and Linux computers, including advanced instruction sets like AVX512 and AMX. For RISC-V architecture chips, several optimization features are available as well. If you have a graphics card from NVIDIA, AMD, or Moore Threads, the project includes custom kernels to run models faster on those GPUs. There is also support for Vulkan and SYCL, which are alternative graphics and compute platforms.

A key feature is quantization. This means the project can compress models to use less memory and run faster, at several precision levels ranging from 1.5-bit up to 8-bit integer formats. This compression allows large models to run on machines with limited memory. There is also a hybrid mode that splits the work between your CPU and GPU, which helps when a model is too large to fit entirely on the graphics card alone.

The project includes a command-line tool called llama-cli that loads a model file and lets you interact with it. There is also llama-server, which starts a local web server that mimics the OpenAI API, so other software can connect to it as if it were talking to OpenAI's services. You can install the software through package managers like Homebrew or conda-forge, download pre-built binaries from the releases page, or build it from source. The server also supports multimodal input, meaning it can process images alongside text. There is a new web interface for the server as well.

llama.cpp supports a long list of model families, including LLaMA, Mistral, Falcon, Gemma, Qwen, Phi, and many others. Editor plugins exist for VS Code, Vim, and Neovim that use llama.cpp for code completion as you type. The full README is longer than what was shown.