loki

Loki is a system for collecting, storing, and searching logs from software running across many machines. The pitch in the description is "like Prometheus, but for logs": Prometheus is a popular tool for tracking numerical metrics over time, and Loki applies the same style of thinking to log lines instead. It is built to be horizontally scalable, highly available, and multi-tenant, meaning many teams or projects can share one installation.

The key idea that makes Loki different from other log tools is that it does not perform full text indexing on the contents of the logs. Instead, it stores the logs in a compressed form and only indexes a small set of labels for each log stream. That keeps it simpler to operate and cheaper to run, because building and storing a full text index is one of the most expensive parts of a typical log system. The labels Loki uses are the same kind of labels you already use with Prometheus, so you can move back and forth between metrics and logs using a shared vocabulary. It is described as a particularly good fit for Kubernetes pod logs, where labels are picked up automatically.

A Loki-based stack has three pieces: an agent called Alloy that gathers logs and ships them in, Loki itself which stores the logs and answers queries, and Grafana as the front end you use to query and view the results. You would reach for it when you are running production services and want a logging backend that integrates naturally with a Prometheus-and-Grafana setup without the cost of a full-text log search engine. It is written in Go.