pebble
RocksDB/LevelDB inspired key-value database in Go
Pebble is the storage engine powering CockroachDB, a Go library that handles reading and writing key-value data to disk with crash recovery, compaction, and point-in-time snapshots.
Pebble is a storage engine, which is the part of a database system responsible for actually reading and writing data to disk. It is built in Go by the team behind CockroachDB, a distributed database. Pebble takes inspiration from two well-known storage engines: LevelDB and RocksDB, meaning it shares their general approach of organizing data into sorted layers on disk, but it rewrites that approach in Go with a narrower feature set tuned for CockroachDB's specific needs.
At its core, Pebble stores key-value pairs, which are simple pairs of a lookup key and its associated data. Applications can read and write individual entries, scan ranges of keys, take snapshots of the data at a point in time, and delete ranges of entries efficiently. The library handles all the complexity of keeping data consistent on disk, managing write-ahead logs to survive crashes, and periodically compacting data to keep reads fast.
Pebble does not try to replicate every feature from RocksDB. It explicitly leaves out things like column families, certain table formats, and transaction support. The README warns that using Pebble with a database originally created by RocksDB can silently corrupt data if that database used features Pebble does not support, so mixing them is not recommended.
The project has been used in production inside CockroachDB since 2020 and became the default storage engine in CockroachDB version 20.2. It ships with nightly benchmark results published publicly so performance regressions can be caught early. The source code is written in Go and is open source.
Where it fits
- Embed Pebble as the storage layer in a custom Go database to get crash-safe reads and writes without building it yourself.
- Use Pebble's range scan API to efficiently iterate over sorted keys in a Go application.
- Take point-in-time snapshots so reads see a consistent view of data even while writes are happening.
- Replace LevelDB or RocksDB as the storage backend in a Go project that needs a production-tested alternative.