juicefs

JuiceFS is a distributed file system that lets applications read and write files using familiar file-system operations, while the actual data is stored in cloud object storage (such as Amazon S3) and metadata is stored in a database like Redis or MySQL. The goal is to let you use massive amounts of cloud storage as if it were a local disk attached to your machine, without changing your existing application code.

The system has three parts: a client that runs on the machine reading and writing files, an object storage backend that holds the actual file data, and a metadata engine that tracks file names, sizes, directory structure, and other attributes. Files are split into chunks and blocks before being stored in object storage, which is why the stored files do not look like your original files when you browse the storage bucket directly.

JuiceFS is compatible with POSIX, which means most Linux and macOS programs that work with files can use it without any code changes. It also provides a Hadoop-compatible Java SDK for use with big data tools, an S3-compatible gateway for tools that expect an S3 API, and a Kubernetes driver for container workloads. Multiple clients can mount the same file system at once, and changes made by one client are immediately visible to others.

Key technical properties include low-latency access, data compression using LZ4 or Zstandard, encryption in transit and at rest, file locking compatible with standard POSIX and BSD lock conventions, and the ability to scale throughput by adding more object storage capacity.

Getting started requires choosing a metadata engine (Redis is the most common), configuring object storage, and then installing the JuiceFS client. The project is released under the Apache 2.0 license and offers documentation, a quick-start guide, and a Discord community.