metaflow
Build, Manage and Deploy AI/ML Systems
A Python framework from Netflix for building and running AI and machine learning workflows that scales from laptop experiments to cloud production automatically, tracking every run, artifact, and model version along the way.
Metaflow is a Python framework built at Netflix for developing and running AI and machine learning systems. It is designed to cover the full lifecycle of a project: starting from experimenting in a notebook on your laptop, scaling up to large parallel jobs on cloud computing clusters, and eventually deploying to a production scheduler that runs the workflow automatically. Today it is maintained by a company called Outerbounds and is used in production at organizations including Amazon, DoorDash, Goldman Sachs, and Ramp. Netflix itself runs over 3,000 AI and machine learning projects on it.
The core idea is that you write Python code defining a workflow as a series of steps. Each step can run locally for fast testing or be pushed to remote cloud compute automatically, including GPU-enabled machines. Metaflow tracks every run, every artifact, and every model version so you can look back at past experiments or compare results across runs. It also manages dependencies so that the Python environment your code needs travels with it into the cloud.
Deploying a finished workflow to production is done with a single command. Metaflow integrates with production-grade orchestration systems that handle scheduling and reliability, and it can react to external events to trigger new runs automatically. It works on AWS, Azure, and GCP.
Installation is through pip or conda. The README points to a free interactive sandbox at outerbounds.com/sandbox as the quickest way to try it without local setup, and links to a tutorial that walks through creating your first workflow step by step. A Slack community is available for support.
Where it fits
- Write a machine learning workflow once and have it run locally for fast testing, then automatically scale to cloud GPU machines for full training.
- Track every experiment run, artifact, and model version so you can compare past results or reproduce any run exactly.
- Deploy a finished ML pipeline to production with a single command so it runs on a schedule or reacts to external events.
- Manage Python environment dependencies automatically so the same code runs identically on your laptop and in the cloud.