maestro

Maestro is Netflix's internal tool for scheduling and running large-scale data and machine learning workflows, now released as open source. At Netflix it schedules hundreds of thousands of workflows and millions of individual jobs every day, serving data scientists, engineers, content producers, and business analysts across the company.

The core idea is workflow orchestration: you define a sequence of steps that need to run in a specific order, and Maestro handles running them, tracking their progress, dealing with retries, and making sure everything completes reliably even when traffic spikes. Netflix describes it as a workflow-as-a-service platform, meaning teams use it without having to manage the underlying infrastructure themselves.

The system is built in Java and uses Spring Boot as its foundation. Getting it running locally requires Java 21, Gradle, and Docker. From there you build it with Gradle, start the server, and interact with it through a REST API. The README walks through curl commands to create a sample workflow, trigger it to run, and check its status. There is also a module for running Maestro with AWS services (using SQS and SNS for event messaging), and a separate extension service that handles additional features like foreach step views.

Netflix has published several technical blog posts explaining the design in depth, including one describing a 100x improvement to the workflow engine's speed and another covering integration with Apache Iceberg for incremental data processing. Those posts provide the full architecture context that the README itself does not repeat.

The project is intended for teams running data pipelines at scale who want a managed orchestration layer rather than building scheduling logic themselves.