ACEStep ORG

ACE-Step
A Step Towards Music Generation Foundation Model

Project |
Hugging Face |
ModelScope |
Space Demo |
Discord |
Technical Report |
ACE-Step v1.5

Table of Contents

• [✨ Features](#-features)
• [📦 Installation](#-installation)
• [🚀 Usage](#-usage)
• [📱 User Interface Guide](#-user-interface-guide)
• [🔨 Train](#-train)

📝 Abstract

We introduce ACE-Step, a novel open-source foundation model for music generation that overcomes key limitations of existing approaches and achieves state-of-the-art performance through a holistic architectural design. Current methods face inherent trade-offs between generation speed, musical coherence, and controllability. For instance, LLM-based models (e.g., Yue, SongGen) excel at lyric alignment but suffer from slow inference and structural artifacts. Diffusion models (e.g., DiffRhythm), on the other hand, enable faster synthesis but often lack long-range structural coherence.

ACE-Step bridges this gap by integrating diffusion-based generation with Sana’s Deep Compression AutoEncoder (DCAE) and a lightweight linear transformer. It further leverages MERT and m-hubert to align semantic representations (REPA) during training, enabling rapid convergence. As a result, our model synthesizes up to 4 minutes of music in just 20 seconds on an A100 GPU—15× faster than LLM-based baselines—while achieving superior musical coherence and lyric alignment across melody, harmony, and rhythm metrics. Moreover, ACE-Step preserves fine-grained acoustic details, enabling advanced control mechanisms such as voice cloning, lyric editing, remixing, and track generation (e.g., lyric2vocal, singing2accompaniment).

Rather than building yet another end-to-end text-to-music pipeline, our vision is to establish a foundation model for music AI: a fast, general-purpose, efficient yet flexible architecture that makes it easy to train sub-tasks on top of it. This paves the way for developing powerful tools that seamlessly integrate into the creative workflows of music artists, producers, and content creators. In short, we aim to build the Stable Diffusion moment for music.

📢 News and Updates

• 🎉 2026.01.28: Released ACE-Step v1.5 - Our latest and most advanced model is now available!

• 📃 2025.06.02: Released ACE-Step Technical Report (PDF).

• 🎮 2025.05.14: Add Stable Audio Open Small sampler pingpong. Use SDE to achieve better music consistency and quality, including lyric alignment and style alignment. Use a better method to re-implement Audio2Audio

• 🎤 2025.05.12: Release RapMachine and fix lora training issues

- See [ZH_RAP_LORA.md](./ZH_RAP_LORA.md) for details. Audio Examples: https://ace-step.github.io/#RapMachine - See [TRAIN_INSTRUCTION.md](./TRAIN_INSTRUCTION.md) for detailed training instructions.

• 🔥 2025.05.10: Memory Optimization Update

- Reduced Max VRAM to 8GB, making it more compatible with consumer devices - Recommended launch options:

bash
    acestep --torch_compile true --cpu_offload true --overlapped_decode true

Windows need to install triton:

pip install triton-windows


• 📢 2025.05.09: Graidio Demo support Audio2Audio. ComfyUI: [Ace_Step_4x_a2a.json](./assets/Ace_Step_4x_a2a.json)

• 🚀 2025.05.08: ComfyUI_ACE-Step node is now available! Explore the power of ACE-Step within ComfyUI. 🎉

!image

• 🚀 2025.05.06: Open source demo code and model

✨ Features

🎯 Baseline Quality

🌈 Diverse Styles & Genres

• 🎸 Supports all mainstream music styles with various description formats including short tags, descriptive text, or use-case scenarios
• 🎷 Capable of generating music across different genres with appropriate instrumentation and style

🌍 Multiple Languages

• 🗣️ Supports 19 languages with top 10 well-performing languages including:

- 🇺🇸 English, 🇨🇳 Chinese, 🇷🇺 Russian, 🇪🇸 Spanish, 🇯🇵 Japanese, 🇩🇪 German, 🇫🇷 French, 🇵🇹 Portuguese, 🇮🇹 Italian, 🇰🇷 Korean

• ⚠️ Due to data imbalance, less common languages may underperform

🎻 Instrumental Styles

• 🎹 Supports various instrumental music generation across different genres and styles
• 🎺 Capable of producing realistic instrumental tracks with appropriate timbre and expression for each instrument
• 🎼 Can generate complex arrangements with multiple instruments while maintaining musical coherence

🎤 Vocal Techniques

• 🎙️ Capable of rendering various vocal styles and techniques with good quality
• 🗣️ Supports different vocal expressions including various singing techniques and styles

🎛️ Controllability

🔄 Variations Generation

• ⚙️ Implemented using training-free, inference-time optimization techniques
• 🌊 Flow-matching model generates initial noise, then uses trigFlow's noise formula to add additional Gaussian noise
• 🎚️ Adjustable mixing ratio between original initial noise and new Gaussian noise to control variation degree

🎨 Repainting

• 🖌️ Implemented by adding noise to the target audio input and applying mask constraints during the ODE process
• 🔍 When input conditions change from the original generation, only specific aspects can be modified while preserving the rest
• 🔀 Can be combined with Variations Generation techniques to create localized variations in style, lyrics, or vocals

✏️ Lyric Editing

• 💡 Innovatively applies flow-edit technology to enable localized lyric modifications while preserving melody, vocals, and accompaniment
• 🔄 Works with both generated content and uploaded audio, greatly enhancing creative possibilities
• ℹ️ Current limitation: can only modify small segments of lyrics at once to avoid distortion, but multiple edits can be applied sequentially

🚀 Applications

🎤 Lyric2Vocal (LoRA)

• 🔊 Based on a LoRA fine-tuned on pure vocal data, allowing direct generation of vocal samples from lyrics
• 🛠️ Offers numerous practical applications such as vocal demos, guide tracks, songwriting assistance, and vocal arrangement experimentation
• ⏱️ Provides a quick way to test how lyrics might sound when sung, helping songwriters iterate faster

📝 Text2Samples (LoRA)

• 🎛️ Similar to Lyric2Vocal, but fine-tuned on pure instrumental and sample data
• 🎵 Capable of generating conceptual music production samples from text descriptions
• 🧰 Useful for quickly creating instrument loops, sound effects, and musical elements for production

🔮 Coming Soon

🎤 RapMachine

• 🔥 Fine-tuned on pure rap data to create an AI system specialized in rap generation
• 🏆 Expected capabilities include AI rap battles and narrative expression through rap
• 📚 Rap has exceptional storytelling and expressive capabilities, offering extraordinary application potential

🎛️ StemGen

• 🎚️ A controlnet-lora trained on multi-track data to generate individual instrument stems
• 🎯 Takes a reference track and specified instrument (or instrument reference audio) as input
• 🎹 Outputs an instrument stem that complements the reference track, such as creating a piano accompaniment for a flute melody or adding jazz drums to a lead guitar

🎤 Singing2Accompaniment

• 🔄 The reverse process of StemGen, generating a mixed master track from a single vocal track
• 🎵 Takes a vocal track and specified style as input to produce a complete vocal accompaniment
• 🎸 Creates full instrumental backing that complements the input vocals, making it easy to add professional-sounding accompaniment to any vocal recording

📋 Roadmap

• [x] Release training code 🔥
• [x] Release LoRA training code 🔥
• [x] Release RapMachine LoRA 🎤
• [x] Release evaluation performance and technical report 📄
• [ ] Train and Release ACE-Step V1.5
• [ ] Release ControlNet training code 🔥
• [ ] Release Singing2Accompaniment ControlNet 🎮

🖥️ Hardware Performance

We have evaluated ACE-Step across different hardware setups, yielding the following throughput results:

| Device | RTF (27 steps) | Time to render 1 min audio (27 steps) | RTF (60 steps) | Time to render 1 min audio (60 steps) |
| --------------- | -------------- | ------------------------------------- | -------------- | ------------------------------------- |
| NVIDIA RTX 4090 | 34.48 × | 1.74 s | 15.63 × | 3.84 s |
| NVIDIA A100 | 27.27 × | 2.20 s | 12.27 × | 4.89 s |
| NVIDIA RTX 3090 | 12.76 × | 4.70 s | 6.48 × | 9.26 s |
| MacBook M2 Max | 2.27 × | 26.43 s | 1.03 × | 58.25 s |

We use RTF (Real-Time Factor) to measure the performance of ACE-Step. Higher values indicate faster generation speed. 27.27x means to generate 1 minute of music, it takes 2.2 seconds (60/27.27). The performance is measured on a single GPU with batch size 1 and 27 steps.

📦 Installation

1. Clone the Repository

First, clone the ACE-Step repository to your local machine and navigate into the project directory:

bash
git clone https://github.com/ace-step/ACE-Step.git
cd ACE-Step

2. Prerequisites

Ensure you have the following installed:

• Python: Version 3.10 or later is recommended. You can download it from python.org.
• Conda or venv: For creating a virtual environment (Conda is recommended).

3. Set Up a Virtual Environment

It is highly recommended to use a virtual environment to manage project dependencies and avoid conflicts. Choose one of the following methods:

Option A: Using Conda

1. Create the environment named ace_step with Python 3.10:

bash
    conda create -n ace_step python=3.10 -y

2. Activate the environment:

bash
    conda activate ace_step

Option B: Using venv

1. Navigate to the cloned ACE-Step directory.

2. Create the virtual environment (commonly named venv):

bash
    python -m venv venv

3. Activate the environment: * On Windows (cmd.exe):

bash
        venv\Scripts\activate.bat

* On Windows (PowerShell):

powershell
        .\venv\Scripts\Activate.ps1

*(If you encounter execution policy errors, you might need to run Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope Process first)* * On Linux / macOS (bash/zsh):

bash
        source venv/bin/activate

4. Install Dependencies

Once your virtual environment is activated: a. (Windows Only) If you are on Windows and plan to use an NVIDIA GPU, install PyTorch with CUDA support first:

bash
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126

(Adjust cu126 if you have a different CUDA version. For other PyTorch installation options, refer to the official PyTorch website). b. Install ACE-Step and its core dependencies:

bash
pip install -e .

The ACE-Step application is now installed. The GUI works on Windows, macOS, and Linux. For instructions on how to run it, please see the [Usage](#-usage) section.

🚀 Usage


🔍 Basic Usage

bash
acestep --port 7865

⚙️ Advanced Usage

bash
acestep --checkpoint_path /path/to/checkpoint --port 7865 --device_id 0 --share true --bf16 true

• If --checkpoint_path is set and models exist at the path, load from checkpoint_path.
• If --checkpoint_path is set but models do not exist at the path, auto download models to checkpoint_path.
• If --checkpoint_path is not set, auto download models to the default path ~/.cache/ace-step/checkpoints.

If you are using macOS, please use --bf16 false to avoid errors.

🔍 API Usage

If you intend to integrate ACE-Step as a library into your own Python projects, you can install the latest version directly from GitHub using the following pip command.

Direct Installation via pip:

1. Ensure Git is installed: This method requires Git to be installed on your system and accessible in your system's PATH. 2. Execute the installation command:

bash
    pip install git+https://github.com/ace-step/ACE-Step.git

It's recommended to use this command within a virtual environment to avoid conflicts with other packages.

🛠️ Command Line Arguments

• --checkpoint_path: Path to the model checkpoint (default: downloads automatically)
• --server_name: IP address or hostname for the Gradio server to bind to (default: '127.0.0.1'). Use '0.0.0.0' to make it accessible from other devices on the network.
• --port: Port to run the Gradio server on (default: 7865)
• --device_id: GPU device ID to use (default: 0)
• --share: Enable Gradio sharing link (default: False)
• --bf16: Use bfloat16 precision for faster inference (default: True)
• --torch_compile: Use torch.compile() to optimize the model, speeding up inference (default: False).

- Windows need to install triton:

pip install triton-windows

• --cpu_offload: Offload model weights to CPU to save GPU memory (default: False)
• --overlapped_decode: Use overlapped decoding to speed up inference (default: False)

📱 User Interface Guide

The ACE-Step interface provides several tabs for different music generation and editing tasks:

📝 Text2Music Tab

1. 📋 Input Fields:
- 🏷️ Tags: Enter descriptive tags, genres, or scene descriptions separated by commas
- 📜 Lyrics: Enter lyrics with structure tags like [verse], [chorus], and [bridge]
- ⏱️ Audio Duration: Set the desired duration of the generated audio (-1 for random)

2. ⚙️ Settings:
- 🔧 Basic Settings: Adjust inference steps, guidance scale, and seeds
- 🔬 Advanced Settings: Fine-tune scheduler type, CFG type, ERG settings, and more

3. 🚀 Generation: Click "Generate" to create music based on your inputs

🔄 Retake Tab

• 🎲 Regenerate music with slight variations using different seeds
• 🎚️ Adjust variance to control how much the retake differs from the original

🎨 Repainting Tab

• 🖌️ Selectively regenerate specific sections of the music
• ⏱️ Specify start and end times for the section to repaint
• 🔍 Choose the source audio (text2music output, last repaint, or upload)

✏️ Edit Tab

• 🔄 Modify existing music by changing tags or lyrics
• 🎛️ Cho

…

Collapse