Prompt2Effect

Prompt2Effect is a research code release that explores how to automatically generate special visual effects for AI generated videos just from a written description, instead of needing a separately trained model for every single effect. It does this by training a smaller helper neural network, called a HyperNetwork, that reads a text prompt describing an effect and predicts a small set of weight adjustments that can be applied to a larger existing video generation model to produce that effect.

This particular codebase reproduces the method on Wan2.1, a publicly available image to video generation model, using a collection of about fifty existing effect specific weight sets shared by another team as reference examples to learn from. The authors note that results may differ somewhat from the original research paper since this is a reproduction built on a different underlying model.

The project describes two stages of training. The first stage, which is the main focus of this repository and follows the original paper closely, trains the HyperNetwork to reproduce the behavior of those fifty reference effect weight sets directly, which the authors describe as fast and stable. The second stage goes further than the original paper: it takes the first stage's result, applies its predicted weights into the frozen video model, and trains the whole system end to end using the video model's own generation loss on real video and effect pairs. The authors are explicit that this second stage is an experimental extension that has not yet been verified to work correctly from start to finish.

Using this project requires access to the specific Wan2.1 model weights and the fifty reference effect weight files, along with a Python environment with the listed dependencies installed. The included scripts cover preparing text embeddings for each effect, running the first stage of training, running the experimental second stage, and generating a video with the resulting effect model. A roadmap section lists releasing the trained first stage weights and supporting files as still planned, upcoming work.