OpenRLHF-GRPO

OpenRLHF-GRPO implements a training algorithm called Group Relative Policy Optimization, or GRPO, built on top of an existing framework named OpenRLHF. GRPO is used to align large language models, meaning it adjusts a model's behavior to better match desired outputs, and the README notes it is the core alignment method used by DeepSeek-R1. Unlike the more common PPO approach, GRPO does not require training a separate reward model. Instead, it samples several responses to the same prompt, ranks them against each other within that group, and uses the relative ranking to calculate how much to adjust the policy directly.

The core addition in this repository is a new trainer file of around 600 lines that implements the full GRPO training process inside the OpenRLHF codebase. It removes the reward model dependency in favor of the in-group ranking approach, supports a configurable group size such as 4, 8, or 16 sampled responses per prompt, and includes a tunable penalty for how far the updated policy drifts from the original one. A one-command shell script starts training and integrates with DeepSpeed's ZeRO-2 and ZeRO-3 memory optimization stages.

The README reports benchmark comparisons against a PPO baseline: GPU memory use down 35 percent, time per training iteration down 40 percent, overall training time 1.8 times faster, and a win rate of 96.2 percent against the PPO baseline's 100 percent on the AlpacaEval 2.0 benchmark, meaning GRPO trains substantially faster and cheaper while giving up a small amount of measured quality.

Getting started means installing the listed Python dependencies with pip and running the provided training script. The repository is small, consisting mainly of the trainer implementation, one example script, and a requirements file, and it links to the original DeepSeek-R1 paper and the OpenRLHF project for background. It is released under the Apache 2.0 license.