persistent-rnn
Fast Recurrent Networks Library
A C++/CUDA library that makes small-batch recurrent neural network computation up to 15x faster on specific NVIDIA GPUs by caching weights in GPU registers.
What This Project Does
This is a high-performance library that speeds up a specific type of AI computation called recurrent neural networks (RNNs) on NVIDIA GPUs. If you're training or running models that process sequences—like text, time-series data, or video—this library can make that computation dramatically faster, especially when you're processing small batches of data at once.
The key insight behind the project is clever use of GPU memory. Modern GPUs have thousands of tiny processing cores, each with its own fast memory called a register file. Most RNN libraries ignore this and keep reloading the same weights (the learned parameters) from slower main memory over and over. This library instead keeps those weights cached in the register files, so they don't need to reload. The result: about 15 times faster performance compared to standard GPU libraries when processing small batches (like 4 samples at a time).
Who Would Use This
You'd want this library if you're building systems that run RNNs on GPUs in scenarios where batch sizes are small. Real-world examples: an inference server responding to individual user requests in real-time, a robotics system processing sensor streams, or any application where you care more about latency than throughput. If you're already comfortable using CUDA and GPU libraries, integrating this should feel familiar—the API is designed to match the style of NVIDIA's standard cuDNN library.
Important Constraints
The README is honest about tradeoffs. This library only supports basic RNN layers, not the more advanced variants like LSTMs that are common in modern models. It also only works on a few specific older NVIDIA GPU models (TitanX, GeForce 1080, GP100), has limits on how large a layer can be, and requires your data to be formatted in a particular way. These constraints reflect the library's narrow focus: extreme performance for one specific use case, rather than broad compatibility. If your needs don't fit these constraints, you'd need to use a more general-purpose library instead.
Where it fits
- Speed up a real-time inference server that processes RNN requests one at a time
- Run sequence models on a robotics system processing live sensor streams with low latency
- Replace a standard cuDNN RNN call with this library for small-batch, latency-sensitive workloads
- Benchmark GPU register-caching techniques against standard RNN libraries