bullmq

BullMQ is a job queue library that lets applications hand off work to be done later or in the background, rather than doing everything immediately in response to a user action. The typical scenario is something like: a user submits a form that triggers a slow task, such as sending emails, resizing images, or generating a report. Instead of making the user wait, the application puts a job into a queue and a separate worker process picks it up and does the work at its own pace. BullMQ handles the queue, the delivery of jobs to workers, retries if a job fails, and notifications when jobs complete.

It is built on top of Redis, which is an in-memory data store commonly used for caching and fast key-value operations. Redis provides the persistence and coordination that makes the queue reliable: jobs are not lost if a worker crashes, and multiple workers can pull from the same queue without doing the same job twice. The library is described as fast and designed with atomicity in mind, meaning operations either fully complete or do not happen at all.

BullMQ supports a variety of patterns beyond a simple first-in-first-out queue. Jobs can be scheduled to run at a future time, given priorities, grouped into batches, or organized into parent-child trees where a parent job waits for its child jobs to finish before it runs. There is also support for rate limiting, concurrency controls, and job progress tracking.

The primary language is TypeScript for Node.js, and the README points to separate client libraries for Python, Elixir, and PHP applications that want to add jobs to the same queue. For teams who want a visual way to monitor queues, the company behind BullMQ offers a paid web dashboard called Taskforce.sh. The core library itself is open source. Organizations including Microsoft and NestJS are listed as users in the README.