huey

Huey is a lightweight task queue library for Python. A task queue is a system that lets your application hand off work to be done in the background, rather than making a user wait while the server completes a slow operation. You mark a Python function as a task, and when you call it, the function is added to a queue and processed separately by a worker process.

The library supports several storage backends for holding queued tasks: Redis (and its forks Valkey and Redict), SQLite, the file system, or plain in-memory storage. Redis is the primary option the project was designed around, but the others exist for situations where Redis is not available or not desired.

Beyond basic queuing, Huey supports scheduling tasks to run at a specific future time or after a set delay, and setting up recurring tasks that run on a schedule similar to cron jobs. It also handles automatic retries when a task fails, task priorities, storing the results of completed tasks so you can retrieve them later, timeouts, rate limiting, and more advanced patterns like chains (one task runs after another) and fan-out groups where one task triggers many others.

The worker process that actually runs the queued tasks is started separately from your application using a command-line tool included with the library. You can configure it to run multiple worker processes, threads, or lightweight greenlets depending on whether your tasks are mostly CPU-heavy or network/IO-heavy.

Huey's API is designed to be simple: you add a decorator to a function, and calling that function queues it instead of running it immediately. The README includes short code examples showing how to queue a function, retry on failure, schedule a future task, and set up a nightly recurring job. Documentation is available at huey.readthedocs.io.