resque

Resque is a tool for Ruby applications that moves slow, heavy tasks to the background so your app stays fast. If your website needs to do something time-consuming—like building a file archive, updating a search index, or warming a cache—you don't want the user staring at a loading screen while it happens. Instead, Resque lets you put that work on a to-do list, and background "workers" handle it later.

At a high level, the system works like a kitchen order line. When your app needs to do a slow task, it writes down a small job ticket and hands it to a queue. Workers act like line cooks: they pick up tickets, do the work, and check for more. You can have multiple queues for different types of work, and you can run workers across multiple machines to handle the load.

Anyone building a Ruby on Rails app or similar Ruby service would use this. For example, if you run an e-commerce site, you might want to generate a large monthly sales report. Instead of making an admin wait for the page to load, you queue the report generation. A worker builds the report in the background and emails it when it's done. At GitHub, where this tool originated, they use it for dozens of tasks like building downloadable archives and firing off webhooks.

A notable design choice is how the system expects things to break. The workers are built to assume that background jobs will occasionally lock up, run too long, or consume too much memory. To handle this, a worker creates a separate child process for each job. When the job is done, the child process exits, freeing up memory. If a job goes off the rails, the system can easily kill the misbehaving job without skipping a beat.

It also includes a built-in web dashboard. This gives you a simple interface to see what your workers are currently doing, which queues are backing up, and—crucially—a place to review and retry jobs that failed.