Why we need Job Queues
In Node.js, the event loop is your most precious resource. If you process a 5MB image or generate a PDF inside an API route, your server will stop responding to other users. Job queues like BullMQ solve this by offloading work to background workers while providing 100% durability.
Atomicity and Lua Scripts
BullMQ is superior to custom Redis-list implementations because it uses **Lua scripts** for every operation. When you move a job from `waiting` to `active`, it's an atomic transaction. Even if your worker crashes mid-move, the job isn't lost—it stays in Redis until the `lockDuration` expires, and then it's automatically retried.
The Power of Sandboxed Workers
The biggest mistake developers make is running jobs in the same process as the worker. If a job has a memory leak or blocks the CPU, it kills the worker.
The professional solution is Sandboxed Workers. By providing a path to a separate file (e.g., `worker.js`), BullMQ spawns a child process for each job. This isolates the main worker process, allowing it to remain responsive and heartbeating with Redis even if the job itself is under extreme load.
BullMQ Flows: Orchestrating Trees
Imagine a video platform: you need to Encode, Generate Thumbnails, and then Notify User.
Using `FlowProducer`, you can add a tree of jobs. The \"Notify User\" job (parent) remains in a `waiting-children` state and is only triggered by Redis once all encoder and thumbnail jobs (children) have returned a success state.