The Problem with In-Memory Async Tasks
Running heavy tasks,video encoding, report generation, or multi-step AI calls,inside HTTP request lifecycles blocks the Node.js event loop and risks silent task failure whenever a server restarts. BullMQ coupled with Redis provides an atomic, distributed queue architecture built for 100% durability.
1. Atomic Transitions via Lua Scripts
BullMQ avoids race conditions across distributed worker clusters by executing all queue state transitions through Redis Lua scripts. Moving a job from waiting to active, renewing a lock token, or failing a job is evaluated atomically on the Redis server, guaranteeing that no job is executed twice by concurrent workers.
2. Sandboxed Workers: CPU Isolation
Standard BullMQ workers run in the same Node.js process as your application. If a task hits an unhandled exception or blocks the CPU with heavy computation, it can freeze queue heartbeats and mark healthy jobs as stalled.
Sandboxed workers solve this by isolating jobs into dedicated child processes. The parent worker coordinates with Redis while child processes execute the workload:
import { Worker } from "bullmq";
import path from "node:path";
// Sandboxed worker points to an isolated file executed in a child process
const processorPath = path.join(__dirname, "report-processor.js");
const worker = new Worker("reportQueue", processorPath, {
connection: { host: "127.0.0.1", port: 6379 },
concurrency: 5,
useWorkerThreads: true,
});
worker.on("completed", (job) => {
console.log(`Job ${job.id} completed successfully`);
});3. Retries, Backoff, and Dead Letter Queues
Network glitches and third-party rate limits are inevitable in production. BullMQ provides built-in exponential backoff:
await queue.add("generateReport", data, {
attempts: 4,
backoff: {
type: "exponential",
delay: 2000, // 2s, 4s, 8s, 16s
},
removeOnComplete: 1000,
removeOnFail: false, // Keep failed jobs for telemetry investigation
});
