Background Jobs & Queues

Pathrule2 Rules • 2 Memories • 1 Skill

Queue-backed work is delivered at-least-once, which means every job can and eventually will run more than once. This pattern bundles the rules, memories, and review checklist that keep your workers idempotent, retried with backoff and jitter, drained on dead-letter queues you actually watch, and enqueued atomically with the data they describe.

Suggested path map

Pathrule places each piece on the matching path, so your assistant only sees it where it belongs. This is the scoping you get on import; you can adjust it in your workspace.

/ workspace root

background-jobs-queues-review

src/

jobs/

Every job handler must be idempotent

Enqueue atomically with the transactional outbox

workers/

Retry with backoff and route permanent failures to a DLQ

queues/

BullMQ conventions for this service

Rules

Every job handler must be idempotent/src/jobshighstrictAssume at-least-once delivery: re-running a job with the same input must not change the final state.

1	Queue delivery is at-least-once, so write every handler so the same input produces the same end state whether it runs once or five times.
2
3	- Carry a stable idempotency key on the job payload (for example `orderId` or a UUID minted at enqueue time), not a value generated inside the worker.
4	- Guard side effects with a unique constraint or a dedup record (`INSERT ... ON CONFLICT DO NOTHING`, or a `processed_jobs` row) checked inside the same transaction as the work.
5	- Make external calls idempotent too: pass the same key to providers (Stripe `Idempotency-Key`, conditional writes) so a redelivery is a no-op.
6	- Keep the dedup record's TTL longer than the queue's full retry window, or a late redelivery will slip past the guard.

Retry with backoff and route permanent failures to a DLQ/src/workershighadvisoryUse exponential backoff with jitter, a bounded attempt count, and a dead-letter queue that is monitored.

1	Retries must back off and stop, and exhausted jobs must land somewhere a human will see them.
2
3	- Configure `attempts` plus `backoff: { type: 'exponential', delay }` in BullMQ; add jitter so synchronized failures do not retry in lockstep.
4	- Classify errors: throw `UnrecoverableError` for permanent failures (malformed payload, missing referenced row) so they skip the remaining attempts instead of burning them.
5	- On final failure move the job to a dead-letter queue and emit a metric or alert; never let it sit silently in the `failed` set forever.
6	- Set `removeOnComplete` and a bounded `removeOnFail` so Redis does not grow unbounded, but keep enough failed history to debug and replay.

Memories

BullMQ conventions for this service/src/queuesVersions, scheduler API, and connection rules we standardize on for BullMQ.

1	We run BullMQ v5.x (current line as of mid-2026) on a dedicated Redis instance, separate from the cache.
2
3	- Use `Worker` and `Queue` from `bullmq`; share one `ioredis` connection per process with `maxRetriesPerRequest: null` (required by BullMQ blocking commands).
4	- Schedule recurring work with `queue.upsertJobScheduler(schedulerId, repeat, template)`; the old `Repeat`/`add({ repeat })` API is deprecated and removed in v6.
5	- Set `Worker` `concurrency` deliberately and add a rate limiter (`limiter: { max, duration }`) for jobs that call rate-limited providers.
6	- Job IDs are the dedup unit: pass a deterministic `jobId` to drop duplicate enqueues, and remember BullMQ keeps a completed job's ID only while its record lives.
7	- See /src/workers for retry and DLQ policy and /src/jobs for the idempotency contract every handler must honor.

Enqueue atomically with the transactional outbox/src/jobsNever enqueue inside a DB transaction; write an outbox row and let a relay publish it.

1	Enqueueing to Redis or SQS from inside a database transaction is a dual-write bug: the commit can succeed while the enqueue fails, or vice versa, leaving data and jobs out of sync.
2
3	- Within the business transaction, insert an `outbox` row describing the job instead of calling the queue directly. The row commits atomically with the data it depends on.
4	- A relay worker polls the outbox with `SELECT ... FOR UPDATE SKIP LOCKED` to claim a batch, enqueues each job, and marks the row dispatched. Multiple relays can run in parallel without double-claiming.
5	- Because the relay can crash after enqueue but before marking the row, dispatch is at-least-once too. That is fine: the downstream job is already idempotent (see /src/jobs idempotency rule).
6	- Use a stable outbox row ID as the job's idempotency key so a re-dispatched row maps to the same job.

Skills

background-jobs-queues-review/rootPre-merge checklist for any new or changed queue job, worker, or enqueue path.

1	---
2	name: background-jobs-queues-review
3	description: Review checklist for background jobs and queues. Use before merging any new or changed job handler, worker configuration, scheduler, or enqueue path to confirm idempotency, retry/backoff policy, dead-letter handling, and atomic enqueue.
4	---
5
6	# Background Jobs & Queues review
7
8	- [ ] The handler is idempotent: the same payload produces the same end state when run more than once.
9	- [ ] A stable idempotency key rides on the payload (not generated inside the worker) and guards side effects via a unique constraint or dedup record.
10	- [ ] The dedup record's TTL outlives the queue's full retry window.
11	- [ ] `attempts` is bounded and `backoff` is exponential with jitter, not a fixed or zero delay.
12	- [ ] Permanent failures throw `UnrecoverableError` (or equivalent) so they skip remaining retries instead of burning them.
13	- [ ] Exhausted jobs move to a dead-letter queue and emit a metric or alert; nothing fails silently.
14	- [ ] `removeOnComplete` and a bounded `removeOnFail` are set so Redis does not grow without limit.
15	- [ ] Recurring jobs use `upsertJobScheduler`, not the deprecated `Repeat`/`add({ repeat })` API.
16	- [ ] The Redis connection uses `maxRetriesPerRequest: null` and worker `concurrency`/rate limits are set deliberately.
17	- [ ] Enqueues that depend on a DB write go through a transactional outbox, not a direct enqueue inside the transaction.
18	- [ ] Workers shut down gracefully (`worker.close()`) on SIGTERM so in-flight jobs finish or requeue cleanly on deploy.

Why this pattern

Queues deliver at-least-once, so naive jobs run twice and corrupt data, retry storms hammer dependencies, and failed jobs vanish silently.

Built for Backend and platform teams running Node.js queue workers (BullMQ, SQS, RabbitMQ) behind an API or event stream..

Keeps your assistant from:

Duplicate side effects from a job that re-runs after a retry or redelivery
Retry storms that hammer a downstream dependency because there is no jitter or failure-type classification
Permanently failed jobs disappearing into the failed set with no dead-letter queue or alert

View as markdown

License: Apache-2.0
Version: 1.0.0
Updated: 2026-06-09