# Pathrule Pattern: Background Jobs & Queues (1.0.0) # ::pathrule:package:background-jobs-queues ### [RULE] Every job handler must be idempotent (path: /src/jobs) Queue delivery is at-least-once, so write every handler so the same input produces the same end state whether it runs once or five times. - Carry a stable idempotency key on the job payload (for example `orderId` or a UUID minted at enqueue time), not a value generated inside the worker. - Guard side effects with a unique constraint or a dedup record (`INSERT ... ON CONFLICT DO NOTHING`, or a `processed_jobs` row) checked inside the same transaction as the work. - Make external calls idempotent too: pass the same key to providers (Stripe `Idempotency-Key`, conditional writes) so a redelivery is a no-op. - Keep the dedup record's TTL longer than the queue's full retry window, or a late redelivery will slip past the guard. --- ### [RULE] Retry with backoff and route permanent failures to a DLQ (path: /src/workers) Retries must back off and stop, and exhausted jobs must land somewhere a human will see them. - Configure `attempts` plus `backoff: { type: 'exponential', delay }` in BullMQ; add jitter so synchronized failures do not retry in lockstep. - Classify errors: throw `UnrecoverableError` for permanent failures (malformed payload, missing referenced row) so they skip the remaining attempts instead of burning them. - On final failure move the job to a dead-letter queue and emit a metric or alert; never let it sit silently in the `failed` set forever. - Set `removeOnComplete` and a bounded `removeOnFail` so Redis does not grow unbounded, but keep enough failed history to debug and replay. --- ### [MEMORY] BullMQ conventions for this service (path: /src/queues) We run BullMQ v5.x (current line as of mid-2026) on a dedicated Redis instance, separate from the cache. - Use `Worker` and `Queue` from `bullmq`; share one `ioredis` connection per process with `maxRetriesPerRequest: null` (required by BullMQ blocking commands). - Schedule recurring work with `queue.upsertJobScheduler(schedulerId, repeat, template)`; the old `Repeat`/`add({ repeat })` API is deprecated and removed in v6. - Set `Worker` `concurrency` deliberately and add a rate limiter (`limiter: { max, duration }`) for jobs that call rate-limited providers. - Job IDs are the dedup unit: pass a deterministic `jobId` to drop duplicate enqueues, and remember BullMQ keeps a completed job's ID only while its record lives. - See /src/workers for retry and DLQ policy and /src/jobs for the idempotency contract every handler must honor. --- ### [MEMORY] Enqueue atomically with the transactional outbox (path: /src/jobs) Enqueueing to Redis or SQS from inside a database transaction is a dual-write bug: the commit can succeed while the enqueue fails, or vice versa, leaving data and jobs out of sync. - Within the business transaction, insert an `outbox` row describing the job instead of calling the queue directly. The row commits atomically with the data it depends on. - A relay worker polls the outbox with `SELECT ... FOR UPDATE SKIP LOCKED` to claim a batch, enqueues each job, and marks the row dispatched. Multiple relays can run in parallel without double-claiming. - Because the relay can crash after enqueue but before marking the row, dispatch is at-least-once too. That is fine: the downstream job is already idempotent (see /src/jobs idempotency rule). - Use a stable outbox row ID as the job's idempotency key so a re-dispatched row maps to the same job. --- ### [SKILL] background-jobs-queues-review (path: /) --- name: background-jobs-queues-review description: Review checklist for background jobs and queues. Use before merging any new or changed job handler, worker configuration, scheduler, or enqueue path to confirm idempotency, retry/backoff policy, dead-letter handling, and atomic enqueue. --- # Background Jobs & Queues review - [ ] The handler is idempotent: the same payload produces the same end state when run more than once. - [ ] A stable idempotency key rides on the payload (not generated inside the worker) and guards side effects via a unique constraint or dedup record. - [ ] The dedup record's TTL outlives the queue's full retry window. - [ ] `attempts` is bounded and `backoff` is exponential with jitter, not a fixed or zero delay. - [ ] Permanent failures throw `UnrecoverableError` (or equivalent) so they skip remaining retries instead of burning them. - [ ] Exhausted jobs move to a dead-letter queue and emit a metric or alert; nothing fails silently. - [ ] `removeOnComplete` and a bounded `removeOnFail` are set so Redis does not grow without limit. - [ ] Recurring jobs use `upsertJobScheduler`, not the deprecated `Repeat`/`add({ repeat })` API. - [ ] The Redis connection uses `maxRetriesPerRequest: null` and worker `concurrency`/rate limits are set deliberately. - [ ] Enqueues that depend on a DB write go through a transactional outbox, not a direct enqueue inside the transaction. - [ ] Workers shut down gracefully (`worker.close()`) on SIGTERM so in-flight jobs finish or requeue cleanly on deploy.