A dead-letter queue is not a backlog to empty blindly. It is evidence that normal delivery or processing stopped for specific messages. First preserve the failure context, then classify the cause, repair it, and replay only a bounded set while watching both the source and destination systems.
Stabilize before redriving
Pause automatic redrive if it is repeatedly cycling failures. Record the queue or subscription, source system, destination, message count, age of the oldest item, first and last failure times, recent deployments, and current retention settings. Capture message identifiers, delivery count, dead-letter reason, and bounded metadata without copying unnecessary sensitive payloads into tickets or chat.
Amazon SQS uses a redrive policy and `maxReceiveCount` to decide when a message moves to a dead-letter queue. AWS warns that setting the count too low can move a message after too few attempts and recommends enough retries for resilience (AWS documentation).
Azure Service Bus also retains messages that cannot be delivered or processed in a dead-letter subqueue. Its documentation notes that messages remain there until a receiver explicitly retrieves and completes them; there is no automatic cleanup of the dead-letter queue (Microsoft Learn). Confirm the actual service behavior rather than assuming every queue expires or purges failed items the same way.
Classify the failure
Inspect a small sample from the oldest, newest, and most common failure groups. Classify each as malformed or unsupported data, expired message, missing dependency, authentication or permission failure, rate limit, destination outage, schema or version mismatch, business-rule rejection, duplicate, or unknown. Group by the smallest stable signature, such as error code plus consumer version, instead of treating every item as a separate incident.
Check whether the producer emitted an invalid event, the transport exhausted delivery attempts, or the consumer rejected a valid event. That distinction determines where to fix the problem. Preserve at least one safe example and the related logs before changing code or deleting messages.
Check queue policy and retention
Verify that the dead-letter queue belongs to the intended source and uses a compatible queue type. Review receive count, visibility or lock duration, message time-to-live, dead-letter retention, redrive permissions, and alert thresholds. AWS documents a redrive allow policy that controls which source queues may use a dead-letter queue, while Azure gives each queue or subscription its own dead-letter subqueue.
Estimate the time remaining before evidence or messages can be lost. If retention is close to expiring, preserve approved diagnostic records and prioritize the oldest recoverable work. Do not extend retention or copy production data to a new location without checking privacy and data-handling requirements.
Repair and prove the consumer
Reproduce one failure in a safe environment using a sanitized payload or provider test event. Fix the producer, consumer, permissions, configuration, or dependency that caused it. Then run valid, invalid, duplicate, and out-of-order cases. Confirm that a permanently bad event is rejected with an actionable reason and that a transient failure retries without duplicating side effects.
Before replay, verify idempotency at the business action: order creation, invoice posting, email send, inventory update, or record mutation. A technically unique queue delivery can still represent a duplicate business event.
Redrive in a controlled batch
Start with one or a few representative messages. Watch success, repeat failures, processing latency, destination capacity, and downstream results. Increase the replay rate only after the small batch reconciles. AWS supports controlling redrive velocity and starts with the oldest messages, but redriven and newly produced traffic can interleave at the destination (AWS redrive guidance). Plan reconciliation accordingly.
Stop the replay if the same failure signature returns, downstream capacity degrades, or side effects cannot be reconciled. Never purge the queue simply to clear an alert.
Close with evidence
Record the root cause, affected interval, number recovered, number intentionally discarded with authorization, remaining count, code or configuration change, replay window, and prevention action. Add an alert for dead-letter arrivals and age, not only total depth.
Completion check
Triage is complete when the cause is identified or explicitly left unknown, retention risk is controlled, the repair passes negative and duplicate tests, a bounded redrive succeeds, downstream outcomes reconcile, and no failed message was silently lost or replayed without an owner.
Sources
- Using dead-letter queues in Amazon SQS — Amazon Web Services
- Service Bus dead-letter queues — Microsoft Learn
- Configure a dead-letter queue redrive in Amazon SQS — Amazon Web Services