Safe Retries for SQS and Lambda Consumers

Posted on In

Retries are easy to add. With SQS and Lambda you do not even add them: they are on from the first deploy. Safe retries are the hard part, and they are your job, not the platform's.

This post walks through where the second run of a message comes from, how to keep one bad message from replaying a whole batch, how to make the second run harmless, and how to keep the messages that never succeed in a state you can actually replay.

Where the second run comes from

A standard SQS queue delivers each message at least once. "At least" is doing real work in that sentence: the same message can reach your function twice even when nothing failed, and it will certainly come back when something did.

Animated: a message is received, the handler fails, the visibility timeout ends, the message is received again, and after the maximum receive count it moves to the dead-letter queue.

When Lambda polls a batch, SQS does not delete the messages. It hides them for the queue's visibility timeout. Lambda deletes them only when your function returns successfully. If the function throws, or runs past its own timeout, nothing deletes them, and when the visibility timeout ends they are visible again with their receive count one higher.

Two settings decide how this loop behaves:

  • The visibility timeout. AWS recommends at least six times the function timeout (plus the batch window, if you use one), so Lambda has time to retry a batch that was throttled. Lambda refuses a function timeout longer than the visibility timeout outright.
  • The redrive policy's maxReceiveCount. After that many receives, SQS moves the message to the dead-letter queue. AWS recommends at least 5, which gives Lambda a few chances to retry before a message lands in the DLQ.

The point to take away is that a second run is normal behaviour, not an incident. Design for it from the first handler.

One bad message should not replay the batch

Lambda hands your function a batch of messages. By default, the batch succeeds or fails as a unit: if the function throws because one message is bad, every message in the batch returns to the queue, including the ones that were already processed.

Animated: one message in a batch of five fails. Without partial batch responses all five come back; with them, only the failed one does.

The fix is partial batch responses. Turn on ReportBatchItemFailures on the event source mapping, catch failures per message inside the handler, and return the IDs of the messages that failed in batchItemFailures. Lambda deletes the rest. Only the failures come back.

A few details matter:

  • Return the failures, do not throw. If the function throws, Lambda treats the whole batch as failed even with the setting on.
  • An empty list means everything succeeded. A response Lambda cannot read, or an item ID that is not in the batch, counts as a failure of the whole batch.
  • On a FIFO queue, order is the point. When one message fails, return it and every message after it in the batch, so nothing later in the group runs ahead of it.

Partial responses shrink how much runs twice. They do not make the second run safe. That is the next part.

Make the second run harmless

A message that runs twice should leave the same result as a message that ran once. That property is idempotency, and the useful question is where the check lives.

  • In the handler: easy to write, but every new handler has to remember it.
  • In a shared library: one place to get it right. Powertools for AWS Lambda has an idempotency utility that records each key in a persistence store such as DynamoDB. It cannot see inside your database transaction, though.
  • At the database, with a unique constraint on the idempotency key: the last line of defence, because it holds even when the code is wrong.

I put the unique constraint in first and treat the handler or library check as an optimisation that saves work. A retry that hits the constraint is then a normal result: the work is already done, so the handler reports success and the message is deleted.

Pick the key before any side effect happens. A message ID is not always enough, because the same business event can arrive in two different messages. A key built from the event itself (an order ID plus the action, for example) survives that.

Side effects outside your database need the same care. A payment or an email cannot be rolled back with your transaction, so pass the idempotency key to the provider when its API accepts one, and record that you sent it.

Dead letters you can actually replay

A dead-letter queue full of messages is only useful if you can tell them apart. SQS moves a message to the DLQ unchanged: the body survives, the reason it failed does not.

So log, next to every failure, what you will need later: the error type and message, the version of the consumer that failed it, the receive count, and a correlation ID back to the original request. With that you can sort the DLQ before replaying it: fixed by the new code, needs a data fix, or safe to drop.

SQS can move messages from a DLQ back to the source queue (DLQ redrive, in the console or through the API). Replay the "fixed by the new code" group only, in small batches, and watch for the same error coming back. A replay that fails again is the DLQ telling you the fix did not cover what you thought it covered.

The short version

  • Expect every message to run more than once. It is how the queue works.
  • Set the visibility timeout to at least six times the function timeout (plus any batch window), and maxReceiveCount to at least 5.
  • Turn on ReportBatchItemFailures and return failed IDs instead of throwing.
  • On FIFO queues, return the failed message and everything after it.
  • Choose the idempotency key before any side effect, and back it with a unique constraint.
  • Log enough with each failure to sort the DLQ, then replay in small batches after a fix.