In this article, we will learn how would we temporarily send messages into an SQS dead-letter queue. In our previous article, we cover everything about SQS itself. Take a moment and read It, if you didn’t.
What SQS Dead Letter Queue resolve?
The SQS Dead Letter Queue also help us to decouple our application. Let’s see one example of a user sending orders via one frontend server, storing those orders in an SQS queue, and then the backend server pulling from that queue.
The user order has just been sent.
Now, what would occur if there was an issue? For example, if the user made an error, like putting a wrong address in that order, where they accidentally put their email address in the address field, and the frontend application didn’t validate that field?
SQS Receive Message Loop
The message will be delivered without any issue to the queue, the backend server would pull the message, and it would fail to process it accurately, and the message would return to the queue again. So, then other backend servers would pull that same message, have the same problem with it. So it would return and remove it from the queue, but the visibility timeout would reach the maximum period, and the message would appear again.
And this process would continue several times until we reach the maximum retention window for our queue, which is at most 14 days (depending on your configuration). Then the message would be removed, and we would lose that message forever.
Analyzing that problem, how can we resolve it? You may be thinking that we will present a vast and complex solution. But, honestly, it’s a lot easier than you think. So let’s see how we can resolve it using a simple approach: dead-letter queue.
What is a SQS dead-letter queue?
A dead-letter queue is, it’s just another regular SQS queue that we can temporarily move messages into. So rather than keeping that message decline in that primary SQS queue, once we reach the number of retries that we’ve specified, the message will get transferred to that dead-letter queue.
The advantage of the dead-letter queue is that we can transfer those inappropriate messages, and we don’t simply have to keep them in that SQS queue eternally.
How to create a Dead-Letter Queue
So, let’s go ahead and create our first queue. Now, if we’re going to create a dead-letter queue, we need to have that queue created before making our primary queue. (Click on Image to expand)
For your dead-letter queue, you might want to increase that to a 14-day maximum. For all other options, you can leave them with the default values.
Our Dead-Letter-Queue is now up and running. So let’s make our primary queue real fast.
Here, we can use all default values, except for the dead-letter queue:
Select Enabled, and then choose that ARN, that Amazon Resource Name, for the queue that we just created (Dead-Letter-Queue).
Remember to create the dead-letter queue first and assign it later on the primary queue. The dead-letter queue is not unique. It is just another regular queue, performing as a reject folder for the failure message.
Attention to the: Maximum Receives
SQS Maximum Receives
The “Maximum Receives” means the maximum number of retries effectively. So how many times can we read that message? And it pops back up in the queue before it gets sidelined into the dead-letter queue.
Probably you want to set this to be higher than 1, but for our learning process, set 1 for it. So we can see how the workflow is working efficiently.
Now, let’s try to send and receive some messages, and see the workflow.
On our TestQueue, click on “Send and receive messages.”
So, let’s write in a message, “Hi Bits Lovers!” and click Send.
Now, we interact with this queue, and let’s check for new messages.
Let’s go forward and poll for that message, click on “Poll for Messages”.
Let’s observe the receive count number. Notice that the “Receive count” is 1, which means “how many times has it been received?”
If we click on the message, we will be able to read that message “Hello Bits Lovers.”
Now, if we try to click on polling again:
No messages were found. What happened?
We received the message once. So, hit that receive count limit of 1 that we’d set for the TestQueue, and the message was moved out of this queue into the Dead-Letter-Queue, which we created and configured dead-letter-queue.
Now, let’s go and check our Dead-Letter-Queue.
Note that TestQueue has zero messages available, and we can see now the Dead-Letter-Queue currently has one available message in it.
So, let’s go ahead and check that out. After clicking the Poll button, we receive the message. So, if you click on that message, we can read: “Hi Bits Lovers!” But also notice something important. It says it’s been received four times because, in my case, I clicked three times on the poll button.
That received count carries over with the messages. So, we can always check how many times we received the message.
Hopefully, that gives us a practical concept of what it looks like to transmit and receive messages and the goal of that dead-letter queue to sideline the message, so it doesn’t just get read and processed repeatedly by our backend server.
SQS Dead Letter Queue Redrive
In December 2021, AWS launched a new ability to improve the SQS dead letter queue: SQS Dead Letter Queue Redrive to allow SQS to manage the lifecycle of unconsumed messages stored in Dead Letter Queue.
What SQS Redrive Does?
So the DLQ Redrive goal is moving standard unconsumed messages out of a dead-letter queue back to its source queue(s).
We can even examine the message attributes and corresponding metadata available for unprocessed messages in the dead-letter queue. Dead-letter queue redrive is an asynchronous workflow. It only redrives the messages available in the dead-letter queue when we start the redrive task. Also, if new messages are sent to the dead-letter queue, we can begin another redrive task once the last task is finished.
Our first exam advice: the dead-letter queue are the most promising sideline, and in fact, is the only one where we can take our SQS messages with problems and temporarily put them aside.
We always use the term “temporarily” is because, as we noticed, dead-letter queues aren’t different from the regular queue.
Dead-letter Queues Monitoring
They’re simply normal SQS queues with all the same characteristics and configurations as our primary queue. But they’re merely a spot to hold our message. Remember that we can set up that CloudWatch alarm to watch that queue depth because if our queue begins to fill up, we will be of an aware big problem. And, without this alarm, we wouldn’t have any knowledge that the queue was growing. So, with that in mind, we could scale horizontally our backend server to match with higher demand.
So, it is crucial to choose answers on the exam that contain this extra step.
Also, set up those traditional alarms, like queue depth for auto-scaling for our primary SQS queue, but also implement alarms for the queue depth
of that dead-letter queue to provide us in advance if the situation goes out of our control.
SQS Dead-letter queues aren’t unique.
Despite the name, it’s just a typical SQS queue, so don’t believe in it as some particular design or anything along those lines. It does the same routines that we’ve already seen with SQS. It’s just placed as that secondary queue.
It has that identical retention window.
This is essential to memorize for the exam because you might catch questions about keeping messages in this queue for more than 14 days. You should instantly identify that answer is incorrect because it’s simply a standard queue. It means 14 days is the most prolonged period for a message can keep in a dead-letter queue.
It also works with SNS
Also, we can place an SQS dead-letter queue for SNS topics. Like our standard SQS queues, if a message in SNS fails to deliver, we can sideline it in that SQS dead-letter queue for additional processing and analysis to figure out what happened with that message.
Consistency between SQS queues and dead-letter queues, it’s crucial:
So, for example, the dead-letter queue of a FIFO queue must also be a FIFO queue. Likewise, the dead-letter queue of a standard queue must also be a standard queue.
Also, we must use the same AWS account to make the SQS dead-letter queue and the other queues that transmit messages to the dead-letter queue. Furthermore, dead-letter queues must live in the same region as the different queues used.
Besides SQS Dead-Letter
Also, to expand your acknowledge about SQs, check the article that we talk about SQS Queue FIFO, and learn how easy it’s to create your first FIFO service.
SQS Dead Letter Queue belongs to the collections of AWS services, where we can reach High Availability.
Tor resume what we have learned, what are the benefits of SQS dead-letter queues?
The primary duty of a dead-letter queue is to manage the lifecycle of unconsumed messages. In addition, a dead-letter queue allows us to set aside and isolate messages that a backend server can’t process accurately to decide why their processing didn’t succeed. For example, setting up a dead-letter queue lets us do the following:
- Decide whether we have provided your sufficient consumer time to execute and process messages.
- Investigate the contents of messages transmitted to a dead-letter queue to analyze applications or hardware problems.
- Inspect logs for anomalies that might have caused messages to be transferred to a dead-letter queue.
- Set the alarm for any messages transmitted to a dead-letter queue.
When should we use an SQS dead letter queue?
- Do utilize SQS dead-letter queue to reduce the number of messages that can be received but can’t be processed.
- Do use dead-letter queues with standard queues. It would help if we always took advantage of dead-letter queues when our applications don’t rely on ordering. SQS Dead-letter queues can support us in troubleshooting wrong message transmission operations.
When should NOT use an SQS dead letter queue?
- Don’t utilize a dead-letter queue with a FIFO queue if we don’t desire to lose the exact order of messages or processes.
- Please don’t use a dead-letter queue with standard queues when we desire to keep retrying to process a message indefinitely. For instance, don’t use a dead-letter queue if one application must wait for a condition to become ready or available.