AWS.SimpleQueueService.BatchEntryIdsNotDistinct

[benkeil]

When using non FIFO queues, it happens quite often that I receive the same message more than once. That's ok during processing, but when broadway wants to acknowledge the messages (which are in the same batch) if throws an error and move the messages to the DLQ.

Could we just filter out duplicates during acknowledging?

[josevalim]

Hi @benkeil! I have moved this to the SQS repo. I think we can support a new option ack type, called :ack_duplicate. here are the docs: https://github.com/dashbitco/broadway_sqs/blob/master/lib/broadway_sqs/producer.ex#L28

Would you link to send a PR for that?

[benkeil]

I will do

[benkeil]

Not sure how to handle the configuration part. If we add the new type :ack_duplicate you can set e.g. on_success: :ack and on_failure: :ack_duplicate which makes no sense. Wouldn't it make more sense to leave the options with :ack and :noop and add a new option :ack_mode with :default and :duplicate?

[josevalim]

@benkeil why do you say it makes no sense? Maybe it has no practical use but in theory there is no problem in allowing success to not ack duplicates and allow failures to ack so.

[josevalim]

To give such an example, consider this:

1. You want to always mark messages as acked
2. Your pipeline checks for the SQS message ID and it raises if the IDs are duplicate

In this case, you want to have exactly on_success: :ack, on_failure: :ack_duplicate, because you know dupes will fail (and therefore should ack) and you expect successes to never have dupes.

[benkeil]

Don't get your point. Either you want to ack or you don't. And if you want to ack you can't ack the same message id twice in the same batch, so you must skip it to not get errors. Also from a Business Logic side it makes no sense, because it's the same message.

[josevalim]

I see your point. :ack_duplicate is a poor name. We can change the name but I still think it should be an option that goes alongside :ack | :noop because that's what provides most flexibility.

It is also important to notice that we can't see it is a duplicate until we ack, so it is more like :ack_and_ignore_duplicate_errors.

[benkeil]

You see at the very beginning if it is a duplicate on the message id. Theoretically you can skip the processing at all.
What is the purpose of being so flexible? It is not possible to ack these messages in the same batch. You could ack the failed messages separate from the succes messages, but what if you receive the message 3 times?

[josevalim]

Oh, I see. If they are duplicate in the same batch, can it be the producer could have filtered them too? In any case, I think having a separate option called “filter_duplicate_ids_in_batch” will suffice. Should it be false or true by default?

[benkeil]

I think if you misconfigured the queue with a too low visibility_timeout, this could also happen.
So I would suggest activate it by default but log an info/warning and/or add the number of duplicates to the metrics.
What exactly happens if you filter them out on SQS side?

[josevalim]

So I would suggest activate it by default but log an info/warning and/or add the number of duplicates to the metrics.

Let's go this approach (logging), we don't even need an option. Thanks for all the discussion so far!

[patrickberkeley]

For what it's worth, we too are running into this issue.

[matheuscumpian]

Hello guys,

I was trying to understand this problem (this issue and #49), just to see if I have understand the discussion, the idea here is to filter out the duplicate messages during the acknowledgment and log a warning on the filtered messages by default, it is it?