Answer by Oskarro for Effective strategy to avoid duplicate messages in...
There are several options to protect yourself against this problem:Idempotence in event services: In Event Driven Development systems such as RabbitMQ and Kafk you should ensure that event services are...
View ArticleAnswer by Ivan Timishenko for Effective strategy to avoid duplicate messages...
As mentioned above, good solution is add id to every consumed message add store indexes of last n messages on consumer database.This solution is described in the book Microservices Patterns by Chris...
View ArticleAnswer by Deepthi for Effective strategy to avoid duplicate messages in...
Other alternative solutions would be:Produce a message with a unique key into the topic testOn the consumer side, push an entry with the unique key into DB once processing is done successfully. Before...
View ArticleAnswer by Chris Halcrow for Effective strategy to avoid duplicate messages in...
There's a relatively new 'Transactional API' now in Kafka that can allow you to achieve exactly once processing when processing a stream. With the transactional API, idempotency can be built in, as...
View ArticleAnswer by Dean Jain for Effective strategy to avoid duplicate messages in...
Whatever done on producer side, still the best way we believe to deliver exactly once from kafka is to handle it on consumer side:Produce msg with a uuid as the Kafka message Key into topic T1consumer...
View ArticleAnswer by peihan for Effective strategy to avoid duplicate messages in apache...
I agree with RaGe's deduplicate on the consumer side. And we use Redis to deduplicate Kafka message.Assume the Message class has a member called 'uniqId', which is filled by the producer side and is...
View ArticleAnswer by RaGe for Effective strategy to avoid duplicate messages in apache...
This is what Kafka FAQ has to say on the subject of exactly-once:How do I get exactly-once messaging from Kafka?Exactly once semantics has two parts: avoiding duplication during data production and...
View ArticleAnswer by kuujo for Effective strategy to avoid duplicate messages in apache...
The short answer is, no.What you're looking for is exactly-once processing. While it may often seem feasible, it should never be relied upon because there are always caveats.Even in order to attempt to...
View ArticleEffective strategy to avoid duplicate messages in apache kafka consumer
I have been studying apache kafka for a month now. I am however, stuck at a point now. My use case is, I have two or more consumer processes running on different machines. I ran a few tests in which I...
View Article
