Answer by Dean Jain for Effective strategy to avoid duplicate messages in apache kafka consumer

Whatever done on producer side, still the best way we believe to deliver exactly once from kafka is to handle it on consumer side:

1. Produce msg with a uuid as the Kafka message Key into topic T1
2. consumer side read the msg from T1, write it on hbase with uuid as rowkey
3. read back from hbase with the same rowkey and write to another topic T2
4. have your end consumers actually consume from topic T2