Kafka: Consumer offsets

Kafka stores the offests of messages at which a consumer group is consumed in a topic called __consumer_offsets.

Consumer can choose when to commit the offsets.

a. At most once: Offsets are committed as soon as the message is received. By ay chance when the message is lost during processing, there is no way to get the message. In this semantic, a message should be delivered maximum only once. It's acceptable to lose a message rather than delivering a message twice

b. At least once: Offsets are committed once the message is processed. If the processing goes wrong, then the messages can read again. Duplicate processing of messages should be handled by consumer.

c. Exactly once: Message must be delivered exactly once and no message should be lost.

Previous                                                    Next                                                    Home