Kafka: Reset offsets (Re read the data)

Sometimes you may want to re-read the data from topic. In this post, I am going to explain how can you do that.

Using ‘kafka-consumer-groups.sh’ command, we can reset the offsets.

--reset-offsets: Use this option to reset the offsets

This options one consumer group at the time, and instances should be       

inactive.

You can choose one of the below rest options.

a. --to-datetime

b. --by-period

c. --to-earliest,

d. --to-latest,

e. --shift-by,

f. --from-file,

g. --to-current

Scope of the reset can be set to --all-topics or --topic.

Let’s experiment --reset-offsets option like below.

Create a topic

kafka-topics.sh --bootstrap-server localhost:9092 --topic myFirstTopic --create --partitions 3 --replication-factor 1

$kafka-topics.sh --bootstrap-server localhost:9092 --topic myFirstTopic --create --partitions 3 --replication-factor 1
$
$kafka-topics.sh --bootstrap-server localhost:9092 --list
__consumer_offsets
myFirstTopic

Create a consumer

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myFirstTopic --group myFirstConsumerGroup

This consumer is part of myFirstConsumerGroup.

Create a producer that send messages to myFirstTopic

kafka-console-producer.sh --broker-list localhost:9092 --topic myFirstTopic --producer-property acks=1

Send some messages.

$kafka-console-producer.sh --broker-list localhost:9092 --topic myFirstTopic --producer-property acks=1
>message 1
>message 2
>message 3

You can see same messages are read by consumer.

$kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myFirstTopic --group myFirstConsumerGroup
message 1
message 2
message 3

Let’s stop the consumer (CTRL + C).

Let’s reset the offsets by executing below command.

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group myFirstConsumerGroup --reset-offsets --to-earliest --execute --topic myFirstTopic

$kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group myFirstConsumerGroup --reset-offsets --to-earliest --execute --topic myFirstTopic

GROUP                          TOPIC                          PARTITION  NEW-OFFSET     
myFirstConsumerGroup           myFirstTopic                   0          0              
myFirstConsumerGroup           myFirstTopic                   2          0              
myFirstConsumerGroup           myFirstTopic                   1          0              
$
$kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myFirstTopic --group myFirstConsumerGroup
message 1
message 2
message 3

As you see above console messages, once offsets are reset, new consumer starts reading the messages from starting.

Previous                                                    Next                                                    Home