Kafka CLI: Add consumers to a group

Using --group option, you can create a consumer group.

Syntax

kafka-console-consumer --bootstrap-server {serverDetails} --topic {topicName} --group {groupName}

Example

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myFirstTopic --group myFirstConsumerGroup

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myFirstTopic --group myFirstConsumerGroup

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myFirstTopic --group myFirstConsumerGroup

In the above example, I created 3 consumers which are part of myFirstConsumerGroup.

Let’s try with an example.

Topic with three partition and consumer group with one consumer

Step 1: Create a topic with three partitions.

kafka-topics.sh --bootstrap-server localhost:9092 --topic myFirstTopic --create --partitions 3 --replication-factor 1

$kafka-topics.sh --bootstrap-server localhost:9092 --list
__consumer_offsets
myFirstTopic

Step 2: Create a consumer which is part of consumer group ‘myFirstConsumerGroup’ 

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myFirstTopic --group myFirstConsumerGroup

Step 3: Let’s produce some messages to topic.

kafka-console-producer.sh --broker-list localhost:9092 --topic myFirstTopic --producer-property acks=1

$kafka-console-producer.sh --broker-list localhost:9092 --topic myFirstTopic --producer-property acks=1
>message 1
>message 2

Since there is only one consumer in the group ‘myFirstConsumerGroup’, it only consumes all the messages.

$kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myFirstTopic --group myFirstConsumerGroup
message 1
message 2

Let’s add 2^nd consumer to the group myFirstConsumerGroup

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myFirstTopic --group myFirstConsumerGroup

Now send 5 more messages to the producer.

$kafka-console-producer.sh --broker-list localhost:9092 --topic myFirstTopic --producer-property acks=1
>message 1
>message 2
>message 3
>message 4
>message 5
>message 6
>message 7

Consumer 1 received messages 3 and 6

$kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myFirstTopic --group myFirstConsumerGroup
message 1
message 2
message 3
message 6

Consumer 2 received messages 4, 5 , and 7.

$kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myFirstTopic --group myFirstConsumerGroup
message 4
message 5
message 7

Let’s add 2 more consumers

Now there are 4 consumers and 3 partitions. If there are more consumers than partitions, then only n (n represent number of partitions) consumers are active at any point of time. Other consumers will be inactive.

When I produced below messages.

>message 8

>message 9

>message 10

>message 11

>message 12

>message 13

Consumer 2 received

message 10

message 13

Consumer 3 received

message 8

message 11

Consumer 4 received

message 9

message 12

In this case, consumer 1 is inactive.

Previous                                                    Next                                                    Home