Kafka CLI: commands cheetsheet

Description	Syntax	Example
Create Topic	kafka-topics.sh --bootstrap-server {bootStratServerDetails} --topic {topicName} --create --partitions {numberOfPartitions} --replication-factor {replicationFactor}	kafka-topics.sh --bootstrap-server localhost:9092 --topic myFirstTopic --create --partitions 3 --replication-factor 1
Describe topic	kafka-topics.sh --bootstrap-server {bootStratServerDetails} --topic {topicName} --describe	kafka-topics.sh --bootstrap-server localhost:9092 --topic myFirstTopic --describe
Delete topic	kafka-topics.sh --bootstrap-server {bootStrapServerDetails} --topic mySecondTopic --delete	kafka-topics.sh --bootstrap-server localhost:9092 --topic mySecondTopic --delete
List all topics	kafka-topics.sh --bootstrap-server {bootStrapServerDetails} --list	kafka-topics.sh --bootstrap-server localhost:9092 --list
Send message to kafka topic	kafka-console-producer.sh --broker-list {kafkaServiceDetails} --topic {topicName}	kafka-console-producer.sh --broker-list localhost:9092 --topic myFirstTopic
Send additional properties to a message	kafka-console-producer.sh --broker-list {kafkaServiceDetails} --topic {topicName} --producer-property {prop1}={value1} {prop2}={value2}	kafka-console-producer.sh --broker-list localhost:9092 --topic myFirstTopic --producer-property acks=1
Consume messages from a topic	kafka-console-consumer --bootstrap-server {serverDetails}--topic {topicName}	kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myFirstTopic
Consume all messages from topic (beginning of topic)	kafka-console-consumer --bootstrap-server {serverDetails}--topic {topicName} --from-beginning	kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myFirstTopic --from-beginning
Add a consumer to group	kafka-console-consumer --bootstrap-server {serverDetails}--topic {topicName} --group {groupName}	kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myFirstTopic --group myFirstConsumerGroup
Print all consumer groups	kafka-consumer-groups.sh --bootstrap-server {serverDetails} --list	kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
Describe consumer group	kafka-consumer-groups.sh --bootstrap-server {serverDetails} --describe --group {groupName}	kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group myFirstConsumerGroup
Reset Offsets	kafka-consumer-groups.sh --bootstrap-server {serverDetails} --group {groupName} --reset-offsets --to-earliest --execute --topic {topicName}	kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group myFirstConsumerGroup --reset-offsets --to-earliest --execute --topic myFirstTopic
Get kafka version	kafka-topics.sh --version	kafka-topics.sh --version

Previous                                                    Next                                                    Home

Kafka CLI: Add consumers to a group

Using --group option, you can create a consumer group.

Syntax

kafka-console-consumer --bootstrap-server {serverDetails} --topic {topicName} --group {groupName}

Example

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myFirstTopic --group myFirstConsumerGroup

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myFirstTopic --group myFirstConsumerGroup

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myFirstTopic --group myFirstConsumerGroup

In the above example, I created 3 consumers which are part of myFirstConsumerGroup.

Let’s try with an example.

Topic with three partition and consumer group with one consumer

Step 1: Create a topic with three partitions.

kafka-topics.sh --bootstrap-server localhost:9092 --topic myFirstTopic --create --partitions 3 --replication-factor 1

$kafka-topics.sh --bootstrap-server localhost:9092 --list
__consumer_offsets
myFirstTopic

Step 2: Create a consumer which is part of consumer group ‘myFirstConsumerGroup’ 

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myFirstTopic --group myFirstConsumerGroup

Step 3: Let’s produce some messages to topic.

kafka-console-producer.sh --broker-list localhost:9092 --topic myFirstTopic --producer-property acks=1

$kafka-console-producer.sh --broker-list localhost:9092 --topic myFirstTopic --producer-property acks=1
>message 1
>message 2

Since there is only one consumer in the group ‘myFirstConsumerGroup’, it only consumes all the messages.

$kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myFirstTopic --group myFirstConsumerGroup
message 1
message 2

Let’s add 2^nd consumer to the group myFirstConsumerGroup

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myFirstTopic --group myFirstConsumerGroup

Now send 5 more messages to the producer.

$kafka-console-producer.sh --broker-list localhost:9092 --topic myFirstTopic --producer-property acks=1
>message 1
>message 2
>message 3
>message 4
>message 5
>message 6
>message 7

Consumer 1 received messages 3 and 6

$kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myFirstTopic --group myFirstConsumerGroup
message 1
message 2
message 3
message 6

Consumer 2 received messages 4, 5 , and 7.

$kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myFirstTopic --group myFirstConsumerGroup
message 4
message 5
message 7

Let’s add 2 more consumers

Now there are 4 consumers and 3 partitions. If there are more consumers than partitions, then only n (n represent number of partitions) consumers are active at any point of time. Other consumers will be inactive.

When I produced below messages.

>message 8

>message 9

>message 10

>message 11

>message 12

>message 13

Consumer 2 received

message 10

message 13

Consumer 3 received

message 8

message 11

Consumer 4 received

message 9

message 12

In this case, consumer 1 is inactive.

Previous                                                    Next                                                    Home