Kafka: Consumers and consumer groups

Consumers read the data from a topic. Data within a partition read sequentially.

Can consumer read the data from multiple partitions?

Yes

Consumer Groups

You can group multiple consumers to one group. Each consumer group has a unique id and a consumer group can subscribed to one or more topics. A message from topic gets delivered to only one consumer in consumer group.

As you see above image, there are two consumer groups.

a.   Consumer Group 1

b.   Consumer Group 2

Consumer Group 1 has two consumers, where consumer 1 read messages from partition 0 and partition2. Consumer 2 read messages from partition 1.

Consumer Group 2 has 3 consumers, where consumer 1 read data from partition 9, consumer 2 read data from partition 1 and consumer 3 read data from partition 2.

What if there are more consumers than partitions?

Some consumers will be inactive.

As you see above diagram, there are 3 partitions and 5 consumers, so consumer 1, 2 and 3 are actively reading the data from partitions, whereas consumer 4 and 5 are inactive.

Previous                                                    Next                                                    Home