Kafka: Producers

Producers write messages to kafka topics. Every message has a key associated with it. If producer send a message without a key, then the messages from a producer will go to the partitions in round robbin fashion.

As you see above image,

a.   Message 1 goes to Topic A partition 0

b.   Message 2 goes to Topic A partition 1

c.    Message 3 goes to Topic A partition 2

Message keys

Producer can attach a key to the message. A key can be a number, string etc., Messages with same key can always go to same partition. Kafka use the message key to identify the partition. Once a partition is identified, then the messages with same key always go to the same partition, kafka guarantees that.

For example, you own a car rental company and want to track all the car navigation details using kafka. All the cars send their location (longitude, latitude) details to kafka topic periodically.

There are 7 cars: C1, C2, C3, C4, C5, C6, C7.

C1, C2 send messages with key 1

C3, C4 send messages with key 2

C5, C6, C7 send messages with key 3.

c1, m1, k1 : c1 send message m1 with key k1

c2, m1, k1 : c2 send message m1 with key k1 etc.,

All the messages with key k1, will go to partition 0, with key k2, will go to partition1 and with key k3 will go to partition 2.

Acknowledging the messages

If producer want an acknowledgement of data that he is written, then producer can choose to receive data acknowledgement option

(ack=0, ack=all, ack=1).

Durability Level	Description
ack=0	Producer will not wait for the acknowledgement
ack=1	Producer will wait for the leader acknowledgement. It is the default durability level.
ack=all	Producer wait for leader and replicas acknowledgement.

Note

As long as no new partitions added to the topic, messages with the same key will always go to the same partition.

Previous                                                    Next                                                    Home