Kafka: Topic replication and partition leaders

We can replicate a topic in kafka cluster by specifying replication factor.

Why to replicate a topic?

To achieve fail-over capability, we need to maintain topic duplications.

Let's see it with an example.

As you see above diagram, there is one topic ‘Topic 0’ with 2 partitions.

a.   Partition 0

b.   Partition 1

Partition 0 is replicated in Broker 1 and Broker 2.

Partition 1 is replicated in Broker 2 and Broker 3.

What if one of the broker is down?

Since partitions are replicated, even though one broker is down, we can get the data from other broker.

For example, assume Broker 2 is down.

Even though Broker 2 is down, Partition 0 is served from Broker 1 and Partition 1 is served from Broker 3.

Partition Leader

Kafka identifies one leader for every partition. Producers send the data to a broker that is leader for current partitions. Data is replicated to other brokers from the broker who is leader of the partitions.

For example, as you see above image, Broker 1 is chosen as leader for topic 0 partition 0 and Broker 2 is chosen as leader for Topic 0 Partition 1.

What if a Partition Leader went down?

Kafka chooses another leader who owns the partition replica.

For example, when Broker 1 is down, then kafka can select Broker 2 as a leader for Topic 0 Partition 0.

Previous                                                    Next                                                    Home