Kafka: Topic and partitions

Topic, partitions, Producers, Consumers and brokers are the key terms in Kakfa. In this post, I am going to explain what is topic and partitions.

Topic

Topic represents a stream of data. In simple terms source systems send messages to topic and receiver systems read messages from topic.

Each topic in kafka is identified by unique name and you can create as many topic as you want.

Partitions

A topic is divided into multiple partitions. Each message in a partition has an unique identifier called offset. Offset is is an incremental id. For example first message is in partition 0 is assigned with id 0, 2^nd message to 1 and 3^rd message to 2……and so on.

For example, in the above case, kafka topic has 4 partitions (0, 1, 2, and 3). Each partition is independent. For example, partition 0 has 4 messages, whereas partition 1 has 2 messages.

Once a message is written to partition, it is immutable (will never change).

Can I specify number of partitions while creating a topic?

Yes, you can specify number of partitions while creating a topic and you can change the number of partitions later point of time.

Is the data in kafka partitions stays inifinite time?

No, data stored in partition for a limited time (default one week).

Previous                                                    Next                                                    Home