In this post, I am going to explain how Hadoop store the blocks across racks to achieve high availability.
What is a Rack?
Rack is a collection of data nodes or computes in Hadoop cluster which are located in the same data centre or geographical location.
What is rack awareness?
The process of making Hadoop aware of which data nodes are part of which racks and how these data nodes communicate within/outside of the racks is called rack awareness.
Where should Hadoop use this rack awareness information?
NameNode maintains all the data nodes and their respective rack ids. For example, When a client wants to store some information into Hadoop cluster, NameNode suggest the closest data nodes to the client location.
Rack awareness is used to make the system fault tolerant
Let me explain it with an example. Hadoop applies below policies while persisting the data into Hadoop cluster.
a. A Data node must not have the more than one replica of same block
For example, in the above scenario, Block B1 is kept twice on the data node 2, which is invalid in Hadoop.
a. Not more than 2 replicas placed in the same rack.
In the above example, block B1 is replicated thrice in rack 1, which is invalid in Hadoop.
Why Hadoop apply above rules?
Hadoop default replication factor is 3. Suppose if you store all the three replications of block B1 in rack 1 (India). What if the data centre in India went down due to some natural calamity. You will lose the data.
How Hadoop persist the data blocks using this rack awareness?
Suppose a client from India want to write some block ‘B1’ to HDFS cluster.
a. First replica of the block B1 is persisted on the closest data node to the client.
b. Second replica of block B1 is stored in different rack.
c. Third replica of block B1 is stored in same rack (rack in step b) but on different data node.
Why can’t replicas store in different racks?
You may ask why can’t B1 replicas store in different racks like
a. First replica of B1 in Rack 1
b. Second replica of B1 in Rack 2
c. Third replica of B2 in Rack 3
Hadoop replication policy is designed such a way to reduce network latency while performing read/write operations. If you store each block on different rack, it increase the latency while writing.
Finally, if a client ‘C1’ from India want to write 3 blocks, one possibility of allocation can be like this.
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/RackAwareness.html
Previous Next Home
No comments:
Post a Comment