Configuring rack awareness

There will always be failures in clusters, such as hardware issues with servers, racks, switches, power supplies, and so on.

To make sure that there is no single point of failure across the entire Hadoop infrastructure, and to ensure that the contention of resources is in a distributed manner, rack awareness plays an important role. Rack awareness is a concept in which Namenode is made aware of the layout of servers in a cluster, thus making intelligent decisions on block placement.

Getting ready

For the following steps, we assume that the cluster that is up and running with Datanodes is in a healthy state. We will log in to the Namenode and make changes there.

How to do it...

1. ssh to Namenode and edit the hdfs-site.xml file to add the following property to it:

   <property>
   <name>topology.script.file.name</name>
   <value>/opt/cluster/topology.sh</value>
   </property>

2. Make sure that the topology.sh file is readable by the user hadoop.

3. Create...