YARN containers and resource allocations
In YARN, there are many configuration parameters, which control the memory available to AM, containers, or the total memory that can be allocated for MapReduce or JVM heap size to be used and the number of CPU cores to be used for a job. This is covered in more detail in Chapter 8, Performance Tuning, but a rough idea is to have one core for each container and each Mapper container should have a memory of about 1 GB and the Reducer should have a memory twice the size of the Mapper. In addition to this, each node must have about 20% spare for the operating system and Hadoop daemons.
Getting ready
For this recipe, you will again need a running cluster and should have completed the previous recipes to make sure the cluster is working fine in terms of HDFS and YARN.
How to do it...
Connect to the
master1.cyrus.commaster node and switch to userhadoop.Navigate to the directory
/opt/cluster/hadoop/etc/hadoop.Edit the configuration file
yarn-site.xml,to make...
