Issues in Kerberos enabled environment

This section lists the issues in the kerberos enabled environment and their workarounds.

Bad local directories in Yarn

If Yarn shows an alert for bad local directories when IBM Spectrum Scale is integrated, and if the Yarn service check failed then Yarn does not have the correct permission to access the local mounted directories created by IBM Spectrum Scale. Click Ambari > Yarn > Configs > Advanced > Node Manager, and review the yarn.nodemanager.local-dirs for the local directory values.

Workaround

Fix the local directory permissions on all nodes to have yarn:hadoop user ID and group ID permissions. Restart all services, or go back to the previous step and continue with the process.
For example, 
# Local directories under /opt/mapred
/dev/sdf1 on /opt/mapred/local1 type ext4 (rw,relatime,data=ordered)
/dev/sdg1 on /opt/mapred/local2 type ext4 (rw,relatime,data=ordered)
/dev/sdh1 on /opt/mapred/local3 type ext4 (rw,relatime,data=ordered)

# Check the directories under /opt/mapred
In /opt/mapred directory:
drwxrwxrwx 6 root root 4096 Mar  8 23:19 local3
drwxrwxrwx 6 root root 4096 Mar  8 23:19 local2
              drwxrwxrwx 6 root root 4096 Mar  8 23:19 local1

# Workaround:
# Change permission from root:root to yarn:hadoop for all the local* directories under /opt/mapred
# for all the nodes.

Under /opt/mapred directory:
chown yarn.hadoop local*

drwxrwxrwx 6 yarn hadoop 4096 Mar  8 23:19 local3
drwxrwxrwx 6 yarn hadoop 4096 Mar  8 23:19 local2
              drwxrwxrwx 6 yarn hadoop 4096 Mar  8 23:19 local1

# Restart all services (Or go back to your previous step and continue with the process).

Nodemanager failure due to device busy

Nodemanager fails to start due to local directory error:
OSError: [Errno 16] Device or resource busy: '/opt/mapred/local1'

To fix this issue:

  1. Go to Yarn > Configs > Search for yarn.nodemanager.local-dirs.
  2. Check the values for the Yarn local directories.
  3. The correct local directory values must contain the Yarn directory in the local directory path. For example: 
    yarn.nodemanager.local-dirs=”/opt/mapred/local1/yarn,/opt/mapred/local2/yarn,/opt/mapred/local3/yarn”
  4. If the <local-dir>/yarn is not specified in yarn.nodemanager.local-dirs, add the path, and save the configuration.

Journal nodes not installed in the native HDFS HA

If you are unintegrating from a Kerberos-enabled NameNode HA mode environment to native HDFS, you may sometimes find JournalNodes components missing in HDFS. In such cases, manually install the journal nodes.