Service fails to start

Solution:

After Integrate_Transparency or Unintegrate_Transparency in HA mode, if the HDFS service or its components (for example, NameNodes) do not come up during start, then do the following:

• Check if the zkfc process is up by running the following command on each NameNode host:

  # ps -eaf | grep zkfc

  If the zkfc process is up, kill the zkfc process from the NameNode host by running the kill -9 command on the pid.

• Once the zkfc process is not running in any NameNode host, go into the HDFS service dashboard and do a Start the HDFS service.

2017-04-04 15:42:49,901 - ========== Check for changes to the configuration. ===========
2017-04-04 15:42:49,901 - Updating remote.copy needs service reboot.
2017-04-04 15:42:49,901 - Values don't match for gpfs.remote.copy. running_config[gpfs.remote.copy]:
sudo wrapper in use; gpfs_config[gpfs.remote.copy]: /usr/bin/scp
2017-04-04 15:42:49,902 - Updating remote.shell needs service reboot.
2017-04-04 15:42:49,902 - Values don't match for gpfs.remote.shell. running_config[gpfs.remote.shell]:
/usr/lpp/mmfs/bin/sshwrap; gpfs_config[gpfs.remote.shell]: /usr/bin/ssh
2017-04-04 15:42:49,902 - Shutdown all gpfs clients.
2017-04-04 15:42:49,902 - Run command: sudo /usr/lpp/mmfs/bin/mmshutdown -N k-001,k-002,k-003,k-004
2017-04-04 15:44:03,608 - Status: 0, Output:
Tue  4 Apr 15:42:50 CEST 2017: mmshutdown: Starting force unmount of GPFS file systems
k-003.gpfs.net:  mmremote: Invalid current working directory detected: /usrhome/am_agent

To resolve this issue: Change the permissions of the home directory of the GPFS™ non-root user to at least 711.

Example: For the /usrhome/am_agent directory, set the directory with at least a 711 permission set or rwx--x—x.

Where, 7= rwx for the user itself, 1= x for the group, 1= x for others; x will allow users to cd into the home directory.

This is because the IBM Storage Scale command does a cd into the home directory of the user. Therefore, the permission should be set to at least 711.

For additional failures, see What to do when the Accumulo service start or service check fails?.

Solution:

Solution:

Check the Kafka configuration log directory to see if the Kafka log directory contains the IBM Storage® Scale shared mount point (/<gpfs mount point>/kafka-logs). Remove the share mount point from the directory list and restart the Kafka service. For more information, see Adding Services.

Solution:

If IBM Spectrum Scale is integrated, Hbase Master fails to start or goes down. This could be because of stale znodes in Zookeeper created for Hbase.

The sequence of steps when this error occurs is to enable NameNode HA in native HDFS then integrate IBM Spectrum Scale service to use HDFS Transparency and later enable Kerberos. During the period when Kerberos is enabling, zkfc fails to start. This leads to both NameNodes being in the standby mode. Therefore, HDFS cannot be used. As a result, the Start All services fails.

Solution:

Restart HDFS or restart all services from Ambari.

For Mpack version 2.7.0.0, the NameNode and DataNode might fail to start with the following error message when the data directory (gpfs.data.dir) is specified through the Ambari IBM Spectrum Scale UI: Failed to replace mapreduce.tar.gz with Transparency jars.

Solution:

Follow these steps to set the mapreduce.tar.gz into a proper directory for the NameNode/DataNode to start:

mkdir -p /<gpfs.mnt.dir>/<gpfs.data.dir>/hdp/apps/<hdp-version>/mapreduce/

cp /usr/hdp/<hdp-version>/hadoop/mapreduce.tar.gz
/<gpfs.mnt.dir>/<gpfs.data.dir>/hdp/apps/<hdp-version>/mapreduce/mapreduce.tar.gz

mkdir -p /bigpfs/datadir1/hdp/apps/2.6.5.0-292/mapreduce/ 
cp /usr/hdp/2.6.5.0-292/hadoop/mapreduce.tar.gz 
/bigpfs/datadir1/hdp/apps/2.6.5.0-292/mapreduce/mapreduce.tar.gz

Restart the failed HDFS components.

There is a timing issue during the formatting of the zookeeper directory which is shared by both ZKFC in HA mode on which ZKFC should be started first.

Solution:

Rerun the Start All operation to get the services back up.

The zkfc might fail to start with Can't set priority for process error if IBM Spectrum Scale is first added to an HA enabled HDP cluster before adding Kerberos. The hdfs_jaas.conf file might not be generated during the kerberos enablement action.

Solution:

1. Create the hdfs_jaas.conf file in the /etc/hadoop/conf/secure directory, on both the NameNodes.
   For example,

   # cat /etc/hadoop/conf/secure/hdfs_jaas.conf
   
   Client {
         com.sun.security.auth.module.Krb5LoginModule required
         useKeyTab=true
         storeKey=true
         useTicketCache=false
         keyTab="/etc/security/keytabs/nn.service.keytab"
         principal="nn/c902f09x13.gpfs.net@IBM.COM";
   };

   Note: Ensure that you change the keyTab and principal values based on your environment.
2. If /etc/hosts is used for hostname resolution instead of DNS in your environment, use the FQDN hostname in /etc/hosts.
   Ensure that the output from the command hostname matches the following:

   1. Hostname specified in the Ambari wizard.
   2. IP/hostname used for DNS.

   Check the same for all the hosts in the cluster and restart HDFS.

ERROR namenode.NameNode (NameNode.java:main(1774)) - 
Failed to start namenode.java.io.FileNotFoundException: 
No valid image files found at 
org.apache.hadoop.hdfs.server.namenode.FSImageTransactionalStorageInspector.
getLatestImages(FSImageTransactionalStorageInspector.java:165)

This is because the dfs.namenode.name.dir (default path: /hadoop/hdfs/namenode) directory is empty.

Solution:

Solution:

Increase the NameNode Heap Size to at least 2GB in Ambari HDFS configuration and restart the NameNodes.

Solution:

Stop and start the Zeppelin Notebook service to get out of the hang situation.

Error message:

2019-05-08 13:34:44,595 WARN zookeeper.ClientCnxn (ClientCnxn.java:startConnect(1014)) - 
SASL configuration failed: javax.security.auth.login.LoginException: Zookeeper client cannot 
authenticate using the Client section of the supplied JAAS configuration: 
'/usr/hdp/3.1.0.0-78/hadoop/conf/secure/hdfs_jaas.conf' because of a RuntimeException: 
java.lang.SecurityException: java.io.IOException: /usr/hdp/3.1.0.0-78/hadoop/conf/secure/hdfs_jaas.conf 
(No such file or directory) Will continue connection to Zookeeper server without SASL authentication, 
if Zookeeper server allows it.

2019-05-06 06:10:09,974 ERROR client.ZooKeeperSaslClient (ZooKeeperSaslClient.java:createSaslToken(388)) - 
An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed 
[Caused by GSSException: No valid credentials provided (Mechanism level: Ticket expired (32) - PROCESS_TGS)]) 
occurred when evaluating Zookeeper Quorum Member's  received SASL token. 
Zookeeper Client will go to AUTH_FAILED state.

2019-05-06 06:10:09,974 ERROR zookeeper.ClientCnxn (ClientCnxn.java:run(1059)) - SASL authentication with Zookeeper 
Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided 
(Mechanism level: Ticket expired (32) - PROCESS_TGS)]) occurred when evaluating Zookeeper Quorum Member's  
received SASL token. Zookeeper Client will go to AUTH_FAILED state.

2019-05-06 06:10:10,081 ERROR ha.ActiveStandbyElector (ActiveStandbyElector.java:fatalError(719)) - 
Unexpected Zookeeper watch event state: AuthFailed

2019-05-06 06:10:10,081 ERROR ha.ZKFailoverController (ZKFailoverController.java:fatalError(379)) - 
Fatal error occurred:Unexpected Zookeeper watch event state: AuthFailed

2019-05-06 06:10:10,081 FATAL tools.DFSZKFailoverController (DFSZKFailoverController.java:main(197)) - 
DFSZKFailOverController exiting due to earlier exception java.io.IOException: 
Couldn't determine existence of znode '/hadoop-ha/nn'

2019-05-06 06:10:10,083 INFO  util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting with status 1: 
java.io.IOException: Couldn't determine existence of znode '/hadoop-ha/nn'

2019-05-06 06:10:10,085 INFO  tools.DFSZKFailoverController (LogAdapter.java:info(49)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DFSZKFailoverController at dn01-dat.gpfs.net/30.1.1.15
************************************************************/

Solution:

The default KDC version on RHEL7.5 has a known bug. You need to upgrade the krb-server packages to 1.1.15.1-19 + version.

For additional information, see 2nd generation HDFS Protocol troubleshooting DataNode reports exceptions after Kerberos is enabled on RHEL7.5.

In HDP 3.0: The Timeline Service 2.0 in Yarn fails to start.

Solution:

There is a new implementation of Timeline service in HDP 3.0 named Timeline Service 2.0. It can run in 2 modes (Embedded mode or System service mode) depending on the cluster capacity. To check which mode is being set, filter the search for is_hbase_system_service_launch under YARN configuration. If this value is checked, it is running in the system service mode. If it is running in the system service mode, follow the set of best practices from the Enable System Service Mode.

The following important steps should be performed after Integrating/UnIntegrating the IBM Spectrum Scale service and Enabling/Disabling Kerberos: Remove ats-hbase before switching between clusters.