HDFS Transparency protocol troubleshooting

This topic contains information on troubleshooting the Second generation HDFS Transparency protocol issues.

Note:
  • For HDFS Transparency 3.1.0 and earlier, use the mmhadoopctl command.
  • For CES HDFS (HDFS Transparency 3.1.1 and later), use the corresponding mmhdfs and mmces commands.
  • gpfs.snap --hadoop is used for all HDFS Transparency versions.
  • From HDFS Transparency 3.1.0-6 and 3.1.1-3, ensure that the gpfs.ranger.enabled field is set to scale. The scale option replaces the original true/false values.
  1. Enable Debugging

    Gather the following for problem determination:

    • IBM Storage Scale and HDFS Transparency version
    • NameNodes (Primary & HA), DataNodes and application service logs
    • gpfs.snap with the --hadoop [-a] option (The --hadoop option is only available for IBM Storage Scale version 4.2.2 and later). For more information, see the Data gathered for hadoop on Linux® topic under Troubleshooting > Collecting details of the issues > CLI commands for collecting issue details > Using the gpfs.snap command > Data gathered by gpfs.snap on Linux for protocols path in IBM® Storage Scale documentation.
      Note:
      • The gpfs.snap --hadoop captures only the logs in the default log settings as specified in the Data gathered for hadoop on Linux topic in the IBM Storage Scale: Problem Determination Guide.
      • From IBM Storage Scale 5.0.5, gpfs.snap --hadoop is able to capture the HDFS Transparency logs from the user configured directories.

    To enable the debug information for HDFS Transparency version 3.0.0-x/2.7.3-x/2.7.2-x, set the following fields to DEBUG as seen in the Fields box below into the log4j.properties file.

    If you are using Ambari and you want to enable the log fields, add the fields through the Ambari GUI HDFS service and restart the HDFS service.

    If you are not using Ambari and you want to enable the log fields, change the fields in the <GPFS_CONFIG_PATH>/log4j.properties file and run the /usr/lpp/mmfs/bin/mmhadoop connector syncconf <GPFS_CONFIG_PATH> command and then restart the HDFS transparency. Restart the HDFS Transparency by running the following commands: 
    /usr/lpp/mmfs/bin/mmhadoopctl connector stop; 
/usr/lpp/mmfs/bin/mmhadoopctl connector start

    For HDFS Transparency 2.7.3-x, the <GPFS_CONFIG_PATH> is mmhadoopctl connector syncconf /usr/lpp/mmfs/hadoop/etc/hadoop.

    For HDFS Transparency 3.0.x, the <GPFS_CONFIG_PATH> is mmhadoopctl connector syncconf /var/mmfs/hadoop/etc/hadoop.

    Log and configuration location:

    For HDFS Transparency version 3.0.0-x:
    • With Hortonworks HDP 3.0, the HDFS Transparency logs are located at /var/log/hadoop/root by default.
    • For Open Source Apache, the HDFS Transparency logs are located at /var/log/transparency.
    • Configuration is moved from /usr/lpp/mmfs/hadoop/etc to /var/mmfs/hadoop/etc.

    For HDFS Transparency version 2.7.3-x with HortonWorks HDP 2.6.x or BI IOP 4.2.5, the HDFS Transparency logs are located at /var/log/hadoop/root by default.

    For HDFS Transparency version 2.7.2-x with IBM BigInsights® IOP 4.0/4.1/4.2.x, the HDFS Transparency logs are located at /usr/lpp/mmfs/hadoop/logs by default.

    For HDFS Transparency version 2.7.x-x with Open Source Apache, the HDFS Transparency logs are located at /usr/lpp/mmfs/hadoop/logs.

    Fields:
    
log4j.logger.BlockStateChange=DEBUG

log4j.logger.org.apache.hadoop.hdfs.StateChange=DEBUG

log4j.logger.org.apache.hadoop.hdfs.server.namenode.GPFSNative=DEBUG

log4j.logger.org.apache.hadoop.hdfs.server.namenode=DEBUG

log4j.logger.org.apache.hadoop.hdfs.protocol.datatransfer=DEBUG

log4j.logger.org.apache.hadoop.hdfs.server.namenode.top.metrics=ERROR

log4j.logger.org.apache.hadoop.hdfs.server.namenode.top.window=ERROR

log4j.logger.org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager=INFO

log4j.logger.org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault=INFO

log4j.logger.org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl=DEBUG

log4j.logger.org.apache.hadoop.hdfs.server=DEBUG

log4j.logger.org.apache.hadoop.hdfs.DFSClient=DEBUG

log4j.logger.org.apache.hadoop.ipc=DEBUG

log4j.logger.org.apache.hadoop.fs=DEBUG

log4j.logger.org.apache.hadoop.conf.Configuration.deprecation=WARN

    The logs are located at /usr/lpp/mmfs/hadoop/logs.

    To enable the debug information for HDFS Transparency version 2.7.0-x, set the following fields in the /usr/lpp/mmfs/hadoop/etc/hadoop/log4j.properties file to DEBUG:
    log4j.logger.org.apache.hadoop.gpfs.server=DEBUG

log4j.logger.org.apache.hadoop.ipc=DEBUG

log4j.logger.org.apache.hadoop.hdfs.protocol=DEBUG

log4j.logger.org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtocol=DEBUG

log4j.logger.org.apache.hadoop.hdfs.StateChange=DEBUG

    Dynamically changing debug log settings

    Find the active NameNode and port to set the daemon logging level dynamically. The setting will be in effect until HDFS restarts or you reset the Debug Log Level to INFO.

    Run the following command:
    hadoop daemonlog -setlevel <Active Namenode>:<Active Namenode port> <Daemon to set Log level> <Debug Log Level>
    For example:
    • To get the debug level:
      # hadoop daemonlog -getlevel c902f08x01.gpfs.net:50070 org.apache.hadoop.hdfs.server.namenode

      Connecting to http://c902f08x01.gpfs.net:50070/logLevel?log=org.apache.hadoop.hdfs.server.namenode.

      Submitted Class Name: org.apache.hadoop.hdfs.server.namenode

      Log Class: org.apache.commons.logging.impl.Log4JLogger

      Effective Level: INFO

    • To set the debug level:
      # hadoop daemonlog -setlevel c902f08x01.gpfs.net:50070 org.apache.hadoop.hdfs.server.namenode DEBUG

      Connecting to http://c902f08x01.gpfs.net:50070/logLevel?log=org.apache.hadoop.hdfs.server.namenode&level=DEBUG.

      Submitted Class Name: org.apache.hadoop.hdfs.server.namenode

      Log Class: org.apache.commons.logging.impl.Log4JLogger

      Submitted Level: DEBUG

      Setting Level to DEBUG ...

      Effective Level: DEBUG

    Get jps and jstack information

    Run jps to get the PID of the daemon and run jstack PID of daemon and pipe to a file to get the output.

    For example:
    # jps | grep NameNode
15548 NameNode

# jstack -l 15548 > jstack_15548_NameNode.output

    Debug Time issues

    Set the debug flags from DEBUG to ALL. Do not change the other flags.

    Debugging HDFS NameNode using failover

    If you cannot stop NameNode in a running cluster and you are not able to dynamically change the debug setting for the NameNode in the cluster due to the security settings, then change the xml file and manually perform the NameNode failover.

    For example, the security settings were set:
    hadoop.http.authentication.simple.anonymous.allowed=true
hadoop.http.authentication.type=simple
    Note: Manually editing the xml files in HDP with Mpack environment will be void after HDFS restarts because Ambari saves the configuration in the database.
    For example, during Ranger debugging:
    • Checked if user group is in dfs.permissions.superusergroup
    • Checked xasecure.add-hadoop-authorization = true
    • Checked dfs.namenode.inode.attributes.provider.class = org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer and dfs.permissions.enabled = true
    • GPFS directory is set to 700
    • Not able to dynamically set the debug command: 
      hadoop daemonlog -setlevel [active namenode]:50070 org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer DEBUG
    Edit /var/mmfs/hadoop/etc/hadoop/log4j.properties to add in the DEBUG flag on the standby NameNode.
    log4j.logger.org.apache.hadoop.security.UserGroupInformation=DEBUG
log4j.logger.org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer=DEBUG
    #On the standby NameNode, stop
    /usr/lpp/mmfs/hadoop/bin/hdfs --config /var/mmfs/hadoop/etc/hadoop --daemon stop namenode
    #Check is stopped
    jps
    #Start standby NameNode to pick up the debug changes
    /usr/lpp/mmfs/hadoop/bin/hdfs --config /var/mmfs/hadoop/etc/hadoop --daemon start namenode
    #Failover primary NameNode to standby
    hdfs haadmin -failover nn2 nn1
    #Recreate issue and look into the NameNode log
    2020-02-17 07:47:06,331 ERROR util.RangerRESTClient ….
  2. A lot of "topN size for comm" in NameNode logs
    The NameNode log contains a lot of topN size entries as shown below:
    2016-11-20 10:02:27,259 INFO  window.RollingWindowManager 
(RollingWindowManager.java:getTopUsersForMetric(247)) - topN size for command rename is: 0

2016-11-20 10:02:27,259 INFO  window.RollingWindowManager 
(RollingWindowManager.java:getTopUsersForMetric(247)) - topN size for command mkdirs is: 1

......

    Solution:

    In the log4j.properties file, set the following field:
    log4j.logger.org.apache.hadoop.hdfs.server.namenode.top.window=WARN
  3. webhdfs is not working

    Solution:

    On the node running the NameNode service, check whether the port (50070 by default) is up. If it is up, check if the dfs.webhdfs.enabled is set to true in your configuration. If not, configure it to true in your hadoop configuration and sync it to the HDFS transparency nodes.

  4. Could not find or load main class org.apache.hadoop.gpfs.tools.GetConf

    Solution:

    Check all the nodes to see whether any of the bash variables (HADOOP_HOME, HADOOP_HDFS_HOME, HADOOP_MAPRED_HOME, HADOOP_COMMON_HOME, HADOOP_COMMON_LIB_NATIVE_DIR, HADOOP_CONF_DIR, HADOOP_SECURITY_CONF_DIR) were exported. If it was exported, unexport or change it to another name. Setting these variables will result in the failure of some of the HDFS Transparency commands.

  5. org.apache.hadoop.hdfs.server.namenode.GPFSRunTimeException: java.io.IOException: Invalid argument: anonymous

    If hive.server2.authentication is configured as LDAP or Kerberos enabled, then the anonymous user is not used by Hive. However, the default setting for hive.server2.authentication is set to NONE. Therefore, no authentication will be done for Hive's requests to the Hiveserver2 (meta data). This means that all the requests will be done as the anonymous user.

    This exception is only seen when you run HIVE (HIVE supports anonymous authentication):
    2016-08-09 22:53:07,782 WARN  ipc.Server (Server.java:run(2068)) - 
IPC Server handler 143 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.mkdirs 
from 192.0.2.10:37501 Call#19 Retry#0

org.apache.hadoop.hdfs.server.namenode.GPFSRunTimeException: java.io.IOException: Invalid argument: anonymous

        at org.apache.hadoop.hdfs.server.namenode.SerialNumberManager.getUserSerialNumber(SerialNumberManager.java:59)

        at org.apache.hadoop.hdfs.server.namenode.INodeWithAdditionalFields$PermissionStatusFormat.toLong(INodeWithAdditionalFields.java:64)

        at org.apache.hadoop.hdfs.server.namenode.INodeWithAdditionalFields.<init>(INodeWithAdditionalFields.java:116)

        at org.apache.hadoop.hdfs.server.namenode.INodeDirectory.<init>(INodeDirectory.java:77)

        at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.unprotectedMkdir(FSDirMkdirOp.java:234)

        at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.createSingleDirectory(FSDirMkdirOp.java:191)

        at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.createChildrenDirectories(FSDirMkdirOp.java:166)

        at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:97)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3928)

        at org.apache.hadoop.hdfs.server.namenode.GPFSNamesystem.mkdirs(GPFSNamesystem.java:1254)

        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:993)

        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622)

        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)

        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)

        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:422)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)

        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)

Caused by: java.io.IOException: Invalid argument: anonymous

        at org.apache.hadoop.hdfs.server.namenode.GPFSNative.getUid(Native Method)

        at org.apache.hadoop.hdfs.server.namenode.SerialNumberManager.getUserSerialNumber(SerialNumberManager.java:51)

        ... 20 more

    Solutions:

    There are two solutions to fix this issue:
    1. Create the user and group "anonymous" with the same uid/gid on all the nodes.
    2. Configure HIVE hive.server2.authentication as LDAP or Kerberos assuming cluster is already LDAP/Kerberos enabled.
  6. javax.security.sasl.SaslException when Kerberos is enabled
    '''

16/09/21 01:40:30 WARN ipc.Client: Exception encountered while connecting to the server : 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid 
credentials provided (Mechanism level: Failed to find any Kerberos tgt)]


Operation failed: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException:
 GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]; Host Details : local host is: "c8f2n13.gpfs.net/192.0.2.11"; 
destination host is: "c8f2n13.gpfs.net":8020;

'''

'''

16/09/21 01:42:20 WARN ipc.Client: Exception encountered while connecting to the server : 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

'''

    Solution:

    For IBM BigInsights IOP 4.0/4.1/4.2, run the kinit -kt /etc/security/keytabs/nn.service.keytab nn/c8f2n13.gpfs.net@IBM.COM command.

    Note: Replace the hostname c8f2n13.gpfs.net with the node on which you will run the knit command.

    For other Hadoop distro, you must check the configuration dfs.namenode.kerberos.principal value.

  7. NameNode failed to start because the null pointer was encountered when SSL was configured with HDFS Transparency version 2.7.2-0.

    The NameNode will fail to start when SSL is configured because a null pointer exception is encountered due to missing ssl files for HDFS Transparency version 2.7.2-0.

    Hadoop NameNode log:
    STARTUP_MSG: Starting NameNode
.....
2016-11-29 18:51:45,572 INFO  namenode.NameNode (LogAdapter.java:info(47)) - 
registered UNIX signal handlers for [TERM, HUP, INT]

2016-11-29 18:51:45,575 INFO  namenode.NameNode (NameNode.java:createNameNode(1438)) - 
createNameNode []

...

2016-11-29 18:51:46,417 INFO  http.HttpServer2 (NameNodeHttpServer.java:initWebHdfs(86)) - 
Added filter 'org.apache.hadoop.hdfs.web.AuthFilter' (class=org.apache.hadoop.hdfs.web.AuthFilter)

2016-11-29 18:51:46,418 INFO  http.HttpServer2 (HttpServer2.java:addJerseyResourcePackage(609)) - 
addJerseyResourcePackage: 
packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
pathSpec=/webhdfs/v1/*

2016-11-29 18:51:46,434 INFO  http.HttpServer2 (HttpServer2.java:openListeners(915)) - 
Jetty bound to port 50070

2016-11-29 18:51:46,462 WARN  mortbay.log (Slf4jLog.java:warn(76)) - 
java.lang.NullPointerException

2016-11-29 18:51:46,462 INFO  http.HttpServer2 (HttpServer2.java:start(859)) - 
HttpServer.start() threw a non Bind IOException

java.io.IOException: !JsseListener: java.lang.NullPointerException

at org.mortbay.jetty.security.SslSocketConnector.newServerSocket(SslSocketConnector.java:516)

at org.apache.hadoop.security.ssl.SslSocketConnectorSecure.newServerSocket(SslSocketConnectorSecure.java:46)

at org.mortbay.jetty.bio.SocketConnector.open(SocketConnector.java:73)

at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:914)

at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:856)

at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:142)

at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:773)

at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:647)

at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:832)

at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:816)

at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1509)

at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1575)

2016-11-29 18:51:46,465 INFO  impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(211)) - 
Stopping NameNode metrics system...

2016-11-29 18:51:46,466 INFO  impl.MetricsSinkAdapter (MetricsSinkAdapter.java:publishMetricsFromQueue(141)) - 
timeline thread interrupted.

2016-11-29 18:51:46,466 INFO  impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(217)) - 
NameNode metrics system stopped.

2016-11-29 18:51:46,466 INFO  impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(607)) - 
NameNode metrics system shutdown complete.

2016-11-29 18:51:46,466 ERROR namenode.NameNode (NameNode.java:main(1580)) - Failed to start namenode.

java.io.IOException: !JsseListener: java.lang.NullPointerException

at org.mortbay.jetty.security.SslSocketConnector.newServerSocket(SslSocketConnector.java:516)

at org.apache.hadoop.security.ssl.SslSocketConnectorSecure.newServerSocket(SslSocketConnectorSecure.java:46)

at org.mortbay.jetty.bio.SocketConnector.open(SocketConnector.java:73)

at org.apache.hadoop.http.HttpServer2.openListeners(HttpServer2.java:914)

at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:856)

at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:142)

at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:773)

at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:647)

at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:832)

at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:816)

at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1509)

at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1575)

2016-11-29 18:51:46,468 INFO  util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1

2016-11-29 18:51:46,469 INFO  namenode.NameNode (LogAdapter.java:info(47)) - SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at <NameNodeHost>/<IPADDRESS>

************************************************************/

    Solution:

    The workaround solution for HDFS Transparency version 2.7.2-0 with SSL configured is to copy the /etc/hadoop/conf/ssl-client.xml and the /etc/hadoop/conf/ssl-server.xml files into /usr/lpp/mmfs/hadoop/etc/hadoop on all the nodes.

  8. org.apache.hadoop.ipc.RemoteException(java.io.IOException): blocksize(xxxxx) should be an integral mutiple of dataBlockSize(yyyyy)

    HDFS Transparency + IBM Storage Scale, the blocksize of file system could only be 64KB, 128KB, 256KB, 512KB, 1MB, 2MB, 4MB, 8MB and 16MB. Therefore, the dataBlockSize(yyyyy) must be the blocksize of your IBM Storage Scale file system. If your hadoop workloads take blocksize which is not integral multiple of dataBlockSize, you will see this issue. Typically, such as Accumulo:

    Accumulo TServer failed to start with the below error.
    Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): blocksize(1181115904) should be an integral mutiple of dataBlockSize(1048576)

       at org.apache.hadoop.hdfs.server.namenode.GPFSDetails.verifyBlockSize(GPFSDetails.java:230)

       at org.apache.hadoop.hdfs.server.namenode.GPFSNamesystem.startFile(GPFSNamesystem.java:254)

       at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:632)

       at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:397)

       at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)

       at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)

       at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)

       at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)

       at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)

       at java.security.AccessController.doPrivileged(Native Method)

       at javax.security.auth.Subject.doAs(Subject.java:422)

    Solution:

    For Accumulo, add the configuration in Accumulo > Configs > Custom accumulo-site and set tserver.wal.blocksize = <value-of-your-gpfs-filesystem> and then restart the service.

  9. Got exception: java.io.IOException No FileSystem for scheme: hdfs

    Running a service failed.

    The service log shows Got exception: java.io.IOException No FileSystem for scheme: hdfs message.

    The issue might occur because of a Maven-assembly issue (refer to hadoop No FileSystem for scheme: file) with duplicated file system in the classpath.

    Solution:

    If this exception is seen, add fs.hdfs.impl = org.apache.hadoop.hdfs.DistributedFileSystem into the core-site.xml to resolve the issue.

  10. java.io.IOException: Couldn't clear parent znode /hadoop-ha/<HA-cluster-ID> when NameNode HA is enabled.

    This exception can be seen when you start the HDFS Transparency NameNode through the Ambari GUI or from the command line which indicates that there are some files or directories under /hadoop-ha/<HA-cluster-ID> that are preventing the NameNode from starting up.

    Solution:

    Manually remove the directory by running <zookeeper-home-dir>/bin/zkCli.sh -server <one-zookeeper-server-hostname>:<network-port-number> rmr /hadoop-ha from any one Hadoop node. For example, the command will be /usr/iop/4.2.0.0/zookeeper/bin/zkCli.sh -server c16f1n02:2181 rmr /hadoop-ha for IOP 4.2 with hostname c16f1n02 and default zookeeper network port number 2181.

  11. datanode.DataNode (BPServiceActor.java:run(840)) - ClusterIds are not matched, existing.

    If a DataNode did not come up and the DataNode's log (located under /usr/lpp/mmfs/hadoop/logs) contains the above error message, then the HDFS Transparency DataNode might have been at one time used for a different HDFS Transparency cluster.

    Solution:

    Run the /usr/lpp/mmfs/hadoop/sbin/initmap.sh <your-file-system> diskmap nodemap clusterinfo command on the node and try to start it again. The command will update the following three files and will place the node into the current HDFS Transparency cluster:
    • /var/mmfs/etc/clusterinfo4hdfs
    • /var/mmfs/etc/diskid2hostname
    • /var/mmfs/etc/nodeid2hostname
  12. All blocks to be queried must be in the same block pool
    When you run workloads (such as Impala or IBM BigSQL), you might hit the following exception:
    All blocks to be queried must be in the same block pool: 
BP-gpfs_data,/gpfs_data/CDH5/user/root/POC2/2014-07.txt:blk_318148_0 and 
LocatedBlock{BP-gpfs_data,/gpfs_data/CDH5/user/root/POC2/2014-09.txt:blk_318150_0; 
getBlockSize()=134217728; corrupt=false; offset=0; 
locs=[DatanodeInfoWithStorage[192.0.2.12:50010,,DISK]]} are from different pools.

    Solution:

    Change the dfs.datanode.hdfs-blocks-metadata.enabled field to false in hdfs-site.xml and restart HDFS Transparency and Impala/BigSQL.

  13. Exception: Failed to load the configurations of Core GPFS (/<gpfs mount point>)
    When you start the HDFS Transparency NameNode or DataNode, the daemon is not up and you get the following exceptions:
    Exception: Failed to load the configurations of Core GPFS (/<gpfs mount point>) :
org.apache.hadoop.hdfs.server.namenode.GPFSDetails.refreshCoreGPFSConfig()

    Solution:

    Check the /var/mmfs/etc/clusterinfo4hdfs, /var/mmfs/etc/diskid2hostname and /var/mmfs/etc/nodeid2hostname on all the Transparency nodes. If they are of 0 size, run /usr/lpp/mmfs/hadoop/sbin/initmap.sh <your-file-system-name> diskmap nodemap clusterinfo on all the Transparency nodes.

  14. UID/GID failed with illegal value “Illagal value: USER = xxxxx > MAX = 8388607"

    Solution:

    If you have installed Ranger and need to leverage Ranger capabilities, you need to make the UID/GID less than 8388607.

    If you do not need Ranger, then follow these steps to disable Ranger from HDFS Transparency:
    1. Stop HDFS Transparency
      mmhadoopctl connector stop -N all
    2. On the NameNode, set the gpfs.ranger.enabled = false in /usr/lpp/mmfs/hadoop/etc/hadoop/gpfs-site.xml
      <property>
  <name>gpfs.ranger.enabled</name>
  <value>false</value>
</property>
    3. Sync the HDFS Transparency configuration
      mmhadoopctl connector syncconf /usr/lpp/mmfs/hadoop/etc/hadoop
    4. Start HDFS Transparency
      mmhadoopctl connector start -N all
      Note: For HDFS Transparency 2.7.3-3 and later, when Ranger is enabled, uid greater than 8388607 is supported.
  15. Issues that can be encountered if the IBM Storage Scale ACL is not set properly
    1. hadoop fs -ls command failed with Operation not supported message.
      Even when the Hadoop command failed, the POSIX command can be executed successfully.
      # hadoop fs -ls /

      ls: Operation not supported: /bigpfs/mapred

    2. hdfs dfs -mkdir and hdfs dfs -ls commands are not working after installation.
      The hdfs dfs -mkdir command fails with NullPointerExecption and the hdfs dfs -ls / command fails with No such file or directory message.
      # /usr/lpp/mmfs/hadoop/bin/hdfs dfs -mkdir -p /user
      mkdir: java.lang.NullPointerException
      # /usr/lpp/mmfs/hadoop/bin/hdfs dfs -ls /

      ls: `/': No such file or directory

      Solution:

      These issues occur when the ACL mode on the IBM Storage Scale file system is set to nfs4 instead of all.

      If IBM Storage Scale CES (SMB and NFS) is used, ACL semantics are required to be set to nfsv4.
      Note: When HDFS is used do not set the ACL semantics to nfsv4 because HDFS and IBM Storage Scale HDFS Transparency support only POSIX ACLs. Therefore, the -k option can be set to value posix or all.
      To display the type of authorization currently set on the file system, issue the following command:
      # /usr/lpp/mmfs/bin/mmlsfs bdafs -k
flag                  value                    description
------------------- ------------------------ -----------------------------------
-k                     nfs4                      ACL semantics in effect

      If the value is nfs4, change it to all.

      To change from nfs4 to all, issue the following command:
      # /usr/lpp/mmfs/bin/mmlsfs bdafs -k
flag                value                    description
------------------- ------------------------ -----------------------------------
-k                   all                         ACL semantics in effect
      Note: -k all means that any supported ACL type is permitted. This includes traditional GPFS (posix) and NFS V4 and Windows ACLs (nfsv4).

      When you are in a HDFS Transparency environment, ensure that the -k value is set to all.

      Restart HDFS Transparency after ACL is set to ALL.

  16. User permission denied when Ranger is disabled

    If Kerberos is enabled and Ranger flag is disabled, then the user might get permission denied errors when accessing the file system.

    Solution:

    Check the Kerberos principal mapping hadoop.security.auth_to_local field in /usr/lpp/mmfs/hadoop/etc/hadoop/core-site.xml or in Ambari under HDFS Config to ensure that the NameNode and DataNode are mapped to root instead of hdfs.

    For example, change

    From:
    RULE:[2:$1@$0](dn@COMPANY.DIV.COM)s/.*/hdfs/ 
RULE:[2:$1@$0](nn@COMPANY.DIV.COM)s/.*/hdfs/
    To:
    RULE:[2:$1@$0](dn@COMPANY.DIV.COM)s/.*/root/ 
RULE:[2:$1@$0](nn@COMPANY.DIV.COM)s/.*/root/
    Then restart the HDFS service in Ambari or HDFS Transparency by executing the following commands:
    /usr/lpp/mmfs/bin/mmhadoopctl connector stop

/usr/lpp/mmfs/bin/mmhadoopctl connector start
  17. IBM Storage Scale NSD is not able to be recovered in FPO.

    In an FPO environment, the IBM Storage Scale NSD is not able to be recovered after the node got expelled or the NSD went down and the auto recovery failed.

    Note: This can also occur while performing a STOP ALL/Start ALL from Ambari.

    If you see the Attention: Due to an earlier configuration change the file system is no longer properly replicated message on executing the mmlsdisk <fs> command, mmrestripefs -R command is needed to re-replicate the files after all the disks are brought back.

    IBM Storage Scale will also do a special check of file system recovery log file. To guarantee the consistency of the file system, if the number of failure groups with down disk is greater than or equal to the current log file replication, IBM Storage Scale will also panic the file system.

    Solution:

    1. Run mmrestripefs <fs> -R to restore all the files (including FS recovery log files) to their designated degree of replication as they were not properly replicated when there were too many down disks. This will also help to avoid the re-occurrence that FS panic on one single metadata disk failure.
    2. After the command completes its execution, run the following command to double check the log file's replica restored to the expected value 3. (FPO replica is set to 3 usually)
      # mmdsh -N all mmfsadm dump log | egrep "Dump of LogFile|nReplicas"

      Dump of LogFile at 0x3FFEE900A990 stripe group mygpfs state 'open for append' type 'striped': nReplicas 3 logGroupInconsistent 0 whichPartial 0 whichDesc 0

      Note:
      • The example shows that the nReplicas is set to 3.
      • Upgrade to GPFS 5.0.1.2 or later. In the newer release, GPFS will not trigger mmrestripefs -r during the auto recovery if there is not enough FGs to satisfy the needed replica. This can help in avoiding the same issue as here. A side effect is it will not try the auto-recovery (includes the attempts to start disk) if not enough FGs are left. Therefore, if there are disk down after many NSD servers restart, you might need to manually run the mmchdisk start to start the disks when the NSD servers are back.
      • Do not execute mmstripefs -r manually if there are not enough FG left to satisfy the desired replica. If you have already executed it, then run mmrestripefs -R after the down disks are brought back up.

    To avoid this issue, follow the FPO Maintenance procedure.

    If you are using Ambari, do not perform a STOP ALL, START ALL or shutdown/restart of the IBM Storage Scale service from the Ambari GUI.

    See the following topics in the IBM Storage Scale KC under the File Placement Optimizer subsection under the Administering section, to perform the maintenance of FPO cluster. Temporarily disable auto recovery before the planned maintenance. Especially do not let the disks fail and the automatic recovery gets initiate.
    • Restarting a large IBM Storage Scale cluster
    • Upgrade FPO
    • Rolling upgrade
    • Handling disk failures
    • Handling node failures
  18. DataNode reports exceptions after Kerberos is enabled on RHEL7.5
    If your Kerberos KDC or your Kerberos client are on RHEL7.5 at version 1.15.1-18 (default version shipped in RHEL7.5), you might hit the following exceptions when you start the DataNodes:
    2018-08-06 14:10:56,812 WARN  ipc.Client (Client.java:run(711)) - 
Couldn't setup connection for dn/140.bd@EXAMPLE.COM to 139.bd/**.**.**.**:8020
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: 
No valid credentials provided (Mechanism level: Ticket expired (32) - PROCESS_TGS)]
    at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)

    Solution:

    Upgrade the Kerberos KDC (krb5-libs and krb5-server) from version 1.15.1-18 to version 1.15.1-19 or upgrade the Kerberos client (krb5-workstation and krb5-libs) from version 1.15.1-18 to version 1.15.1-19.

  19. NameNode fails to start - Cannot find org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer

    With the Ranger enabled, the NameNode fails to start and HDFS Transparency shows the cannot find org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer error.

    Solution:

    The hadoop-env.sh file did not have the proper Ranger setup. See Apache Ranger on how to add the Apache Ranger files properly within the hadoop-env.sh file as well as ensure that the 4 ranger-*.xml files are copied to the proper HDFS Transparency directory.

    If you are using Ambari, restart the HDFS service. Otherwise, restart HDFS Transparency by executing the /usr/lpp/mmfs/bin/mmhadoopctl connector stop/start command.

    For example:

    The /usr/hdp/current/ranger-hdfs-plugin path was not used and the /usr/hdp/<your-HDP-version>/ranger-hdfs-plugin path was used. Therefore, set up the hadoop-env.sh file to use the path in your environment.

  20. Read from HDFS interface hangs (for example, hadoop dfs -get, hadoop dfs -copyToLocal)

    When you are reading data from the HDFS interface, the read command will hang. If you are using Ambari, the service checks from the Ambari GUI will be timed out.

    Debug output from the command shows timeout after 60s:
    DEBUG hdfs.DFSClient: Connecting to datanode x.x.x.x:50010
DEBUG sasl.SaslDataTransferClient: SASL encryption trust check: 
localHostTrusted = false, remoteHostTrusted = false
DEBUG sasl.SaslDataTransferClient: SASL client skipping handshake in 
unsecured configuration for addr = /x.x.x.x, datanodeId = DatanodeInfoWithStorage[xxxxx]
WARN hdfs.DFSClient: Exception while reading from .....

java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel 
to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/x.x.x.x:37058 
remote=/x.x.x.x:50010]
        at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)

    However, when the HDFS Transparency debug is set, no exceptions are seen in the logs. Increasing the timeout value of the HDFS client socket does not resolve the issue.

    Note: Writing in the HDFS interface works. The issue is only while reading from the HDFS interface.

    Solution:

    Configure dfs.datanode.transferTo.allowed as false in the hdfs-site.xml and restart HDFS Transparency. When dfs.datanode.transferTo.allowed is true by default, some transfers on the socket might hang on some platforms (OS/JVM).

    If you are using Ambari, ensure that you have set dfs.datanode.transferTo.allowed to false in the HDFS service configuration. If the field does not exist, add a new field and restart the HDFS service.

  21. Hive LLAP queries failed

    By default, LLAP uses inode paths to access data from the file system using the hive.llap.io.user.fileid.path=true setting which will not work on the IBM Storage Scale file system.

    Solution:

    Configure hive.llap.io.use.fileid.path=false to have LLAP access the file from file path instead of inode number.

    If you are using Ambari, then go to Ambari GUI > Hive > Configs - Advanced Tab.
    1. Search for hive.llap.io.use.fileid.path field.
    2. If you find it and it is not set to false, change it to false. If you do not find it, add hive.llap.io.use.fileid.path=false under Custom hive-site Add Property.
    3. Save configuration and restart Hive.
  22. DataNode failed with error Failed to refresh configuration

    The DataNode will not be able to come up if the initmap.sh internal generated configuration files are stale or incorrect.

    The following error can be seen:
     ERROR datanode.DataNode (DataNode.java:secureMain(2922)) - Exception in secureMain
java.io.IOException: Failed to refresh configurations
       at org.apache.hadoop.hdfs.server.namenode.GPFSDetails.refreshCoreGPFSConfig(GPFSDetails.java:613)
       at org.apache.hadoop.hdfs.server.namenode.GPFSDetails.init(GPFSDetails.java:175)
       at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:477)
       at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2821)
       at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2713)
       at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2766)
       at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2915)
       at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2939)
Caused by: java.io.IOException: /var/mmfs/etc/hadoop/clusterinfo4hdfs.fpo is outdated because initmap.sh failed probably.
       at org.apache.hadoop.hdfs.server.namenode.GPFSFs.collectClusterInfo(GPFSFs.java:414)
       at org.apache.hadoop.hdfs.server.namenode.GPFSFs.collectInfo(GPFSFs.java:551)
       at org.apache.hadoop.hdfs.server.namenode.GPFSDetails.refreshCoreGPFSConfig(GPFSDetails.java:608)
       ... 7 more

    Solution:

    The DataNode internal configuration files need to be regenerated. If possible, restart the NameNode, or restart the Standby NameNode if HA is configured, or touch those internal configuration files and then restart the DataNode from the Ambari GUI or from the command line.

    For more information, see Cluster and file system information configuration.

  23. Viewfs hadoop dfs -copyFromLocal -l fails
    With ViewFs configuration, running the copyFromLocal -l will generate failure. 
    $ hadoop fs -copyFromLocal -f -l /etc/hosts 
/TestDir/Test_copyFromLocal.1545110085.28040/copyFromLocal -copyFromLocal: Fatal internal error

org.apache.hadoop.fs.viewfs.NotInMountpointException: getDefaultBlockSize on empty path is invalid
at org.apache.hadoop.fs.viewfs.ViewFileSystem.getDefaultBlockSize(ViewFileSystem.java:695)
at org.apache.hadoop.fs.FilterFileSystem.getDefaultBlockSize(FilterFileSystem.java:420)
at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.create(CommandWithDestination.java:505)
at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:484)
at org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(CommandWithDestination.java:407)
at org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:342)
at org.apache.hadoop.fs.shell.CopyCommands$CopyFromLocal.copyFile(CopyCommands.java:357)
at org.apache.hadoop.fs.shell.CopyCommands$CopyFromLocal.copyFileToTarget(CopyCommands.java:365)
at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:277)
at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:262)
at org.apache.hadoop.fs.shell.Command.processPathInternal(Command.java:367)
at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:331)
at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:304)
at org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:257)
at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:286)
at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:270)
at org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:228)
at org.apache.hadoop.fs.shell.CopyCommands$Put.processArguments(CopyCommands.java:295)
at org.apache.hadoop.fs.shell.CopyCommands$CopyFromLocal.processArguments(CopyCommands.java:385)
at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:120)
at org.apache.hadoop.fs.shell.Command.run(Command.java:177)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:328)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:391)

    Solution:

    HDP SPEC-56 has a fix For IBM to distribute to customer. The fix is to replace the jar files in the environment.

    Contact IBM service if you want the fix for SPEC-56/RTC Defect 20924.

  24. numNode=0 due to time synchronization issue

    When numNode is 0, the number of DataNode available for use is 0.

    The NameNode Log will show the following error:
    Caused by: org.apache.hadoop.hdfs.server.namenode.GPFSRunTimeException: 
110259f803894787b8d364019aa4106f fileid=259093 block=blk_259093_0, numNodes=0
  at org.apache.hadoop.hdfs.server.blockmanagement.GPFSBlockManager.getStorages(GPFSBlockManager.java:102)
  at org.apache.hadoop.hdfs.server.blockmanagement.GPFSBlockManager.createLocatedBlock(GPFSBlockManager.java:220)

    Solution:

    Check if the Scale cluster time are in sync.

    Run: mmdsh -N all "date +%m%d:%H%M%S-%s"

    If not, ensure that the time are in sync.

  25. How to enable audit logging in HDFS Transparency
    Solution:
    1. Manually enable the audit log by editing the hadoop-env.sh file.
      • Edit the /var/mmfs/hadoop/etc/hadoop/ hadoop-env.sh file to include the following entry:
        export HDFS_AUDIT_LOGGER=INFO,RFAAUDIT
      • Synchronize the configuration files by using the mmhadoopctl command (for HDFS Transparency 3.1.0-x and earlier) or upload the configuration files into CCR by using the mmhdfs command (for CES HDFS Transparency 3.1.1-x).
      • Restart HDFS Transparency.
      • Verify looking into the ${hadoop.log.dir}/hdfs-audit.log file for audit information such as:
        INFO FSNamesystem.audit: allowed=true   
cmd=getfileinfo src=
cmd=listStatus  src
(auth:KERBEROS)
    2. Manually enable the audit log by editing the log4j.properties file.
      • Edit the /var/mmfs/hadoop/etc/hadoop/log4j.properties file to change the following entry:
        From
        hdfs.audit.logger=INFO,NullAppender
        To
        hdfs.audit.logger=INFO,RFAAUDIT
      • Synchronize the configuration files by using the mmhadoopctl command (for HDFS Transparency 3.10-x and earlier) or upload the configuration files into CCR by using the mmhdfs command (for CES HDFS Transparency 3.1.1-x).
      • Restart HDFS Transparency.
      • Verify by looking into the ${hadoop.log.dir}/hdfs-audit.log file for the audit information such as:
        INFO FSNamesystem.audit: allowed=true   
cmd=getfileinfo src=
cmd=listStatus  src
(auth:KERBEROS)
    3. If you are using Ambari, then go to HDFS service > Configs > Advanced tab > Advanced hdfs-log4j and set the hdfs.audit.logger=INFO,RFFAAUDIT, save the configuration and restart the HDFS service.
      Advanced hdfs-log4j
      #
# hdfs audit logging
#
hdfs.audit.logger=INFO,RFAAUDIT
log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=${hdfs.audit.logger}
log4j.additivity.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=false
  26. Ulimit issues
    1. Java™ exception: All datanodes DatanodeInfoWithStorage are bad error

      Solution:

      If you see similar exception errors of java.io.IOException: All datanodes DatanodeInfoWithStorage are bad when running the map/reduce jobs, you must increase your ulimit -n and ulimit -u to 64K.
      15/12/30 07:09:16 INFO mapreduce.Job: Task Id : attempt_1450797405784_0281_m_000005_0, Status : FAILED

c902f10x09: Error: java.io.IOException: All datanodes DatanodeInfoWithStorage
[192.0.2.13:50010,DS-1ca06221-194c-47d2-82d0-8b602a64921b,DISK] are bad. Aborting...

c902f10x09:     at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1218)

c902f10x09:     at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1016)

c902f10x09:     at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:560)
    2. Job getting IOException BlockMissingException could not obtain block error and cannot read the file using HDFS until HDFS Transparency restarts. The file can be read using POSIX.
      Error:
      org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block:
...
Caused by: org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block
...

      Solution:

      Check the DataNode for errors:
      IOException "don't match block error", then search previous messages for that block blk_<value> under sees "FileNotFoundException" and "Too many open files" error previously, then this means the DataNode does not have enough open files limit value.

2020-03-17 12:22:46,638 INFO  datanode.DataNode (DataXceiver.java:writeBlock(1023)) - opWriteBlock BP-XXXXXXXX:blk_5079204_0 received exception java.io.FileNotFoundException: /dev/null (Too many open files)

2020-03-17 13:06:50,384 ERROR datanode.DataNode (DataXceiver.java:run(326)) - <HOST>.com:1019:DataXceiver error processing READ_BLOCK operation  src: /<IP>:49771 dst: /<IP>:1019
java.io.IOException:  Offset 0 and length 60986628 don't match block BP-XXXXXXXX:blk_5079204_0 ( blockLen 0 )
  at org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:429)
  at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:684)
  at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:163)
  at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:111)
  at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:293)
  at java.lang.Thread.run(Thread.java:748)

      The issue occurs when there are not enough open file limit set. The FileNotFoundException exception is thrown and the internal data structure is left in an uninitiated state. The block length is not initialized to the correct value and is still at 0. Therefore, when the block was read later, the sanity check failed in HDFS Transparency.

      To fix this issue, increase the open file limit.

      To increase the open file limit, see OS tuning for all nodes in HDFS Transparency.

  27. HDFS Transparency fails to start if the Java version is upgraded.

    When you are starting HDFS Transparency in CES HDFS or non-CES HDFS you get the following error:

    ERROR: JAVA_HOME /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el7_7.x86_64 does not exist.

    Solution:

    If the Java version was upgraded, or if a kernel patch or OS upgrade, upgraded the Java version, then the Java path is changed to the updated version path. Therefore, the JAVA_HOME setting for the user profile (.bashrc) and the JAVA_HOME in /var/mmfs/hadoop/etc/hadoop/hadoop-env.sh need to be updated to reflect the updated JAVA directory.

    The updated Java directory can be seen under /etc/alternatives/java.
    # ls -l /etc/alternatives/jre lrwxrwxrwx 1 root root 62 Jan 21 10:05 /etc/alternatives/jre -> 
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre
    This path has to be set in /var/mmfs/hadoop/etc/hadoop/hadoop-env.sh and then the user profile (.bashrc):
    export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre
    Instead of using the versioned path to the JRE, you can also use the version agnostic symbolic link that is automatically updated with every upgrade:
    export JAVA_HOME=/etc/alternatives/jre
  28. In the NameNode log, when you are appending a file, you get NullPointerException on the file operations. The exception stack trace includes the following:
    java.lang.NullPointerException
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget()
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget()
        at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4AdditionalDatanode()
        at org.apache.hadoop.hdfs.server.namenode.GPFSNamesystemV0.getAdditionalDatanode()
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode()
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAdditionalDatanode()
(...)

    Solution:

    The problem occurs when there are fewer DataNodes available than the value of the specified replication factor (dfs.replication). Add more DataNodes to the cluster or reduce the value of the dfs.replication parameter to resolve the error.

    To get the existing replication value, run the following command: 
    mmhdfs config get hdfs-site.xml -k dfs.replication
    The output for the above command is displayed as follows:
    dfs.replication=3
    To reduce the replication value, run the following command: 
    mmhdfs config set hdfs-site.xml -k dfs.replication=2
  29. Write operations (put/copyFromLocal) fail with the following error message:

    File [FILENAME] could only be written to 0 of the X minReplication nodes. There are Y datanode(s) running and no node(s) are excluded in this operation.

    Solution:

    The problem occurs when the dfs.datanode.du.reserved value is set too large. Ensure that it is set to the recommended value or reduce the value to resolve the error.

  30. NameNode fails to start in HDFS Transparency 3.1.1-11, 3.1.1-12, 3.2.2-2 or 3.2.2-3 with the following stack trace:
    2023-01-25 17:58:22,926 ERROR namenode.NameNode (NameNode.java:main(1828)) - Failed to start namenode.
java.lang.NoClassDefFoundError: org/codehaus/jackson/Versioned
...
Caused by: java.lang.ClassNotFoundException: org.codehaus.jackson.Versioned

    Solution:

    Add jackson-core-asl-1.9.13.jar from the Maven repository, or from HDFS Transparency version 3.1.1-10 or 3.2.2-1, to the following paths on every HDFS node in the cluster.
    1. /usr/lpp/mmfs/hadoop/share/hadoop/hdfs/lib/
    2. /usr/lpp/mmfs/hadoop/share/hadoop/common/lib/

    The jackson-core-asl-1.9.13.jar has an MD5 checksum of 319c49a4304e3fa9fe3cd8dcfc009d37.

  31. To list all the HDFS encryption zones in an IBM Storage Scale file system using the GPFS policy engine, use the following steps to create a policy rule that matches the encryption zone directory structure and prints the names of the directories containing the system.raw.hdfs.crypto.encryption.zone attribute. These steps are a workaround as the hdfs crypto -listZones command is not supported in HDFS Transparency.
    It is important to note that these steps should be executed on the node where the file system is locally mounted:
    1. Create a policy rule file that matches the encryption zone directory structure (for example, list_encryption_zones.pol):
      RULE 'dirsRule' LIST 'dirs'
DIRECTORIES_PLUS
SHOW(varchar(mode) || ' ' || varchar(XATTR('system.raw.hdfs.crypto.encryption.zone')))
where (XATTR('system.raw.hdfs.crypto.encryption.zone') IS NOT NULL)

      In this rule, the system.raw.hdfs.crypto.encryption.zone extended attribute matches any directory in IBM Storage Scale to identify encryption zones.

    2. To run this policy rule, save it to a file (follow step 1) and apply the policy to the file system using the following command: 
      #/usr/lpp/mmfs/bin/mmapplypolicy GPFS -P list_encryption_zones.pol -I defer -f /tmp/encryption_zone
      Note: It is important to keep in mind that the GPFS policy engine applies policies to the local file system on the node where the file system is mounted. To ensure accurate and consistent policy application, it is recommended to run policies on the node where the file system is locally mounted.
      [root@]# cat list_encryption_zones.pol
RULE 'dirsRule' LIST 'dirs'
DIRECTORIES_PLUS
SHOW(varchar(mode) || ' ' || varchar(XATTR('system.raw.hdfs.crypto.encryption.zone')))
where (XATTR('system.raw.hdfs.crypto.encryption.zone') IS NOT NULL)

[root@]# mmapplypolicy GPFS -P list_encryption_zones.pol -I defer  -f /tmp/encryption_zone
[I] GPFS Current Data Pool Utilization in KB and %
Pool_Name                  KB_Occupied           KB_Total     Percent_Occupied
system                        393849856         1992294400        19.768657484%
[I] 199168 of 307968 inodes used: 64.671654%.
[W] Attention: In RULE 'dirsRule' LIST name 'dirs' appears but there is no corresponding "EXTERNAL LIST 'dirs' EXEC ... OPTS ..." rule to specify a program to process the matching files.
[I] Loaded policy rules from list_encryption_zones.pol.
Evaluating policy rules with CURRENT_TIMESTAMP = 2023-05-04@12:07:36 UTC
Parsed 1 policy rules.
RULE 'dirsRule' LIST 'dirs'
DIRECTORIES_PLUS
SHOW(varchar(mode) || ' ' || varchar(XATTR('system.raw.hdfs.crypto.encryption.zone')))
where (XATTR('system.raw.hdfs.crypto.encryption.zone') IS NOT NULL)
[I] 2023-05-04@12:07:37.298 Directory entries scanned: 195159.
[I] Directories scan: 191829 files, 3294 directories, 36 other objects, 0 'skipped' files and/or errors.
[I] 2023-05-04@12:07:39.739 Parallel-piped sort and policy evaluation. 195159 files scanned.
[I] 2023-05-04@12:07:39.772 Piped sorting and candidate file choosing. 2 records scanned.
[I] Summary of Rule Applicability and File Choices:
Rule#        Hit_Cnt         KB_Hit         Chosen      KB_Chosen         KB_Ill    Rule
     0              2              0              2              0              0    RULE 'dirsRule' LIST 'dirs' DIRECTORIES_PLUS SHOW(.) WHERE(.)

[I] Filesystem objects with no applicable rules: 195138.

[I] GPFS Policy Decisions and File Choice Totals:
Chose to list 0KB: 2 of 2 candidates;
Predicted Data Pool Utilization in KB and %:
Pool_Name                  KB_Occupied           KB_Total     Percent_Occupied
system                        393879552         1992294400        19.770148026%
[I] 2023-05-04@12:07:39.776 Policy execution. 0 files dispatched.
[I] A total of 0 files have been migrated, deleted or processed by an EXTERNAL EXEC/script;
    0 'skipped' files and/or errors.
    Encryption zones directories list will be collected in /tmp/encryption_zone.list.dirs directory.
    [root@]# cat /tmp/encryption_zone.list.dirs
10041 1253024653 0  drwxr-xr-xkey -- /gpfs/datadir_regr32-01/zone
179202 968077442 0  drwxr-xr-xkey -- /gpfs/datadir_regr32-01/test_zone
  32. To address any failures that may occur during the installation or upgrade of HDFS versions 3.1.1-15 or 3.2.2-6, follow these steps before a retry.
    1. Execute one the following cleanup commands to remove the RPM package from all nodes.

      To remove gpfs.hdfs-protocol-3.1.1-15.x86_64 (HDFS 3.1.1-15), issue the next command:

      mmdsh -N all "rpm -e gpfs.hdfs-protocol-3.1.1-15.x86_64"

      To remove gpfs.hdfs-protocol-3.2.2-6.x86_64 (HDFS 3.2.2-6), issue the next command:

      mmdsh -N all "rpm -e gpfs.hdfs-protocol-3.2.2-6.x86_64"
    2. Delete the Hadoop directory at the specified location on all nodes by using the next command:
      mmdsh -N all "rm -rf /usr/lpp/mmfs/hadoop/share/hadoop"
  33. Debug, trace, and logs.

    Solution:

    To check the state of the CES HDFS cluster, see the mmhealth command documentation in IBM Storage Scale: Command and Programming Reference Guide.

    To determine the status of the CES HDFS NameNodes state, run the following command:
    /usr/lpp/mmfs/hadoop/bin/hdfs haadmin -checkHealth -scale -all

    For more information, see the hdfs haadmin command.

    For HDFS Transparency, see HDFS Transparency protocol troubleshooting on how to Enable Debugging.

  34. CES HDFS Transparency cluster failed to start.
    
mmces service enable HDFS
or
mmces service start hdfs -a
    Solution:
    Note: Run /usr/lpp/mmfs/hadoop/bin/hdfs namenode -initializeSharedEdits, if the NameNode failed to start with the following exception:
    2019-11-22 01:02:01,925 ERROR namenode.FSNamesystem (FSNamesystem.java:<init>(911)) - GPFSNamesystem initialization failed.
java.io.IOException: Invalid configuration: a shared edits dir must not be specified if HA is not enabled.
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:789)
        at org.apache.hadoop.hdfs.server.namenode.GPFSNamesystemBase.<init>(GPFSNamesystemBase.java:49)
        at org.apache.hadoop.hdfs.server.namenode.GPFSNamesystem.<init>(GPFSNamesystem.java:74)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:706)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:669)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:731)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:968)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:947)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1680)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1747)
  35. Mapreduce container job exit with return code 1.

    Solution:

    If Container exited with a non-zero exit code 1. Error file: prelaunch.err occurred while running the Mapreduce workloads, add the following property into the mapred-site.xml to resolve the issue:
    <property>
   <name>mapreduce.application.classpath</name>
   <value>/usr/hadoop-3.1.2/share/hadoop/mapreduce/*, /usr/hadoop-3.1.2/share/hadoop/mapreduce/lib/*</value>
</property>
  36. mmhdfs hdfs status shows node is not a DataNode.
    The command mmhdfs hdfs status shows the following errors: 
    c16f1n13.gpfs.net:  This node is not a datanode
mmdsh: c16f1n13.gpfs.net remote shell process had return code 1.

    Solution:

    Remove the localhost value from the host.

    On the worker node, run:
    mmhdfs worker remove localhost
  37. All the NameNodes status shows standby after mmhdfs start/stop/restart commands.

    Solution:

    Use the mmces service command to start/stop NameNodes so that the proper state is reflected for the NameNodes.

    If the mmhdfs start/stop/restart command was executed against the NameNodes, run the mmces service start/stop hdfs to fix the issue.

  38. hdfs dfs -ls or another operation fails with a StandbyException.
    Running the hdfs dfs -ls command fails with a StandbyException exception:
    [root@scale12 transparency]# /usr/lpp/mmfs/hadoop/bin/hdfs dfs -ls /HDFS
2020-04-06 16:26:25,891 INFO retry.RetryInvocationHandler: 
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): 
Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:88)
at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:2010)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1447)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3129)
at org.apache.hadoop.hdfs.server.namenode.GPFSNamesystem.getFileInfo(GPFSNamesystem.java:494)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1143)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:939)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
, while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over scale12/192.0.2.21:8020 after 1 
failover attempts. Trying to failover after sleeping for 1157ms.
^C2020-04-06 16:26:27,097 INFO retry.RetryInvocationHandler: java.io.IOException: 
The client is stopped, while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo 
over scale11/192.0.2.20:8020 after 2 failover attempts. Trying to failover after
sleeping for 2591ms.
    Both the NameNodes are in standby and the CES has failed to select one as active. To verify, run the following command:
    /usr/lpp/mmfs/hadoop/bin/hdfs haadmin -getAllServiceState
    scale01:8020 standby scale02:8020 standby

    Solution:

    1. Check the NameNode that should be active by running the following command: 
      /usr/lpp/mmfs/bin/mmhealth node show -v HDFS_NAMENODE -N cesNodes
    2. For one of the nodes, the output shows the hdfs_namenode_wrong_state event.
    3. ssh to that node and set it manually to active by running the following command: 
      /usr/lpp/mmfs/hadoop/bin/hdfs haadmin -transitionToActive -scale
    4. Wait for 30 seconds and verify if the NameNode is now active by running the following commands: 
      /usr/lpp/mmfs/hadoop/bin/hdfs haadmin -getAllServiceState

      and

      /usr/lpp/mmfs/bin/mmhealth node show -v HDFS_NAMENODE -N cesNodes
  39. CES HDFS Transparency fails to start if the Java version is upgraded.

    Solution:

    For information on troubleshooting this issue, see HDFS Transparency fails to start if the Java version is upgraded.

  40. The mmhdfs command cannot recognize the FQDN hostnames if the NameNodes or DataNodes were added with short hostname.

    If IBM Storage Scale and HDFS Transparency are set up with short hostname then there is no issue with using a short hostname.

    If IBM Storage Scale is set up with FQDN and HDFS Transparency is set up with short hostname then mmhdfs does not recognize the node as a NameNode or DataNode.

    For example, the mmhdfs hdfs status command will state that this is not a NameNode and will exit with a return code 1.

    Solution:

    Set up Transparency to use FQDN by updating the hdfs-site.xml to set the NameNodes to FQDN and the worker file hostnames to FQDN.

  41. Multi-HDFS cluster deployment through IBM Storage Scale 5.1.1.0 installation toolkit is not supported.

    Solution:

    If you want to create multi-hdfs clusters on the same IBM Storage Scale, perform the following:
    1. Clear the installation toolkit HDFS metadata, by running the following command: 
      /spectrumscale config hdfs clear
    2. Follow Adding a new HDFS cluster into existing HDFS cluster on the same GPFS cluster using install toolkit.
      Note: Ensure that the creation of the new HDFS fields are unique from already existing HDFS cluster. The installation toolkit will not be able to check if there are duplicate values. The installation toolkit HDFS metadata will be regenerated after the CES HDFS cluster is deployed but will only contain the new HDFS cluster information.
  42. mmhealth node shows CES in Degraded state.
    When you are creating a CES HDFS cluster, mmhealth node shows CES -v as degraded and with hdfs_namenode_wrong_state message.
    [root@scale-31 ~]# mmhealth node show CES -v
Node name:      scale-31.openstacklocal
Component         Status        Status Change            Reasons
-------------------------------------------------------------------------------------------------------------
CES               DEGRADED      2021-05-05 09:52:29      hdfs_namenode_wrong_state(hdfscluster3)
  AUTH            DISABLED      2021-05-05 09:49:28      -
  AUTH_OBJ        DISABLED      2021-05-05 09:49:28      -
  BLOCK           DISABLED      2021-05-05 09:49:27      -
  CESNETWORK      HEALTHY       2021-05-05 09:49:58      -
    eth1          HEALTHY       2021-05-05 09:49:44      -
  HDFS_NAMENODE   DEGRADED      2021-05-05 09:52:29      hdfs_namenode_wrong_state(hdfscluster3)
  NFS             DISABLED      2021-05-05 09:49:25      -
  OBJECT          DISABLED      2021-05-05 09:49:28      -
  SMB             DISABLED      2021-05-05 09:49:26      -

[root@scale-31 ~]# mmhealth event show hdfs_namenode_wrong_state
Event Name:              hdfs_namenode_wrong_state
Event ID:                998178
Description:             The HDFS NameNode service state is not as expected (e.g. is in STANDBY but is supposed to be ACTIVE or vice versa)
Cause:                   The command /usr/lpp/mmfs/hadoop/sbin/mmhdfs monitor checkHealth -Y returned serviceState which does not match the expected state when looking at the assigned ces IP attributes
User Action:             N/A
Severity:                WARNING
State:                   DEGRADED

[root@scale-31 ~]# hdfs haadmin -getAllServiceState
scale-31.openstacklocal:8020                       active
scale-32.openstacklocal:8020                       standby
[root@scale-31 ~]#

[root@scale-31 ~]# mmces address list
Address     Node                      Ces Group          Attributes
----------- ----------------------- ------------------ ------------------
192.0.2.0   scale-32.openstacklocal   hdfshdfscluster3   hdfshdfscluster3
192.0.2.1   scale-32.openstacklocal   none               none
192.0.2.2   scale-32.openstacklocal   none               none
192.0.2.3   scale-31.openstacklocal   none               none
192.0.2.4   scale-31.openstacklocal   none               none
192.0.2.5   scale-31.openstacklocal   none               none
[root@scale-31 ~]#

    The issue here is that the CES IP is assigned to the Standby NameNode instead of the Active NameNode.

    Solution:

    The following are the three solutions for this problem:
    • Manually set the active NameNode to standby on the node by running the /usr/lpp/mmfs/hadoop/bin/hdfs haadmin -transitionToStandby -scale command. Then on the other node, set the standby NameNode to active by running the /usr/lpp/mmfs/hadoop/bin/hdfs haadmin -transitionToActive -scale command.
    • Move the CES IP to the active NameNode by running the mmces address move --ces-ip <CES IP> --ces-node <node name> command.
    • Restart the CES HDFS NameNodes by running the following commands: 
      mmces service stop HDFS -a
mmces service start HDFS -a
  43. Kerberos principal update not taking effect on changing KINIT_PRINCIPAL in hadoop-env.sh.

    Solution:

    The CES HDFS Kerberos information is cached at /var/mmfs/tmp/krb5cc_ces. Delete this file to force the update.

  44. If Kerberos was configured on multiple HDFS Transparency clusters using a common KDC server and the supplied gpfs_kerberos_configuration.py script, kinit with the hdfs user principal fails for all the clusters except the most recent one.

    The kerberos configuration script gpfs_kerberos_configuration.py, generates a keytab fie for the hdfs user under the /etc/security/keytabs/hdfs.headless.keytab default path. The kinit error occurs because the gpfs_kerberos_configuration.py script updated the keytab file and invalidated the copies of the keytab on the previous cluster.

    Solution:

    From the most recent HDFS Transparency cluster that the script was run, copy the keytab file to all the other HDFS Transparency cluster nodes where the script was run.

    For example:

    If Hadoop cluster A ran the gpfs_kerberos_configuration.py script which created the hdfs user principal and Hadoop cluster B ran the gpfs_kerberos_configuration.py script which then updated the original hdfs user keytab, copy the hdfs keytab from Hadoop cluster B to Hadoop cluster A to ensure that the Hadoop cluster A kinit works properly.

    This limitation has been fixed in HDFS Transparency 3.1.1.6.

  45. DataNodes are down after system reboot.

    Solution:

    HDFS Transparency DataNodes may not start automatically after a system reboot. As a workaround, you can manually start the DataNodes after the system reboot by using the following command from one of the CES nodes as root:
    #/usr/lpp/mmfs/hadoop/sbin/mmhdfs hdfs-dn start
  46. HDFS administrative commands, such as hdfs haadmin and hdfs groups cannot be executed from HDFS clients where Kerberos is enabled. The HDFS client ensures that the CES-HDFS user principle has the CES-HOST name instead of the NameNode hostname. The administrative commands fail with the following error:
    Caused by: java.lang.IllegalArgumentException: Server has invalid Kerberos principal:
nn/c88f2u33.pokprv.stglabs.ibm.com@HADOOP.COM, expecting:
nn/c88f2u31b.pokprv.stglabs.ibm.com@HADOOP.COM
at org.apache.hadoop.security.SaslRpcClient.getServerPrincipal(SaslRpcClient.java:337)
at org.apache.hadoop.security.SaslRpcClient.createSaslClient(SaslRpcClient.java:234)
at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:160)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:622)
at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:413)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:822)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:818)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:818)
... 15 more
    To resolve this, we have to add the following key in the core-site.xml file on the client:
    hadoop.security.service.user.name.key.pattern=*
    While using Cloudera Manager:
    1. Go to Clusters > IBM Spectrum Scale > Configuration > Cluster-wide Advanced Configuration Snippet (Safety Valve) for the core-site.xml file.
    2. Add the hadoop.security.service.user.name.key.pattern=* parameter and restart related services.