Troubleshooting InfoSphere Information Server on Hadoop
Use the information in this section to help you understand, isolate, and resolve issues with InfoSphere® Information Server on Hadoop.
This section provides descriptions of possible problems and the steps to correct them.
Message or symptom | Action |
---|---|
You receive the following error message when
you use a configuration file that statically defines nodes (fastname): If a host is down and you are using a static
configuration, you will see a similar message with State: LOST . |
To avoid host name issues, use a dynamic configuration file. Using a configuration file with statically defined nodes can cause incorrect node names or node names that do not match the node name that is expected by Hadoop, which is usually the long host name. |
You receive the following error message after
you run an InfoSphere
DataStage® job: The version of the shared libraries
that is loaded during InfoSphere
DataStage job
run time is incorrect. |
Network file system (NFS), Version 4 has a defect that loads the wrong shared libraries at run time. It affects any massively parallel processing (MPP) environment that uses NFS, Version 4 to share InfoSphere Information Server installation directories. To avoid this issue, use NFS Version 3. |
The wrong host name is returned by operating system commands. | Configure both YARN and InfoSphere Information Server to have the hostname and hostname -f commands return the fully qualified host name of the system. |
When you run large InfoSphere
DataStage jobs,
you receive the following error message: java/langOutOfMemoryError . |
Increase the heap size of the Java™ Virtual Machine (JVM) that is used by HDFS by setting the LIBHDFS_OPTS parameter to -Xmx1024m. This setting can be used to change the heap size to be higher or lower. |
When you run a parallel engine job from the InfoSphere
DataStage client
or when running $APT_ORCHHOME/etc/yarn_conf/start-pxyarn.sh from
the command line, you receive the following error message: The error is logged in the following directory: /tmp/yarn_client.out. |
This message is generated because the bin directory that contains the Hadoop YARN command is not in your PATH environment variable. Add this directory to your PATH environment variable or set the HADOOP_HOME or HADOOP_YARN_HOME environment variables so these two directories can be searched. Set these two environment variables at the job or project level if you run a parallel job from the InfoSphere DataStage client. Set the environment variables on the command prompt if you run the start-pxyarn.sh script from the command line. |
You see the following message in the InfoSphere
DataStage job
log:
|
In the /tmp/yarn_client.out file,
look for this error in /bin/yarn: If
you see this error, the bin directory that contains
the Hadoop YARN command is not in your PATH environment. Add this
directory to your PATH environment variable or
set the HADOOP_HOME or HADOOP_YARN_HOME environment
variables so these two directories can be searched. Set these two
environment variables at the job or project level if you run a parallel
job from the InfoSphere
DataStage client.
Set the environment variables on the command prompt if you run the start-pxyarn.sh script
from the command line. |
You receive the following message:
|
Set the APT_USE_IPV4 environment variable to true. |
You receive the following message:
|
The InfoSphere DataStage administrator must be able to access the path that was provided for the APT_YARN_USER_CACHED_CRED_PATH environment variable. Verify that the InfoSphere DataStage administrator has access to the path. |
You receive the following error message when
you run InfoSphere
DataStage jobs: These jobs request
zero containers from YARN. For example, you might be trying to import
data. |
Use the Peek operator in the InfoSphere DataStage job flow so that a container is requested from the Application Master. The issue occurs when zero containers are requested. |
You receive the following error message when
you open the InfoSphere
DataStage Designer
client or InfoSphere
DataStage Director
client:
|
This issue occurs when the LD_LIBRARY_PATH file path begins with /usr/lib64. Verify that the library paths that InfoSphere Information Server uses do not start with /usr/lib64. If one or more of the paths start with /usr/lib64, then move this part of the path name to the middle or end of the file path. |
Your Hadoop first-in-first-out (FIFO) scheduling is not working. | Hadoop first-in-first-out (FIFO) scheduling is not supported by InfoSphere Information Server. Static configuration files, node pools, and data locality features in InfoSphere Information Server are incompatible with FIFO scheduling. |
You receive the following error message in the yarnclient.out log
when you run an InfoSphere
DataStage job:
|
Add the user that does not have the appropriate
permissions to write to the /user directory to
the group that has permission to write to the directory. To add the
user to the group, issue the following command:
|
You receive the following error message when
you run an InfoSphere
DataStage parallel
job:
|
Increase the value that you set for the APT_YARN_MSG_TIMEOUT environment variable. |
You receive the following error message in the InfoSphere
DataStage job
log:
|
Add the file path to the Hadoop native library
to the LD_LIBRARY_PATH environment variable.
For example:
|
You receive the following error message when
you run an InfoSphere QualityStage® job:
|
Ensure that you correctly set permissions to access files. |
You receive the following error message:
|
This happens because the YARN client is not able to receive communication from the Application Master that it tried to start. Check the yarn_client.out file to find out why is it not able to listen to the Application Master. It could be because the Application Master is not started because of any of the following reasons:
|
You receive the following error message when
you run an InfoSphere
DataStage job
on SuSE Linux:
|
This error indicates that the version of the Java Development Kit (JDK) that is installed with InfoSphere Information Server is different than the version that is installed with the YARN client that you are using with InfoSphere Information Server. To resolve this issue, update the value for the environment variable APT_YARN_JDK_LIBRARY_PATH to specify the Java Development Kit libraries that belong to the same Java Development Kit that is used by the YARN client. You can modify this setting in the dsenv file which is located in the following default directory: /opt/IBM/InformationServer/Sever/DSEngine/. For example, update the APT_YARN_JDK_LIBRARY_PATH environment variable to /usr/jdk64/java-1.8.0-openjdk-1.8.0/jre/lib/amd64. |
You receive the following error message:
|
The error message indicates that your system JAVA is not version 1.7 or later. Take one of the following
actions:
|
You are using NFS mounting and are experiencing performance problems. | NFS mounting is an option that works for smaller clusters and is supported by InfoSphere Information Server on Hadoop. However, when the clusters get bigger, NFS mounting can cause performance problems and cause a single point of failure for the cluster, so it is not typically used in production systems. Consider using another method for coping the binaries, such as HDFS. For additional information, see Copying binaries to Hadoop nodes. |