Troubleshooting problems when you create InfoSphere DataStage projects

You can encounter failures during the installation of IBM® InfoSphere® Information Server with the Administrator client or the dsadmin command.

Symptoms

Creating a project fails during the installation of InfoSphere Information Server, when you use the Administrator client, or when you use the dsadmin command.

Causes

  • Incorrectly configured repository database
  • Leftover metadata in repository database from a previously failed project creation
  • Unable to create log file on the InfoSphere DataStage® Server
  • Incorrectly configured locale on the InfoSphere DataStage Server
  • Incorrectly configured locale on the InfoSphere DataStage Server
  • Failed to load JVM into theInfoSphere DataStage Server process (dsapi_slave)
  • Firewall configuration
  • Trusted authentication between InfoSphere DataStage Server system and the Domain system failed
  • InfoSphere DataStage was not installed on the Domain system
  • Locale regional settings that are customized on the Client system
  • A full disk, partition, or user quota that is reached on InfoSphere DataStage Server system
  • Project creation fails at Initializing demo files... within the Administrator client
  • Stack Execution Disable (SED) is enabled on AIX®
  • Unable to increase the table space for the XMETA
  • Error updating secondary indexes

Diagnosing the problem

The dstage_wrapper_trace_N.log indicates where the problem lies. These log files might require customer support, but some errors can be interpreted and the problem can be resolved.

Locate the error message. The error message location is based on how the project creation was started.

Installation
Search in the installation logs at /opt/IBM/InformationServer/logs/ for RUN BP DSR_QUICKADD.B. The error message is a few lines that follow this text.
Administrator client
The error message is displayed in a message box.
dsadmin command
The error message is displayed on the console.

After you locate the error message, you can find more detailed error information in the log file on the InfoSphere DataStage server system. You can find that log at the following file path: /home/_Credential_Mapped_Username_/ds_logs/dstage_wrapper_trace_N.log

  • N is a number 1 - 20 for the log files for the last 20 InfoSphere DataStage sessions
  • _Credential_Mapped_Username_ is the InfoSphere DataStage Server user name that credentials are mapped to in web Administration console.

These logs go to ${user.home}/ds_logs where ${user.home} is the home directory of the credential mapped user or the user that runs the installation for projects that are created during installation. On Windows systems, the user home directory is C:\Documents and Settings\_Credential_Mapped_UserName_

Sometimes tracing the dsrpcd and child processes can produce useful information from which IBM support can narrow down the causes of a project creation problem. Operating systems differ in how to trace processes but a few examples are shown here:

AIX: "truss -o /tmp/dsrpcd_truss.out -fp <dsrpcdPID&gt;"RedHat: "strace -fp <dsrpcdPID&gt; &gt; /tmp/dsrpcd_strace.out"

These commands attach to the running dsrpcd process and record all of the system calls that are made by that process and its child process during subsequent client/server sessions. For example, a call to create a project from the Administrator client or the dsadmin command line is recorded.

To produce extra diagnostic information for the JVM initialization after all of its libraries are successfully loaded, you can enable JVM startup tracing. Add the following lines to /opt/IBM/InformationServer/Server/DSEngine/dsenv:

XMOG_TRACE_LEVEL=TraceVerbose
XMOG_TRACE_FILE=/tmp/xmogtrace.txt
export XMOG_TRACE_LEVEL
export XMOG_TRACE_FILE

On Windows, these tracing options can be set as System Environment Variables by using the System Control panel. Remember to restart the InfoSphere DataStage Server engine processes after you add these variables, and to remove these environment variables after they are no longer needed.

Enable repository database tracing. To enable tracing of the code that populates the repository database, follow these steps:

  1. Create a file on the InfoSphere DataStage Server system in /opt/IBM/InformationServer/ASBNode/conf/ called NewRepos.debug.properties. The file name is case-sensitive.
  2. In the file, add the following three lines:
    log4j.logger.com.ascential.dstage=DEBUG
log4j.logger.com.ibm.datastage=DEBUG
NewRepos.spy.trace=true

The dstage_wrapper_trace_N.log contains extra tracing information the next time that you create a project. Ensure that you delete the NewRepos.debug.properties file when finished. In addition, spy trace files, such as dstage_wrapper_spy_N.log, are produced in the same directory as the log files. These files contain a detailed record of low-level method calls and can grow large.

Running project creation manually. The project creation code runs in the context of a dsapi_slave process that does not have any console output. Locate the full RUN BP DSR_QUICKADD.B command line from the domain installation log files on /opt/IBM/InformationServer/logs/.

Use the following commands to run the project creation code so that you can view the console output:

Linux and UNIX

  1. cd /opt/IBM/InformationServer/Server/DSEngine
  2. ./dsenv
  3. bin/uvsh
  4. RUN BP DSR_QUICKADD.B <<i>arguments from log file</i>&gt;
  5. QUIT

Windows

  1. cd /opt/IBM/InformationServer/Server/DSEngine
  2. bin/uvsh
  3. RUN BP DSR_QUICKADD.B <arguments from log file> <newProjectName> C:\IBM\InformationServer\Server\Projects\<newProjectName> CREATE
  4. QUIT

Resolving the problem

Incorrectly configured repository database.

8.0.x message: Error creating DR elements, Error was -1", "Invalid node name: %1

8.1 and later message: DSR.ADMIN: Error creating DR elements, Error was Unique constraint violation.

These types of errors usually occur because the repository database returns an error when you attempt to make an update. The dstage_wrapper_trace_N.log file might contain more specific details about the exact database error.

There might also be a database log, depending on what type of database the repository is running in, which contains more information. DB2® has the db2diag tool, which can be run to find out the exact reason why an update failed. The typical failures are running out of disk space, and memory configuration problems.

For repository database errors it is important to confirm that the database was created with the scripts supplied on the installation media. These scripts configure important database parameters that might cause project creation problems.

It is also important that the database was created with the correct character set. The database creation script documentation on the installation media typically specifies UTF16/32. If a different character set was used, some of the metadata that is stored can become corrupted or might cause unexpected primary key violations. If the wrong character set was used, the product must be reinstalled.

For errors at this level the WebSphere® Application Server logs might contain more information. The files SystemOut.log and SystemErr.log can be found in the following directory: ...WebSphere/AppServer/profiles/_profile_name_/logs/server1/

Leftover metadata in repository database from a previously failed project creation

8.0.x message: Error creating DR elements, Error was -1

This problem occurs only on 8.0.x systems and can be identified by looking in the dstage_wrapper_trace_N.log file for a unique constraint violation error. This can occur when a project creation failed and did not remove all of its metadata from the repository. Even though the project cannot be seen in InfoSphere DataStage, attempting to create a project of the same name results in this error.

To work around this problem, you can create a project with a different name. Alternatively, IBM support can provide a tool and instructions for how to remove the leftover data from the repository.

Unable to create log file on the DataStage Server

8.0.x message: Error creating DR elements, Error was -1

8.1 and later message: DSR.ADMIN: Error creating DR elements, Error was log4j:ERROR setFile(null,true) call failed.

Just before the metadata repository is populated with the default project contents, a log file is created on the InfoSphere DataStage Server system in /home/_Credential_Mapped_Username_/ds_logs/. If this log file cannot be created, the project creation fails. On Windows computers, the user home directory is at C:\Documents and Settings\_Credential_Mapped_UserName_

The usual reasons why this log was not created are either because the user has no home directory at all or they do not have appropriate permissions on it.

Incorrectly configured locale on the InfoSphere DataStage Server

8.0.x message: "Error creating DR elements, Error was -1"

8.1 and later message: "DSR.ADMIN: Error creating DR elements, Error was Unmatched quotation marks"

This problem is ultimately caused by bad locale configuration on the DataStage Server system. This problem manifests itself because the host name command is run during project creation. Instead of returning the correct host name, the command returns an error that the process couldn't set locale correctly.

Failed to load JVM into the InfoSphere DataStage Server process (dsapi_slave)

8.0.x message: The connection is broken (81002)

The JVM (Java™ virtual machine) can fail to load for several reasons. If it does fail to load, the dsapi_slave process is stopped, which results in broken connection errors on the client such as error 81002.

A core file might be produced which can be used to determine what caused the process to be stopped.

Possible causes of this problem are:

  • The LIBPATH (or equivalent) is too long and caused a buffer overflow. This can be confirmed by using the Administrator client to run the env command with the Command button. If the contents of LIBPATH are duplicated, then it is probable that dsenv are sourced twice. The dsenv does not need to be sourced when it starts the InfoSphere DataStage Server engine processes with the uv -admin -start command.
  • Incompatible or missed patches on the Client, Server, and Domain systems. By looking in the version.xml file of each system you can confirm what patches are installed. Ensure that patches are installed on all appropriate systems.
  • Environment variables such as LDR_CNTRL were added or modified in the IBM/InformationServer/Server/DSEngine/dsenv file. Generally speaking LDR_CNTRL settings in dsenv must not be modified unless otherwise directed by IBM.
  • Incompatible operating system kernel parameters.
Firewall configuration

8.0.x message: Error creating DR elements, Error was -1

8.1 and later message: DSR.ADMIN: Error creating DR elements, Error was com.ascential.xmeta.exception.ServiceException

The InfoSphere DataStage Server system must communicate with the domain system, and certain ports must be open between these systems if they are on separate servers.

This problem can be confirmed by looking in the dstage_wrapper_N.log file for the following error: Connection refused:host=<hostname>,port=2809. Ensure that the firewall is correctly configured and use telnet <hostname> <port> from the InfoSphere DataStage Server system to confirm that the port is accessible.

The necessary firewall configuration can be found in the installation guide.

Trusted authentication between InfoSphere DataStage Server system and the Domain system failed

8.0.x message: Error creating DR elements, Error was -1

8.1 and later message: DSR.ADMIN: Error creating DR elements, Error was Mapping failed to copy attributes: MetaTable -> DSTableDefinition (EObject: null, MetaTable)

The InfoSphere DataStage Server system authenticates with the Domain system by a process called trusted authentication. This process uses a secure certificate exchange rather than explicit user name and password authentication. If the process fails, the project creation fails. Trusted authentication failure is identified by multiple exceptions in the InfoSphere DataStage Server ds_logs that says "Null session".

This can fail for a number of reasons:

  • If the InfoSphere DataStage Server is installed onto a Windows system at C:\IBM\InformationServer. If you install the clients into a different directory, the certificate exchange might fail. If the certificate exchange fails, the project creation fails. See technote #1409412 and APAR JR34441 for more information.
  • The number of trusted sessions reaches a maximum limit, so a new session cannot be started. This is identified by an entry in the WebSphere logs that says the limit is exceeded. If so, restarting WepSphere Application Server clears everything so that new sessions can be created and project creation can succeed.
DataStage was not installed on the Domain system

8.0.x message: Error creating DR elements, Error was -1

8.1 and later message: DSR.ADMIN: Error creating DR elements, Error was Mapping failed to copy attributes: MetaTable -> DSTableDefinition (EObject: null, MetaTable)

When you install the Domain and InfoSphere DataStage Server onto different physical servers, the installation of DataStage Server fails to create projects that are specified in the installer if InfoSphere DataStage is not installed onto the Domain. These errors can be found in the installation logs.

Furthermore, attempting to create projects that use the Administrator client or command line fails. In both these cases, the exceptions state that The package with URI "http:///1.1/DataStageX.ecore" is not registered.

InfoSphere DataStage can be added to the Domain system by rerunning the installer, selecting InfoSphere DataStage and clearing the other components.

Locale or regional settings that are customized on the Client system

8.0.x message: Error creating DR elements, Error was -1

8.1 and later message: Invalid Node Name %1

If the regional language settings are modified to use a customized short date format (for example ddd dd/MM/yyyy) it can cause the DataStage Administrator client to send the wrong date information to the DataStage Server, causing project creation to fail. A patch for this issue is available under APAR JR34770.

Full disk, partition, or user quota that is reached on InfoSphere DataStage Server system

8.0.x message: Error creating DR elements, Error was -1

8.1 and later message: DSR.ADMIN: Error creating DR elements, error was log4j: ERROR failed to flush writer.

The project creation operation creates a log file on the InfoSphere DataStage Server system, called dstage_wrapper_trace_N.log, in the path indicated at the beginning of this document. The log creation fails when the disk partition is full or the user to which credentials are mapped to reaches their disk quota. Free up space as necessary and try the operation again.

Project creation fails at Initializing demo files... within the Administrator client

8.5 message: Errors were detected during project creation that might render project <name> unstable. Caused by: DSR.ADMIN: Error creating DR elements, Error was <date timestamp> java.utils.prefs.FileSystemPreferences$2 run.

This error states that there was a problem with being able to write Java preference data.

One of the following items causes these problems:

  • SE (Security Enhanced) Linux is enabled.
  • The user ID that is trying to create the project does not have a local home directory to write to

If SELinux is enabled, disable it.

To determine if SELinux is installed and in enforcing mode, you can do one of the following actions:

  • Check the /etc/sysconfig/selinux file
  • Run the sestatus command
  • Check the /var/log/messages file for SELinux notices (Notice format might differ between RHEL 4 and RHEL 5.)

To disable SELinux, you can do one of the following actions:

  • Set it in permissive mode and run the setenforce 0 command as a superuser
  • Modify /etc/sysconfig/selinux and restart the system

If there is no home directory for the user ID, create a local home directory with write permissions. (766) and have the group as part of the local dstage group.

Stack Execution Disable (SED) is enabled on AIX

8.0.x message: Error creating DR elements, Error was -1

8.1 and later message: Unable to confirm the JVM can be loaded into the DataStage server process 'DSR_CREATE.PROJECT.B TestJVM' failed

If Stack Execution Disable (SED) is enabled in AIX, the JIT compiler fails when you try to run code it generated in the process data area. This error occurs with all of the InfoSphere DataStage executable items that include embedded JVMs. The solution to this problem is to turn off the SED at the system level and restart the system. To turn off the SED, use the sedmgr -m off command.

Unable to increase the table space for the XMETA

8.0.x message: Error creating DR elements, Error was -1

8.1 and later message: DSR.ADMIN: Error creating DR elements, Error was unable to save

The SystemOut.log might show the following error: 
Oracle¨ORA-01653: unable to extend table XMETAUSER.LOGGING_LOGGINGEVENT1466CB5F by 128 in tablespace XMETA

If DB2 is used for the XMETA look in the <db2instance_home>/sqllib/db2dump/db2diag.log file for errors.

To resolve this problem, increase the table space, and try the operation again. It might be necessary to manually delete any partially created project, which can be done by following the material here: http://www-01.ibm.com/support/docview.wss?uid=swg27021312

Error updating secondary indexes

Error message: DSR.ADMIN: Error updating secondary indexes. Status code = -135 DSJE_ADDPROJECTFAILED

A known cause for the error error updating secondary indices is one or more missing I_* directories in the /opt/IBM/InformationServer/Server/Template directory. If there is another InfoSphere DataStage engine installation with the same version and patch level available, it is possible copy the Template directory from the working engine and use it to replace the Template directory on the broken engine. However, backup the existing Template directory first. If a template directory is taken from a working engine of a different patch level, some of the patches on the broken engine might be rendered ineffective.

Parent topic: Troubleshooting job design issues