Troubleshooting problems when creating InfoSphere DataStage projects

Symptoms

Creating a project fails during the installation of InfoSphere Information Server, when using the Administrator client, or when using the dsadmin command.

Causes

Diagnosing the problem

The dstage_wrapper_trace_N.log indicates where the problem lies. These log files might require customer support, but some errors can be interpreted to attempt further diagnostics.

Locate the error message. The error message location is based on how the project creation was started.

Installation
Search in the installation logs (for example, /opt/IBM/InformationServer/logs/) for "RUN BP DSR_QUICKADD.B". The error message is a few lines that follow this text.
Administrator client
The error message is displayed in a message box.
dsadmin command
The error message is displayed on the console.

After you locate the error message, there might be more detailed error information in the following log file on the DataStage Server system in the following variable path: /home/_Credential_Mapped_Username_/ds_logs/dstage_wrapper_trace_N.log

These logs go to ${user.home}/ds_logs where ${user.home} is the home directory of the credential mapped user or the user that runs the installation for projects that are created during installation. On Windows systems, the user home directory is C:\Documents and Settings\_Credential_Mapped_UserName_

Sometimes tracing the dsrpcd and child processes can produce useful information from which IBM support can narrow down the causes of a project creation problem. Operating systems differ in how to trace processes but a few examples are shown here:

AIX: "truss -o /tmp/dsrpcd_truss.out -fp <dsrpcdPID&gt;"RedHat: "strace -fp <dsrpcdPID&gt; &gt; /tmp/dsrpcd_strace.out"

These commands attach to the running dsrpcd process and record all of the system calls that are made by that process and its children during subsequent client-server sessions. For example, a call to create a project from the Administrator client or the dsadmin command line is recorded.

To produce extra diagnostic information for the JVM initialization after all of its libraries are successfully loaded you can enable JVM startup tracing. Add the following lines to /opt/IBM/InformationServer/Server/DSEngine/dsenv:

XMOG_TRACE_LEVEL=TraceVerbose
XMOG_TRACE_FILE=/tmp/xmogtrace.txt
export XMOG_TRACE_LEVEL
export XMOG_TRACE_FILE

On Windows these tracing options can be set as System Environment Variables by using the System Control panel. Remember to restart the DataStage Server engine processes after adding these variables, and to remove these environment variables after they are no longer needed.

Enable repository database tracing. To enable tracing of the code that populates the repository database follow these steps:

  1. Create a file on the DataStage Server system in /opt/IBM/InformationServer/ASBNode/conf/ called NewRepos.debug.properties. The file name is case sensitive.
  2. In the file, add the following three lines:
    log4j.logger.com.ascential.dstage=DEBUG
log4j.logger.com.ibm.datastage=DEBUG
NewRepos.spy.trace=true

The dstage_wrapper_trace_N.log will then contain extra tracing information the next time a project creation is attempted. Ensure that you delete the NewRepos.debug.properties file when finished. In addition, spy trace files, such as dstage_wrapper_spy_N.log, are produced in the same directory as the log files. These files contain a detailed record of low-level method calls and can grow large.

Running project creation manually. The project creation code runs in the context of a dsapi_slave process which does not have any console output. Locate the full "RUN BP DSR_QUICKADD.B" command line from the domain installation log files on /opt/IBM/InformationServer/logs/.

Use the following commands to run the project creation code so that you can view the console output:

Linux and UNIX

  1. cd /opt/IBM/InformationServer/Server/DSEngine
  2. ./dsenv
  3. bin/uvsh
  4. RUN BP DSR_QUICKADD.B <<i>arguments from log file</i>&gt;
  5. QUIT

Windows

  1. cd /opt/IBM/InformationServer/Server/DSEngine
  2. bin/uvsh
  3. RUN BP DSR_QUICKADD.B <arguments from log file> <newProjectName> C:\IBM\InformationServer\Server\Projects\<newProjectName> CREATE
  4. QUIT

Resolving the problem

Incorrectly configured repository database.

8.0.x message: "Error creating DR elements, Error was -1", "Invalid node name: %1"

8.1 and later message: "DSR.ADMIN: Error creating DR elements, Error was Unique constraint violation."

These types of errors usually occur because the repository database returns an error when attempting to make an update. The dstage_wrapper_trace_N.log file may contain more specific details about the exact database error.

There might also be a database log, depending on what type of database the repository is running in, which contains more information. For example, DB2 has the db2diag tool which can be run to find out the exact reason why an update failed. Typical failures are: out of disk space, memory configuration problems and so on

For repository database errors it is important to confirm that the database was created with the scripts supplied on the installation media. These scripts configure important database parameters which if missed might cause project creation problems.

It is also important that the database was created with the correct character set, as per the database creation script documentation on the installation media (typically UTF16/32). If a different character set was used some of the metadata stored can become corrupted or might cause unexpected primary key violations. If the wrong character set was used the product needs to be reinstalled.

For errors at this level the WebSphere Application Server logs might contain additional information. The files SystemOut.log and SystemErr.log can be found in the following directory: ...WebSphere/AppServer/profiles/_profile_name_/logs/server1/

Leftover metadata in repository database from a previously failed project creation

8.0.x message: "Error creating DR elements, Error was -1"

This problem occurs only on 8.0.x systems and can be identified by looking in the dstage_wrapper_trace_N.log file for a "unique constraint violation" error. This can occur when a project creation failed and did not remove all of its metadata from the repository. Even though the project cannot be seen in DataStage, attempting to create a project of the same name results in this error.

To work around this problem, you can simply create a project with a different name. Alternatively, IBM support can provide a tool and instructions for how to remove the leftover data from the repository.

Unable to create log file on the DataStage Server

8.0.x message: "Error creating DR elements, Error was -1"

8.1 and later message: DSR.ADMIN: Error creating DR elements, Error was log4j:ERROR setFile(null,true) call failed.

Just before the metadata repository is populated with the default project contents, a log file is created on the DataStage Server system in /home/_Credential_Mapped_Username_/ds_logs/. If this log file cannot be created, the project creation will fail. On Windows computers, the user home directory is usually C:\Documents and Settings\_Credential_Mapped_UserName_

The usual reasons why this log was not created are either because the user has no home directory at all or they do not have appropriate permissions on it.

Incorrectly configured locale on the DataStage Server

8.0.x message: "Error creating DR elements, Error was -1"

8.1 and later message: "DSR.ADMIN: Error creating DR elements, Error was Unmatched quotation marks"

This problem is ultimately caused by bad locale configuration on the DataStage Server system. This manifests itself because the "hostname" command is run during project creation, and instead of returning the correct host name it returns a string such as "couldn't set locale correctly".

Failed to load JVM into the DataStage Server process (dsapi_slave)

8.0.x message: "(The connection is broken (81002))"

The JVM (Java Virtual Machine) can fail to load for several reasons. If it does fail to load, the dsapi_slave process is terminated, which results in broken connection errors on the client such as error 81002.

A core file might be produced which can be used to determine what caused the process to be terminated.

Possible causes of this problem are:

  • The LIBPATH (or equivalent) is too long and caused a buffer overflow. This can be confirmed by using the Administrator client to run the "env" command with the Command button. If the contents of LIBPATH are duplicated then it is probable that dsenv are sourced twice. The dsenv does not need to be sourced when starting the DataStage Server engine processes with the uv -admin -start command.
  • Incompatible or missed patches on the Client, Server, and Domain systems. By looking in the version.xml file of each system you can confirm what patches are installed. Ensure that patches are installed on all appropriate systems.
  • Environment variables such as LDR_CNTRL were added or modified in the IBM/InformationServer/Server/DSEngine/dsenv file. Generally speaking LDR_CNTRL settings in dsenv must not be modified unless otherwise directed by IBM.
  • Incompatible operating system kernel parameters.
Firewall configuration

8.0.x message: "Error creating DR elements, Error was -1"

8.1 and later message: DSR.ADMIN: Error creating DR elements, Error was com.ascential.xmeta.exception.ServiceException The DataStage Server system needs to communicate with the domain system, which means that certain ports need to be open between these systems if they are on separate machines.

This problem can be confirmed by looking in the dstage_wrapper_N.log file for the following error: Connection refused:host=<hostname>,port=2809. Ensure that the firewall is correctly configured and use telnet <hostname> <port> from the DataStage Server machine to confirm the port is accessible.

The necessary firewall configuration can be found in the installation guide.

Trusted authentication between DataStage Server system and the Domain system failed

8.0.x message: "Error creating DR elements, Error was -1"

8.1 and later message: "DSR.ADMIN: Error creating DR elements, Error was Mapping failed to copy attributes: MetaTable -> DSTableDefinition (EObject: null, MetaTable)"

The DataStage Server system authenticates with the Domain system by a process called trusted authentication. This process uses a secure certificate exchange rather than explicit user name and password authentication. If the process fails, the project creation fails. Trusted authentication failure is identified by multiple exceptions in the DataStage Server ds_logs that says "Null session".

This can fail for a number of reasons:

  • If the DataStage Server is installed onto a Windows system (say C:\IBM\InformationServer), installing the clients into a different directory (say C:\IBM\InformationServer2) causes the certificate exchange to fail, ultimately causing the project creation to fail. See technote #1409412 and APAR JR34441 for more information.
  • The number of trusted sessions reaches a maximum limit, so a new session cannot be started. This is identified by an entry in the WebSphere logs that says the limit is exceeded. If so, restarting WepSphere Application Server clears everything so that new sessions can be created and project creation can succeed.
DataStage was not installed on the Domain system

8.0.x message: "Error creating DR elements, Error was -1"

8.1 and later message: "DSR.ADMIN: Error creating DR elements, Error was Mapping failed to copy attributes: MetaTable -> DSTableDefinition (EObject: null, MetaTable)"

When installing the Domain and DataStage Server onto different physical systems, the installation of DataStage Server fails to create projects specified in the installer if DataStage is not installed onto the Domain. These errors can be found in the installation logs.

Furthermore, attempting to create projects that use the Administrator client or command line fails. In both these cases, the exceptions state that The package with URI "http:///1.1/DataStageX.ecore" is not registered.

DataStage can be added to the Domain system by rerunning the installer, selecting DataStage and clearing the other components.

Locale or regional settings customized on the Client system

8.0.x message: "Error creating DR elements, Error was -1"

8.1 and later message: "Invalid Node Name %1"

If the regional language settings are modified to use a customized short date format (for example "ddd dd/MM/yyyy") it can cause the DataStage Administrator client to send the wrong date information to the DataStage Server, causing project creation to fail. A patch for this issue is available under APAR JR34770.

Disk / partition full or user quota reached on DataStage Server system

8.0.x message: "Error creating DR elements, Error was -1"

8.1 and later message: "DSR.ADMIN: Error creating DR elements, error was log4j: ERROR failed to flush writer."

The project creation operation creates a log file on the DataStage Server system, called dstage_wrapper_trace_N.log, in the path indicated at the beginning of this document. The log creation fails when the disk /partition is full or the user to which credentials are mapped to reaches their disk quota. Free up space as necessary and try the operation again.

Project creation fails at "Initializing demo files..." within the Administrator client

8.5 message: "Errors were detected during project creation that might render project <name> unstable.

Caused by: DSR.ADMIN: Error creating DR elements, Error was <date timestamp> java.utils.prefs.FileSystemPreferences$2 run.

This error states that there was a problem with being able to write Java preference data.

One of the following items causes these problems:

  • SE (Security Enhanced) Linux is enabled.
  • The user ID that is trying to create the project does not have a local home directory to write to

If SELinux is enabled, disable it.

To determine if SELinux is installed and in enforcing mode, you can do one of the following actions:

  • Check the /etc/sysconfig/selinux file
  • Run the sestatus command
  • Check the /var/log/messages file for SELinux notices (Notice format might differ between RHEL 4 and RHEL 5.)

To disable SELinux, you can do one of the following actions:

  • Set it in permissive mode and run the setenforce 0 command as a superuser
  • Modify /etc/sysconfig/selinux and reboot the machine

If there is no home directory for the user ID, create a local home directory with write permissions. (766) and have the group as part of the local dstage group.

Stack Execution Disable (SED) is enabled (AIX only)

8.0.x message: "Error creating DR elements, Error was -1"

8.1 and later message: "Unable to confirm the JVM can be loaded into the DataStage server process 'DSR_CREATE.PROJECT.B TestJVM' failed"

If Stack Execution Disable (SED) is enabled in AIX, the JIT compiler fails when trying to run code it generated in the process data area. This occurs with all of the DataStage executable items that have embedded JVMs. The solution to this problem is to turn off the SED at the system level and reboot the machine. To turn off the SED use the command: sedmgr -m off

Unable to increase the table space for the metadata repository (XMETA)

8.0.x message: "Error creating DR elements, Error was -1"

8.1 and later message: "DSR.ADMIN: Error creating DR elements, Error was unable to save"

The SystemOut.log might show the following error: 
Oracle¨ORA-01653: unable to extend table XMETAUSER.LOGGING_LOGGINGEVENT1466CB5F by 128 in tablespace XMETA

If DB2 is used for the metadata repository (XMETA) look in the <db2instance_home>/sqllib/db2dump/db2diag.log for errors.

To resolve this problem, increase the table space, and try the operation again. It might be necessary to manually delete any partially created project, which can be done by following the material here: http://www-01.ibm.com/support/docview.wss?uid=swg27021312

Error updating secondary indexes

Error message: "DSR.ADMIN: Error updating secondary indexes. Status code = -135 DSJE_ADDPROJECTFAILED"

A known cause for the error error updating secondary indices is one or more missing I_* directories in the /opt/IBM/InformationServer/Server/Template directory. If there is another DataStage engine installation (of the same version and patch level) available it is possible copy the Template directory from the working engine and use it to replace the Template directory on the broken engine. However be careful the backup the existing Template directory first. If a template directory is taken from a working engine of a different patch level, some of the patches on the broken engine might be rendered ineffective.

Parent topic: Troubleshooting job design issues