Troubleshoot IBM DataStage

Getting help and support for DataStage

If you have problems or questions when you use DataStage, you can get help by searching for information or by asking questions through a forum. You can also open a support ticket.

When you ask a question on the forums, tag your question so that it is seen by the DataStage development teams.

For questions about the service and getting started instructions, use the forum at https://stackoverflow.com/questions/tagged/datastage.

General

• Jobs fail with "Resource temporarily unavailable" error
• Jobs fail because SQL and Before SQL statements run in incorrect order
• The mailx command fails to run in before-job and after-job subroutines without SMTP server info
• Properties selections not preserved if you deselect "Use DataStage properties"
• Routine fails when CEL function ds.getUserStatus is run on an external flow
• Job fails when loading a large Excel file
• Cannot add DataStage component after system power outage
• Compilation of DataStage flow fails if the Java classpath is parameterized
• Exported flows generate JSON connection files that contain plaintext passwords
• Issues browsing database tables with columns that contain special characters
• Incorrect inferences assigned to a schema read by the Asset Browser
• Using sequential files as a source
• Error running jobs with a parquet file format
• Migration pod getting evicted for exceeding its ephemeral storage limits
• Error occurs during the upgrade of the Cloud Pak for Data from 5.0.0 to 5.0.1 version
• Compilation of a flow that contains the transformer stage is timed out
• Ephemeral local storage usage exceeds the total limit of a container

Connectors

• Netezza® connector: Duplicate records occur when partitioned reads are enabled
• MySQL connector: Jobs might fail if you use Write mode "Update" for the target without a primary key
• FTP connector: The home directory path is prepended to the path
• Jobs fail with error "The connector could not establish a connection to Db2® database"
• Job with source data from a SAP OData connector fails
• Cannot run transactional SQL on data from Apache Hive version 2.0 or earlier
• IBM Db2 for DataStage connection with SSL certificate fails with "Protocol specific error code 414" error
• Error parameterizing the credential field for a flow connection in IBM Cloud® Object Storage
• PostgreSQL connector times out on large tables
• Jobs with an Apache Hive connection that has ZooKeeper discovery enabled fail
• Schema changes that originate in data from the HTTP connector can cause the job to fail
• Cannot preview data from the generic JDBC connector
• Not able to create successful database connection that uses the SSL certificate
• Snowflake connector: Jobs are failing with "java/lang/OutOfMemoryError" error
• Snowflake connector: Jobs are failing with "Fork failed: Resource temporarily unavailable" error

Runtime

• Large ISX imports produce gateway timeout and compilation errors
• Out of memory issue for the DataStage operator
• Jobs queue when compute pods fail to start
• Message handler changes are not applied to a job run
• Job queuing
• Restarting container pods
• PXRruntime instance that is deployed on physical location cannot appear
• No existing jobs getting stuck or display in a Starting or Running state
• Synchronization error across pods with mounted NFS path in PXRuntime

ds-metrics

• Running ds-metrics on FIPS-clusters for 5.0.3 version fails
• Cannot enable ds-metrics FIPS-tolerant or FIPS-enabled clusters for 5.1.0 version
• Job_run_log stores only non-INFO level log messages
• Metrics database cannot be cleared out fully after deleting the ds_metrics schema
• ds-metrics display short error messages in the service logs

General

Connectors

Snowflake connector: Jobs are failing with "java/lang/OutOfMemoryError" error

The log displays the following error message:

java/lang/OutOfMemoryError", exception "Failed to create a thread: retVal -1073741830, errno 11"

Workaround: Increase the heap size on the Output or Input tab for your Snowflake connector.

Snowflake connector: Jobs are failing with "Fork failed: Resource temporarily unavailable" error

The log displays the following error messages:

<SCLoadAudIdLd.sf_write__JOB_EXECUTION_LOG__Ins,1> Error: Unable to create iprofiler thread11/6/2024 06:13:03 WARNING IIS-DSEE-USBP-00002 <Sf_STG_VEH_ALFA,0> Error: Unable to create iprofiler thread

and

WARNING IIS-DSEE-USBP-00002 <sc_AFT_COST_CENTRE_CD.sf_write__AUTO_FINANCE_TYPE__Ins,1>
Type=Segmentation error vmState=0x00000000
WARNING IIS-DSEE-USBP-00002 <sc_AFT_COST_CENTRE_CD.sf_write__AUTO_FINANCE_TYPE__Ins,1> J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001

Workaround: Increase PID Limits inside the OpenShift cluster. Set podsPIDLimit to 16,384, which is a maximum value for PID limits. For information on how to increase PID Limits, see Red Hat Customer Portal.

Runtime

Large ISX imports produce gateway timeout and compilation errors

Importing a large ISX file can cause gateway timeout errors, and importing one with many Build stages can cause compilation errors due to lack of resources.

Workaround: Modify the configurations for DataStage and px-runtime to allocate more resources. See the following recommended custom configurations.

For more information, see Customizing resources.

Edit the DataStage custom resource to modify the migration resources:

oc edit datastage datastage -n ${PROJECT_CPD_INST_OPERANDS}

Inside the spec: section of the yaml add the below content:

spec:
  custom:
    resources:
      components: 
        migration:
          replicas: 1
          limits:
            cpu: "6"
            memory: 20Gi
          requests:
            cpu: "2"
            memory: 6Gi

Update PXRuntime resources:

# retrieve pxruntime cr
oc -n ${PROJECT_CPD_INST_OPERANDS} get pxruntime

Edit the custom resource:

oc -n ${PROJECT_CPD_INST_OPERANDS} edit pxruntime <cr-name>

Inside the spec: section of the yaml add the below content:

spec:
  custom:
    resources:
      components:
        pxruntime:
          limits:
            cpu: "4"
            memory: 8Gi
          requests:
            cpu: "2"
            memory: 2Gi

Out of memory issue for the DataStage operator

When more than 5 PX runtime instances are deployed on the cluster, the operator may run out of memory. To resolve this issue, update the CSV to increase the memory limits:

Retrieve the DataStage CSV:

oc -n ${PROJECT_CPD_INST_OPERATORS} oc get csv | grep datastage

Patch the DataStage CSV to increase the operator memory from 1Gi to 2Gi:

oc -n ${PROJECT_CPD_INST_OPERATORS} patch csv <DataStage-CSV-name> --type='json' -p='[{"op": "replace", "path": "/spec/install/spec/deployments/0/spec/template/spec/containers/0/resources/limits/memory", "value": "2Gi" }]'

Jobs queue when compute pods fail to start

If the runtime instance compute pods don't start, all jobs will run on the px-runtime pod. Resource limits lead to jobs being queued.

Workaround: Fix any issues that are preventing the compute pods from starting.

Message handler changes are not applied to a job run

The message handler is cached for 3 minutes after it is fetched. Immediate changes to the message handler will not affect the job run.

Workaround: Wait a few minutes to run the job.

Job queuing

When you submit a job, it may not start immediately. Instead, it gets a queued state. This is the way the system manages resources and prioritizes work.

Why your job run queued?

Reason	Description
Resource limitations	Your job waits for the necessary resources (CPU, GPU, memory) to become available. It happens when: Other jobs use all available capacity. Your job requires more resources than are currently free.
Concurrency limits	Some systems enforce limits on the number of jobs that can run simultaneously. Your job queues until other job finish.
Priority and scheduling	Lower-priority jobs may be queued while higher-priority jobs are run first.
Maintenance or downtime	The system can be undergoing maintenance or updates, which reduces available capacity or pauses job execution.
Queue backlog	High traffic or large numbers of submitted jobs can create a backlog. Jobs start when the earlier jobs are completed.

Resolution: The DataStage Workload Manager (WLM) ships with a default configuration that allows running 5 jobs simultaneously. If you have more resources, you should scale pxruntime to a larger instance. For more information, see Customizing hardware configurations for DataStage service instances with the command line. After modifying the scale configuration, it's important to update the WLM configuration file to enable the execution of more jobs simultaneously. So, if you need to run another concurrent jobs, make sure to adjust the RunJob settings in the XML file.

Restarting container pods

If you notice that compute pods have different ages, then you should consider younger containers as recently restarted.

ds-px-default-ibm-datastage-px-compute-0                          1/1     Running     0               169m
ds-px-default-ibm-datastage-px-compute-1                          1/1     Running     0               5h51m
ds-px-default-ibm-datastage-px-compute-2                          1/1     Running     0               2d7h

Workaround: Container restart can cause some job failures. For example, you can notice the following error message in the job log:

##I IIS-DSEE-TLCP-00013 2024-06-19 01:53:18(000) <main_program> SSLConnection(16252672:411,248): Error from SSL read bytesRead=0 ssl_error=5 ssl_error_string= errno=0 errno_string=Success
##E IIS-DSEE-TFPM-00330 2024-06-19 01:53:18(001) <main_program> The Section Leader on node node4 has terminated unexpectedly.
##I IIS-DSEE-TLCP-00013 2024-06-19 01:53:18(002) <main_program> SSLConnection(15598048:411,248): Error from SSL read bytesRead=-1 ssl_error=5 ssl_error_string= errno=104 errno_string=Connection reset by peer
##E IIS-DSEE-TFPM-00330 2024-06-19 01:53:18(003) <main_program> The Section Leader on node node2 has terminated unexpectedly.
##I IIS-DSEE-TLCP-00013 2024-06-19 01:53:18(004) <main_program> SSLConnection(16202192:411,248): Error from SSL read bytesRead=-1 ssl_error=5 ssl_error_string= errno=104 errno_string=Connection reset by peer
##E IIS-DSEE-TFPM-00330 2024-06-19 01:53:18(005) <main_program> The Section Leader on node node3 has terminated unexpectedly.
##W IIS-DSEE-TFPM-00647 2024-06-19 01:53:23(000) <main_program> APT_PMwaitForSectionLeadersCleanup: non-zero status 2 from APT_PMpollUntilZero. If this message persists try increasing the poll timeout seconds by setting APT_PM_CLEANUP_TIMEOUT.
##I IIS-DSEE-TFSR-00115 2024-06-19 01:53:28(000) <main_program> Starting job postRun

Container restarts due to multiple reasons, for example:

• Too many jobs are running concurrently. The container have no CPU cycle to respond on OpenShift health probe. You can detect it by running oc adm top pods | grep -i datastage and checking the CPU usage. If the CPU usage is very high, then you need to add more resources or lower the concurrency in wlm.config.xml.
• Running a large job exhausts all available memory. This causes container to run into OOMKilled. You can detect it by analyzing output from oc get events | grep -i OOM.
• Ephemeral storage is used up. By default, pxruntime cr has limited temporary storage and compute pods to 10GB. See:

[root@solo1 761]# oc get pxruntime -o yaml
apiVersion: v1
items:
- apiVersion: ds.cpd.ibm.com/v1
  kind: PXRuntime
  metadata:
    annotations:
      ansible.sdk.operatorframework.io/verbosity: "3"
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"ds.cpd.ibm.com/v1","kind":"PXRuntime","metadata":{"annotations":{"ansible.sdk.operatorframework.io/verbosity":"3"},"name":"ds-px-default","namespace":"ds"},"spec":{"description":"The default DataStage runtime instance","ephemeralStorageLimit":"10Gi","license":{"accept":true},"parameters":{"scaleConfig":"small","storageClass":"nfs-client"},"version":"5.0.0","zenCloudPakInstanceId":"dda5c8d1-9e57-472b-90b9-ce27adbe28ea","zenControlPlaneNamespace":"ds","zenServiceInstanceId":1715711527523198,"zenServiceInstanceNamespace":"ds","zenServiceInstanceOwnerUID":1000331001}}
    creationTimestamp: "2024-05-14T18:32:08Z"
    finalizers:
    - ds.cpd.ibm.com/finalizer
    generation: 50
    name: ds-px-default
    namespace: ds
    resourceVersion: "38308991"
    uid: e9d537c6-5d6b-459a-8ea4-dc7fe4c9b5d5
  spec:
    additional_storage:
    - mount_path: /mnts/pipline
      pvc_name: volumes-pipline-pvc
    - mount_path: /mnts/user-ing-flows
      pvc_name: volumes-user-ing-flows-pvc
    description: The default DataStage runtime instance
    ephemeralStorageLimit: 10Gi

If you want to remove ephemeral storage limit or change it to higher value, run oc edit pxruntimeor oc edit sts ds-px-default-ibm-datastage-px-compute. If you have other DataStage runtime instances, then you need to make the same modifications to those instances.

Migration service can face similar problem. To resolve it run oc edit deployment datastage-ibm-datastage-migration, then modify ephemeral-storage: 4Gi to have more space.

PXRuntime instance that is deployed on physical location cannot appear

While deploying remote data plane on the clusters, you can run into the issue of not appearing PXRruntime instance, which was deployed on physical location. It happens because the connection is being closed before completing the request.

Workaround: The timeout setting for the load balancer needs to be increased. For more information, see: Changing load balancer timeout settings.

Jobs get stuck, nonexistent jobs display in Starting/Running state

When your stale jobs are running, you can have a problem with deleting them from the UI. You can face similar problem with the projects because they can possibly using resources (for example memory). To clean up those processes, use one of the following commands:

cpdctl asset delete --asset-id ASSET-ID --purge-on-delete=true`

cpdctl dsjob jobrunclean {{--project PROJECT | --project-id PROJID} | {--space SPACE | --space-id SPACEID}} {--name NAME | --id ID} [--run-id RUNID] [--dry-run] [--threads n] [--all-spaces] [--before YYYY-MM-DD:hh:mm:ss]

Note: Using those commands cleans up all active jobs in a project. Make sure to stop running new jobs.

Synchronization error across pods with mounted NFS path in PXRuntime

If you use NFS for storage, synchronization issue can occur when one compute pod writes a file to an NFS-mounted path and another compute pod reads the file immediately. Although the file exists in the NFS path, read operations sometimes fail because of the delayed visibility across pods.

Workaround: Add the actimeo=0 parameter to the persistent volume (PV) configuration.

cat <<EOF |oc apply -f -
apiVersion: v1
kind: PersistentVolume
metadata:
  name: sample-nfs-pv
spec:
  capacity:
    storage: 100Gi
  accessModes:
  - ReadWriteMany
  mountOptions:
  - actimeo=0
  nfs:
    path: /data/sample-pv    # all worker nodes on the cluster should be allowed 
    server: <NFS server IP>  # to mount the path specified on the NFS server
  persistentVolumeReclaimPolicy: Retain
EOF 

For more information, see Setting up an NFS mount.

ds-metrics

Cannot enable ds-metrics on FIPS-tolerant or FIPS-enabled clusters for 5.1.0 version

ds-metrics displays the following error in the logs:

java.lang.NoClassDefFoundError: liquibase.snapshot.SnapshotIdService (initialization failure)
liquibase.exception.UnexpectedLiquibaseException: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: MD5, provider: SUN,
class: sun.security.provider.NativeMD5)
java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: MD5, provider: SUN, class: sun.security.provider.NativeMD5)
java.security.ProviderException: Error in Native Digest

Workaround: Initialize your database and configure ds-metrics through environment variables. For more information on enabling ds-metrics, see Storing and persisting metrics.

Job_run_log stores only non-INFO level log messages

Workaround: Set the environment variable METRICS_SEND_FULL_LOG to true in the px-runtime instance. If the variable is set to true, the full job run log is sent to metrics and stored in the job_run_log table. To set the environment variable, see Storing and persisting metrics.

Metrics database cannot be cleared out fully after deleting the ds-metrics schema

If you want to clear out the database that was initialized by ds-metrics, you need to delete the databasechangelog and databasechangeloglock tables in the default public schema.

Workaround: Use the following command to clear out the metrics database:

drop schema if exists ds_metrics cascade;
drop table if exists public.databasechangelog;
drop table if exists public.databasechangeloglock;

ds-metrics displays short error messages in the service logs

You may face error messages at the start of a job run, for example:

org.hibernate.engine.jdbc.spi.SqlExceptionHelper E logExceptions Batch entry 0 /* insert for
com.ibm.ds.metrics.api.models.JobRun */insert into ds_metrics.job_run (conductor_pid,config_file,controller_id,create_time,duration,instance_id,job_id,last_update_time,partition,queue_name,run_name,run_status,start_time,stop_time,user_status,run_id) values ((NULL),(NULL),(NULL),('2024-10-02 17:54:56.52134+00'),(NULL),('ds-px-default'),('324cac2e-8eac-4452-af75-81ee357fcb82'),('2024-10-02
17:54:56.521442+00'),('2'::int4),('Medium'),('name'),(NULL),(NULL),(NULL),(NULL),('00b84b48-47ea-4a6e-9343-1ed68afb7179')) was aborted: ERROR: duplicate key
value violates unique constraint "job_run_pkey" Detail: Key (run_id)=(00b84b48-47ea-4a6e-9343-1ed68afb7179) already exists. Call getNextException to see other errors in the batch.

Workaround: ds-metrics handles the errors, which can be ignored.