Managing a remote engine for DataStage Anywhere

DataStage® Anywhere supports maintenance, updates, and other data considerations with remote runtime engines.

Maintenance

With DataStage-aaS Anywhere, the control plane remains hosted on and managed by IBM Cloud.

Manage your data plane through the remote engine. For more information on managing a remote engine, see DataStage Remote Engine using Docker. To update your remote engine with the automated scripts, download the container image to your internal registries and deploy it. The scripts include information on controls including creating, running, cleaning, and upgrading a remote engine.

Scaling

You can add or remove remote engines to scale deployments throughout the month. There is no deployment limit, but you are charged for the maximum amount of VPCs deployed each month whether or not they are used.

Disaster recovery

Deploy additional remote engines to support disaster recovery.

Data observability

You can put an observability solution in place within your container management platform. Databand is integrated with DataStage Anywhere and can monitor DataStage pipelines.

Storage

The DataStage operator mounts default storage to the remote engine's Kubernetes pods. To add additional storage with persistent volumes, see Setting up an NFS mount in DataStage.

Enabling alternative Cloud Object Storage location for remote engine logs

By default job run logs for the remote engine are pushed to the default bucket in IBM Cloud® Object Storage (COS). You can enable an alternate COS location for storing the job run logs.

To disable pushing job run logs to IBM Cloud Object Storage default bucket for the Kubernetes deployment, use the following command:

kubectl -n <namespace> set env deployment/<instance-name>-ibm-datastage-px-runtime DISABLE_REMOTE_LOG_PUSH=true

To enable pushing logs to the alternative COS location for the Kubernetes deployment, use the following command that creates the secret containing new COS location:

kubectl -n <namespace> create secret generic datastage-log-cos-location \
--from-literal=CUSTOM_S3_BUCKET_NAME=<bucket-name> \
--from-literal=CUSTOM_S3_REGION=<region> \
--from-literal=CUSTOM_S3_ENDPOINT=<endpoint> \
--from-literal=CUSTOM_S3_ACCESS_KEY=<access-key> \
--from-literal=CUSTOM_S3_SECRET_KEY=<secret-key>

This command triggers pod restart. If you use the disabling command first, you must restart the pod manually.

Importing / exporting assets

To avoid job environment configuration issues, use the DataStage specific import / export function.

Create new project and update project settings to bind it to the remote engine.
Use cpdctl dsjob to export assets from the original project.
Use cpdct dsjob to import assets to the new project that is bound to the remote engine.

Setting proxy information

Container deployment supports proxy information. To set the proxy information to a remote engine for DataStage Anywhere, set the following environment variable in the container:

REMOTE_HTTPS_PROXY=http://username:password@host:port

Proxy information is not available for Kubernetes deployment.

DataStage directory structure

Projects


/ds-storage/PXRuntime/Projects
/ds-storage/PXRuntime/Projects/project-id

Flows

Flows directory where compiled artifacts for the flow are stored based on flow id:


/ds-storage/PXRuntime/Projects/project-id/flows
/ds-storage/PXRuntime/Projects/project-id/flows/flow-id
/ds-storage/PXRuntime/Projects/project-id/flows/flow-id/lib
/ds-storage/PXRuntime/Projects/project-id/flows/flow-id/scripts

Jobs

Jobs directory where runtime artifacts are stored for each job based on job id and each run based on run id:


/ds-storage/PXRuntime/Projects/project-id/jobs/job-id
/ds-storage/PXRuntime/Projects/project-id/jobs/job-id/runs/run-id

Custom resource name limitation

When you set up custom resource (CR), the CR name must have less than 28 characters.

Configuring parameters on a job or project level

You can manually override parameters on a job or project level by using the cpdctl command in the remote engine pod.

On a job level, you can provide the following environment variable: APT_PARAM_VALUE_FILE with a parameter file to override as value. For example:

APT_PARAM_VALUE_FILE=/ds-storage/param.txt

On a project level, run the following command: /ds-storage/PXRuntime/Projects/<projectId>/.local_jpfile in the suggested directory.

Enabling metrics for the remote engine

To enable metrics collection, deploy the remote engine with the following environment variable by using the --env-vars option of the dsengine.sh deployment script:

ENABLE_DS_METRICS=true

The setting applies to the entire remote engine runtime and all jobs and projects that use it. Project settings control where metrics are stored, but they do not enable or disable metrics collection.

If the remote engine is already running, stop and remove the container, then redeploy it with ENABLE_DS_METRICS=true. No job recompilation is required, and the remote engine ID is preserved.

To verify that metrics collection is enabled, check the container startup logs and confirm that the logs indicate that metrics are enabled, show the target region, and report a successful connection to the control plane.

You can also configure the optional DS_METRICS_CLIENT_THREADS environment variable to control the number of threads that populate the metrics database:

DS_METRICS_CLIENT_THREADS=<value>

The default value is 10. Increase this value only if you consistently observe a delay of more than one minute between job completion and metrics appearing in the metrics database. This setting applies to the entire remote engine runtime and cannot be configured per project.

For a list of supported remote engine environment variables, see the Remote engine environment variable reference.