Accessing Spark logs for ingestion jobs

This topic provides the steps required to locate and view Spark logs associated with submitted ingestion jobs within IBM® watsonx.data. By accessing these logs, you can gain valuable insights into the execution details, potential error messages related to the ingestion process, and troubleshooting ingestion jobs.

Before you begin

For a Failed or Finished status, you must start the Spark history server to get the ingestion log details. For more information, see Accessing the Spark history server.

Procedure

  1. Log in to watsonx.data console.
  2. From the navigation menu, select Data manager and click Ingestion history tab.
  3. Locate the specific ingestion Job ID you want to examine.
  4. Click on the Spark log ID associated with the ingestion job to get log details based on the status of the ingestion job as follows:
    • Starting: If the ingestion job is in Starting status, clicking the Spark log ID may not provide immediate details. You might see a message indicating that the application ID is not available yet.
    • Running: If the job status is Running, clicking the Spark log ID will generally take you to the Spark UI. You can monitor the ingestion job progress, resource usage, and other details in real-time through this web interface. This Spark UI information will be temporarily available only when the job is in the Running status.
    • Failed or Finished: If the job has Failed or Finished, clicking the Spark log ID will navigate you to the Spark history UI. This interface provides a summary of the ingestion job execution, including its timeline, resource usage, and any error messages.
    Note: If an ingestion job encountered an issue and failed early in its execution and did not generate a Spark application ID, it is likely that the job did not progress far enough for a Spark log ID to be created. Therefore, the Spark log ID will not be available to provide details about the ingestion process.