This topic provides the steps required to locate and view Spark logs associated with
submitted ingestion jobs within IBM®
watsonx.data. By
accessing these logs, you can gain valuable insights into the execution details, potential error
messages related to the ingestion process, and troubleshooting ingestion jobs.
Before you begin
For a Failed or Finished status, you must start the
Spark history server to get the ingestion log details. For more information, see Accessing the Spark history server.
Procedure
- Log in to watsonx.data
console.
- From the navigation menu, select Data manager and click
Ingestion history tab.
- Locate and click open the specific ingestion Job log you want to
examine.
- Click Spark application history to get log details based on the
status of the ingestion job as follows:
- Starting: If the ingestion job is in Starting
status, clicking the Spark application history may not provide immediate
details. You might see a message indicating that the application ID is not available yet.
- Running: If the job status is Running, clicking
the Spark application history will generally take you to the Spark UI. You
can monitor the ingestion job progress, resource usage, and other details in real-time through this
web interface. This Spark UI information will be temporarily available only when the job is in the
Running status.
- Failed or Finished: If the job is
Finished or Failed, clicking the Spark
application history will navigate you to the Spark history UI. This interface provides a
summary of the ingestion job execution, including its timeline, resource usage, and any error
messages.
- Cancelled: If the job is Cancelled,
Spark application history is disabled.
Note: If an ingestion job encountered an issue and failed early in its execution and did not
generate a Spark application ID, it is likely that the job did not progress far enough for a Spark
log ID to be created. Therefore, the Spark log ID will not be available to provide details about the
ingestion process.