Configuration scenarios and examples

This section provides configuration scenarios that demonstrate how you can leverage Spark configurations in watsonx.data.

Configuring Spark log level information
Review the applications that run and identify the issues that are present by using the logs that the watsonx.data Spark application generates. The standard logging levels available are ALL, TRACE, DEBUG, INFO, WARN, ERROR, FATAL, and OFF. By default, the watsonx.data Spark application logs at the Spark INFO level. You can configure the logging level to display relevant, and fewer verbose messages.
spark.local.dir configuration parameter
Enabling NVIDIA GPU acceleration in watsonx.data
Enable Spark log and event log rotation
Configure and validate Spark log rotation and cleanup behavior at both engine level and application level to ensure logs are rotated based on time or size, and old logs are cleaned up automatically.

Parent topic: Managing custom configurations and environment variables

Parent topic: Working with Spark