Custom configuration properties and environment variables for Spark
Applies to :
Spark engine
Apache Gluten accelerated Spark engine
Custom Configuration Parameters (Spark Configurations)
Properties set using spark.conf.set() or --conf flag that
control Spark application behavior. These are application-specific, set at runtime.
| Spark configuration | Default value |
|---|---|
ae.spark.driver.log.level
|
Possible values:
For information about how to configure the parameter, see Configuring Spark log level information. |
ae.spark.executor.log.level |
Change the log level of executor. Possible values: DEBUG, INFO, WARN, ERROR, OFF For information about how to configure the parameter, see Configuring Spark log level information. |
ae.kubernetes.spec.topologySpreadConstraints |
Base64 encoded value of topologySpreadConstraints supported by k8s. For information about how to configure the parameter, see Setting custom configurations. |
ae.kubernetes.spec.tolerations |
Base64 encoded value of tolerations supported by k8s. |
ae.kubernetes.spec.nodeSelector |
Provide list of labels to identify the nodes to schedule pods ON. |
ae.spark.application.priority |
To set the priority of the Spark applications. Can be used only when ibm cpd schedular is enabled. Higher the value, higher the priority. |
ae.spark.log.dir |
Set the log directory for the Spark applications. |
spark.driver.resource.gpu.amount |
Number of GPUs required for the Spark driver. For information about how to configure the parameter, see Enabling NVIDIA GPU acceleration. |
spark.driver.resource.gpu.vendor |
Set the value to nvidia to indicate GPU vendor.For information about how to configure the parameter, see Enabling NVIDIA GPU acceleration. |
spark.executor.resource.gpu.amount |
Number of GPUs required for each Spark executor. For information about how to configure the parameter, see Enabling NVIDIA GPU acceleration. |
spark.executor.resource.gpu.vendor |
Set the value to nvidia.com.For information about how to configure the parameter, see Enabling NVIDIA GPU acceleration. |
spark.eventLog.enabled |
TRUE |
spark.executor.extraClassPath |
/home/spark/space/assets/data_asset/*:/home/spark/user_home/dbdrivers/*:/cc-home/_global_/dbdrivers/*:/home/spark/shared/user-libs/spark2/*:/home/spark/user_home/dbdrivers/*:/home/spark/shared/user-libs/common/*:/home/spark/shared/user-libs/connectors/*:/opt/ibm/connectors/parquet-encryption/*:/opt/ibm/third-party/libs/spark2/*:/opt/ibm/third-party/libs/common/*:/opt/ibm/third-party/libs/connectors/*:/opt/ibm/spark/external-jars/* |
spark.executer.memory |
1 G |
spark.executer.cores |
1 |
(custom) ae.spark.executor.count |
1 |
(custom) ae.spark.application.priority |
1 |
spark.driver.extraClassPath |
/home/spark/space/assets/data_asset/*:/home/spark/user_home/dbdrivers/*:/cc-home/_global_/dbdrivers/*:/home/spark/shared/user-libs/spark2/*:/home/spark/user_home/dbdrivers/*:/home/spark/shared/user-libs/common/*:/home/spark/shared/user-libs/connectors/*:/opt/ibm/connectors/parquet-encryption/*:/opt/ibm/third-party/libs/spark2/*:/opt/ibm/third-party/libs/common/*:/opt/ibm/third-party/libs/connectors/*:/opt/ibm/spark/external-jars/* |
spark.driver.memory |
1024 M |
spark.driver.cores |
1 |
spark.local.dir |
/tmp/spark/scratch See spark.local.dir configuration parameter for details. |
spark.master.ui.port |
8080 |
spark.worker.ui.port |
8081 |
spark.ui.port |
4040 |
spark.history.ui.port |
18080 |
spark.ui.enabled |
TRUE |
spark.ui.killEnabled |
FALSE |
spark.eventLog.dir |
file:///home/spark/spark-events |
spark.ui.reverseProxy |
TRUE |
spark.ui.showConsoleProgress |
TRUE |
spark.shuffle.service.port |
7337 |
spark.r.command |
/opt/ibm/conda/R/bin/Rscript |
spark.hadoop.fs.s3a.fast.upload |
TRUE |
spark.hadoop.fs.s3a.multipart.size |
33554432 |
spark.hadoop.fs.stocator.scheme.list |
cos |
spark.hadoop.fs.stocator.cos.scheme |
cos |
spark.hadoop.fs.stocator.glob.bracket.support |
TRUE |
spark.hadoop.fs.stocator.cos.impl |
com.ibm.stocator.fs.cos.COSAPIClient |
spark.hadoop.fs.cos.impl |
com.ibm.stocator.fs.ObjectStoreFileSystem |
spark.hadoop.fs.s3a.impl |
org.apache.hadoop.fs.s3a.S3AFileSystem |
spark.authenticate |
FALSE |
spark.network.crypto.enabled |
FALSE |
spark.network.crypto.keyLength |
256 |
ae.spark.autoscale.enable |
Signals that the application will autoscale based on the application's demand and any other
autoscaling configurations that are set. If specified at instance level, all applications in the
instance will autoscale. Default value:false. For information about how to configure the parameter, see Autoscaling. |
spark.dynamicAllocation.initialExecutors |
Specifies the initial number of executors to be created, irrespective of any demand made by the Spark application. Default value:0 |
spark.dynamicAllocation.minExecutors |
Specifies the minimum number of executors to be maintained, irrespective of any demand by the Spark application. Default value:0 |
spark.dynamicAllocation.maxExecutors |
Specifies the maximum number of executors to be created irrespective of any demand by the Spark Application. Default value:2 |
ae.spark.autoscale.scaleupDemandPercentage |
Specifies the percentage of the executors demanded by Spark Application that should be fulfilled by the application auto-scaler. For example, if at a certain stage a Spark application requests 10 executors and the value for this configuration is set to 50 percent, the application auto-scaler will scale up the number of executors by 5 executors. The default value of this configuration is 100 percent which essentially means that any number of executors requested by an application's demand can be added by the application auto-scaler. Default value:100 |
ae.spark.autoscale.scaledownExcessPercentage |
Specifies the percentage of the idle executors held up by the Spark application that are to be removed by the application auto-scaler. For example, if after completing a certain stage in a Spark application, there are 20 idle executors held up by the application and the value for this configuration is set to 25 percent, the application auto-scaler will scale down the executors in the application by 4 executors. The default value for this configuration is 100 percent which means that the application auto-scaler can scale down all idle executors. Default value:100 |
ae.spark.autoscale.frequency |
The frequency in seconds at which the application auto-scaler scales up executors or scales down executors in an application. Default value:10s |
spark.sql.extensions |
To enable the Spark access control extension, you must set the Spark configuration to
authz.IBMSparkACExtension. For information about how to configure the parameter,
see Enforcing data access policies for Spark runtime using
Spark access control extension. |
idempotency_key |
Use the idempotency_key parameter in the Spark application payload to avoid
triggering duplicate job submissions. This ensures that repeated requests with the same key are
treated as a single request. Idempotency_key cannot be greater than 64 characters
in length. For information about how to configure the parameter, see Configuring Idempotency key in payload. |
timeout_in_seconds |
Use the timeout_in_seconds parameter in the payload to specify the maximum
execution time while submitting an application. For information about how to configure the
parameter, see Configuring maximum runtime time in
payload. |
max_retries
|
Use the max_retries parameter in the payload to specify the maximum
execution time while submitting an application. For information about how to configure the
parameter, see Configuring maximum retries in
payload. |
min_retry_interval_in_seconds |
Use min_retry_interval_in_seconds field in the payload to specify Retry
feature. For information about how to configure the parameter, see Configuring minimum retries in payload. |
ae.spark.kubernetes.container.image |
Use these configurations to add system packages, Python libraries, or JAR files required by your workloads. For information about how to configure the parameter, see Submitting a Spark application by using custom runtime image. |
init_scripts |
Initialization scripts are shell scripts that run on every cluster node at startup before the
Spark driver or executor launch. They are especially useful when your Spark application requires
custom packages, environment variables, or perform any one‑time spark runtime configuration. For
information about how to configure the parameter, see Seamless
package management using init scripts. |
spark.executor.logs.rolling.strategyspark.executor.logs.rolling.time.intervalspark.executor.logs.rolling.maxRetainedFiles |
Configure Spark log rotation based on time, and old logs are cleaned up automatically. For information about how to configure the parameter, see Enable Spark log and event log rotation. |
spark.eventLog.rolling.enabledspark.eventLog.rolling.maxFileSize |
Configure Spark log rotation based on size, and old logs are cleaned up automatically. With
spark.eventLog.rolling.enabled=true and maxFileSize configured,
event logs are correctly split into multiple files (events_1, events_2, …). For information about
how to configure the parameter, see Enable Spark log
and event log rotation. |
spark.history.fs.cleaner.enabled |
Configure history server log clean up automatically. For information about how to configure the parameter, see Enable Spark log and event log rotation. |
Environment variables
System-level variables that configure Spark's runtime environment and JVM settings. The cluster-wide configurations set before Spark starts.
The following table lists the environment variables and their defaults:
| Environment variable | Default value |
|---|---|
| SPARK_DIST_CLASSPATH | /home/spark/space/assets/data_asset/*:/home/spark/user_home/dbdrivers/*:/cc-home/_global_/dbdrivers/*:/opt/ibm/connectors/idax/*:/opt/ibm/connectors/cloudant/*:/opt/ibm/connectors/db2/*:/opt/ibm/connectors/others-db-drivers/*:/opt/ibm/connectors/wdp-connector-driver/*:/opt/ibm/connectors/wdp-connector-jdbc-library/*:/opt/ibm/connectors/stocator/*:/opt/ibm/connectors/s3/*:/opt/ibm/image-libs/common/*:/opt/ibm/image-libs/spark2/*:/opt/ibm/third-party/libs/batch/*:/opt/ibm/spark/external-jars/* |
| SPARK_LOCAL_DIRS | /tmp/spark/scratch |
| SPARK_MASTER_WEBUI_PORT | 8080 |
| SPARK_MASTER_PORT | 7077 |
| SPARK_WORKER_WEBUI_PORT | 8081 |
| CLASSPATH | /home/spark/user_home/dbdrivers/*:/opt/ibm/connectors/idax/*:/opt/ibm/connectors/cloudant/*:/opt/ibm/connectors/db2/*:/opt/ibm/connectors/others-db-drivers/*:/opt/ibm/connectors/wdp-connector-driver/*:/opt/ibm/connectors/wdp-connector-jdbc-library/*:/opt/ibm/connectors/stocator/*:/opt/ibm/connectors/s3/*:/opt/ibm/image-libs/common/*:/opt/ibm/image-libs/spark2/*:/opt/ibm/third-party/libs/batch/* |
| LD_LIBRARY_PATH | /opt/ibm/connectors/dsdriver/dsdriver/lib:/opt/ibm/connectors/others-db-drivers/oracle/lib:/opt/ibm/jdk/jre/lib/architecture/server:/opt/ibm/jdk/jre/lib/architecture/:/usr/local/lib:/lib64 |
| RUNTIME_PYTHON_ENV | python310 |
| PYTHONPATH | /home/spark/space/assets/data_asset:/home/spark/user_home/python-3:/cc-home/_global_/python-3:/home/spark/shared/user-libs/python:/home/spark/shared/conda/envs/python/lib/python/site-packages:/opt/ibm/conda/miniconda/lib/python/site-packages:/opt/ibm/third-party/libs/python3:/opt/ibm/image-libs/python3:/opt/ibm/image-libs/spark2/xskipper-core.jar:/opt/ibm/image-libs/spark2/spark-extensions.jar:/opt/ibm/image-libs/spark2/metaindexmanager.jar:/opt/ibm/image-libs/spark2/stmetaindexplugin.jar:/opt/ibm/spark/python:/opt/ibm/spark/python/lib/py4j-0.10.7-src.zip |
| R_LIBS_USER | /home/spark/space/assets/data_asset:/home/spark/shared/user-libs/R:/opt/ibm/third-party/libs/R:/opt/ibm/conda/R/lib64/R/library/:/opt/ibm/spark/R/lib:/opt/ibm/image-libs/R |