Event log directory and file permissions

Complete this task to set permissions for your Apache Spark event logs for future reading through the Spark history service.

About this task

The event log directory stores the event logs when enabled for your applications. These files are then used by the Spark history server to reconstruct the application’s web UI.

Procedure

Complete the following steps to enable event logging and point the history server to the event logs.

Create the event log directory:
```
mkdir -p /var/spark/events
```
Ensure the owner and group of the directory correspond to the user who is running the Spark history server:
```
chown SPARKID:SPKGRP /var/spark/events
```
Ensure the permissions of the event log directory are set properly:
```
ls -l /var/spark/events
```
Change permissions of the event log directory if necessary. Apache Spark advises a default of 777:
```
chmod ### /var/spark/events
```
Set the sticky bit on the event log directory. This permits users to delete only the files from this directory which they own:
```
chmod +t /var/spark/events
```
Add the following statement to your spark-defaults.conf file, where ### is the permissions you’d like the application to create the file with. Default is 770:
```
spark.zos.app.eventLog.permission ###
```