Configuring z/OS Spark client authentication
Complete the following tasks to enable authentication for connections to the Spark master port.
z/OS® Spark client authentication is enabled by default. Spark does not function properly until you complete the setup for client authentication or disable the client authentication function.
The z/OS UNIX System Services APAR OA57666 is required for z/OS Spark client authentication to work properly.
spark-defaults.conf
file.spark.zos.master.authenticate false
spark-sql
command line
interface, might not work properly. This is because the directories that are created by the Spark cluster might not be
permissible to the Spark
driver.- Application Transparent Transport Layer Security (AT-TLS)
- This is the default Spark client authentication method that uses digital certificates along with AT-TLS. You need to set up digital certificates for the Spark cluster and its users, as well as an AT-TLS policy.
- Trusted Partner
- If all connections to the master port are internal, then you can consider using the Trusted
Partner client authentication method, which doesn't require client certificates. However, this
method continues to use AT-TLS for server authentication. A connection is internal if both endpoints
belong in the same sysplex, the data flowing through the connection is never exposed outside of the
sysplex, and the link or interface that is used is one of the following types:
- CTC
- HiperSockets interface (iQDIO)
- MPCPTP (including XCF and IUTSAMEH)
- OSA-Express QDIO with CHPID type OSX or OSM
- Loopback
- Both connection partners are owned by the same multihomed stack
Trusted Partner requires additional security configuration for the cluster and its users.
For more information about internal connections, see
Sysplex-specific connection routing information
in z/OS Communications Server: IP Programmer's Guide and Reference.
ATTLS
or
TrustedPartner
) in the spark-defaults.conf
file. For
example:spark.zos.master.authenticate.method ATTLS
The APAR PI89136 is
required to use Trusted Partner client authentication method.- Master's client authentication does not match Worker's
- Master's client authentication method does not match Worker's
About Application Transparent Transport Layer Security (AT-TLS)
AT-TLS is a z/OS Communications Server feature that transparently implements the TLS protocol in the TCP layer of the stack. As defined by the TLS protocol, AT-TLS uses digital certificates to authenticate the server and optionally the client, and encrypts the data that is flowing between the server and the client.
During client authentication, the Spark master acts as a server and accepts connections from the Spark worker and Spark users, which act as clients. Once the Spark master validates the client’s digital certificate, a secure connection will be established and all subsequent data-flow between the server and the client will be encrypted.
For more information about AT-TLS, see Application Transparent Transport Layer Security data
protection
in z/OS Communications Server: IP Configuration Guide.
Using AT-TLS as the client authentication method
- Communication between the client and the server is encrypted.
- A trusted relationship is established.
- A client certificate is matched to a local z/OS user ID.
Using Trusted Partner as the client authentication method
Using AT-TLS as the client authentication method requires a digital certificate for each user that is connecting to the Spark master port. If you know that all connections to the master port are internal, you can consider using the Trusted Partner client authentication method instead, which doesn't require client certificates. However, this method continues to use AT-TLS for server authentication.
- Creating and configuring digital certificates and key rings
- Configuring Policy Agent
- Defining security authorization for Policy Agent
- Creating the Policy Agent configuration files
- Configuring PROFILE.TCPIP for AT-TLS
- Defining the AT-TLS policy rules
- Starting and stopping Policy Agent
- Configuring additional authorities and permissions for the Spark cluster
- Starting the Spark cluster