Scripting with Python for Spark
IBM® SPSS® Modeler can execute Python scripts using the Apache Spark framework to process data. This documentation provides the Python API description for the interfaces provided.
Prerequisites
- If you plan to execute Python/Spark scripts against IBM SPSS Analytic Server, you must have a connection to Analytic Server, and Analytic Server must have access to a compatible installation of Apache Spark. Refer to your IBM SPSS Analytic Server documentation for details about using Apache Spark as the execution engine.
- If you plan to execute Python/Spark scripts against IBM SPSS Modeler Server (or the local server included
with IBM SPSS Modeler Client), you must configure
it to use your Python installation by adding the following new option to
options.cfg:
For example:# Set to the full path to the python executable (including the executable name) to enable use of PySpark. eas_pyspark_python_path, ""
eas_pyspark_python_path, "C:/Your_Python_Install/python.exe"
- The IBM SPSS Modeler installation includes a Spark distribution but not a Python distribution. If you want to use the Machine Learning Library (MLlib), you must install a version of Python that includes NumPy.
- When installing Python, make sure all users have permission to access the Python installation.
The IBM SPSS Analytic Server context object
import spss.pyspark.runtime
asContext = spss.pyspark.runtime.getContext()
sparkContext = asc.getSparkContext()
sqlContext = asc.getSparkSqlContext()
Refer to your Apache Spark documentation for information about the Spark context and the SQL context.
Accessing data
inputData = asContext.getSparkInputData()
asContext.setSparkOutputData(outputData)
outputData = sqlContext.createDataFrame(rdd)
Defining the data model
A node that produces data must also define a data model that describes the fields visible downstream of the node. In Spark SQL terminology, the data model is the schema.
A Python/Spark script defines its output data model in the form of a pyspsark.sql.types.StructType object. A StructType describes a row in the output data frame and is constructed from a list of StructField objects. Each StructField describes a single field in the output data model.
inputSchema = inputData.schema
field = StructField(name, dataType, nullable=True, metadata=None)
Refer to your Spark documentation for information about the constructor.
You must provide at least the field name and its data type. Optionally, you can specify metadata to provide a measure, role, and description for the field (see Data metadata).
DataModelOnly mode
IBM SPSS Modeler needs to know the output data model for a node, before the node is executed, in order to enable downstream editing. To obtain the output data model for a Python/Spark node, IBM SPSS Modeler executes the script in a special "data model only" mode where there is no data available. The script can identify this mode using the isComputeDataModelOnly method on the Analytic Server context object.
if asContext.isComputeDataModelOnly():
inputSchema = asContext.getSparkInputSchema()
outputSchema = ... # construct the output data model
asContext.setSparkOutputSchema(outputSchema)
else:
inputData = asContext.getSparkInputData()
outputData = ... # construct the output data frame
asContext.setSparkOutputData(outputData)
Building a model
A node that builds a model must return to the execution context some content that describes the model sufficiently that the node which applies the model can recreate it exactly at a later time.
Model content is defined in terms of key/value pairs where the meaning of the keys and the values is known only to the build and score nodes and is not interpreted by Modeler in any way. Optionally the node may assign a MIME type to a value with the intent that Modeler might display those values which have known types to the user in the model nugget.
asContext.setModelContentFromString(key, value, mimeType=None)
value = asContext.getModelContentToString(key)
asContext.setModelContentFromPath(key, path)
Note that in this case there is no option to specify a MIME type because the bundle may contain various content types.
path = asContext.createTemporaryFolder()
path = asContext.getModelContentToPath(key)
Error handling
from spss.pyspark.exceptions import ASContextException
if ... some error condition ...:
raise ASContextException("message to display to user")