Create a notebook in Watson Studio Local

Create the notebook file

To create or get a notebook file to add to the project:

1. From your project assets view, click the add notebook link.
2. In the Create Notebook window, specify the method to use to create your notebook.
3. You can create a blank notebook, upload a notebook file from your file system, or upload a notebook file from a URL. The notebook you create or select must be a .ipynb file.
4. Specify the rest of the details for your notebook.
5. Click Create Notebook.

Alternatively, you can copy a sample notebook from the community page. The sample notebooks are based on real-world scenarios and contain many useful examples of computations and visualizations that you can adapt to your analysis needs. To work with a copy of the sample notebook, click the Open Notebook icon () and specify your project and the Spark service for the notebook.

For information about the notebook interface, see parts of a notebook.

Create the SparkContext

A SparkContext is required if you want to analyze data using Spark. The SparkContext can either point to the Spark running in cluster mode or Spark running in local mode within the user environment.

Python 2.7, Python 3.5, Python 3.6

Watson Studio provides five different Spark environments that differ in Spark's version and the deploy mode.

A default SparkContext is set up in a variable called sc for Python 2.7, 3.5 and GPU notebooks when a user environment starts up.

If you choose the Python 2.7 with Watson Studio Spark 2.0.2 or Python 3.5 with Watson Studio Spark 2.2.1 kernel, sc points to Spark running in cluster mode. If you choose the Python 2.7 or Python 3.5 or Python 3.6 kernel, sc points to Spark running in local mode within the user environment. You can run the sc.getConf().getAll() command to see the configuration properties of the SparkContext. The configuration properties of the default SparkContext cannot be modified.

AutoStartJupyterSC

If your data scientists don't intend to use Spark for analytics, the cluster admin can disable the creation of the default SparkContext by setting the AutoStartJupyterSC property to false. Complete these steps to set the property.

1. SSH to the Watson Studio Local cluster.
2. Run kubectl exec to a usermgmt pod.
3. Set the configuration AutoStartJupyterSC to false in the /user-home/global/config/config.properties file to disable the creation of the SparkContext. The default value for AutoStartJupyterSC is true.
4. Notify data scientists to restart their Jupyter environment.

import pyspark
conf = pyspark.SparkConf()
conf.setMaster("local[*]")
conf.set("spark.driver.extraClassPath", "/dbdrivers/*")
conf.set("spark.executor.memory", "4g")
conf.set("spark.driver.memory", "4g")
sc = pyspark.SparkContext(conf=conf)
sc.getConf().getAll()

By default, SparkContext is not set up for R notebooks.

Watson Studio Local users can modify one of the following templates to create a SparkContext setup for their R notebooks:

sparklyr library

For Spark 2.0.2:

library(sparklyr)
library(dplyr)
sc <- spark_connect(master = "spark://spark-master-svc:7077")

For Spark 2.2.1:

library(sparklyr)
library(dplyr)
sc <- spark_connect(master = "spark://spark-master221-svc:7077")

SparkR library

For Spark 2.0.2, use master="spark://spark-master-svc:7077". For Spark 2.2.1, use master="spark://spark-master221-svc:7077".

library(SparkR)
sc <- sparkR.session(master="spark://spark-master-svc:7077",
           appName="notebook-R",
           enableHiveSupport=FALSE,
           sparkEnvir=list(
            spark.port.maxRetries="100",
            spark.dynamicAllocation.enabled="true",
            spark.shuffle.service.enabled="true",
            spark.dynamicAllocation.executorIdleTimeout="300",
            spark.executor.memory="4g",
            spark.cores.max="2",
            spark.dynamicAllocation.initialExecutors="1",
spark.driver.extraJavaOptions="-Djavax.net.ssl.trustStore=/user-home/_global_/security/customer-truststores/cacerts",
spark.executor.extraJavaOptions="-Djavax.net.ssl.trustStore=/user-home/_global_/security/customer-truststores/cacerts"
      )
)

Set Watson Studio Local Spark resources

Based on the user cases, you might need to change the resources allocated for the Spark application. The default settings of Watson Studio Local Spark are as follows:

By default, Watson Studio Local uses three Spark workers on the compute nodes. If you add more compute nodes, one additional Spark worker will be started on each added compute node.

To change the resources for the Spark application in the notebook:

First, stop the pre-created sc and then create a new spark context with the proper resource configuration. Python example:

sc.stop()
from pyspark import SparkConf, SparkContext
conf = (SparkConf()
    .set("spark.cores.max", "15")
    .set("spark.dynamicAllocation.initialExecutors", "3")
    .set("spark.executor.cores", "5")
    .set("spark.executor.memory", "6g"))
sc=SparkContext(conf=conf)

Then you can verify the new settings by running the following command in a cell using the new sc:

for item in sorted(sc._conf.getAll()):
print(item)

Note that the resource settings also apply to running notebooks for scheduling jobs.

See Spark Configuration for more information.

Analyze data in the notebook

Now you're ready for the real work to begin!

Typically, you'll install any necessary libraries, load the data, and then start analyzing it. You and your collaborators can prepare the data, visualize data, make predictions, make prescriptive recommendations, and more.