Working with Spark

You can use watsonx.data Spark to run workloads to achieve Spark use cases.

watsonx.data on IBM Software Hub

Introduction to watsonx.data Spark
This section provides an overview about watsonx.data Spark engines, guides you on how to get started with your Spark application, and outlines the complete workflow for developing and managing Spark applications.
Spark application runtime
The Spark application runtime enables you to run Spark applications,written in Python or Scala to perform workloads such as data engineering, analysis, or data management outside interactive environments. These applications can be submitted through multiple modes using console, cpdctl, and REST API.
Query Server runtime
Spark query server allows you to start Spark Hive based thrift server that provides the interface to collect, store, query and analyse all your lakehouse data from Spark engine.
Working with watsonx.ai Jupyter Notebook
watsonx.data integrates with watsonx.ai to allow a web-based working experience with Jupyter Notebook. You can use the watsonx.ai interface to build your own code in the Jupyter Notebook, and run it by using watsonx.data Spark as the runtime environment.
Development Environments – Spark lab experience
Monitoring and debugging
Data management