Batch processing

When you add your model deployment in the Watson OpenScale service, you can enable batch processing to manage data for model evaluations. With batch processing, you can run evaluations with your own data pipeline to help process large datasets.

The process of deploying a model can require large amounts of data. To help you manage data for model evaluations, Watson OpenScale provides two methods that you can use to decide how you want to process your data and generate results. When you select a model deployment that you want to add, you can choose to use the system-managed or self-managed method to configure model evaluations. If you use the system-managed method, you must use an API or the Watson OpenScale Python SDK to log your model transactions in Watson OpenScale. When you use the self-managed method, you can log your model transactions in your own data warehouse and run evaluations with an Apache Spark analytics engine.

To enable batch processing, you must use one of the following options to prepare your deployment environment:

• Preparing the batch processing environment in IBM Analytics Engine
• Preparing the batch processing environment in IBM Analytics Engine on zLinux
• Preparing the batch processing environment on the Hadoop Ecosystem

After you prepare your deployment environment, you can configure batch processing in Watson OpenScale. You can also use the Watson OpenScale Python SDK to configure batch processing with the IBM Analytics Engine powered by Apache Spark environment.

Parent topic: Evaluating AI models with Watson OpenScale