Architecture of IBM Optim Data Privacy

Edit online

Understanding the architecture of IBM® Optim Data Privacy helps administrators to plan deployments, troubleshoot issues, and optimize performance across hybrid environments.

This section describes the architecture of IBM Optim Data Privacy, including its modular components, deployment model, and integration points.

IBM Optim Data Privacy is a containerized, cloud-native solution designed to support secure, scalable, and automated test data provisioning. The architecture is composed of loosely coupled services that communicate by using REST APIs.

IBM Optim supports both UI-driven and API-driven workflows and integrates with enterprise authentication systems, external schedulers, and a variety of data sources.

Component architecture

The platform consists of five core services, each deployed as a container:

Table 1. IBM Optim Data Privacy component services
Service Description
IBM Optim UI A React-based web interface built with IBM Carbon Design System. Provides workflow creation, monitoring, and reporting. Also known as the IBM Optim Hub.
IBM Optim Core API The central orchestration layer. Handles all requests from the UI and external APIs, manages workflows, and communicates with the runtime.
IBM Optim Runtime Executes data operations such as extraction, masking, and loading. Includes an Apache Spark runtime for large-scale processing.
IBM Optim MetastoreDB Stores metadata including user profiles, workflow definitions, and connection details. Uses PostgreSQL or EDB.
IBM Optim Keycloak Implements Keycloak for authentication and authorization. Integrates with enterprise identity providers (LDAP, SAML, OIDC, Kerberos).
IBM Optim Catalog Manages file storage table metadata by using the Apache Iceberg catalog service. Provides centralized metadata management, reliable transaction handling, and improved support for concurrent operations across multiple clients and processing engines.
IBM Flight A connector library based on the Apache Arrow Flight protocol that provides a single, unified interface for applications to read, write, and query various data sources.

All services are stateless except for the MetastoreDB, which requires persistent storage.

Data flow and workflow execution

IBM Optim Data Privacy follows a modular data flow.

[Source database] → [Extract] → [Transform/Mask] → [Load] → [Target/Test database]

Each step is orchestrated by the IBM Optim Core API and executed by the IBM Optim Runtime. The data flow maintains referential integrity and supports data masking.

Consider the following example workflow:
  1. A user logs in and is validated by using Keycloak.
  2. The user creates a workflow in the UI or by using the API.
  3. IBM Optim Core API validates the request and stores metadata.
  4. IBM Optim Runtime connects to the source system and extracts data.
  5. Data is masked and transformed as needed.
  6. Data is loaded into the target environment.
  7. Reports and logs are generated and stored.
Figure 1. Architecture of IBM Optim Data Privacy
This image shows the general architectural layout and flow of the Optim product. The flow is also described in the surrounding text.

Integration points

IBM Optim Data Privacy integrates with:

  • Enterprise identity providers by using Keycloak