Storage considerations for IBM Optim Data Privacy

Edit online

To install IBM® Optim Data Privacy, you must have a supported persistent storage solution that is accessible to your application deployment.

This section provides guidance on planning and managing storage for IBM Optim Data Privacy deployments. It covers storage architecture, sizing strategies, performance optimization, and data retention policies to help ensure reliable and scalable operations across hybrid environments.

IBM Optim Data Privacy is designed to handle large volumes of structured and semi-structured data across multiple environments. Storage planning is critical to ensure:

  • Efficient data extraction, transformation, and loading
  • Secure handling of sensitive data
  • High availability and performance of containerized services
  • Compliance with data retention and privacy regulations

Review the following sections to determine what storage is right for you.

Storage architecture

IBM Optim Data Privacy uses a modular, container-based architecture. Each service within IBM Optim has distinct storage requirements:

Table 1. Storage architecture
Service Storage type Purpose
IBM Optim Runtime Ephemeral/log Workflow execution, temporary data
IBM Optim MetastoreDB PostgreSQL database Metadata storage (users, workflows, connections)
IBM Optim Core API Ephemeral/log API logs and request handling
IBM Optim Spark Ephemeral/log Large-scale data processing

Recommended storage sizing guidelines

IBM Optim storage requirements vary based on data volume, number of workflows, and retention policies.

The following table provides baseline sizing recommendations:

Table 2. Storage sizing
Deployment size Metadata storage Workflow storage
Small (development/test) 20 GB 50 GB
Medium (quality testing) 50 GB 200 GB
Large (production) 100 GB 500 GB+
Tip: Use SSD-backed volumes for workflow and archive storage to improve I/O performance.

Separating storage volumes

Use separate dedicated volumes for different IBM Optim storage areas to prevent disk space issues from causing system failures.

If storage areas are shared with each other or with the host operating system disk, generated files filling up the disk can cause the MetastoreDB or the host operating system to fail. To prevent these failures, configure separate dedicated volumes for different types of IBM Optim data.

Use separate dedicated volumes for the following storage areas:
IBM Optim MetastoreDB (metadata)

The MetastoreDB directory stores critical metadata required for IBM Optim operations. This volume must be isolated to ensure metadata integrity and prevent database failures.

Logs and reports (workflow data)

Workflow data includes operational logs and generated reports. These files can grow rapidly during normal operations and should be stored on a dedicated volume.

Tip: Use separate logical drives (partitions) at minimum. Physical disks are not required. Partitions provide adequate isolation.

With this configuration, if workflow or test data volumes get full, you can still access the host system and IBM Optim application to clear space and resolve the problem.

Using mounted or extended storage volumes (SAN and NAS)

If your environment uses external storage such as a SAN or NAS drive that is mounted to the Linux host, additional steps are required before you can use those volumes for IBM Optim directories.

Storage paths configured in the .env file must use the host-side mount path as shown in the Mounted on column of the df -h command output — not the raw device path. For example, if your archive storage is mounted at /mnt/optim, set:

OPTIM_LOG_DIR=/mnt/optim/logs

Before running the setup script, complete the following steps for each extended or non-root disk that will store IBM Optim data:

  1. Ensure the disk is formatted with a supported file system (for example, xfs or ext4) and mounted at its intended host path.
  2. If you intend to store Podman container images on a non-root disk, create the required Podman directories on that disk and assign ownership to the non-root installation user. Run the following commands as root, replacing /mydata with your actual mount point and optim with your non-root installation user: 
    sudo mkdir -p /mount-point/podman/data/containers/storage
sudo mkdir -p /mount-point/podman/config/containers
sudo chown -R optim:optim /mount-point/podman
    Important: The non-root installation user must have write permission on the additional disk before you export the Podman storage variables. Without this step, Podman silently falls back to the root default storage paths regardless of the exported variables.
  3. After creating the directories, export the following environment variables in the same shell session as the non-root installation user, replacing /mydata with your actual mount point: 
    export XDG_CONFIG_HOME=/mount-point/podman/config
export XDG_DATA_HOME=/mount-point/podman/data
    Important: These variables must be set in the same shell session in which you run the setup script. They cannot be set inside the script itself because child bash sessions do not inherit them and the values will fall back to the root defaults.
Tip: Run df -h on the host to verify that your extended volumes are recognized by the operating system and to confirm the correct mount paths to use in the .env file.

Storage performance

To ensure optimal performance of your IBM Optim storage volumes, follow these best practices:

  • Use dedicated volumes for IBM Optim Runtime and Spark containers.
  • Enable read/write caching for frequently accessed datasets.
  • Configure I/O throttling to prevent resource contention during concurrent workflows.
  • Monitor disk usage and latency using built-in reporting and external tools.

Storage security

All IBM Optim data at rest must be encrypted using enterprise-grade encryption standards. Additional security measures include the following strategies:

  • Role-based access to storage volumes
  • Integration with Vault for credential and secret management
  • Masking and de-identification of sensitive data before storage

Backup and recovery

To protect against data loss, follow these best practices:

  • Schedule regular backups of MetastoreDB and workflow logs.
  • Use snapshot-based backups for container volumes.
  • Store backups in geographically redundant locations.
  • Validate recovery procedures quarterly.

For more information, see Backing up and restoring.

Planning for growth

Monitor IBM Optim storage utilization trends and set alerts for threshold breaches to proactively scale resources.

Storage planning should account for the following growth areas:

  • Increasing data volumes from new applications
  • Additional workflows and environments
  • Future support for synthetic data and virtualization