Known issues and limitations in IBM Optim Data Privacy

General limitations

The following known issues apply to general product usage, including usability and integration scenarios.

• Installation script summary shows the incorrect Keycloak port
• After you log into IBM Optim, the user interface fails to load
• Notifications are cleared after one day
• The IBM Optim user interface fails to load after running a high number of flows
• Limited UI customization
• CI/CD integration requires external API management
• When viewing job logs by using the IBM Optim API or the Jobs dashboard, logs can be truncated if they are too long
• Additional limitations

Installation script summary shows the incorrect Keycloak port

When you attempt to access Keycloak by using the URL provided by the installation script output, the connection fails. The installation script output shows the incorrect port number. The script points to {url}:8443/admin, but the correct URL is {url}:7780/admin.

If you have edited the default values during installation setup, your deployment details might be different.

Resolving the problem

To resolve this issue, open Keycloak by using the correct URL and port 7780. For example: optim.kc.example.com:7780/admin

After you log into IBM Optim, the user interface fails to load

When you start the IBM Optim user interface, you can encounter a blank page. Refreshing the page does not resolve the problem.

Resolving the problem

To resolve this issue, stop and then start the IBM Optim containers.

• If your IBM Optim deployment is on an online (connected) system, run the following commands:

  podman-compose -f optim-compose-icr.yaml down

  podman-compose -f optim-compose-icr.yaml up -d

• If your IBM Optim deployment is on an offline (airgapped) system, run the following commands:

  podman-compose -f optim-compose.yaml down

  podman-compose -f optim-compose.yaml up -d

The IBM Optim user interface fails to load after running a high number of flows

The IBM Optim user interface can fail to load after running a high number of flows. This issue occurs when disk space is insufficient. When disk space is full, jobs cannot run.

Resolving the problem

To resolve this issue, clear some disk space. If the issue persists, increase the amount of available disk space.

Notifications are cleared after one day

Notification messages are only available in the notifications menu for one day. Even if the notification is unread, it disappears from the notifications list after one day.

Limited UI customization

Branding and layout customization options are limited. You cannot modify the product logo, color scheme, or page layouts.

Resolving the problem

Use role-based access and dashboard filters to tailor views for different user groups. Submit enhancement requests through IBM Support for UI customization needs.

CI/CD integration requires external API management

You must manage API orchestration and authentication externally.

Resolving the problem

Use gateway tools to manage authentication and routing. Store API credentials securely and automate calls by using DevOps scripts.

When viewing job logs by using the IBM Optim API or the Jobs dashboard, logs can be truncated if they are too long

When you retrieve logs by using the /v1/job/spark/status/:id endpoint or the Jobs dashboard, the returned log can be truncated if it surpasses a certain length. The beginning of the log is lost, and you can only see the end.

Instead of the full log, the API shows a truncated log with a response that starts with Bytes 94211-196611 of 196611.

Resolving the problem

View the full job log by navigating to the OPTIM_LOG_DIR directory defined during IBM Optim installation. For more information about viewing logs, see Monitoring and logging IBM Optim Data Privacy.

Additional limitations

In addition to the known issues documented in this section, IBM Optim also has the following general limitations:

• Concurrent archive execution is limited to two active jobs per environment. Additional jobs are queued.
• You cannot bulk select multiple tables for archiving.
• Compare reports might not flag all numeric precision or truncation discrepancies.
• There is no pre-validation capability to verify database privileges before running an archive flow.

Feature limitations

IBM Optim Data Privacy includes the following feature limitations.

• Incomplete feature parity compared to traditional IBM InfoSphere Optim
• Custom data masking formats created by using the API are not shown on the custom formats page in the UI
• When applied to numeric columns, the Format Preserving masking policy can produce values shorter than the requested output length
• #known-issues__optim-relationships-limitations
• Data localization enforcement
• Finding related tables when you create access definitions
• Exporting data with cyclical foreign key relationships
• Only tables, basic indexes, primary keys, and foreign keys are exported
• Source tables omitted from the Insert Definition cause runtime to apply unsafe defaults
• Predefined Hash Policy hashes INTEGER inputs as their string representation
• Standard Credit Card Policy behavior for invalid input values
• Standard Credit Card Policy for Extended Credit Card can cause a job to fail with a data truncation error
• Standard SSN and SIN masking policy behavior for invalid input values when using the RepeatableFormatFabricationProcessor
• Masking jobs created in IBM Optim 1.2 or earlier can fail with an InvalidRequestException error when imported into version 2.0

Incomplete feature parity compared to traditional IBM InfoSphere® Optim

Certain traditional Optim features such as automated data discovery, synthetic data generation, and data virtualization are not available in this release of IBM Optim Data Privacy.

Custom data masking formats created by using the API are not shown on the custom formats page in the UI

When you use the flow/create-nodes API to create and apply a custom format masking policy, you cannot later use the same custom format from the Workflow > Custom formats page in the IBM Optim UI.

The issue occurs because the create-nodes API directly applies the custom format to the flow being created. This action is not the same as creating a reusable custom masking format template.

Resolving the problem

Use the IBM Optim UI to define a custom format masking template. For more information, see Defining custom data masking formats.

When applied to numeric columns, the Format Preserving masking policy can produce values shorter than the requested output length

Normally, the predefined Format Preserving policy accepts an optional output length and behaves as follows: shorter lengths truncate the value, longer lengths add padding, and 0 preserves the source length. However, this behavior does not always hold true for numeric columns. A masked numeric value can be shorter than the configured output length if the masked result begins with one or more leading zeros. Because numeric values do not preserve leading zeros, the database drops those zeros when it writes the value to the target, reducing the effective length of the masked value.

This limitation affects only numeric column types. It does not affect all values in a column, but only those where the masked output coincidentally begins with one or more leading zeros.

Resolving the problem

Define a custom format for the numeric column type and apply it to achieve masking by using the standard policy instead of the Format Preserving policy.

Data localization enforcement

Data residency enforcement depends on external infrastructure.

Resolving the problem

Deploy IBM Optim within region-specific cloud zones and configure storage policies to restrict data movement. Use cloud provider-native tools to enforce localization.

Finding related tables when you create access definitions

When you define an access definition, finding related tables is limited to detecting primary key to foreign key relationships. It does not include unique constraints.

Resolving the problem

Manually add related tables when only unique constraints exist.

Exporting data with cyclical foreign key relationships

You cannot export source tables that have cyclical foreign key relationships.

Resolving the problem

Consider restructuring the schema or breaking cycles before you export the data.

Only tables, basic indexes, primary keys, and foreign keys are exported

Database objects other than tables, basic indexes, primary keys, and foreign keys, such as views and triggers, are not included in the data export process.

Source tables omitted from the Insert Definition cause runtime to apply unsafe defaults

When you configure a target database node in the flow builder, the Insert Definition lets you select which source tables from the Access Definition to map to target tables. If you omit one or more source tables from the Insert Definition, the runtime does not validate the omission at submission time. Instead, the runtime applies the following assumptions for each unmapped table:

• The target schema and table name are the same as those in the source.
• The target connection is the first target connection found in the flow JSON.

For example, if your Access Definition contains 4 tables from a Db2 source that uses schema DB2INST1, and your Insert Definition maps only 1 of those tables, the runtime attempts to create the remaining 3 tables by using schema DB2INST1 on the target database. If the target is a PostgreSQL database where that schema does not exist, the job fails immediately with errors similar to the following:

OPTTR0222E Table creation failed for DB2INST1.CUSTOMER
CDICO2009E: Table "DB2INST1"."CUSTOMER" could not be created:
SQL error: [IBM][PostgreSQL JDBC Driver][PostgreSQL] schema "DB2INST1" does not exist.
(error code: DATA_IO_ERROR)

The job fails within seconds of starting and produces no warning at flow-save time. If the Std logs tab in the job log viewer does not show an error message, select the Error logs tab. Runtime errors of this type appear in the Error logs tab only. The Std logs tab contains Apache Spark startup and infrastructure output and does not include job-level runtime errors.

Resolving the problem

To resolve this issue, open the Insert Definition node in the flow builder by right-clicking and selecting Edit. Add the missing source tables and specify the correct target schema and connection for each table. Ensure that every source table from the Access Definition has a corresponding entry in the Insert Definition before you run the flow.

Predefined Hash Policy hashes INTEGER inputs as their string representation

When the Predefined Hash Policy is applied to an INTEGER column, the input value is converted to its string representation before hashing. This means that numerically equivalent but differently formatted values can produce different hash outputs. For example, the values 0 and 00 are numerically equal but produce different hash values because they are hashed as the strings "0" and "00" respectively.

The word removal and case transformation options (uppercase, lowercase) available in the policy configuration apply only to VARCHAR data types. These options have no effect when the policy is applied to INTEGER columns.

Resolving the problem

Ensure that integer columns contain consistently formatted values before applying the Predefined Hash Policy. If repeatable hashing is required across runs, use the same string representation for all equivalent values.

Standard Credit Card Policy behavior for invalid input values

The Standard Credit Card Policy has the following limitations when it processes invalid input:

• Credit card numbers that fail the Luhn check or have an incorrect length for the detected card type (Visa, Mastercard, American Express, or Discover) are returned as-is without masking. No error is raised and no indication is provided that the value was not masked.
• Input values that contain non-numeric characters cause the masking job to fail with an error.

Resolving the problem

Before you run a masking job that uses the Standard Credit Card Policy for Extended Credit Card, ensure that the target column is wide enough to accommodate the masked output. Set the target column length to at least 44 characters to prevent truncation errors. Additionally, ensure that all source data contains valid credit card numbers. Invalid or randomly generated input values can trigger this issue even when the target column length is sufficient.

Standard Credit Card Policy for Extended Credit Card can cause a job to fail with a data truncation error

CDICO2033E: Character data truncation detected for the field column_name.
The length of the value is n and the length of the target column is m.

The Standard Credit Card Policy for Extended Credit Card is designed to mask card numbers up to 19 digits. The hash-based output produced by this policy is longer than a standard 16-digit credit card number and can exceed the column width defined in the target database schema.

Resolving the problem

Before you run a masking job that uses the Standard Credit Card Policy for Extended Credit Card, ensure that the target column is wide enough to accommodate the masked output. Set the target column length to at least 44 characters to prevent truncation errors. Additionally, ensure that all source data contains valid credit card numbers. Invalid or randomly generated input values can trigger this issue even when the target column length is sufficient.

Standard SSN and SIN masking policy behavior for invalid input values when using the RepeatableFormatFabricationProcessor

When you use the RepeatableFormatFabricationProcessor with the UsaSocialSecurityNumber or CanadianSIN format, the following limitations apply:

• Invalid input values are masked instead of being returned unchanged. For example, an invalid SSN such as 999999999 is masked to a fabricated value rather than passed through as-is. Support for retaining invalid values is planned for a future release.
• The output format is not guaranteed to match the input format. Separators such as spaces or dashes can appear in the output even when the input contained no separators. For example, the input 999999999 (no separators) can produce output such as 560-21 5896 (with separators added).

Resolving the problem

To ensure consistent separator formatting in the output, use the predefined SSN masking policy instead of the standard UsaSocialSecurityNumber format. The predefined policy includes post-processing to enforce dash formatting when the includeDashes option is enabled. Note that the predefined policy supports dashes only; space-separated output is not available.

Masking jobs created in IBM Optim 1.2 or earlier can fail with an InvalidRequestException error when imported into version 2.0

When you import an IBM Optim masking job that was created in version 1.2.0 or earlier into version 2.0.0, the job can fail at runtime with the following error:

com.ibm.optim.tdm.runtime.exception.InvalidRequestException: Invalid javaScript field(s):
.mask_policies[0].policy.pipelines[0].nodes[1].app_data.processor.configuration.preProcessorConfiguration.javaScript:
unrecognised javaScript value

This issue affects jobs that include a predefined masking policy. The internal representation of predefined masking policies changed between version 1.2 and version 2.0. When the runtime parses the imported job model, it encounters an unrecognized value in the javaScript field and rejects the job before any data processing begins.

Resolving the problem

After you import the job, recreate the predefined masking policies in the masking node by completing the following steps for each affected masking node in the flow:

1. In the flow builder, right-click the masking node and select Edit.
2. Click Next to navigate to the Set Policies page.
3. Locate and delete each predefined masking policy.
4. Recreate the predefined masking policies by selecting the appropriate policy for each column.
5. Save the masking node and run the flow.

Database limitations

The following known issues apply to database operations.

• Limitations with building queries
• Non-numeric data type handling
• Keyword restriction
• The system renames constraints to avoid constraint name collisions
• Functional expressions in index definitions are not fully supported
• Limited varchar size during Informix target table creation
• Writing data to a Snowflake target connection fails
• SQL Server can encounter deadlock errors when used as a target database
• Constraint creation fails if SQL Server is the target and the source uses a RESTRICT constraint
• SQL Server limitations
• Apache Hive is limited when acting as a data source or target
• When extracting data from SAP ASE to other database platforms, foreign key constraint names cannot be retrieved from JDBC metadata
• When transferring data to SAP ASE tables that contain large object (LOB) columns, the IBM Optim job fails
• MongoDB is limited when acting as a data source
• No target collection created automatically when MongoDB is the target database
• MongoDB is limited when acting as a data target
• Apache Cassandra and DataStax are limited when acting as a source or target
• Oracle rows not found are counted as deleted in the Delete Process Report
• Oracle limitations
• IBM Db2 can implicitly create a missing target schema when a flow is submitted through the API
• IBM Db2 for i does not support lowercase or mixed-case source table names

Limitations with building queries

Only a limited set of operators is available in the query builder.

Resolving the problem

The query builder supports the following operators: equals (=), not equals (!=), greater than (>), less than (<), and LIKE. For complex queries requiring additional operators, use custom SQL queries instead.

Non-numeric data type handling

Non-numeric data (based on the column data type) is only correctly parsed if enclosed in single quote characters.

Resolving the problem

Enclose non-numeric data in single quotes to ensure correct parsing.

Keyword restriction

The string "INVALID" is a reserved value used by the UI component to indicate an incomplete state. It cannot be used as a valid query value.

Resolving the problem

Use an alternative string value in your queries. Do not use "INVALID" as a query parameter or filter value.

The system renames constraints to avoid constraint name collisions

IBM Optim renames every recreated constraint to use a generated UUID in the name. This avoids constraint name collisions caused by name scopes in different databases and manages the different character limits for constraint names.

This behavior helps to ensure compatibility across all supported data sources.

Functional expressions in index definitions are not fully supported

IBM Optim cannot fully extract or reconstruct index definitions that include functional or expression-based columns. Indexes that use expressions such as sorting directives (DESC, ASC) or SQL functions such as UPPER() and LOWER() are not fully supported during metadata extraction and reconstruction. Instead of capturing the complete functional definition of an index column, IBM Optim retrieves only a flattened list of column identifiers and omits or misinterprets the associated expressions.

This limitation applies to all supported database types. The following problems can occur:

• Functional indexes are recreated incorrectly or fail during deployment.
• Generated DDL statements might reference invalid or non-existent identifiers. For example, a statement might contain the string upper instead of the function call UPPER(column_name).
• Unique or constraint-based indexes that rely on functional expressions might not enforce the intended rules after migration.
• Behavior varies depending on the source and target database combination.

Resolving the problem

Until full support is available, use one or more of the following workarounds to reduce the risk of migration errors:

1. Avoid source schemas that use functional or expression-based indexes when you use IBM Optim.
2. Replace functional indexes with equivalent computed or physical columns where your database supports this option.
3. Manually recreate functional indexes in the target database after migration is complete.

Limited varchar size during Informix target table creation

Maximum varchar size has been exceeded

Resolving the problem

To work around this issue, ensure that your source varchar data does not exceed 255 characters if you are exporting to a target Informix database.

Writing data to a Snowflake target connection fails

Caused by: java.util.NoSuchElementException: No value found for 'LOAD_DELETE'
	at scala.Enumeration.$anonfun$withName$1(Enumeration.scala:159)
	at scala.collection.immutable.HashMap.getOrElse(HashMap.scala:717)
	at scala.Enumeration.withName(Enumeration.scala:159)
	at com.ibm.connect.spark.ConnectorOptions.$anonfun$targetWriteMode$9(ConnectorOptions.scala:147)
	at scala.collection.StrictOptimizedIterableOps.map(StrictOptimizedIterableOps.scala:100)
	at scala.collection.StrictOptimizedIterableOps.map$(StrictOptimizedIterableOps.scala:87)
	at scala.collection.mutable.ArrayBuffer.map(ArrayBuffer.scala:42)

Resolving the problem

To work around this issue, complete the following steps:

1. Add the following line to the IBM Optim .env file:

   OPTIM_FLIGHT_ENABLE=true

2. Stop and then start the IBM Optim containers to allow the setting to take effect.
   • If your IBM Optim deployment is on an online (connected) system, run the following commands:

     podman-compose -f optim-compose-icr.yaml down

     podman-compose -f optim-compose-icr.yaml up -d

   • If your IBM Optim deployment is on an offline (airgapped) system, run the following commands:

     podman-compose -f optim-compose.yaml down

     podman-compose -f optim-compose.yaml up -d

3. Run the IBM Optim flow to write to the Snowflake target.

SQL Server can encounter deadlock errors when used as a target database

When you use Microsoft SQL Server as a target database in IBM Optim operations, you can encounter intermittent deadlock errors during merge operations with related tables. The error occurs because SQL Server uses a default locking mechanism when multiple Spark executors write simultaneously.

[IBM][SQLServer JDBC Driver][SQLServer]Transaction (Process ID XX) was deadlocked
on lock resources with another process and has been chosen as the deadlock victim.
SQLSTATE: 40001

Resolving the problem

To prevent this issue, a database administrator can enable Read Committed Snapshot Isolation (RCSI) on the SQL Server target database. Complete this configuration before running IBM Optim jobs.

To enable RCSI:

1. Connect to your SQL Server instance by using SQL Server Management Studio or sqlcmd.
2. Enable RCSI on your target SQL Server database:

   ALTER DATABASE [YourDatabaseName] SET READ_COMMITTED_SNAPSHOT ON;

   Replace [YourDatabaseName] with the actual database name.
3. Verify that RCSI is enabled:

   SELECT 
       name, 
       is_read_committed_snapshot_on 
   FROM sys.databases 
   WHERE name = 'YourDatabaseName';

   A successful result is that is_read_committed_snapshot_on returns a value of 1 (enabled).

Constraint creation fails if SQL Server is the target and the source uses a RESTRICT constraint

SQL Server does not support the delete rule RESTRICT. If SQL Server is used as target connection in a flow and the source connection uses the RESTRICT rule in a constraint, then the constraint creation will fail on the target.

Resolving the problem

Do not use SQL Server as a target connection if your source connection uses RESTRICT in a constraint.

SQL Server limitations

SQL Server has the following limitations when used as a source or target connection:

• Tables with special characters in table names might not be discovered consistently.
• Tables containing BLOB/VARBINARY(MAX) columns might not be fully supported by Data Discovery.
• Tables containing extensive DATETIMEOFFSET or timezone-specific data might not be discovered correctly.
• Viewing tables with very large CLOB or large-volume datasets (for example, 10M+ records) might cause UI performance issues.
• TIME data types are currently displayed with a default date component (1970-01-01).
• Newlines and multiple spaces in character data might not be preserved in Data Explorer and Archive Browse views.
• DATE values might be displayed as TIMESTAMP values in the user interface.
• Binary data types (VARBINARY/RAW) might not be rendered in a user-friendly format.
• Floating-point values with precision greater than approximately 15 digits might experience precision loss during archive and restore operations.
• Restored floating-point columns might not retain the original source data type metadata.

Apache Hive is limited when acting as a data source or target

Apache Hive has the following limitations when used as a source or target connection:

• It only supports Varchar or Blob types with a length of 65535. If Apache Hive is the target and any source table includes columns of these types that exceed the maximum, the job fails.
• It converts all names to lowercase. When Apache Hive is used as a target, you cannot keep the original case format from the source database.
• It does not support any constraints or relations, including primary keys. As a result, IBM Optim cannot load related tables from Apache Hive.
• It only supports the append and replace conflict_resolution options. Only the append option is available from the IBM Optim user interface.
• It only supports the ORC table format.

When extracting data from SAP ASE to other database platforms, foreign key constraint names cannot be retrieved from JDBC metadata

The behavior is a known limitation of the SAP ASE JDBC driver. The driver does not populate foreign key constraint names in JDBC metadata, even though these constraints are explicitly defined and enforced within the database.

When you use the SAP ASE JDBC driver, the method DatabaseMetaData.getImportedKeys() returns NULL values for the FK_NAME column in the resulting ResultSet.

As a result, data extraction tools or applications that rely on JDBC metadata to extract schema definitions are unable to identify foreign key constraint names when you source schemas from SAP ASE.

Resolving the problem

To work around this issue, manually recreate foreign keys after the IBM Optim job completes. Because foreign key names cannot be obtained through JDBC metadata, foreign key constraints must be manually recreated in the target database after the table migration is complete.

To recreate the foreign key constraints:

1. Run the IBM Optim job to extract data from SAP ASE to the target database.
2. Identify foreign key relationships from the source system, either by using system catalogs or database scripts.
3. Manually recreate the foreign key constraints in the target database.

For example, if you are using SAP ASE as the source connection and IBM Db2® as the target, first run the IBM Optim, then recreate the foreign keys in Db2 by using the following command syntax:

ALTER TABLE TABLE_NAME
ADD CONSTRAINT <FK_NAME>
FOREIGN KEY (COLUMN_NAME)
REFERENCES <REFERENCE_TABLE_NAME> (REFERENCE_COLUMN_NAME);

When transferring data to SAP ASE tables that contain large object (LOB) columns, the IBM Optim job fails

When the target connection is SAP ASE in an IBM Optim job and conflict_resolution is set to merge, then tables that contain large text columns (CLOB, TEXT, or LONGVARCHAR) cause the job to terminate during the insertion phase. Other tables within the same job might not complete successfully.

java.sql.SQLException: [IBM][Sybase
        JDBC Driver] Data type not supported: 0x24

The behavior is a known limitation of the SAP JDBC driver and is independent of the IBM Optim workflow logic.

Resolving the problem

Two workarounds are available.

• Workaround 1: Set conflict_resolution to append mode instead of merge. This workaround avoids the JDBC driver limitation related to LOB parameter binding. Complete the following steps:
  1. In the IBM Optim flow configuration, set conflict_resolution to append mode instead of merge.
  2. Save the configuration and run the flow.
  This workaround is only suitable when merge behavior is not required. Note the following considerations:

  • Existing rows in the target table are not updated.
  • Duplicate records might be inserted if unique constraints are not enforced.
• Workaround 2: Convert CLOB or TEXT columns to VARCHAR in the source database before running the flow. If the actual data length permits, you can modify the source table schema. For example, for an IBM Db2 source database, complete the following steps:
  1. Determine the maximum data length in the source database.

     SELECT MAX(LENGTH(column_name)) FROM table_name;

  2. If the maximum length is within supported limits (lower or equal to 8000 bytes), convert the column to VARCHAR.

     ALTER TABLE CUSTOMERS
     ALTER COLUMN CREDITCARD_HISTORY SET DATA TYPE VARCHAR(8000);

  3. Run the IBM Optim flow. The flow completes successfully without LOB errors.

MongoDB is limited when acting as a data source

MongoDB has the following limitations when used as a source connection:

• The system does not copy indexes to the target table other than the primary key index.
• The system does not copy constraints to the target table.

Resolving the problem

Manually create the missing indexes and constraints in the target database.

No target collection created automatically when MongoDB is the target database

If a target MongoDB collection does not already exist, the collection is not automatically created.

Resolving the problem

The target collection must be manually created first, with matching table schema as the source.

For example, if the source table is

create table t1 (col1 int);

then the target MongoDB collection needs to be created first with a placeholder document. For example:

db.t1.insert({col1:1})

Inserting a document creates a collection named T1 with a field COL1 that matches the source table's column col1.

MongoDB is limited when acting as a data target

MongoDB has the following limitations when used as a target connection:

• The system does not copy the primary key to MongoDB.
• The system does not copy indexes to MongoDB.
• The system does not copy constraints to MongoDB.
• Rows are always inserted (process option = append).

Resolving the problem

Manually create the missing artifacts in the target database.

Apache Cassandra and DataStax are limited when acting as a source or target

Apache Cassandra and DataStax databases both have the following limitations when used as a source or target connection:

No write conflict resolution (append-only writes)

When Apache Cassandra or DataStax is used as a target, the pipeline supports only append-only writes (TABLE_ACTION_APPEND). The existingTableAction option (for example, merge, replace, truncate, update) and primary-key-based upsert through key_column_names are not supported and are silently ignored. Duplicate row handling must be managed at the CQL or application level.

No SQL views

Apache Cassandra and DataStax do not support SQL views (information_schema.views). Any attempt to read views from a Cassandra or DataStax source keyspace returns an empty result. Source views cannot be used as pipeline input for these connectors.

No foreign keys

Neither Apache Cassandra nor DataStax supports FOREIGN KEY constraints in CQL. During schema migration to a Cassandra or DataStax target, all foreign key definitions from the source schema are dropped. Cross-table referential integrity must be enforced entirely at the application level.

No unique constraints

Standard UNIQUE constraints are not supported in Cassandra or DataStax DDL. Any unique constraints present in the source schema are silently discarded during schema migration. Uniqueness must be ensured through appropriate primary key or partition key design in the data model.

No standard index DDL

Cassandra and DataStax use CQL-specific index syntax (CREATE INDEX ON keyspace.table (column)) which is incompatible with standard SQL CREATE INDEX DDL. No secondary indexes are automatically created during schema migration. Indexes must be created manually using CQL after the pipeline completes.

No ALTER TABLE or schema diffs

Apache Cassandra and DataStax do not support ALTER TABLE ADD CONSTRAINT or FOREIGN KEY through CQL. Incremental schema reconciliation (writeSchemaDiffs) is a no-op for these connectors, meaning schema drift between source and target is not automatically corrected on subsequent pipeline runs.

No database field (keyspace is used instead)

Cassandra and DataStax do not follow the traditional relational database → schema → table hierarchy. The database field in the connection profile is not required and not validated for these connector types. The keyspace field serves as the equivalent of a schema, and the database field must be left blank in the connection profile.

Performance and scalability considerations

The following known issues apply to performance and scalability scenarios.

• Performance variability with large datasets

Performance variability with large datasets

Processing speed varies based on dataset size and schema complexity.

Resolving the problem

Optimize access definitions and use custom objects to limit data scope.

Compliance and security considerations

The following known issues apply to compliance and security scenarios.

• Single Sign-On (SSO) and Multi-Factor Authentication (MFA) configuration required
• Audit logging scope

Single Sign-On (SSO) and Multi-Factor Authentication (MFA) configuration required

You must manually set up SSO and MFA to enable enterprise identity integrations.

Resolving the problem

Follow IBM integration guides for configuring SSO and MFA by using Keycloak for authentication and authorization. Validate configurations in a staging environment before production rollout.

Audit logging scope

Detailed data-level access events might not be captured.

Resolving the problem

Supplement IBM Optim logs with external SIEM tools to capture granular access events. Enable detailed logging in IBM Optim configuration settings where available.

Oracle rows not found are counted as deleted in the Delete Process Report

When Oracle is used as the source database in a Delete Process, rows that are not found are recorded as deleted rows in the Delete Process Report. This behavior differs from other supported database sources, where rows not found are reported by using a separate "Not Found" metric.

This behavior is a known limitation of the Oracle source implementation. No workaround is available. When you review Delete Process Reports for Oracle sources, the deleted row count can include rows that were not present in the source table and were not deleted.

Oracle limitations

Oracle database has the following limitations when used as a source or target connection:

• Oracle RAW data types might not be rendered in a user-friendly format.
• Oracle NUMBER columns might be displayed as strings in Data Explorer.
• Archived Oracle NUMBER columns might be restored as NVARCHAR2 instead of NUMBER.
• Leading zeros in decimal NUMBER values (for example, 0.1234) might not be preserved in display, archive, or restore operations.
• Numeric precision and scale might not be preserved consistently for Oracle NUMBER data types during archive and restore operations.

IBM Db2 can implicitly create a missing target schema when a flow is submitted through the API

IBM Optim Data Privacy does not create schemas in target databases; if the target schema does not exist, the job is expected to fail. However, IBM Db2 has an implicit schema creation feature: when a CREATE TABLE statement references a non-existent schema, Db2 creates it automatically (owned by SYSIBM, with CREATEIN granted to PUBLIC). As a result, a flow that is submitted through the REST API with a non-existent target schema succeeds silently instead of failing.

This behavior only affects users who submit flows by using the REST API with a manually drafted JSON payload. It does not affect flows run from the UI, which requires selecting a schema from existing schemas in the target database.

Resolving the problem

Before submitting a flow by using the API, verify that the target schema in the JSON payload exists in the Db2 database. To check whether a schema was implicitly created (owned by SYSIBM), run:

SELECT SCHEMANAME, OWNER FROM SYSCAT.SCHEMATA WHERE OWNER = 'SYSIBM';

For more information, see the IBM Db2 documentation on implicit schema authority.

IBM Db2 for i does not support lowercase or mixed-case source table names

When you use IBM Db2 for i as a source connection, tables with lowercase or mixed-case names, such as employee_test or FinancialInfo2_test, are not supported. Jobs that include such a table fail.

Resolving the problem

Rename the source table to use an uppercase name before you run a job. For example:

RENAME TABLE MYSCHEMA."FinancialInfo2_test" TO FINANCIALINFO2_TEST;

Indexes and foreign key constraints on the table are retained after the rename.