Oracle XStream
7.3 and later
The Oracle XStream origin uses Oracle XStream to process change data capture (CDC) information that is stored in Oracle redo logs. For information about supported versions, see Supported Systems and Versions in the Data Collector documentation.
The Oracle XStream origin connects to an Oracle XStream outbound server and generates a record for each DML event that is fetched from the server. Use this origin when you want to use Oracle XStream to process high volumes of Oracle change data. To use LogMiner to process change data, try the Oracle CDC origin.
The Oracle XStream origin can use existing XStream components or can provision the necessary XStream components to access the data.
The origin includes the CRUD operation type in a record header attribute so generated records can be easily processed by CRUD-enabled destinations. For an overview of Data Collector changed data processing and a list of CRUD-enabled destinations, see Processing changed data.
When you configure the Oracle XStream origin, you configure the operation mode to use existing components or to provision components, then you define related properties. You specify connection information and optionally configure properties to tune performance. You can also use a connection to configure the origin.
The origin can generate events for an event stream. For more information about dataflow triggers and the event framework, see Dataflow triggers overview.
Before you use the Oracle XStream origin, you must complete the required Oracle and Data Collector prerequisite tasks.
Oracle prerequisites
Complete the following prerequisite tasks in Oracle before you use the Oracle XStream origin:
- Set database parameters
- Create the CDC user
- Create the outbound server (existing XStream components only)
- Start the outbound server (existing XStream components only)
- Prepare the database and tables
- Run verification commands
- CDB or multitenant database - A multitenant container database (CDB) that includes one or more pluggable databases (PDB).
- PDB or pluggable database - A database that is part of a CDB.
Requirements
The following table lists the requirements for using the Oracle XStream origin to process Oracle change capture data in a pluggable database. For other database types, see the Oracle documentation for details on performing similar tasks.
| Requirement | Sample verification command | Expected result |
|---|---|---|
| Supported version of Oracle |
|
Supported version of Oracle. For more information, see Supported Systems and Versions in the Data Collector documentation. |
|
Container database (CDB) |
|
Yes |
| Target pdb exists |
|
Name of the pdb |
| Target pdb is OPEN READ WRITE |
|
READ WRITE |
| Database is in ARCHIVELOG mode | SELECT log_mode FROM v$database; |
ARCHIVELOG |
- Connect to the CDB root container
CDB$ROOTwith SYSDBA privileges:ALTER SESSION SET CONTAINER = CDB$ROOT; - Run the following commands to enable ARCHIVELOG mode:
SHUTDOWN IMMEDIATE; STARTUP MOUNT; ALTER DATABASE ARCHIVELOG; ALTER DATABASE OPEN;
Command tools and syntax
| SQL*Plus syntax | JDBC SQL syntax |
|---|---|
EXEC procedure(); |
BEGIN procedure(); END; |
| Use a slash (/) after PL/SQL blocks | Omit the slash, simply run the PL/SQL blocks |
| Run the entire block of code at once | Execute each statement separately |
Task 1. Set database parameters
Task 2. Create the CDC user
Task 3. Create the outbound server (existing components only)
When you use existing XStream components instead of having the origin provision XStream components, you must create an outbound server. If you configure the origin to provision XStream components, you can skip this task.
When you create the outbound server, you can perform a global capture to include all tables or you can perform a selective capture to include selected tables.
Include all tables
Use the following steps to create an outbound server that uses global capture to include all existing tables.
Include specified tables
Use the following steps to create an outbound server that uses selective capture to include only the listed tables.
Task 4. Start the outbound server (existing components only)
When you use existing XStream components instead of having the origin provision XStream components, the outbound server must be running before you run the pipeline. If you configure the origin to provision XStream components, you can skip this task.
Use the following steps to start the outbound server.
Task 5. Prepare the database and tables
Use the following steps to prepare tables in the database for processing.
Task 6. Run verification commands
Use the following commands to verify that the prerequisite tasks were completed.
Data Collector prerequisite
Complete the following Data Collector prerequisite task to allow the Oracle XStream origin to use Oracle Instant Client libraries to access Oracle.
The Oracle XStream origin uses Oracle Instant Client libraries to access Oracle. Docker deployments include the Instant Client libraries and require no additional prerequisite tasks.
- Download and install Oracle Instant Client to the Data Collector engine machine.
For more information, see the Oracle Instant Client Downloads page.
- From the Oracle Instant Client installation location, copy the
xstream.jarandojdbc<version>.jarfiles, and then install them as external resources for the Oracle XStream origin.Best practice is to use an archive file for the external resources. For information about installing external resources, see the Control Hub documentation.
- Add the following lines to the
$SDC_INSTALLATION/libexec/sdc-env.shfile:export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:"<instantclient_installdir>"export ORACLE_INSTANTCLIENT_HOME="<instantclient_installdir>"
- Install additional dependencies required by Oracle XStream. Use the command appropriate for your
Linux distribution:
For Red Hat Enterprise:
sudo yum install libaioFor Ubuntu or Debian:
sudo apt-get install libaio1 libnsl2
Operation mode
- Provision new XStream components
- The origin provisions all XStream components that are required to process changed data. Use this option for early development or testing.
- Use existing XStream components
- The origin uses existing XStream components to process changed data. Use this option for testing and production to can ensure that the origin processes the exact data set that you want.
Start mode
When the Oracle XStream origin uses existing XStream components to process data, you specify where the origin starts processing by configuring the Start Mode property and related settings. You can configure the origin to start at the current change or instant in time so that it processes all available changes. Or, you can specify a specific change or time to start processing.
The origin uses the Start Mode and any specified initial change or time only after you start the pipeline for the first time or after you reset the origin. At other times, the origin uses the last-saved offset to ensure that it processes all appropriate data.
Note that the Oracle XStream origin processes only change capture data. To process existing data, you might use a JDBC Multitable Consumer origin in a separate pipeline to read table data before you start an Oracle XStream pipeline.
The origin provides the following start modes:
- From the latest change
- The origin processes all changes that occur after you start the pipeline.
- From a specified change
- The origin processes all changes that occurred in the specified change (SCN) and later. When the origin uses the SCN, the origin started processing with the timestamp that is associated with the SCN. If the SCN cannot be found in the redo logs, the origin continues reading from the next higher SCN that is available in the redo logs.
- From a specified datetime
- The origin processes all changes that occurred at the specified datetime and
later. Use the following format:
YYYY-MM-DD HH24:MM:SS.
CRUD operation header attributes
- sdc.operation.type
- The origin evaluates the operation type associated
with each entry that it processes. When appropriate, it writes the
operation type to the
sdc.operation.typerecord header attribute. - oracle.cdc.operation
- The origin also writes the CRUD operation type to an
oracle.cdc.operationattribute. This attribute was implemented in an earlier release and is supported for backward compatibility.
CDC header attributes
schema- Schema where the change occurredoracle.cdc.table- Table where the change occurred.oracle.cdc.xid- Transaction ID for the change.oracle.cdc.scn- System Change Number for the change.oracle.cdc.timestamp- Timestamp of the DDL record.
You can use the record:attribute or
record:attributeOrDefault functions to access the information
in the attributes. For more information about working with record header attributes,
see Working with header attributes.
Field attributes
lob_value_omitted- Set totruewhen the origin encounters an object larger than the configured Max large object size property.nanoSeconds- Includes the nanosecond value from timestamp data.old_value- Includes the original value of the column for columns that were updated.oracle_type- Provides the original Oracle data type for each field.oracle_type_code- Provides the code for the original Oracle data type for each field.
You can use the record:fieldAttribute or
record:fieldAttributeOrDefault functions to access the information
in the attributes. For more information about working with field attributes, see Field attributes.
Event generation
- With the Email executor to send a custom email
after receiving an event.
For an example, see Sending email during pipeline processing.
- With a destination to store event information.
For an example, see Preserving an audit trail of events.
For more information about dataflow triggers and the event framework, see Dataflow triggers overview.
Event records
| Record header attribute | Description |
|---|---|
| sdc.event.type | Event type. Uses one of the following types:
|
| sdc.event.version | Integer that indicates the version of the event record type. |
| sdc.event.creation_timestamp | Epoch timestamp when the stage created the event. |
- DDL-generated events
- The DDL-generated events - alter table, create table, drop table, and truncate table - are created when the origin reads a DDL statement.
- Cleanup success events
- The origin creates a cleanup success event when you stop a pipeline that provisions XStream components, and the provisioned components are successfully shut down and deleted.
- Cleanup failure events
- The origin creates a cleanup failure event when you stop a pipeline that provisions XStream components, and the provisioned components are not successfully shut down and deleted.
Configuring an Oracle XStream origin
Configure an Oracle XStream origin to use XStream to read Oracle CDC data from multiple tables.