SSIJV2 - Documentation Index

Table of Contents

• Welcome
• Documentation home
• Overview of the Data Fabric experience
  • What's new
  • Use cases
    • Data fabric use case
    • Data integration use case
  • Feature differences between Data Fabric experience deployments
    • Feature differences between watsonx.data intelligence deployments
    • Feature differences between watsonx.data integration deployments
    • Feature differences between watsonx.data integration and separate service offerings
  • Platform architecture
    • Asset types and properties
      • Previews
      • Profiles
      • Data quality
      • Relationships
      • Visualizations
      • Activities
    • Searching for assets and artifacts
    • Object storage for workspaces
  • AI agents and augmentations
    • AI augmentations in curation tools
  • Services
    • Regional availability
    • watsonx.data intelligence plans
      • IBM Cloud feature differences between plans
      • AWS feature differences between plans
    • watsonx.data integration plans
    • IBM Cloud services
      • Creating and managing services
      • IBM Cloud Object Storage plans
    • AWS services
    • AWS GovCloud services
  • Known issues and limitations
  • FAQ
    • Browser support
    • Language support
    • Preview releases
    • Notices
    • Accessibility
  • Get help
• Getting started and tutorials
  • Signing up for the Data Fabric experience
  • Getting started with watsonx.data intelligence
    • Getting started with data governance and catalog
    • Getting started with data quality
    • Getting started with data lineage
    • Getting started with Data Product Hub
    • Getting started with Data Intelligence Agent
    • Getting started with watsonx.data intelligence MCP server
      • Installing the local MCP server
      • MCP configuration file and environment variables
      • Configuring IBM Bob
      • Configuring Claude Desktop
      • Configuring watsonx Orchestrate
      • Configuring GitHub Copilot in Visual Studio Code
      • Setting up and using skills
    • Inviting users
  • Getting started with watsonx.data integration
  • Switching between experiences
  • Creating task credentials
  • Generating an API key and bearer token
  • watsonx APIs and SDKs
  • Tutorials
    • Curate high quality data
    • Protect your data
    • Consume your data
    • Replicate data
    • Stream data
    • Transform batch data
    • Observe data
• Projects
  • Shared projects across experiences
  • Creating a project
    • Importing a project
    • Importing project assets
  • Administering projects
    • Managing collaborators
    • Project collaborator roles
    • Marking a project as sensitive
    • Managing task credentials
    • Adding associated services
    • Exporting project assets
    • Enabling folders
    • Switching the experience for a project
    • Defining default settings for tools
      • Data intelligence
        • Providing additional context for SQL generation
      • Data quality
      • Job execution windows
      • Metadata enrichment
        • CSV file for rule-based term assignment
        • Tuning term assignment
        • Custom abbreviation files for name generation
      • Unstructured Data Integration
  • Managing assets in projects
    • Organizing assets with folders
  • Downloading data assets
  • Choosing compute resources for tools
    • Compute options for Data Refinery
    • Compute options for DataStage
    • Compute options for Pipelines
  • Managing compute resources
    • Creating non-standard environment templates
    • Customizing environment templates
      • Examples of customizations (before 25.1)
      • Examples of customizations (from 25.1)
    • Runtime usage
  • Creating and managing jobs
    • Creating jobs in Data Refinery
    • Creating jobs in DataStage
    • Creating jobs for running data quality rules
    • Creating jobs for Pipelines
    • Viewing jobs across projects
  • Adding catalog assets to a project
  • Publishing assets to a catalog
  • Leaving a project
  • Markdown cheatsheet
• Preparing data
  • Adding data to a project
    • Adding very large files to a project
    • Adding connections to projects
    • Adding integrated service connectors
    • Connecting to data behind a firewall
    • Adding data from a connection
    • Adding a dynamic view of data from a connection
      • Adding a query-based asset
        • Providing additional context for text-to-SQL conversions
      • Adding segmented assets
    • Adding a connected folder asset from a connection
  • Connectors
    • AlloyDB for PostgreSQL
    • Amazon Aurora for MySQL
    • Amazon Aurora for PostgreSQL
    • Amazon DynamoDB
    • Amazon RDS for MySQL connection
    • Amazon RDS for Oracle connection
    • Amazon RDS for PostgreSQL connection
    • Amazon Redshift connection
    • Amazon S3 connection
      • Setting up temporary credentials or a Role ARN for Amazon S3
    • Apache Cassandra connection
    • Apache Cassandra for DataStage connection
    • Apache Derby connection
    • Apache HDFS connection
    • Apache Hive connection
    • Apache Impala connection
    • Apache Kafka connection
    • Apache Spark SQL connection
    • AWS Databricks
    • Box connection
    • ClickHouse connection
    • Collibra connection
    • Confluence connection
    • Databricks Delta Lake connection
    • DataStax Astra connection
    • DataStax Enterprise connection
    • DataStax HCD connection
    • Denodo connection
    • Dremio connection
    • Dropbox connection
    • Elasticsearch connection
    • Elastic Cloud connection
    • FTP connection
    • Generic S3 connection
    • Google BigQuery connection
      • Workload identity federation examples
    • Google Cloud Pub/Sub connection
    • Google Cloud Storage connection
    • Google Drive connection
    • Google Looker connection
    • Greenplum connection
    • HTTP connection
    • IBM Cloud Data Engine connection
    • IBM Cloud Databases for MongoDB connection
    • IBM Cloud Databases for MySQL
    • IBM Cloud Databases for PostgreSQL connection
    • IBM Cloud Object Storage connection
      • Controlling access to Cloud Object Storage buckets
    • IBM Cloud Object Storage (infrastructure) connection
    • IBM Cloudant connection
    • IBM Cognos Analytics connection
    • IBM Data Virtualization connection
    • IBM Data Virtualization Manager for z/OS connection
    • IBM DataStage for Cloud Pak for Data connection
    • IBM Db2 connection
    • IBM Db2 for DataStage connection
    • IBM Db2 Big SQL connection
    • IBM Db2 for i connection
    • IBM Db2 for z/OS connection
    • IBM Db2 on Cloud connection
    • IBM Db2 Warehouse connection
    • IBM FileNet P8 connection
    • IBM Informix connection
    • IBM Master Data Management connection
    • IBM MQ connection
    • IBM Netezza Performance Server connection
    • IBM Netezza Performance Server for DataStage connection
    • IBM Planning Analytics connection
    • IBM watsonx.data Milvus connection
    • IBM watsonx.data Presto connection
    • IBM watsonx.data SharePoint connection
    • Iceberg metastore connection
    • Informatica PowerCenter connection
    • JDBC connection
    • JMS connection
    • MariaDB connection
    • Microsoft Azure Blob Storage connection
    • Microsoft Azure Cosmos DB connection
    • Microsoft Azure Data Lake Storage connection
    • Microsoft Azure Databricks connection
    • Microsoft Azure Fabric Warehouse connection
    • Microsoft Azure File Storage connection
    • Microsoft Azure PostgreSQL connection
    • Microsoft Azure SQL Database connection
    • Microsoft Azure Synapse Analytics connection
    • Microsoft OneDrive connection
    • Microsoft Power BI (Azure) connection
    • Microsoft Power BI (Local) connection
    • IBM watsonx.data SharePoint connection
    • Microsoft SharePoint Files connection
    • Microsoft SharePoint connection
    • Microsoft SQL Server connection
    • Microstrategy connection
    • Milvus connection
    • MongoDB connection
    • MySQL connection
    • OData connection
    • ODBC connection
    • OpenLineage connection
    • OpenSearch connection
    • OpenSearch IBM Cloud connection
    • Oracle connection
    • Oracle Database for DataStage connection
    • PostgreSQL connection
    • Presto connection
    • Salesforce.com connection
    • Salesforce API for DataStage connection
    • SAP ASE connection
    • SAP IQ connection
    • SAP OData connection
    • SingleStoreDB connection
    • Slack connection
    • Snowflake connection
    • Splunk connection
    • Statistical Analysis System (SAS) connection
    • Tableau connection
    • Talend Data connection
    • Teradata connection
    • Teradata database for DataStage connection
    • Trino connection
    • Vertica connection
    • Web client connection
  • Adding platform connections
  • Building custom connectors with Connector forge
    • Creating and deploying custom connectors
    • Troubleshooting custom connectors
  • Parametrized connections
  • Data protection with data source definitions
    • Protection solutions for data source definition
    • Connectors that support data source definitions
    • Connectors with hard-coded data source identity properties
    • Roles and asset privacy settings for data source definitions
    • Creating a data source definition
      • Creating a data source definition from the Data source definition list
      • Adding endpoints to a new or existing data source definition
    • Creating a connection from the Data source definitions list
    • Setting connection limits for data source definitions
    • Editing, deactivating, activating, or deleting data source definitions
  • Managing feature groups (beta)
  • Orchestrating tasks with Orchestration Pipelines
    • Getting started with Pipelines
      • Planning a pipeline
    • Creating a pipeline
      • Configuring pipeline nodes
      • Managing default settings
      • Configuring global objects
      • Adding conditions to a pipeline
        • Functions used in pipelines Expression Builder
        • DataStage functions used in pipelines Expression Builder
      • Handling pipeline errors
      • Programming a pipeline
      • Creating custom components
    • Running and saving pipelines
    • Configuration management for Orchestration Pipelines
• Curating and integrating unstructured data
  • Supported connectors for curation of unstructured data
  • Setting up curation flows for unstructured data
    • Designing unstructured data curation flows
    • Document classes
      • Schema requirements
      • Managing document classes
  • Integrating unstructured data documents
    • Creating data integration flows
      • Working with parameters
      • Running a flow with Spark
      • Debugging a flow
    • Data preparation nodes
      • Ingest data
      • Extract data
      • Quality
      • Transform data
      • Generate output
      • Custom nodes
• Curating structured data
  • Supported connectors for discovery, enrichment, and data quality
  • Supported connectors for lineage import
    • Amazon RDS for Oracle lineage configuration
    • Amazon RDS for PostgreSQL lineage configuration
    • Amazon Redshift lineage configuration
    • Apache Hive lineage configuration
    • Google BigQuery lineage configuration
    • Greenplum lineage configuration
    • IBM Cloud Databases for PostgreSQL lineage configuration
    • IBM Cognos Analytics lineage configuration
    • IBM DataStage for Cloud Pak for Data lineage configuration
    • IBM Db2 lineage configuration
    • IBM Db2 for z/OS lineage configuration
    • IBM Db2 on Cloud lineage configuration
    • Informatica PowerCenter lineage configuration
    • Microsoft Azure Databricks lineage configuration
    • Microsoft Azure SQL Database lineage configuration
    • Microsoft Power BI (Azure) lineage configuration
    • Microsoft Power BI Report Server lineage configuration
    • Microsoft SQL Server lineage configuration
    • Microsoft SQL Server Integration Services (SSIS) lineage configuration
    • MicroStrategy lineage configuration
    • OpenLineage lineage configuration
    • Oracle lineage configuration
    • PostgreSQL lineage configuration
    • Qlik Sense lineage configuration
    • SAP BusinessObjects lineage configuration
    • Snowflake lineage configuration
    • Statistical Analysis System (SAS) lineage configuration
    • Tableau lineage configuration
    • Talend lineage configuration
    • Teradata lineage configuration
    • YugabyteDB lineage configuration
  • Importing metadata
    • Designing metadata imports
      • Configuring metadata import for data integration assets
    • Creating metadata imports
    • Managing existing metadata imports
    • Metadata import jobs
  • Enriching your data assets
    • Designing metadata enrichments
      • Term assignment
        • Best practices
      • Random sampling concepts
    • Creating an enrichment asset
    • Managing existing enrichments
    • Managing enrichment jobs
      • Monitoring job runs
    • Reviewing enrichment results
      • Profile details
      • Frequency distributions
    • Reviewing results in a spreadsheet program
    • Making bulk changes to assignments
    • Managing data quality checks
    • Identifying primary keys
    • Identifying relationships
    • Advanced data profiling
    • Publishing enrichment results
  • Managing data quality
    • Data quality dimensions
    • Types of predefined data quality checks
    • Data quality assets
    • Managing data quality definitions
      • Building blocks for rule logic
      • Sample data quality definitions
      • Sample rule expressions
      • Reserved words
    • Managing data quality rules
      • Creating rules from data quality definitions
      • Creating SQL-based rules
        • Sample SQL rules
      • Configuring output settings for rules
    • Assessing data quality with rules
    • Data quality analysis results
      • Data quality scores
    • Data quality SLA compliance and remediation
    • Ensuring data quality with data contracts
    • Automating data quality remediation
  • Refining data
    • Adding data to Data Refinery
    • Validating your data
    • Visualizing your data
    • Managing Data Refinery flows
    • Target connection options
    • GUI operations
    • Interactive code templates
    • Supported data sources for Data Refinery
  • Visualizing your data
    • Chart types
      • 3D charts
      • Bar charts
      • Box plots
      • Bubble charts
      • Candlestick charts
      • Circle packing charts
      • Custom charts
      • Dendrogram charts
      • Dual Y-axes charts
      • Error bar charts
      • Evaluation charts
      • Heat map charts
      • Histogram charts
      • Line charts
      • Map charts
      • Math curve charts
      • Multi-chart charts
      • Multiple series charts
      • Parallel charts
      • Pareto charts
      • Pie charts
      • Population pyramid charts
      • Q-Q plots
      • Radar charts
      • Relationship charts
      • Scatter plots and dot plots
      • Scatter matrix charts
      • Series array charts
      • Sunburst charts
      • t-SNE charts
      • Time plots
      • Theme River charts
      • Tree charts
      • Treemap charts
      • Word cloud charts
    • Global visualization preferences
• Data integration
  • Transforming data with DataStage
    • Finding resource usage
    • Designing flows
      • Asset browser
      • DataStage stages
        • Aggregator
          • Fast path
          • Stage tab
            • Calculation and recalculation dependent properties
        • Bloom Filter
          • Stage tab
        • Change Apply
          • Example data (DataStage)
          • Fast path
        • Change Capture
          • Fast path
          • Stage tab
        • Checksum
          • Adding a Checksum column to your data (DataStage)
          • Properties for Checksum Stage (DataStage)
          • Mapping output columns (DataStage)
          • Specifying execution options (DataStage)
        • Column Export
          • Fast path
          • Stage tab
          • Input tab
          • Output tab
        • Column Generator
          • Fast path
          • Stage tab
        • Column Import
          • Examples (DataStage)
          • Fast path
          • Stage tab
          • Input tab
          • Output tab
        • Combine Records
          • Examples (DataStage)
            • Example 1 (DataStage)
            • Example 2 (DataStage)
          • Fast path
          • Stage tab
            • Properties section
              • Outputs section
              • Combine keys section
              • Options section
            • Advanced section
            • NLS Locale section
        • Compare
          • Fast path
          • Stage tab
        • Compress
          • Fast path
          • Stage tab
        • Copy
          • Fast path
          • Stage tab
        • Decode
          • Fast path
        • Difference
          • Fast path
          • Stage tab
        • Distributed Transaction
        • Encode
          • Fast path
          • Stage tab
        • Excel
          • Configuring the Excel stage as a source
          • Configuring the Excel stage as a target
          • Extracting the data from Microsoft Excel
            • Examples of extracting data from Microsoft Excel files
            • Examples of writing data to Microsoft Excel files
          • Reference
            • Properties reference: Excel
            • Data type conversions from Microsoft Excel to IBM DataStage
            • Data type conversions from DataStage to Microsoft Excel
            • Job abort conditions in Microsoft Excel
            • Consideration about end of wave
        • Expand
          • Fast path
          • Stage tab
        • External Filter
          • Fast path
          • Stage tab
        • External Source
          • Fast path
          • Stage tab
            • Advanced section
            • NLS Map section
          • Output tab
            • Properties section
              • Source section
              • Options section
            • Format section
            • Using RCP With External Source Stages (DataStage)
        • External Target
          • Fast path
          • Stage tab
            • Advanced section
            • NLS Map section
          • Input tab
            • Input link properties section
              • Target section
              • Options section
            • Format section
              • Output tab
          • Using RCP with External Target stages (DataStage)
        • Filter
          • Specifying the filter
            • Input data columns
            • Supported Boolean expressions and operators
              • Order of association (DataStage)
            • String comparison (DataStage)
          • Fast path
          • Stage tab
        • Funnel
          • Fast path
          • Stage tab
            • Link Ordering section
        • Generic
          • Fast path
          • Stage tab
        • Head
          • Fast path
          • Stage tab
        • Hierarchical data
          • Using the Hierarchical Data stage (DataStage)
            • Adding a Hierarchical Data stage to a (DataStage) flow
            • Configuring runtime properties for the Hierarchical Data stage (DataStage)
          • The assembly (DataStage)
            • Input step (DataStage)
            • Output step (DataStage)
            • Assembly Editor (DataStage)
            • Opening the Assembly Editor (DataStage)
          • Schema views
          • Mapping data (DataStage)
            • Working with the mapping table (DataStage)
            • Determining mapping candidates (DataStage)
            • Configuring how mapping candidates are determined (DataStage)
          • XML Composer step (DataStage)
            • XML Composer validation rules (DataStage)
          • XML Parser step (DataStage)
            • XML Parser validation rules (DataStage)
            • Setting default values for types (DataStage)
          • JSON transformation (DataStage)
            • Schema management (DataStage)
              • Opening the Schema Library Manager (DataStage)
              • Working with libraries and resources (DataStage)
              • Creating a JSON schema in the schema library (DataStage)
            • JSON Parser step (DataStage)
              • JSON Parser validation rules (DataStage)
            • JSON Composer step (DataStage)
              • JSON Composer validation rules (DataStage)
          • REST operator in (DataStage)
            • REST step pages
              • General
              • Security
              • Request
              • Response
              • Mappings
            • Output schema of the REST step
          • Passing multiple rows from an XML or JSON file
          • Transformation steps for the Hierarchical Data stage (DataStage)
            • Aggregate step (DataStage)
            • H-Pivot step (DataStage)
            • HJoin step (DataStage)
            • Order Join step (DataStage)
            • Regroup step (DataStage)
            • Sort step (DataStage)
            • Union step (DataStage)
            • V-Pivot step (DataStage)
        • Java Integration
          • Java code integration
            • Setting up your development environment
            • Implementing abstract methods of the Processor class
            • Compiling the Java code
            • Running the Java code on the Parallel Engine, Java Integration stage
            • Accessing Stage Configuration
            • Declaring the Capabilities of the Java code
            • Reading Records from Input Link
            • Writing Records to Output Link
            • Rejecting Records
            • Looking up data in the Sparse lookup mode
            • Data Types
            • Retrieving Column Metadata on the Link
            • Using user-defined properties
            • Runtime column propagation
            • Running the Java code on the conductor node
            • Transferring data from the conductor node to player nodes
            • Logging messages with the Java Integration stage
            • Terminating a job from the Java code
            • Using JavaBeans
            • User Defined Function
          • Designing jobs with the Java Integration stage
            • Configuring Java Integration stage as a source
            • Configuring Java Integration stage as a transformer
            • Configuring Java Integration stage as a target
            • Looking up data by using reference links
            • Setting up column definitions
          • Java libraries
          • Properties reference: Java Integration
        • Join
          • Join versus lookup
          • Fast path
          • Stage tab
        • Lookup
          • Lookup versus Join
          • Fast path
          • Properties
            • Stage tab
            • Input and Output tab
          • Sparse and normal Lookup
        • Make Subrecord
          • Fast path
          • Stage tab
          • Examples (DataStage)
        • Make Vector
          • Examples (DataStage)
            • Example 1 (DataStage)
            • Example 2 (DataStage)
          • Fast path
          • Stage tab
            • Properties section
              • Options section
            • Advanced section (DataStage)
        • Merge
          • Fast path
          • Stage tab
        • Modify
          • Fast path
          • Stage tab
        • Peek
          • Fast path
          • Stage tab
        • Pivot Enterprise
          • Specifying a horizontal pivot operation (DataStage)
            • Specifying a horizontal pivot operation and mapping output columns (DataStage)
            • Example of horizontally pivoting data (DataStage)
          • Specifying a vertical pivot operation (DataStage)
            • Specifying a vertical pivot operation and mapping output columns (DataStage)
            • Example of vertically pivoting data (DataStage)
          • Properties tab
            • Specifying execution options (DataStage)
            • Specifying where the stage runs (DataStage)
            • Specifying partitioning or collecting methods (DataStage)
            • Specifying a sort operation (DataStage)
        • Promote Subrecord
          • Examples (DataStage)
            • Example 1 (DataStage)
            • Example 2 (DataStage)
          • Fast path
          • Stage tab
            • Properties tab
              • Options section
            • Advanced tab
        • Remove Duplicates
          • Fast path
          • Stage tab
        • REST
          • Stage tab
        • Row Generator
          • Fast path
          • Stage tab
          • Output tab
        • Sample
          • Fast path
          • Stage tab
            • Link Ordering
        • Slowly Changing Dimension
          • Job design
          • Purpose codes
          • Surrogate keys
          • Editing
            • Defining the match condition
            • Selecting purpose codes
              • Purpose code definitions
            • Specifying information about a key source
            • Creating derivations for dimension columns
          • Dimension update action
        • Sort
          • Fast path
          • Stage tab
        • Split Subrecord
          • Fast path
          • Stage tab
          • Examples (DataStage)
        • Split Vector
          • Examples (DataStage)
            • Example 1 (DataStage)
            • Example 2 (DataStage)
          • Fast path
          • Stage tab
            • Properties tab
              • Options section
            • Advanced tab
        • Stored Procedure
        • Surrogate Key Generator
          • Creating the key source (DataStage)
          • Deleting the key source (DataStage)
          • Updating the state file (DataStage)
          • Generating surrogate keys (DataStage)
        • Switch
          • Example
          • Fast path
          • Stage tab
        • Tail
          • Fast path
          • Stage tab
        • Transformer
          • Basic concepts
          • Properties
            • Stage variables (DataStage)
            • Loop variables (DataStage)
              • Entering expressions (DataStage)
              • Loop example: converting a single row to multiple rows (DataStage)
              • Loop example: multiple repeating values in a single field (DataStage)
              • Loop example: generating new rows (DataStage)
              • Loop example: aggregating data (DataStage)
            • Surrogate Key tab (DataStage)
            • Link ordering (DataStage)
            • Triggers
            • Advanced (DataStage)
            • Input tab
            • Output tab
              • Runtime column propagation (DataStage)
            • System variables (DataStage)
            • Evaluation sequences for transformer expressions, stage variables, and loop variables (DataStage)
            • Reserved words
          • Parallel transform functions (DataStage)
            • Date and time functions (DataStage)
            • Logical functions (DataStage)
            • Mathematical functions (DataStage)
            • Null handling functions (DataStage)
            • Number functions (DataStage)
            • Raw functions (DataStage)
            • String functions (DataStage)
              • Retrieving substrings
              • Concatenating strings
            • Vector function (DataStage)
            • Type conversion functions (DataStage)
            • Utility functions (DataStage)
            • Operator functions (DataStage)
        • Wave Generator
          • Stage Tab
            • Properties
          • Input tab
            • Output tab
        • Web Service stage
          • Introduction to web services in DataStage
          • Encoding requests and responses
          • Required tasks in the Web Service stage
          • Other tasks
          • Setting up stage properties
          • Setting up input link properties
          • Setting up output link properties
          • Using a Web Service in DataStage
        • Write Range Map
          • Fast path
          • Stage tab
          • Input tab
        • XML Input
          • Stage tab
            • Transformation settings (DataStage)
        • XML Output
          • Using XML Output
            • About transforming tabular data
            • Validating documents and schemas
            • Aggregating input rows on output
            • Writing output to your file system
            • Processing NULLs and empty values
            • Selecting items for the XML Output
            • Setting the format of the XML output
            • Elements: Controlling the order and the repetition
            • Setting up stage properties
            • Setting up input and output link properties
      • DataStage connectors
        • Connecting to a data source in DataStage
        • Supported data sources in DataStage
          • Amazon RDS for PostgreSQL connector
          • Amazon S3 connector
          • Apache HDFS connector
          • Apache Hive connector
          • Apache Kafka connector
          • Apache Impala connector
          • FTP connector
          • Google BigQuery connector
          • Generic S3 connector
          • Greenplum connector
          • IBM Cloud Databases for PostgreSQL connector
          • IBM Data Virtualization Manager for z/OS connector
          • IBM Db2 for DataStage connector
          • IBM Db2 for z/OS connector
          • IBM Informix connector
          • IBM Master Data Management connector
          • Oracle Database for DataStage connector
          • PostgreSQL connector
          • Salesforce API for DataStage connector
          • Snowflake connector
          • Teradata connector
          • Teradata database for DataStage connector
        • File connectors in DataStage
          • Complex Flat File
            • Complex Flat File as a source
              • Defining record ID constraints in DataStage
            • Complex Flat File as a target in DataStage
            • Complex Flat File schema
            • Reject links in DataStage
          • Data set in DataStage
          • File set in DataStage
            • Input tab (DataStage)
            • Output tab (DataStage)
          • Lookup file set in DataStage
          • Sequential file in DataStage
        • Using Before/After SQL Statements
          • Before SQL (DataStage)
          • Before SQL (node) in DataStage
          • After SQL (DataStage)
          • After SQL (node) in DataStage
        • Using stored procedures
          • Syntax
        • Using multiple links
        • Common properties for DataStage connectors
      • Quality stages in DataStage
        • Designing match specifications
          • Adding passes
            • Match comparisons in DataStage
              • Reverse matching
              • ABS_DIFF comparison
              • AN_DINT comparison
              • AN_INTERVAL comparison
              • CHAR comparison
              • CNT_DIFF comparison
              • D_INT comparison
              • D_USPS comparison
              • DATE8 comparison
              • DELTA_PERCENT comparison
              • DISTANCE comparison
              • INT_TO_INT comparison
              • INTERVAL_NOPAR comparison
              • INTERVAL_PARITY comparison
              • LR_CHAR comparison
              • LR_UNCERT comparison
              • MULT_ALIGN comparison
              • MULT_EXACT comparison
              • MULT_RANGE comparison
              • MULT_UNCERT comparison
              • NAME_UNCERT comparison
              • NUMERIC comparison
              • PREFIX comparison
              • PRORATED comparison
              • TIME comparison
              • UNCERT comparison
                • Adding QSM_MAT_UNCERT_VERSION environment variable
              • USPS comparison
              • USPS_DINT comparison
              • USPS_INT comparison
          • Testing passes
          • Matching data in DataStage
            • How matching is done
              • Match types
              • Match column selection in DataStage
              • Match passes in DataStage
              • Blocking in DataStage
            • Weights and record comparisons
        • Investigate
        • Match Frequency
        • One-source Match
        • Two-source Match
        • Standardize
          • Standardize rule sets
            • Objects within rule sets
              • Classifications
              • Lookup tables
              • Output columns
              • Rules
              • Overrides
          • Quality stage pattern action reference
            • Introduction to the Pattern Action language
            • Parsing elements
            • Unconditional patterns
              • Identifying simple pattern classes
            • Conditional patterns
              • Simple conditional values
              • Conditional expressions
              • Using arithmetic expressions
            • Action statements
              • Copying information
              • Referencing dictionary fields from another rule set
              • Moving information
              • Concatenating information
              • Converting information
              • Retyping operands
              • Retyping multiple tokens
              • Patterning
              • Rule set extensions
              • User overrides for domain preprocessor rule sets
              • User overrides for rule sets
              • Setting margins
              • SOUNDEX phonetic coding
              • NYSIIS coding
              • Terminating pattern matching
              • Calling subroutines
              • Writing subroutines
              • Performing actions repetitively
              • Summary of sources and targets
      • Specifying your own stages
        • Defining build stages in DataStage
          • General tab
          • Properties tab
          • Build tab
          • Build stage macros
            • Informational macros (DataStage)
            • Flow-control macros (DataStage)
            • Input and output macros (DataStage)
            • Transfer Macros (DataStage)
            • How your code is executed (DataStage)
            • Inputs and outputs (DataStage)
              • Using multiple inputs (DataStage)
            • Example Build stage in DataStage
          • Header files
            • C++ classes - sorted by header file (DataStage)
            • C++ macros - sorted by header file (DataStage)
        • Defining custom stages
          • Compiling custom stages
        • Defining wrapped stages in DataStage
      • Creating a DataStage component
        • Message handlers
      • Defining data definitions
      • Reusable job design with subflows
        • Local subflows
        • Subflows
      • Parameters and parameter sets
        • Creating and using local parameters in DataStage
        • Creating and using parameter sets
        • Inserting parameters and parameter sets as properties in DataStage
        • Configuring runtime parameters in a flow
        • Configuring runtime parameters in a job
        • PROJDEF parameter set
        • Passing values from parameter sets into jobs by command-line interface
      • Environment variables
        • Managing environment variables
        • Guide to setting environment variables in DataStage
        • Buffering
        • Checkpoint
        • Compiler
        • Db2 Support
        • Debugging
        • Decimal support
        • Disk I/O
        • General Job Administration
        • Look up support
        • Miscellaneous
        • Network
        • NLS support
        • Oracle support
        • Partitioning
        • Reading and writing files environment variables in DataStage
        • Reporting environment variables in DataStage
        • SAS support
        • Sorting environment variables in DataStage
        • Sybase support environment variables in DataStage
        • Teradata support
        • Transport blocks
        • WLM
      • Partitioning and collecting data
        • Writing partitioned data in file-storage connectors
    • Running jobs
      • Settings for the project, flow, and job level
      • Setting up before-job and after-job subroutines
        • Cataloging a Db2 database in the runtime container
      • Scheduling DataStage jobs
      • Compile options with SQL Pushdown in DataStage
      • ELT run mode
        • ELT materialization policies in DataStage
      • Macros
      • Observing DataStage jobs
    • Migrating jobs
      • Migrating connections
        • Migrating a Teradata connection
      • Changing XML file locations after migration
      • Configuring the Hierarchical stage after migration in DataStage
      • Migrating BASIC routines
      • Migrating Web Service Transformer and Web Service Client stages
      • Asset import report (DataStage)
    • Downloading and importing flows
    • Development, testing, and production
      • Deployment spaces
    • DataStage Anywhere
      • Creating a remote engine
      • Managing a remote engine
      • Frequently asked questions for DataStage Anywhere
      • Security for remote engines with DataStage Anywhere
      • Sharing the existing remote engine across projects in watsonx.data integration
      • ODBC support for the remote engine
    • DataStage command-line tools
    • DataStage APIs
    • Orchestrating flows with Orchestration Pipelines
      • Pipeline components for DataStage
      • CEL expressions and limitations
      • Migrating and constructing pipeline flows
        • Referencing files in Bash node
      • Replacing BASIC routines
        • Examples
      • DataStage optimized runner
    • Storing and persisting metrics
    • Sharing DataStage artifacts with all IBM Cloud Object Storage containers
    • High availability and disaster recovery in DataStage
    • DevOps for DataStage
      • Managing assets
      • Getting started with Unit Testing
      • Configuring test data storage
      • Creating a DataStage test case
        • Capturing test data
        • Recapturing test result baseline
      • Editing a DataStage test case
        • DataStage specification format
          • Testing flow by using date/time references
          • Row count comparisons
          • Excluding columns from tests
          • Using Cluster Keys for high volume DataStage tests
      • Running a DataStage test case
      • Verifying a DataStage test results
      • Migrating test cases from older DataStage versions
    • Using the DataStage Assistant
  • Streaming real-time data
    • Administering StreamSets engines and environments
      • Understanding engine types
      • Engine version details
        • Data Collector engine versions
          • Data Collector 7.5.x engine details
            • 7.5.x new features and enhancements
            • 7.5.x upgrade impact
            • 7.5.0 fixed issues
            • 7.5.x known issue
          • Data Collector 7.4.x engine details
            • 7.4.x new features and enhancements
            • 7.4.x upgrade impact
            • 7.4.0 fixed issues
            • 7.4.x known issue
          • Data Collector 7.3.x engine details
            • 7.3.x new features and enhancements
            • 7.3.x upgrade impact
            • 7.3.0 fixed issues
            • 7.3.x known issue
          • Data Collector 7.2.x engine details
            • 7.2.x new features and enhancements
            • 7.2.x upgrade impact
            • 7.2.x known issues
        • Jetstream engine versions
          • Jetstream 1.0.0 release notes
            • 1.0.x new features and enhancements
            • 1.0.x known issues
      • Creating a StreamSets environment
        • Environment configurations
        • Creating an environment
      • Running an engine
        • Prerequisites
          • Account prerequisites
          • Engine workstation prerequisites
          • Jetstream prerequisites
        • Running an engine with Docker or Podman
          • Running an engine with tunneling communication (default)
          • Running an engine with direct communication
        • Running an engine on Kubernetes with Helm (advanced)
          • Helm configurations
          • Running an engine on Kubernetes
          • Managing engine containers
          • Helm chart default values
        • Running multiple engines
          • Job failover guidelines
        • Customizing the engine command
      • Configuring environments
        • Stage libraries
          • Common stage libraries
        • Supported systems and versions
          • Cloud native
          • Protocols
          • Versioned systems
        • Data Collector configuration
          • Blocklist and allowlist for stage libraries
          • Configure Data Collector engine properties
        • Enabling HTTPS host verification
          • Prerequisite tasks
          • Step 1. Create a keystore file
          • Step 2. Configure engines to use the keystore file
          • Step 3. Mount the keystore file
        • Engine log configuration
          • Modifying the log level
          • Customizing the log configuration
        • Defining JVM options
        • Setting up external resources
          • External resource types
          • Archive structure
          • Setting up an archive
          • Updating an archive
        • Resource thresholds
          • CPU load
          • Memory
          • Running jobs
          • Defining resource thresholds
        • Install external libraries
        • Custom stage libraries
          • Including custom libraries in an external resource archive
        • Credential stores
          • Enabling credential stores
          • Configuring credential stores
            • Step 1. Install the credential store stage library
            • Step 2. Create the credential store properties file
            • Step 3. Configure credential store properties
              • Protecting sensitive data in credential store properties
              • Mounting local files and directories
              • AWS Secrets Manager properties
              • Azure Key Vault properties
              • CyberArk properties
              • Google Secret Manager properties
              • Hashicorp Vault properties
              • Java Keystore properties
            • Step 4. Mount the credential store properties file
          • Additional requirements and details
            • Azure Key Vault prerequisites
            • Google authentication requirement
            • Adding secrets to Java Keystore (Java only)
            • jks-credentialstore command reference (Java only)
          • Calling secrets from the flow
        • Defining custom CA certificates
      • Engine communication
        • Switching to direct communication
      • Monitoring engines
        • Viewing engine health and status
        • Monitoring online engines
          • Viewing engine logs
          • Generating support bundles
        • Accessing engine log files
      • Managing engines and environments
        • Freeing disk space on an engine workstation
        • Deleting engines from an environment
        • Deleting an environment
      • Upgrading engines
        • Upgrade an engine
        • Post upgrade tasks
          • Review Azure Synapse SQL flows
          • Upgrade from Oracle 11g release 2
          • Update Web Client stages that use signed JWT tokens
          • Create Kafka topics for Kafka Producer flows
          • Install a license certificate to connect to IBM Db2 for z/OS
          • Upgrade watsonx.data
          • Update flows that use unavailable stages
          • Update flows that use unavailable stage libraries
          • Review flows with Amazon stages and stages with an Amazon staging location
          • Review Pulsar flows
          • Update Snowflake targets that use the Use Snowflake Default Values property
          • Review SQL Server CDC Client and SQL Server Change Tracking sources
          • Review Azure Key Vault credential store configuration
          • Review Databricks cluster Java version
          • Review Databricks cluster storage configuration
    • Creating StreamSets flows
      • Flow concepts and design
        • Data in motion
          • Single and multithreaded flows
          • Delivery guarantee
          • Data Collector data types
        • Designing the data flow
          • Branching streams
          • Merging streams
        • Dropping unwanted records
          • Required fields
          • Preconditions
        • Error record handling
          • Flow error record handling
          • Stage error record handling
          • Example
          • Error records and version
        • Record header attributes
          • Working with header attributes
            • Internal attributes
          • Header attribute-generating stages
          • Record header attributes for record-based writes
            • Generating attributes for record-based writes
        • Field attributes
          • Working with field attributes
          • Field attribute-generating stages
        • Processing changed data
          • CRUD operation header attribute
            • Earlier implementations
          • CDC-enabled stages
          • Data changes in SQL Server or Azure SQL Database
          • CRUD-enabled stages
          • Processing the record
          • Use cases
        • Control character removal
        • Development stages
        • Shortcut keys for flow design
        • Understanding flow states
          • State transition examples
        • Technology preview features
        • Deprecated features
      • Flow configuration
        • Retrying the flow
        • Rate limit
        • Simple and bulk edit mode
        • Runtime values
          • Runtime resources
            • Step 1. Define runtime resources
            • Step 2. Call the runtime resource
          • Runtime properties
            • Step 1. Define runtime properties
            • Step 2. Call the runtime property
        • Parameters
          • Creating a parameter set
          • Adding a parameter set to a flow
          • Using parameters in a flow
          • Defining parameter values for a job
        • Event generation
          • Flow event records
        • SSL/TLS encryption
          • Keystore and truststore configuration
            • Local keystore and truststore
            • Remote keystore and truststore
              • Using a credential store
          • Transport protocols
          • Cipher suites
        • Connections
          • Connection types and support
          • Configuring a connection
        • Available Jetstream stages
        • Security in Amazon stages
          • Assume another role
            • Assume role methods
            • Create the trust policy
            • Configure stages to assume a role
        • Security in Google Cloud stages
          • Default credentials
          • Credentials in a file
          • Credentials in a property
        • Security in Kafka stages
          • Prerequisite tasks
          • SASL authentication credentials
            • Providing PLAIN credentials
            • Providing Kerberos credentials
              • Using a credential store
          • Enabling SSL/TLS encryption
          • Enabling SSL/TLS encryption and authentication
          • Enabling SASL authentication
          • Enabling SASL authentication on SSL/TLS
          • Enabling Confluent Cloud authentication
          • Enabling custom authentication
        • Kafka message keys
          • Storing message keys
          • Message key formats
            • String message keys
            • Avro message keys
          • Passing key values to Kafka
          • Example: Storing and passing Kafka message keys
        • SSL/TLS in CONNX stages
        • Authentication in Salesforce stages
          • Connected app with OAuth prerequisites
        • Considerations for IBM Connectivity Service stages
          • IBM Connectivity Service stage prerequisite
          • Troubleshooting IBM Connectivity Service
        • Connections
          • Connection types and support
          • Configuring a connection
        • Expression configuration
          • Basic syntax
          • Using field names in expressions
            • Field names with special characters
          • Referencing field names and field paths
            • Wildcard use for arrays and maps
          • Field path expressions
            • Supported stages
            • Field path expression syntax
          • Data type coercion
        • Flow validation
        • Configuring a flow
          • Configuring flow properties
          • Error record handling properties
      • Sources
        • Sources
          • Comparing Azure storage sources
          • Comparing HTTP sources
          • Comparing UDP Source sources
          • Comparing WebSocket sources
          • Batch size and wait time
          • Maximum record size
        • Amazon S3
          • Authentication method
          • Common prefix, prefix pattern, and wildcards
          • Multithreaded processing
          • Record header attributes
            • Object metadata in record header attributes
          • Read order
          • Buffer limit and error handling
          • Server side encryption
          • Event generation
            • Event records
          • Data formats
          • Configuring an Amazon S3 source
        • Amazon SQS Consumer
          • Authentication method
          • Queue name prefix
          • Multithreaded processing
          • Including SQS message attributes
            • Including sender attributes
          • Data formats
          • Configuring an Amazon SQS Consumer source
        • Aurora PostgreSQL CDC Client
          • Aurora PostgreSQL prerequisites
            • Enable logical replication
            • Assign the required role
          • JDBC driver
          • Schema, table name, and exclusion patterns
          • Initial change
          • Memory usage
          • SSL/TLS encryption
          • Record contents and generated records
            • Record header attributes
            • Sample records
          • Configuring an Aurora PostgreSQL CDC Client source
        • Azure Blob Storage
          • Prerequisites
          • Common path, path pattern, and wildcards
          • Read order
          • Buffer limit and error handling
          • Multithreaded processing
          • Record header attributes
            • Object metadata in record header attributes
          • Event generation
            • Event records
          • Data formats
          • Configuring an Azure Blob Storage source
        • Azure Data Lake Storage Gen2
          • Prerequisites
            • Retrieve authentication information
          • Common path, path pattern, and wildcards
          • Read order
          • Buffer limit and error handling
          • Multithreaded processing
          • Record header attributes
            • Object metadata in record header attributes
          • Event generation
            • Event records
          • Data formats
          • Configuring an Azure Data Lake Storage Gen2 source
        • Azure Data Lake Storage Gen2 (Legacy)
          • Prerequisites
            • Retrieve authentication information
          • File directory
          • File name pattern and mode
          • Read order
          • Multithreaded processing
          • Reading from subdirectories
            • Post-processing subdirectories
          • First file for processing
          • Record header attributes
          • Event generation
            • Event records
          • Buffer limit and error handling
          • Data formats
          • Configuring an Azure Data Lake Storage Gen2 (Legacy) source
        • Azure IoT/Event Hub Consumer
          • Storage account and container prerequisite
          • Resetting the source in Event Hub
          • Multithreaded processing
          • Data formats
          • Configuring an Azure IoT/Event Hub Consumer source
        • CoAP Server
          • Prerequisites
          • Multithreaded processing
          • Network configuration properties
          • Data formats
          • Configuring a CoAP Server source
        • Couchbase
          • Prerequisites
          • Offset
          • Event generation
            • Event record
          • Multithreaded processing
          • Configuring a Couchbase source
        • Directory
          • File name pattern and mode
          • Read order
          • Multithreaded processing
          • Reading from subdirectories
            • Post-processing subdirectories
          • First file for processing
          • Late directory
          • Record header attributes
          • Event generation
            • Event records
          • Buffer limit and error handling
          • Data formats
          • Configuring a Directory source
        • Elasticsearch
          • Security
          • Batch and incremental mode
          • Query
            • Incremental mode query
          • Scroll timeout
          • Multithreaded processing
          • Configuring an Elasticsearch source
        • File Tail
          • File processing and archived file names
          • Multiple paths and file sets
          • First file for processing
          • Late directories
          • Files matching a pattern - pattern constant
          • Record header attributes
            • Defining and using a tag
          • Multiple line processing
          • File Tail output
          • Event generation
            • Event records
          • Data formats
          • Configuring a File Tail source
        • Google BigQuery
          • Credentials
          • BigQuery data types
            • Datetime conversion
          • Event generation
            • Event record
          • Configuring a Google BigQuery source
        • Google Cloud Storage
          • Credentials
          • Common prefix, prefix pattern, and wildcards
          • Record header attributes
          • Event generation
            • Event records
          • Data formats
          • Configuring a Google Cloud Storage source
        • Google Pub/Sub Subscriber
          • Credentials
          • Multithreaded processing
          • Record header attributes
          • Data formats
          • Configuring a Google Pub/Sub Subscriber source
        • Groovy Scripting
          • Scripting objects
          • Multithreaded processing
          • Accessing record details
          • Type handling
          • Event generation
            • Event record
          • Record header attributes
          • Calling external Java code
          • Granting permissions on Groovy scripts
          • Configuring a Groovy Scripting source
        • HTTP Client
          • Processing mode
          • HTTP method
          • Headers
          • Per-status actions
          • Pagination
            • Page or offset number
            • Result field path
            • Keep all fields
            • Pagination examples
              • Example for link in HTTP header
              • Example for link in response field
              • Example for page number
              • Example for offset number
          • OAuth 2 authorization
            • Example for Twitter
            • Example for Microsoft Azure AD
            • Example for Google
          • Logging request and response data
          • Generated records
          • Data formats
          • Configuring an HTTP Client source
        • HTTP Server
          • Prerequisites
            • Send data to the listening port
            • Include the application ID in requests
          • Multithreaded processing
          • Data formats
          • Record header attributes
          • Configuring an HTTP Server source
        • IBM Db2
          • Prerequisites
          • Multithreaded processing
          • Record header attributes
          • Field attributes
          • IBM Db2 data types
            • Key column data types
          • Troubleshooting IBM Connectivity Service
          • Configuring an IBM Db2 source
        • JavaScript Scripting
          • Scripting objects
          • Multithreaded processing
          • Accessing record details
          • Type handling
          • Event generation
            • Event record
          • Record header attributes
          • Calling external Java code
          • Configuring a JavaScript Scripting source
        • JDBC Multitable Consumer
          • Database vendors and drivers
            • MySQL data types
            • Oracle data types
            • PostgreSQL data types
            • SQL Server data types
            • Unsupported data types
          • Installing the JDBC driver
            • Working with a MySQL JDBC driver
          • Table configuration
            • Schema, table name, and exclusion patterns
            • Offset column and value
            • Reading from views
          • Multithreaded processing modes
            • Multithreaded table processing
            • Multithreaded partition processing
              • Partition processing requirements
              • Multiple offset value handling
              • Best effort: Processing non-compliant tables
          • Non-incremental processing
          • Batch strategy
            • Process all available rows
            • Switch tables
          • Initial table order strategy
          • Processing queue
            • Multithreaded table processing only
            • Multithreaded partition processing only
            • Both multithreaded partition and table processing
          • JDBC attributes
            • JDBC header attributes
            • JDBC field attributes
          • Event generation
            • Event record
          • Configuring a JDBC Multitable Consumer source
        • JDBC Query Consumer
          • Database vendors and drivers
            • MySQL data types
            • Oracle data types
            • PostgreSQL data types
            • SQL Server data types
            • Unsupported data types
          • Installing the JDBC driver
          • Offset column and offset value
          • Full and incremental mode
          • Recovery
          • SQL query
            • SQL query for incremental mode
            • SQL query for full mode
            • Stored procedure in full mode
          • JDBC attributes
            • JDBC header attributes
            • JDBC field attributes
          • Event generation
            • Event record
          • Configuring a JDBC Query Consumer source
        • Jira
          • Event generation
            • Event records
          • OAuth 2 authentication
          • Configuring a Jira source
        • JMS Consumer
          • Installing JMS drivers
          • Additional JMS properties
          • Working with TIBCO EMS
            • Configuring for TIBCO and SSL
          • Data formats
          • Configuring a JMS Consumer source
        • Jython Scripting
          • Scripting objects
            • Thread safety in Jython scripts
          • Multithreaded processing
          • Accessing record details
          • Type handling
          • Event generation
            • Event record
          • Record header attributes
          • Calling external Java code
          • Configuring a Jython Scripting source
        • Kafka Multitopic Consumer
          • Offset management
          • Multithreaded processing
          • Additional Kafka properties
          • Record header attributes
          • Kafka security
          • Data formats
          • Configuring a Kafka Multitopic Consumer source
        • Kinesis Consumer
          • Multithreaded processing
          • Authentication method
          • Additional Kinesis properties
          • Read interval
          • Lease table tags
          • Resetting the Kinesis Consumer source
          • Data formats
          • Reading from DynamoDB or CloudWatch
          • Configuring a Kinesis Consumer source
        • MongoDB Atlas
          • Credentials
          • Offset field and initial offset
          • Specifying field paths
          • Read preference
          • Custom filter
          • Event generation
            • Event records
          • Enabling SSL/TLS
          • MongoDB data types
          • Reading BSON types
          • Configuring a MongoDB Atlas source
        • MongoDB Atlas CDC
          • Credentials
          • Read preferences
          • Generated records
            • CRUD operation and CDC header attributes
          • Enabling SSL/TLS
          • MongoDB data types
          • Reading BSON types
          • Configuring a MongoDB Atlas CDC source
        • MQTT Subscriber
          • Topics
          • Record header attributes
          • Data formats
          • Configuring an MQTT Subscriber source
        • MySQL Binary Log
          • Prerequisites
            • Configure MySQL Server for row-based logging
            • Enable the required authentication
            • Set the row metadata property
            • Install the JDBC driver
          • Initial offset
          • Generated records
          • Processing generated records
          • Tables to include or ignore
          • MySQL data types
          • Configuring a MySQL Binary Log source
        • OPC UA Client
          • Processing mode
          • Providing NodeIds
          • Security
          • Configuring an OPC UA Client source
        • Oracle Bulkload
          • Prerequisite
          • Static and non-static tables
          • Batch processing
          • Schema and table names
          • Multithreaded processing
          • Field attributes
          • Event generation
            • Event record
          • Configuring an Oracle Bulkload source
        • Oracle CDC
          • Comparing Oracle CDC sources
            • Updating Oracle CDC Client flows
          • Prerequisites
            • Task 1. Configure the database archiving mode
            • Task 2. Enable supplemental logging
            • Task 3. Create a user account
            • Task 4. Install required libraries on Data Collector
          • Primary and standby databases
          • Table filters
          • Start mode
          • Record formats
            • Custom record options
          • Parse SQL statements, supported operations, and record attributes
            • CRUD operation header attributes
            • CDC header attributes
            • Other header attributes
            • Default field attributes
            • Annotation field attributes
          • Multithreaded parsing
          • Query fetch size
          • Uncommitted transaction handling
            • Evicted transaction handling
          • Event generation
            • Event records
          • Oracle data types
            • Unsupported data types
          • Data preview with Oracle CDC
          • Configuring an Oracle CDC source
        • Oracle CDC Client
          • LogMiner dictionary source
          • Oracle CDC Client prerequisites
            • Task 1. Enable database archiving mode
            • Task 2. Enable supplemental loggingIf you change this title, update the text reference in Task 1.
            • Task 3. Create a user accountIf you change this title, update the text reference in Task 2.
            • Task 4. Extract a LogMiner dictionary (redo logs)
            • Task 5. Install the driver
          • Available parsers
          • Table configuration
            • Schema, table name, and exclusion patterns
          • Initial change
          • Choosing buffers
            • Local buffer resource requirements
            • Uncommitted transaction handling
          • JDBC fetch options
          • Include nulls
          • Unsupported data types
            • Conditional data type support
          • Processing Blob and Clob columns
          • Parse SQL, supported operations, and generated records
            • CRUD operation header attributes
            • CDC header attributes
            • Other header attributes
            • Field attributes
          • Database time zone
            • Change in daylight saving time
          • Event generation
            • Event records
          • Multithreaded parsing
          • Working with the SQL Parser processor
          • Data preview with Oracle CDC Client
          • Configuring an Oracle CDC Client source
        • Oracle Multitable Consumer
          • Prerequisite
          • Oracle data types
            • Unsupported data types
          • Table configuration
            • Schema, table name, and exclusion patterns
            • Offset column and value
          • Multithreaded processing modes
            • Multithreaded table processing
            • Multithreaded partition processing
              • Partition processing requirements
              • Multiple offset value handling
              • Best effort: Processing non-compliant tables
          • Non-incremental processing
          • Batch strategy
            • Process all available rows
            • Switch tables
          • Initial table order strategy
          • Processing queue
            • Multithreaded table processing only
            • Multithreaded partition processing only
            • Both multithreaded partition and table processing
          • Generated records
            • CRUD operation header attributes
            • JDBC header attributes
            • JDBC field attributes
          • Event generation
            • Event records
          • Configuring an Oracle Multitable Consumer source
        • Oracle XStream
          • Oracle prerequisites
            • Requirements
            • Command tools and syntax
            • Task 1. Set database parameters
            • Task 2. Create the CDC user
            • Task 3. Create the outbound server (existing components only)
              • Include all tables
              • Include specified tables
            • Task 4. Start the outbound server (existing components only)
            • Task 5. Prepare the database and tables
            • Task 6. Run verification commands
          • Data Collector prerequisite
          • Operation mode
          • Start mode
          • CRUD operation header attributes
          • CDC header attributes
          • Field attributes
          • Event generation
            • Event records
          • Configuring an Oracle XStream source
        • PostgreSQL CDC Client
          • PostgreSQL prerequisites
            • Install the logical decoder
            • Assign the required role
          • JDBC driver
          • Schema, table name, and exclusion patterns
          • Initial change
          • Memory usage
          • SSL/TLS encryption
          • Record contents and generated record
            • Record header attributes
            • Sample records
          • Configuring a PostgreSQL CDC Client source
        • Pulsar Consumer
          • Topics selector
          • Multithreaded processing
          • Offset management
          • Record header attributes
          • Schema properties
          • Security
            • Enabling TLS transport encryption
            • Enabling TLS authentication
            • Enabling JWT authentication
            • Enabling OAuth authentication
          • Data formats
          • Configuring a Pulsar Consumer source
        • Pulsar Consumer (Legacy)
          • Topics selector
            • Parallel reads of a topic
          • Offset management
          • Record header attributes
          • Schema properties
          • Security
            • Enabling TLS transport encryption
            • Enabling TLS authentication
            • Enabling JWT authentication
            • Enabling OAuth authentication
          • Data formats
          • Configuring a Pulsar Consumer (Legacy) source
        • RabbitMQ Consumer
          • Queue
          • Record header attributes
          • Data formats
          • Configuring a RabbitMQ Consumer source
        • Redis Consumer
          • Channels and patterns
          • Data formats
          • Configuring a Redis Consumer source
        • REST Service
          • HTTP listening port
          • API gateway
            • Gateway authentication
            • REST API URLs
            • Required request header
          • Security
          • Using application IDs
          • REST API client requirement
          • Multithreaded processing
          • Generated response
            • Sample responses
          • Record header attributes
          • Data formats
          • Configuring a REST Service source
        • Salesforce
          • Querying data
            • Using the SOAP and Bulk API
              • Bulk API example
              • Aggregate functions in SOQL queries
            • Using the Bulk API with PK Chunking
              • PK chunking example
            • Repeat query
            • Processing deleted records
          • Subscribing to notifications
            • Processing PushTopic events
              • PushTopic event record format
            • Processing platform events
            • Processing change events
          • Reading custom objects or fields
          • Salesforce attributes
            • Salesforce header attributes
            • CRUD operation header attribute
            • Salesforce field attributes
          • Event generation
            • Event record
          • Changing the API version
          • Configuring a Salesforce source
        • Salesforce Bulk API 2.0
          • Querying data
            • Bulk API 2.0 queries
          • Full and incremental mode
          • Multithreaded processing
          • Processing deleted records
          • Reading custom objects or fields
          • Salesforce attributes
            • Salesforce header attribute
            • Salesforce field attributes
          • Event generation
            • Event record
          • Changing the API version
          • Configuring a Salesforce Bulk API 2.0 source
        • SAP HANA Query Consumer
          • Installing the JDBC driver
          • Offset column and offset value
          • Full and incremental mode
          • Recovery
          • SQL query
            • SQL query for incremental mode
            • SQL query for full mode
          • JDBC attributes
            • JDBC header attributes
            • JDBC field attributes
          • SAP HANA header attributes
          • Event generation
            • Event record
          • Configuring an SAP HANA Query Consumer source
        • SFTP/FTP/FTPS Client
          • File name pattern and mode
          • Read order
          • First file for processing
          • Credentials
          • Record header attributes
          • Event generation
            • Event records
          • Post processing
          • Data formats
          • Configuring an SFTP/FTP/FTPS Client source
        • Snowflake Bulk
          • Prerequisites
            • Create a Snowflake stage
              • AWS credentials
              • Google Cloud Storage credentials
            • Assign privileges
          • Batch processing
          • Define a role
          • Multithreaded processing
          • Table configurations
          • Error handling
          • Record header attributes
          • Snowflake data types
          • Configuring a Snowflake Bulk source
        • SQL Server CDC Client
          • JDBC driver
          • Supported operations
          • Time windows
          • Table configuration
          • Initial table order strategy
          • Allow late table processing
          • Checking for schema changes
          • Generated record
            • Record header attributes
            • CRUD operation header attributes
            • Field attributes
          • Event generation
            • Event record
          • Configuring a SQL Server CDC Client source
        • SQL Server Change Tracking
          • JDBC driver
          • Prerequisites
            • Enable change tracking
            • Assign permissions
          • Table configuration
          • Initial table order strategy
          • Generated record
            • Record header attributes
            • CRUD operation header attributes
            • Field attributes
          • Event generation
            • Event record
          • Configuring a SQL Server Change Tracking source
        • TCP Server
          • Multithreaded processing
          • Closing connections for invalid data
          • Sending acknowledgements
            • Using expressions in messages
          • TCP modes
          • Data formats
          • Configuring a TCP Server source
        • UDP Multithreaded Source
          • Processing raw data
          • Receiver and worker threads
          • Packet queue
          • Multithreaded flows
          • Metrics for performance tuning
          • Configuring a UDP Multithreaded Source
        • UDP Source
          • Processing raw data
          • Receiver threads
          • Configuring a UDP Source
        • Web Client
          • Ingestion mode
          • HTTP method
            • Expression method
          • Headers
          • Grouping style
          • Event generation
            • Event records
          • Per-status actions
          • Per-timeout actions
          • Pagination
            • Page or offset number
          • OAuth 2 authentication
          • Generated records
          • Data formats
          • Configuring a Web Client source
        • WebSocket Client
          • Read REST response data from Data Collector
          • Generated microservice responses
            • Sample responses
          • Data formats
          • Configuring a WebSocket Client source
        • WebSocket Server
          • Prerequisites
            • Send data to the listening port
            • Include the application ID in requests
          • Multithreaded processing
          • Generated microservice responses
            • Sample responses
          • Data formats
          • Configuring a WebSocket Server source
      • Processors
        • Processors
        • Base64 Field Decoder
          • Configuring a Base64 Field Decoder processor
        • Base64 Field Encoder
          • Configuring a Base64 Field Encoder processor
        • Couchbase Lookup
          • Record header attributes
          • Configuring a Couchbase Lookup processor
        • Data Generator
          • Target field
          • Data formats
          • Configuring a Data Generator processor
        • Data Parser
          • Data formats
          • Configuring a Data Parser processor
        • Delay
          • Configuring a Delay processor
        • Encrypt and Decrypt Fields
          • Supported data types
          • Key provider
            • AWS credentials
          • Cipher suite
          • Encryption contexts
          • Data key caching
          • Encrypt and decrypt records
          • Configuring an Encrypt and Decrypt Field processor
        • Expression Evaluator
          • Output fields and attributes
          • Record header attribute expressions
          • Field attribute expressions
          • Configuring an Expression Evaluator processor
        • Field Flattener
          • Flatten the entire record
          • Flatten specific fields
          • Configuring a Field Flattener processor
        • Field Hasher
          • Hash methods
          • Field separator
          • List, map, and list-map fields
          • Configuring a Field Hasher processor
        • Field Mapper
          • Field values
          • Field names
          • Field paths
            • Aggregation expressions
          • Configuring a Field Mapper processor
        • Field Masker
          • Mask types
          • Configuring a Field Masker processor
        • Field Merger
          • Configuring a Field Merger processor
        • Field Order
          • Missing and extra fields
          • Configuring a Field Order processor
        • Field Pivoter
          • Generated records
          • Configuring a Field Pivoter processor
        • Field Remover
          • Configuring a Field Remover processor
        • Field Renamer
          • Renaming sets of fields
          • Configuring a Field Renamer processor
        • Field Replacer
          • Replacing values with nulls
          • Replacing values with new values
          • Data types for conditional replacement
          • Configuring a Field Replacer processor
        • Field Splitter
          • Not enough splits
          • Too many splits
          • Example
          • Configuring a Field Splitter processor
        • Field Type Converter
          • Valid type conversions
          • Expressions to convert by field name
          • Changing the scale of decimal fields
          • Configuring a Field Type Converter processor
        • Field Zip
          • Merging List data
          • Merging List-Map data
          • Pivoting merged lists
          • Configuring a Field Zip processor
        • Geo IP
          • Supported databases
          • Database file location
          • GeoIP field types
            • Full JSON field types
          • Configuring a Geo IP processor
        • Groovy Evaluator
          • Processing mode
          • Groovy scripting objects
          • Accessing record details
          • Processing list-map data
          • Type handling
          • Event generation
          • Working with record header attributes
          • Accessing whole file format records
          • Calling external Java code
          • Granting permissions on Groovy scripts
          • Configuring a Groovy Evaluator processor
        • HTTP Client
          • HTTP method
            • Expression method
          • Headers
          • Per-status actions
          • Pass records after retry or timeout failures
            • Record header attributes for passed records
          • Pagination
            • Page or offset number
            • Result field path
            • Keep all fields
            • Pagination examples
              • Example for link in HTTP header
              • Example for link in response field
              • Example for page number
              • Example for offset number
          • OAuth 2 authorization
            • Example for Twitter
            • Example for Microsoft Azure AD
            • Example for Google
          • Generated output
            • Response headers
          • Logging request and response data
          • Logging the resolved resource URL
          • Data formats
          • Configuring an HTTP Client processor
        • HTTP Router
          • Configuring an HTTP Router processor
        • JavaScript Evaluator
          • Processing mode
          • JavaScript scripting objects
          • Accessing record details
          • Processing list-map data
          • Type handling
          • Event generation
          • Working with record header attributes
          • Accessing whole file format records
          • Calling external Java code
          • Configuring a JavaScript Evaluator processor
        • JDBC Lookup
          • Database vendors and drivers
            • MySQL data types
            • Oracle data types
            • PostgreSQL data types
            • SQL Server data types
            • Unsupported data types
          • Installing the JDBC driver
          • Lookup cache
            • Using additional threads
            • Retry lookups for missing values
          • JDBC field attributes
          • Configuring a JDBC Lookup processor
        • JDBC Tee
          • Database vendors and drivers
            • MySQL data types
            • PostgreSQL data types
          • Installing the JDBC driver
          • CRUD operation processing
          • Single and multi-row operations
          • Configuring a JDBC Tee processor
        • JSON Generator
          • Configuring a JSON Generator processor
        • JSON Parser
          • Configuring a JSON Parser processor
        • Jython Evaluator
          • Processing mode
          • Jython scripting objects
          • Accessing record details
          • Processing list-map data
          • Type handling
          • Event generation
          • Working with record header attributes
          • Accessing whole file format records
          • Calling external Java code
          • Calling an external Python module
          • Configuring a Jython Evaluator processor
        • Log Parser
          • Log formats
          • Configuring a Log Parser processor
        • MLeap Evaluator
          • Prerequisites
          • MLeap model as a microservice
          • Example: Airbnb model
          • Configuring an MLeap Evaluator processor
        • MongoDB Atlas Lookup
          • Field mappings
          • Lookup cache
          • Credentials
          • Read preference
          • Configuring a MongoDB Atlas Lookup processor
        • PMML Evaluator
          • Prerequisites
            • Installing the PMML stage library
          • PMML model as a microservice
          • Example: Iris classification
          • Configuring a PMML Evaluator processor
        • PostgreSQL Metadata
          • JDBC driver
          • Schema and table names
          • Decimal precision and scale field attributes
          • Caching information
          • PostgreSQL data types
          • Configuring a PostgreSQL Metadata processor
        • Record Deduplicator
          • Comparison window
          • Configuring a Record Deduplicator processor
        • Redis Lookup
          • Data types
          • Lookup cache
          • Configuring a Redis Lookup processor
        • Salesforce Bulk API 2.0 Lookup
          • Lookup cache
          • Salesforce attributes
          • Changing the API version
          • Configuring a Salesforce Bulk API 2.0 Lookup processor
        • Salesforce Lookup
          • Lookup mode
            • Aggregate functions in SOQL queries
          • Lookup cache
          • Salesforce attributes
          • Changing the API version
          • Configuring a Salesforce Lookup processor
        • Schema Generator
          • Using schema header attributes
          • Generated schemas
          • Caching schemas
          • Configuring a Schema Generator processor
        • SQL Parser
          • Using multiple flows
            • Example
          • Resolving the schema
          • Unsupported data types
          • Generated records
            • CRUD operation header attributes
            • CDC header attributes
            • Other header attributes
            • Field attributes
          • Configuring an SQL Parser processor
        • Static Lookup
          • Configuring a Static Lookup processor
        • Stream Selector
          • Default stream
          • Sample conditions for streams
          • Configuring a Stream Selector processor
        • TensorFlow Evaluator
          • Prerequisites
          • Evaluation method
          • Event generation
            • Event records
          • Serving a TensorFlow model
          • Configuring a TensorFlow Evaluator processor
        • Web Client
          • Comparing Web Client and HTTP Client processors
          • HTTP method
            • Expression method
          • Headers
          • Grouping style
          • Event generation
            • Event records
          • Per-status actions
          • Per-timeout actions
          • Pagination
            • Page or offset number
          • OAuth 2 authentication
          • Generated output
          • Data formats
          • Configuring a Web Client processor
        • Whole File Transformer
          • Implementation overview
            • Write Avro flow
            • Parquet conversion flow
          • Memory and storage requirements
          • Generated record
          • Implementation example: Amazon S3 Parquet conversion
            • Write Avro flow example
            • Parquet conversion flow example
            • Runtime
          • Configuring a Whole File Transformer processor
        • Windowing Aggregator
          • Window type, time windows, and information display
            • Rolling windows
            • Sliding windows
          • Calculation components
          • Aggregate functions
          • Event generation
            • Event record root field
            • Event records
          • Monitoring aggregations
          • Configuring a Windowing Aggregator processor
        • XML Flattener
          • Generated records
          • Configuring an XML Flattener processor
        • XML Parser
          • Configuring an XML Parser processor
      • Targets
        • Targets
        • Amazon S3
          • Authentication method
          • Bucket
          • Partition prefix
          • Time basis and data time zone for time-based buckets and partition prefixes
          • Object names
            • Whole file names
          • Add tags to objects
          • Event generation
            • Event records
          • Server-side encryption
          • Data formats
          • Configuring an Amazon S3 target
        • Apache Iceberg for AWS
          • AWS Prerequisites
          • Creating the Iceberg table
            • Partitioning a new table
          • Write mode
          • Schema evolution mode
            • Supported type promotions
          • Amazon server-side encryption
          • Processing timestamp data
          • Iceberg data types
          • Configuring an Apache Iceberg for AWS target
        • Azure Blob Storage
          • Prerequisite
          • Blob types
          • Event generation
            • Event records
          • Data formats
          • Configuring an Azure Blob Storage target
        • Azure Data Lake Storage Gen2
          • Prerequisites
            • Retrieve authentication information
          • Directory templates
          • Time basis
          • Late records and late record handling
          • Timeout to close idle files
          • Recovery
          • Event generation
            • Event records
          • Resolving out of memory errors
          • Data formats
          • Configuring an Azure Data Lake Storage Gen2 target
        • Azure Event Hub Producer
          • Data formats
          • Configuring an Azure Event Hub Producer target
        • Azure IoT Hub Producer
          • Register Data Collector as an IoT Hub device
          • Data formats
          • Configuring an Azure IoT Hub Producer target
        • Azure Synapse SQL
          • Prerequisites
            • Prepare the Azure Synapse instance
            • Prepare the Azure Storage staging area
            • Enable access to the container
          • Load methods
          • Connections and authentication
            • Azure Synapse connection
            • Staging connection
            • Copy statement connection
          • Specifying tables
          • Enabling data drift handling
          • Row generation
            • Missing fields and fields with invalid types
          • Performance optimization
          • Azure Synapse data types
          • CRUD operation processing
          • Configuring an Azure Synapse SQL target
        • Cassandra
          • Batch type
          • Authentication
            • Kerberos (DSE) authentication
          • Cassandra data types
          • Configuring a Cassandra target
        • CoAP Client
          • Data formats
          • Configuring a CoAP Client target
        • Couchbase
          • CRUD operation processing
          • Compare and swap
          • Data formats
          • Configuring a Couchbase target
        • Databricks
          • Prerequisite
          • Load methods
            • CRUD operation processing
            • Primary key location
          • Specifying tables
          • Enabling data drift handling
          • Partitioning tables
          • Performance optimization
          • Staging location
            • Amazon S3 credentials
            • ADLS Gen2 authentication information
            • Google Cloud credentials
          • Row generation
            • Missing fields and fields with invalid types
          • Databricks data types
          • Configuring a Databricks target
        • Elasticsearch
          • Security
          • Time basis and time-based indexes
          • Document IDs
          • CRUD operation processing
          • Configuring an Elasticsearch target
        • Google BigQuery
          • Prerequisites
            • Prepare the Google BigQuery data warehouse
            • Prepare the Google Cloud Storage staging area
          • Credentials
          • Staging file formats
          • Specifying datasets and tables
          • Enabling data drift handling
            • Partitioning new tables
              • Partition configurations
              • Partition options
          • Row generation
            • Missing fields and fields with invalid types
          • Performance optimization
          • CRUD operation processing
          • BigQuery data types
          • Configuring a Google BigQuery target
        • Google Bigtable
          • Prerequisites
            • Install the BoringSSL library
            • Configure credentials
          • Row key
          • Cloud Bigtable data types
          • Column family and field mappings
          • Time basis
          • Configuring a Google Bigtable target
        • Google Cloud Storage
          • Credentials
          • Partition prefix
          • Time basis, data time zone, and time-based partition prefixes
          • Object names
            • Whole file names
          • Event generation
            • Event records
          • Data formats
          • Configuring a Google Cloud Storage target
        • Google Pub/Sub Publisher
          • Credentials
          • Data formats
          • Configuring a Google Pub/Sub Publisher target
        • HTTP Client
          • HTTP method
            • Expression method
          • Headers
          • Number of requests
          • Logging request and response data
          • Send microservice responses
          • OAuth 2 authorization
            • Example for Twitter
            • Example for Microsoft Azure AD
            • Example for Google
          • Data formats
          • Configuring an HTTP Client target
        • IBM Cloud Object Storage
          • Prerequisite
          • Bucket
          • Troubleshooting IBM Connectivity Service
          • Configuring an IBM Cloud Object Storage target
        • IBM Db2
          • Prerequisites
          • Specifying tables
          • Key columns
          • IBM Db2 data types
          • Troubleshooting IBM Connectivity Service
          • Configuring an IBM Db2 target
        • IBM watsonx.data
          • Prerequisite
          • Specifying tables
            • Creating tables
          • watsonx.data data types
          • Troubleshooting IBM Connectivity Service
          • Configuring an IBM watsonx.data target
        • InfluxDB 2.x
          • Configuring an InfluxDB 2.x target
        • JDBC Producer
          • Database vendors and drivers
          • Installing the JDBC driver
          • CRUD operation processing
          • Update and delete operations
          • Single and multi-row operations
          • Configuring a JDBC Producer target
        • Jira
          • Event generation
            • Event records
          • OAuth 2 authentication
          • Configuring a Jira target
        • JMS Producer
          • Installing JMS drivers
          • Include headers
          • Additional JMS properties
          • Working with TIBCO EMS
            • Configuring for TIBCO and SSL
          • Data formats
          • Configuring a JMS Producer target
        • Kafka Producer
          • Broker list
          • Runtime topic resolution
          • Partition strategy
          • Send microservice responses
          • Additional Kafka properties
          • Writing Kafka message headers
          • Kafka security
          • Data formats
          • Configuring a Kafka Producer target
        • Kinesis Firehose
          • Authentication method
          • Delivery stream
          • Data formats
          • Configuring a Kinesis Firehose target
        • Kinesis Producer
          • Authentication method
          • Additional Kinesis properties
          • Send Microservice responses
          • Data formats
          • Configuring a Kinesis Producer target
        • Local FS
          • Directory templates
          • Time basis
          • Late records and late record handling
          • Timeout to close idle files
          • Recovery
          • Event generation
            • Event records
          • Data formats
          • Configuring a Local FS target
        • MongoDB Atlas
          • Credentials
          • Specifying field paths
          • Unordered writes and stopping the flow
          • Define the CRUD operation
            • Performing upserts
          • Enabling SSL/TLS
          • MongoDB data types
          • Writing BSON types
          • Configuring a MongoDB Atlas target
        • MQTT Publisher
          • Topic
          • Data formats
          • Configuring an MQTT Publisher target
        • Named Pipe
          • Prerequisites
            • Create the named pipe
            • Configure the named pipe reader
          • Working with the named pipe reader
          • Data formats
          • Configuring a named pipe target
        • Oracle
          • Prerequisite
          • Enabling data drift handling
            • Generated data types
          • Creating tables
          • CRUD operation processing
          • Configuring an Oracle target
        • Pulsar Producer
          • Schema properties
          • Security
            • Enabling TLS transport encryption
            • Enabling TLS authentication
            • Enabling JWT authentication
            • Enabling OAuth authentication
          • Data formats
          • Configuring a Pulsar Producer target
        • RabbitMQ Producer
          • Queue
          • Data formats
          • Configuring a RabbitMQ Producer target
        • Redis
          • Mode
            • Data types for batch mode
            • Data formats for publish mode
          • Define the CRUD operation
          • Configuring a Redis target
        • Salesforce
          • CRUD operation processing
          • Hard deleting records
          • Field mappings
          • Changing the API version
          • Configuring a Salesforce target
        • Salesforce Bulk API 2.0
          • CRUD operation processing
          • Hard deleting records
          • Field mappings
          • Changing the API version
          • Configuring a Salesforce Bulk API 2.0 Target
        • Send Response to Source
          • Configuring a Send Response to Source target
        • SFTP/FTP/FTPS Client
          • Credentials
          • Event generation
            • Event records
          • Data format
          • Configuring an SFTP/FTP/FTPS Client target
        • SingleStore
          • CRUD operation processing
          • Configuring a SingleStore target
        • Snowflake
          • Sample use case
          • Prerequisites
            • Create a Snowflake stage
              • AWS credentials
              • Google Cloud Storage credentials
            • DESCRIBE prerequisite
            • COPY prerequisites
            • Snowpipe prerequisites
            • MERGE prerequisites
          • Implementation notes
          • Load methods
            • Primary key location
          • Error handling
          • Define a role
          • Performance optimization
          • Row generation
          • Specifying tables
          • Enabling data drift handling
            • Generated data types
          • Creating tables
          • CRUD operation processing
          • Configuring a Snowflake target
        • Snowflake File Uploader
          • Prerequisites
            • Create an internal Snowflake stage (optional)
            • Assign required privileges
          • Implementation notes
          • Define a role
          • Event generation
            • Event records
          • Configuring a Snowflake File Uploader target
        • Splunk
          • Prerequisites
          • Required record format
          • Logging request and response data
          • Configuring a Splunk target
        • Syslog
          • Protocol
          • Using SSL/TLS
          • Enabling SSL
          • Message content
          • Data formats
          • Configuring a Syslog target
        • Tableau CRM
          • Changing the API version
          • Define the operation
            • Metadata JSON
          • Automatic recovery
            • Manual upload recovery
          • Configuring an Tableau CRM target
        • Teradata
          • Prerequisites
            • Installing the Teradata driver
            • COPY prerequisites
            • MERGE prerequisites
          • Load methods
            • Primary key location
          • Performance optimization
          • Specifying tables
          • Enabling data drift handling
            • Generated data types
          • Creating tables
          • Staging location
            • Amazon S3 credentials
            • Azure authentication information
            • Google Cloud credentials
          • Row generation
            • Missing fields and fields with invalid types
          • CRUD operation processing
          • Configuring a Teradata target
        • To Error
        • Trash
        • Web Client
          • Comparing Web Client and HTTP Client targets
          • HTTP method
            • Expression method
          • Headers
          • Grouping style
          • Event generation
            • Event records
          • Per-status actions
          • Per-timeout actions
          • Pagination
            • Page or offset number
          • OAuth 2 authentication
          • Data formats
          • Configuring a Web Client target
        • WebSocket Client
          • Data formats
          • Configuring a WebSocket Client target
      • Executors
        • Executors
        • ADLS Gen2 File Metadata
          • Prerequisites
            • Retrieve authentication information
          • Related event generating stages
          • Changing metadata
            • Specifying the file path
            • Changing the file name or location
            • Defining the owner, group, permissions, and ACLs
          • Creating an empty file
          • Removing a file or directory
          • Event generation
            • Event records
          • Configuring an ADLS Gen2 File Metadata executor
        • Amazon S3
          • Authentication method
          • Create new objects
          • Copy objects
          • Tag existing objects
          • Event generation
            • Event records
          • Server-side encryption
          • Configuring an Amazon S3 executor
        • Databricks Job Launcher
          • Prerequisites
          • Event generation
            • Event records
          • Monitoring
          • Configuring a Databricks Job Launcher executor
        • Databricks Query
          • Prerequisite
          • Spark SQL queries
          • Storage connection
            • ADLS Gen2 authentication information
          • Event generation
            • Event records
          • Configuring a Databricks Query executor
        • Email
          • Prerequisite
          • Conditions
          • Using expressions
          • Configuring an Email executor
        • Google Cloud Storage
          • Credentials
          • Create new objects
          • Copy or move objects
          • Set metadata
          • Event generation
            • Event records
          • Configuring a Google Cloud Storage executor
        • Google BigQuery
          • Prerequisite
          • Credentials
          • SQL queries
          • Event generation
            • Event records
          • Configuring a Google BigQuery executor
        • JDBC Query
          • Database vendors and drivers
          • Installing the JDBC driver
          • SQL queries
          • Event generation
            • Event records
          • Configuring a JDBC Query executor
        • Pipeline Finisher
          • Recommended implementation
          • Related event generating stages
          • Source reset for additional flow runs
          • Notification options
          • Configuring a Pipeline Finisher executor
        • SFTP/FTP/FTPS Client
          • Credentials
          • Configuring an SFTP/FTP/FTPS Client executor
        • Shell
          • Data Collector shell impersonation mode
          • Script configuration
          • Configuring a Shell executor
        • Snowflake
          • Prerequisite
          • Implementation notes
            • Snowflake File Uploader and executor flows
          • Define a role
          • SQL queries
          • Event generation
            • Event record
          • Configuring a Snowflake executor
      • Working with flows
        • Previewing a flow
          • Views
          • Preview data types
          • Writing to targets and executors
          • Modifying preview properties
          • Additional preview details
        • Downloading and importing flows
          • Downloading flows
            • Downloading multiple flows
            • Downloading a single flow from the canvas
          • Importing a flow
      • Dataflow triggers
        • Dataflow triggers overview
        • Flow event generation
          • Using flow events
            • Pass to an executor
        • Stage event generation
          • Using stage events
            • Task execution streams
            • Event storage streams
        • Executors
        • Logical pairings
        • Event records
          • Event record header attributes
        • Summary
      • Solutions
        • Solutions overview
        • Stopping a flow after processing all available data
        • Sending email during flow processing
        • Preserving an audit trail of events
        • Loading data into Databricks Delta Lake
          • Bulk loading data into a Delta Lake table
            • Create the flow and configure the Salesforce source
            • Configure the Expression Evaluator processor
            • Configure the target to bulk load data
            • Run the flow to bulk load data
          • Merging changed data into a Delta Lake table
            • Create the flow and configure the MySQL Binary Log source
            • Configure processors to restructure the record
            • Configure the target to merge changed data
            • Run the flow to merge changed data
        • Drift Synchronization Solution for PostgreSQL
          • Basic implementation and processing
            • Flattening records
          • Requirements
          • Implementation steps
          • Case study
            • JDBC Multitable Consumer source
            • PostgreSQL Metadata processor
            • JDBC Producer target
            • Running the flow
          • PostgreSQL data types
      • Data formats
        • Data formats overview
          • File compression formats
        • Avro data format
          • Reading Avro data
          • Writing Avro data
        • Binary data format
        • Datagram data format
        • Delimited data format
          • Reading delimited data
            • CSV parser
            • Delimited data root field type
          • Writing delimited data
        • Excel data format
        • Log data format
        • NetFlow data processing
          • Caching NetFlow 9 templates
          • NetFlow 5 generated records
          • NetFlow 9 generated records
        • Parquet data format
          • Reading Parquet data
          • Writing Parquet data
        • Protobuf data format prerequisites
        • SDC record data format
        • Text data format with custom delimiters
          • Processing XML data with custom delimiters
        • Whole file data format
          • Basic flow
          • Whole file records
            • Processors for file reference information
          • Defining the transfer rate
          • Writing whole files
            • Access permissions
            • Including checksums in events
        • Reading and processing XML data
          • Creating multiple records with an XML element
            • Using XML elements with namespaces
          • Creating multiple records with an XPath expression
            • Using XPath expressions with namespaces
            • Simplified XPath syntax
            • Sample XPath expressions
            • Predicates in XPath expressions
            • Predicate examples
          • Preserving the root element
          • Including field XPaths and namespaces
          • XML attributes and namespace declarations
          • Parsed XML
        • Writing XML data
          • Record structure requirement
      • Multithreaded flows
        • Multithreaded flow overview
        • How it works
          • Sources for multithreaded flows
          • Processor caching
        • Tuning threads and runners
        • Resource usage
        • Multithreaded flow summary
      • Microservice flows
        • Microservice flows overview
        • Stages for microservice flows
          • Microservice sources
          • Microservice targets
        • Sample flow
        • Creating a microservice flow
      • Data formats by stage
        • Data format support
        • Sources
        • Processors
        • Targets
      • Expression language
        • Expression language
          • Expression completion in properties
            • Tips for expression completion
          • Expression examples
        • Functions
          • Record functions
          • Delimited data record functions
          • Error record functions
          • Base64 functions
          • Credential functions
          • Data generation functions
            • Faker functions
            • Xeger functions
          • Field functions
          • File functions
          • Flow functions
          • Math functions
          • String functions
          • Time functions
          • Timestamp functions
          • Miscellaneous functions
        • Constants
        • Datetime variables
        • Literals
        • Operators
          • Operator precedence
        • Reserved words
      • Regular expressions
        • Regular expressions overview
        • Regular expressions in the flow
        • Quick reference
        • Regex examples
      • Grok patterns
        • Defining grok patterns
        • General grok patterns
        • Date and time grok patterns
        • Java grok patterns
        • Log grok patterns
        • Networking grok patterns
        • Path grok patterns
      • Troubleshooting
        • Accessing error messages
        • Flow basics
          • General validation errors
        • Sources
          • Directory
          • Elasticsearch
          • JDBC sources
          • Oracle CDC Client
          • PostgreSQL CDC Client
          • Salesforce
          • Scripting sources
          • SQL Server CDC Client
        • Processors
          • Encrypt and Decrypt Fields
        • Targets
          • Azure Data Lake Storage
          • Cassandra
          • Elasticsearch target
          • Kafka Producer
        • JDBC connections
          • No suitable driver
          • Cannot connect to database
          • MySQL JDBC driver and time values
        • Performance
    • Running StreamSets jobs
      • Managing jobs and job runs
      • Monitoring job runs
        • Reviewing metrics
        • Tracking history
        • Viewing log messages
        • Capturing snapshots
      • Managing job offsets
        • Sources that maintain offsets
        • Viewing the job offset
        • Resetting the job offset
      • Orchestrating jobs with Orchestration Pipelines
  • Replicating data
    • Running replication jobs
    • Supported Data Replication connections
      • Replicating Amazon RDS for PostgreSQL data
        • Supported PostgreSQL data types
      • Replicating Apache Kafka data
      • Replicating IBM Db2 data
        • Supported IBM Db2 data types
      • Replicating IBM Db2 on Cloud data
        • Supported IBM Db2 on Cloud data types
      • Replicating IBM Db2 Warehouse data
      • Replicating PostgreSQL data
    • Choosing a business goal
      • Configuring the Change Log business goal
        • Change Log types
        • Change Log tables
    • Monitoring replication jobs
    • Managing replication jobs
  • Observing your data
    • Data Observability issue detection and remediation
      • Data Observability alerts
        • Creating alerts
    • Receiving alerts in external applications
      • Creating a Slack alert receiver
      • Creating a Microsoft Teams alert receiver
      • Creating a PagerDuty alert receiver
    • Roles and permissions in Data Observability
    • Using Data Observability
      • Setting up watsonx.data integration and configuring accounts
      • Accessing Data Observability
      • Analyzing detected issues, creating alerts and alert receivers
      • Creating threshold alerts with dynamic ranges
      • Analyzing triggered alerts
  • Integrating data with an agent
    • Setting up a project for an agent
    • Integrating data with Data Integration Agent
    • Integrating data with the MCP server
      • Configuring Claude Desktop
      • Configuring IBM Bob
• Data products
  • Managing Data Product Hub
    • Managing storage
    • Managing the community
    • Managing business domains
    • Managing connections
      • Understanding credentials for connections
    • Managing custom properties
    • Managing data contracts
      • Understanding data contract fields
    • Managing approval processes
      • Workflow template properties
  • Discovering data products
    • Managing your task inbox as a consumer
    • Requesting a new data product
    • Searching for data products
    • Subscribing to a data product
    • Subscribing to a data product that requires approval
    • Flight client example for accessing a data product
  • Publishing a data product
    • Managing your task inbox as a producer
    • Managing your Insights dashboard
    • Creating a data product
      • Creating a data product from a catalog
      • Creating a data product from a project
      • Creating a data product directly from a source
      • Creating a data product from a URL
      • Creating a data product from a YAML file
      • Creating a data product from SQL
        • Creating a data product from a query
        • Creating a data product from a customizable query
        • Creating a data product from a complex query
      • Creating a data product from a custom delivery method
      • Best practices for creating a data product
      • Supported connectors for data products
    • Managing the lifecycle of data products
    • Working with delivery methods
      • Creating a custom delivery method
      • Delivery methods for connectors
• Data governance
  • Planning to implement data governance
    • Planning to set up data cataloging and governance
    • Planning to implement a governance framework
    • Planning to protect data with rules
    • Planning to author data protection rules
    • Planning to curate data
    • Planning to monitor IBM watsonx.data intelligence
  • Managing IBM watsonx.data intelligence
    • Monitoring with Governance insights dashboards
    • Assigning roles and permissions for users
    • Custom properties, relationships, and asset types
      • Creating custom asset types
      • Creating custom properties
      • Creating custom relationships
      • Importing custom properties or relationships from a file
      • Managing custom properties, relationships, and asset types
    • Managing rule settings
    • Migrating data protection rules
    • Managing workflows for governance artifacts
    • Configuring workflows for data quality SLAs
      • Working with data quality remediation tasks
    • Monitoring workflow tasks
    • Detecting anomalies in workflows by using IBM Databand
    • Setting up reporting for IBM watsonx.data intelligence
      • Database requirements
      • Data model
      • Managing reporting
      • Sample reporting queries
      • Reporting tables
        • Workspaces
        • Asset relationships
        • Categories
        • Governance artifacts
        • Artifact relationships
        • Data quality rules
        • Customizations
        • Workflow
        • User Profiles
        • Tags
        • Rules
        • Metadata imports and enrichments
  • Catalogs
    • Administering a catalog
      • Creating a catalog
        • Duplicate asset handling
      • Managing access to a catalog
        • Catalog collaborator roles
      • Accessing orphaned catalogs
      • Changing catalog settings
      • Deleting a catalog
      • Saving searches for catalog assets
    • Catalog assets
      • Finding and viewing an asset in a catalog
      • Browsing asset hierarchies
      • Adding assets to a catalog
        • Adding a data file
        • Adding a connection
        • Adding data from a connection
        • Adding a connected folder asset from a connection
        • Adding COBOL copybook assets
        • Importing and exporting assets and asset metadata with a CSV file
          • CSV file format for importing and exporting asset metadata
          • CSV file format for importing and exporting asset relationships
      • Downloading data assets
      • Editing asset properties
      • Relationships in a catalog
        • Asset relationships
        • Managing relationships in a catalog
        • Exploring relationships
      • Controlling access to an asset
      • Profiling an asset
      • Removing an asset
  • Governance artifacts
    • Finding and viewing governance artifacts
      • Tags
    • Governance artifact properties
    • Versioning of governance artifacts
    • Managing governance artifacts
      • Import methods for governance artifacts
        • Importing artifacts by type with a CSV file
        • CSV import file format
        • Importing all governance artifacts with a ZIP file
      • Exporting governance artifacts
    • Workflows for governance artifacts
    • Categories
      • Predefined categories
      • Designing categories
      • Managing categories
      • Managing category collaborators
      • Category collaborator roles
      • Creating custom category collaborator roles
      • Importing or exporting categories
    • Policies
      • Designing policies
    • Governance rules
      • Designing governance rules
    • Data protection rules
      • Designing data protection rules
        • Filtering rows
        • Mask data
      • Data protection rules enforcement
      • Managing data protection rules
    • Data quality SLAs
      • Designing data quality SLAs
      • Managing data quality SLAs
    • Business terms
      • Designing business terms
      • Predefined business terms
      • Managing business terms
      • Authoring business terms
      • Generating business terms
      • Creating glossaries with MCP
    • Classifications
      • Designing classifications
      • Predefined classifications
    • Data classes
      • Designing data classes
      • Adding matching methods to data classes
      • Predefined data classes
        • Predefined data classes details
    • Reference data
      • Designing reference data sets
      • Creating reference data sets with composite keys
      • Importing files for reference data sets
      • Relationships between reference data sets
      • Predefined reference data sets
    • Reporting queries
    • Knowledge Accelerators
      • Notices
      • KA for Cross Industry
      • KA for Energy and Utilities
      • KA for Financial Services
      • KA for Healthcare
      • KA for Insurance
      • Getting started
        • Artifacts available to import
          • Cross Industry
          • Energy and Utilities
          • Financial Services
          • Healthcare
          • Insurance
      • Components
        • Use of governance artifacts
          • Category areas and subcategories
          • Business terms
          • Relationships
          • Custom attributes
          • Classifications and tags
        • Business Core Vocabulary
          • Subcategories
          • Concept terms
          • Property terms
          • Relationship terms
        • Business Performance Indicators
          • Subcategories
          • Performance analysis terms
          • Measures
        • Industry Alignment Vocabularies
          • Alignment area categories
          • Alignment topic categories
          • Alignment terms
        • Business Scopes
          • Subcategories
          • Business Scopes available for separate import
            • Cross Industry
            • Energy and Utilities
            • Financial Services
            • Healthcare
            • Insurance
        • Reference data sets
        • Data classes
        • Policies and rules
        • Synonyms
      • Using and customizing
  • Masking watsonx.data assets in watsonx.data intelligence
  • Resynchronizing assets and artifacts in the knowledge graph
• Data lineage
  • Lineage for unstructured data
  • Preparing data for data lineage
  • Viewing data lineage
  • Managing data lineage graph
  • Configuring alias assignments
  • OpenLineage integration
    • Mapping OpenLineage events
      • Designing mappings for OpenLineage events
      • Creating mappings for OpenLineage events
      • OpenLineage mappings tutorial
  • Configuring agents for lineage metadata import
  • Exporting data lineage
    • Exporting data lineage to Collibra
      • Configuring Collibra export
    • Exporting data lineage to OpenLineage format
    • Designing lineage export
    • Creating an export asset and managing jobs
• Administration
  • Administration on IBM Cloud
    • Setting up the platform on IBM Cloud
      • Setting up the IBM Cloud account
      • Managing users and access
        • Adding users to the account
        • Levels of user access roles
        • User roles for watsonx.data intelligence on IBM Cloud
        • Creating custom roles
        • IAM access groups
          • Setting up IAM access groups
          • Example IAM access groups
      • Setting up Cloud Object Storage
      • Setting up watsonx.data intelligence
      • Setting up watsonx.data integration
      • Creating the Platform assets catalog
    • Managing the platform on IBM Cloud
      • Customizing the branding of the web browser
      • Monitoring account resource usage
      • Setting up trusted profiles
      • Managing account settings
      • Managing all projects in the account
      • Upgrading services on the platform
      • Managing Cloud Object Storage resources
      • Removing users
      • Stop using services or the platform
    • Security on IBM Cloud
      • Network security
      • Enterprise security
      • Account security
      • Data security
      • Collaborator security
      • Security policies and responsibilities in IBM Cloud
      • Securing connections to services with private service endpoints
      • Configuring firewall access
        • Firewall access for the platform
        • Firewall access for IBM Cloud Object Storage
        • Firewall access for DataStage
        • Firewall access for Redshift
        • Firewall access for Spark
        • Firewall access for StreamSets
  • Administration on AWS
    • Setting up the platform on AWS
      • User roles for watsonx.data intelligence on AWS
    • Managing the platform on AWS
    • Security on AWS
      • Account security on AWS
      • Data security on AWS
      • Collaborator security on AWS
      • Configuring firewall access on AWS
  • Troubleshooting
    • Connections
    • Data Product Hub
    • Data Refinery
    • DataStage
    • IBM Cloud Status
    • IBM Cloud Object Storage for projects
    • IBM Knowledge Catalog
      • Can't provision a second service instance
      • Solving governance artifacts import problems
      • Synchronize the data policy service (DPS) category caches
      • New columns do not appear in preview of a connected asset
    • StreamSets
  • Managing the user API key
  • Managing your settings on IBM Cloud
  • Managing your settings on AWS
  • Determining your roles
  • Activity Tracker Event Routing
  • Managing your cloud account