Install additional stage libraries

Install additional stage libraries to use stages that are not included in the core or common installation of Data Collector. This is an optional step, but core installations typically require installing additional stage libraries.

For a complete list of the stages installed with each stage library, see Available stage libraries.

Important: You must perform additional steps to install the MapR stage libraries, as described in MapR Prerequisites.

You can install additional RPM stage libraries using the Data Collector command line program.

You can install additional tarball stage libraries using the Package Manager within Data Collector, using the stage library panel in the pipeline canvas, or using the Data Collector command line program.

An installation with Cloudera Manager is a full installation that includes all available stage libraries. As a result, you cannot install or uninstall additional stage libraries in a Cloudera Manager installation.

Installing for RPM

Use the following commands to install additional stage libraries for a core RPM installation:

To install one or more stage libraries:
Use the following command to install the stage libraries downloaded to the current directory:
yum localinstall <libraryID>-<version>-1.noarch.rpm <libraryID>-<version>-1.noarch.rpm ...
Use the full name of the library package that you want to install, separating them with commas. Do not include spaces in the command.
For example, to install the Amazon S3 origin and destination, as well as the Kudu destination for Data Collector version 7.0.0, use the following command: 
yum localinstall streamsets-datacollector-aws-lib-7.0.0-1.noarch.rpm streamsets-datacollector-apache-kudu_1_0-lib-7.0.0-1.noarch.rpm
To list the stage libraries installed on the current Data Collector:
Use the following command:
yum list installed | grep streamsets
To uninstall libraries when necessary:
Use the following command:
yum remove <libraryID> <libraryID> ...
Use the full name of the libraries that you want to uninstall, separating them with commas. Do not include spaces in the command.
For example, to uninstall the Amazon S3 origin and destination, use the following command:
yum remove streamsets-datacollector-aws-lib

Installing for Tarball Using Package Manager

You can use Package Manager within Data Collector to install additional stage libraries for a core or common tarball installation.

Complete one of the following steps to display Package Manager:

  • Click the Package Manager icon .

  • Click Add/Remove Stages in the Stage Library panel when viewing a pipeline in the pipeline canvas.

Package Manager lists all available stage libraries and the stages within each stage library. Origins display in blue, processors in orange, destinations in light green, and executors in dark green. Installed stage libraries display a check mark in the Installed column. You can filter the stage libraries by type or you can search for a stage library in the list.

To install an additional stage library, click the More icon for the library, and then click Install. Or to install multiple stage libraries, select the libraries in the list and then click the Install icon . Confirm that you want to install the libraries, and then restart Data Collector for the changes to take effect.

To uninstall a stage library, click the More icon for the library, and then click Uninstall. Or to uninstall multiple stage libraries, select the libraries in the list and then click the Uninstall icon . Confirm that you want to uninstall the libraries, and then restart Data Collector for the changes to take effect.
Note: If Data Collector does not have internet connectivity, you can view the installed stage libraries and can uninstall a stage library. However, you cannot view all stage libraries or install an additional stage library.

For information about the stages installed with each stage library, see Available stage libraries.

Installing for Tarball Using the Stage Library Panel

You can use the stage library panel in the pipeline canvas to install additional stage libraries for a core or common tarball installation.

By default, the stage library panel in the pipeline canvas displays all Data Collector stages, instead of only the installed stages. Stages that are not installed appear disabled, or greyed out. For example, the stage library panel shown below indicates that the Azure origins are not installed:

Stage library panel showing several origins, with Azure origins greyed out

To install an additional stage library, click on a disabled stage. Confirm that you want to install the library, and then restart Data Collector for the changes to take effect.

Note: If Data Collector does not have internet connectivity, you cannot view all stage libraries or install an additional stage library from the stage library panel.

For information about the stages installed with each stage library, see Available stage libraries.

Installing for Tarball Using the Command Line

You can use the stagelibs command to install additional stage libraries for a core or common tarball installation.

The stagelibs command requires that curl version 7.18.1 or later and sha1sum utilities are installed on the machine. Verify that these utilities are installed before running the command.

Use the following commands to install additional tarball libraries:
To view the list of available libraries:
Run the following command from the $SDC_DIST directory:
bin/streamsets stagelibs -list
This provides a list of all available stage libraries and whether they are already installed. For more information about the stages installed with each stage library, see Available stage libraries.
To install one or more stage libraries:
Run the following command from the $SDC_DIST directory:
bin/streamsets stagelibs -install=<libraryID>,<libraryID>,...
Use the full name of the libraries that you want to install, separating them with commas. Do not include spaces in the command.
For example, to install the Amazon S3 origin and destination, as well as the Cassandra destination, use the following command: 
bin/streamsets stagelibs -install\
=streamsets-datacollector-aws-lib,streamsets-datacollector-cassandra_2-lib
When successful, the command line indicates that the stage libraries have been installed as follows: 
Downloading: https://archives.streamsets.com/datacollector/<version>/tarball\
/streamsets-datacollector-aws-lib-<version>-SNAPSHOT.tgz
######################################################################## 100.0%
Downloading: https://archives.streamsets.com/datacollector/<version>/tarball\
streamsets-datacollector-jdbc-lib-<version>-SNAPSHOT.tgz
######################################################################## 100.0%
Downloading: https://archives.streamsets.com/datacollector/<version>/tarball\
streamsets-datacollector-rabbitmq-lib-<version>-SNAPSHOT.tgz
######################################################################## 100.0%

Stage library streamsets-datacollector-aws-lib installed
Stage library streamsets-datacollector-jdbc-lib installed
Stage library streamsets-datacollector-rabbitmq-lib installed
To generate the command required to perform the current installation (optional):
You can use the stagelibs command to generate the command to install the libraries that are installed on the current Data Collector. This allows you to easily replicate the installation elsewhere.
For example, say you installed three libraries above, and then installed another two. You can generate the command required to install all five libraries on additional machines.
To generate an installation script based on the current Data Collector installation, run the following command from the $SDC_DIST directory:
bin/streamsets stagelibs -installScript
The command returns an install command, such as the following: 
=================================================================================
streamsets stagelibs -install=streamsets-datacollector-apache-kafka_0_8_1-lib,\
streamsets-datacollector-aws-lib,streamsets-datacollector-basic-lib,\
streamsets-datacollector-cdh_kafka_1_3-lib,streamsets-datacollector-jdbc-lib,\
streamsets-datacollector-jython_2_7-lib,streamsets-datacollector-rabbitmq-lib
=================================================================================
To uninstall libraries when necessary:
To uninstall a library, run the following command from the $SDC_DIST directory:
bin/streamsets stagelibs -uninstall=<libraryID>,<libraryID>,...
Use the full name of the libraries that you want to uninstall, separating them with commas. Do not include spaces in the command.

Available stage libraries

A full Data Collector installation includes all of the following stage libraries. A core installation includes only some of the following stage libraries and typically requires you to install additional stage libraries. A common installation includes commonly-used stage libraries.

You can install additional stage libraries into either a core or common installation.

The following table describes the stages installed with each stage library:
Stage Library Name Included Stages

6.1 and later

streamsets-datacollector-apache-kafka-lib

 For Kafka.
Includes:
  • Kafka Multitopic Consumer origin
  • Kafka Producer destination
streamsets-datacollector-apache-pulsar-lib For Apache Pulsar versions 2.x.
Includes:
  • Pulsar Consumer origin
  • Pulsar Consumer (Legacy) origin
  • Pulsar Producer destination
streamsets-datacollector-aws-lib For Amazon Web Services 1.11.x.
Includes:
  • Amazon S3 origin
  • Amazon SQS Consumer origin
  • Amazon S3 destination
  • Amazon S3 executor
streamsets-datacollector-aws-secrets-manager-credentialstore-lib For the AWS Secrets Manager credential store.
streamsets-datacollector-azure-keyvault-credentialstore-lib For the Microsoft Azure Key Vault credential store.
streamsets-datacollector-azure-lib For Microsoft Azure.
Includes:
  • Azure Blob Storage origin
  • Azure Data Lake Storage Gen2 origin
  • Azure Data Lake Storage Gen2 (Legacy) origin
  • Azure IoT/Event Hub Consumer origin
  • Azure Blob Storage destination
  • Azure Data Lake Storage Gen2 destination
  • Azure Event Hub Producer destination
  • Azure IoT Hub Producer destination
  • Azure Synapse SQL destination
  • ADLS Gen2 File Metadata executor
streamsets-datacollector-basic-lib
Includes the following origins:
  • CoAP Server
  • Directory
  • File Tail
  • JavaScript Scripting
  • MQTT Subscriber
  • OPC UA Client
  • REST Service
  • TCP Server
  • UDP Multithreaded Source
  • UDP Source
  • WebSocket Client
  • WebSocket Server
Includes the following processors:
  • Base64 Field Decoder
  • Base64 Field Encoder
  • Data Generator
  • Data Parser
  • Delay
  • Expression Evaluator
  • Field Flattener
  • Field Hasher
  • Field Mapper
  • Field Masker
  • Field Merger
  • Field Order
  • Field Pivoter
  • Field Remover
  • Field Renamer
  • Field Replacer
  • Field Splitter
  • Field Type Converter
  • Field Zip
  • Geo IP
  • JavaScript Evaluator
  • JSON Generator
  • JSON Parser
  • Log Parser
  • Record Deduplicator
  • Schema Generator
  • Static Lookup
  • Stream Selector
  • Windowing Aggregator
  • XML Flattener
  • XML Parser
Includes the following destinations:
  • CoAP Client
  • Local FS
  • MQTT Publisher
  • Named Pipe
  • Send Response to Origin
  • Splunk
  • Syslog
  • To Error
  • Trash
  • WebSocket Client
Includes the following executors:
  • Databricks Job Launcher
  • Email
  • Pipeline Finisher
  • Shell
streamsets-datacollector-bigtable-lib For Google Cloud Bigtable.

Includes the Google Bigtable destination.
streamsets-datacollector-cassandra_3-lib For Cassandra 1.2, 2.x, and 3.x.

Includes the Cassandra destination.
streamsets-datacollector-cdp_7_3_1-lib For Cloudera CDP 7.3.1.
Includes:
  • Hadoop FS Standalone origin
  • Kafka Multitopic Consumer origin 6.0
  • HBase Lookup processor
  • Hive Metadata processor
  • Kudu Lookup processor
  • Hadoop FS destination
  • HBase destination
  • Hive Metastore destination
  • Kafka Producer destination 6.0
  • Kudu destination
  • Solr destination
  • HDFS File Metadata executor
  • Hive Query executor
  • MapReduce executor
  • Spark executor
Important: This stage library uses Cloudera Data Platform libraries that have known security vulnerabilities. Consult your security team before installing this library.
streamsets-datacollector-couchbase_3-lib For Couchbase SDK 3.x.
Includes:
  • Couchbase origin
  • Couchbase destination
streamsets-datacollector-crypto-lib For cryptography stages.

Includes the Encrypt and Decrypt Fields processor.
streamsets-datacollector-cyberark-credentialstore-lib For the CyberArk credential store.
streamsets-datacollector-dataformats-lib

Contains parsers and generators for the data formats supported by Data Collector.
streamsets-datacollector-dev-lib For developing and testing pipelines.
Includes:
  • Dev Data Generator origin
  • Dev Random Record origin
  • Dev Raw Data Source origin
  • Dev Snapshot origin
  • Dev Identity processor
  • Dev Random Error processor
  • Dev Record Creator processor
  • To Event destination
Note: Do not use these stages in production pipelines.
streamsets-datacollector-elasticsearch_9-lib For Elasticsearch 9.x.

Includes the Elasticsearch origin and destination.
streamsets-datacollector-file-transfer-lib For SFTP/FTP/FTPS.

Includes the SFTP/FTP/FTPS Client origin, destination, and executor.
streamsets-datacollector-google-cloud-lib For Google Cloud.
Includes:
  • Google BigQuery origin
  • Google Cloud Storage origin
  • Google Pub/Sub Subscriber origin
  • Google BigQuery destination
  • Google Cloud Storage destination
  • Google Pub/Sub Publisher destination
  • Google BigQuery executor
  • Google Cloud Storage executor
streamsets-datacollector-google-secret-manager-credentialstore-lib For the Google Secret Manager credential store.
streamsets-datacollector-groovy_2_4-lib For Groovy version 2.4.
Includes:
  • Groovy Scripting origin
  • Groovy Evaluator processor
streamsets-datacollector-groovy_4_0-lib For Groovy version 4.0.
Includes:
  • Groovy Scripting origin
  • Groovy Evaluator processor
streamsets-datacollector-hpe_edf_7_2-eep_9_2-lib For HPE Ezmeral Data Fabric 7.2.x with EEP 9.2.
Includes:
  • MapR DB CDC Consumer origin
  • MapR DB JSON origin
  • MapR FS Standalone origin
  • MapR Multitopic Streams Consumer origin
  • MapR Streams Consumer origin
  • MapR DB destination
  • MapR DB JSON destination
  • MapR FS destination
  • MapR Streams Producer destination
  • MapR FS File Metadata executor
Important: This stage library uses HPE Ezmeral Data Fabric libraries that have known security vulnerabilities. Consult your security team before installing this library.
streamsets-datacollector-http-lib For HTTP.
Includes:

  • HTTP Client origin, processor, and destination

  • HTTP Router processor

  • HTTP Server origin
streamsets-datacollector-ibm-connectivity-service-lib For IBM Connectivity Service.
  • IBM Cloud Object Storage destination 6.4 and later
  • IBM Db2 origin
  • IBM Db2 destination
  • IBM watsonx.data destination 6.3 and later
streamsets-datacollector-influxdb_2_0-lib For InfluxDB version 2.x.

Includes the InfluxDB 2.x destination.
streamsets-datacollector-jdbc-branded-oracle-lib For Oracle.
Includes:
  • Oracle Multitable Consumer origin
  • Oracle destination
streamsets-datacollector-jdbc-lib For JDBC access to databases.
Includes:
  • JDBC Multitable Consumer origin
  • JDBC Query Consumer origin
  • PostgreSQL CDC Client origin
  • Oracle CDC Client origin
  • SQL Server CDC Client origin
  • SQL Server Change Tracking origin
  • JDBC Lookup processor
  • JDBC Tee processor
  • PostgreSQL Metadata processor
  • SQL Parser processor
  • JDBC Producer destination
  • JDBC Query executor
streamsets-datacollector-jdbc-oracle-lib For Oracle.
Includes:
  • Oracle Bulkload origin
  • Oracle CDC origin
  • Oracle Multitable Consumer origin
streamsets-datacollector-jdbc-sap-hana-lib For JDBC access to SAP HANA databases.

Includes the SAP HANA Query Consumer origin.
streamsets-datacollector-jks-credentialstore-lib For the Java keystore credential store.
streamsets-datacollector-jms-lib For Java Messaging Services (JMS).

Includes the JMS Consumer origin and JMS Producer destination.
streamsets-datacollector-jython_2_7-lib For Jython version 2.7.x.
Includes:
  • Jython Scripting origin
  • Jython Evaluator processor
streamsets-datacollector-kaitai-lib For Kaitai Struct.

Includes the Kaitai Struct Parser processor.
streamsets-datacollector-kinesis-lib For Amazon Kinesis.
Includes:
  • Kinesis Consumer origin
  • Kinesis Firehose destination
  • Kinesis Producer destination
streamsets-datacollector-mleap-lib For MLeap.

Includes the MLeap Evaluator processor.
streamsets-datacollector-mongodb-atlas-lib For MongoDB Atlas and MongoDB Enterprise Server.
Includes:
  • MongoDB Atlas origin
  • MongoDB Atlas CDC origin
  • MongoDB Atlas destination
streamsets-datacollector-mysql-binlog-lib For MySQL binary logs.

Includes the MySQL Binary Log origin.
streamsets-datacollector-orchestrator-lib For the orchestration stages.
Includes:
  • Cron Scheduler origin
  • Start Jobs origin
  • Control Hub API processor
  • Start Jobs processor
  • Wait for Jobs processor
streamsets-datacollector-postgres-aurora-lib For Amazon Aurora PostgreSQL versions 1 through 4.

Includes the Aurora PostgreSQL CDC Client origin.
streamsets-datacollector-rabbitmq-lib For RabbitMQ version 3.5.6.

Includes the RabbitMQ Consumer origin and RabbitMQ Producer destination.
streamsets-datacollector-redis-lib For Redis versions 2.8 and 3.0.
Includes:
  • Redis Consumer origin
  • Redis Lookup processor
  • Redis destination
streamsets-datacollector-salesforce-lib

For Salesforce.

Includes:
  • Salesforce origin
  • Salesforce Bulk API 2.0 origin
  • Salesforce Lookup processor
  • Salesforce Bulk API 2.0 Lookup processor
  • Salesforce destination
  • Salesforce Bulk API 2.0 destination
  • Tableau CRM destination
streamsets-datacollector-sdc-databricks-lib For Databricks.
Includes:
  • Databricks destination
  • Databricks Query executor
streamsets-datacollector-sdc-snowflake-lib For Snowflake.
Includes:
  • Snowflake Bulk origin
  • Snowflake destination
  • Snowflake File Uploader destination
  • Snowflake executor
streamsets-datacollector-singlestore-lib For SingleStore.

Includes the SingleStore destination.
streamsets-datacollector-tensorflow-lib For TensorFlow.

Includes the TensorFlow Evaluator processor.
streamsets-datacollector-teradata-lib For Teradata.

Includes the Teradata destination.
streamsets-datacollector-thycotic-credentialstore-lib For the Thycotic Secret Server credential store.
streamsets-datacollector-vault-credentialstore-lib For the Hashicorp Vault credential store.
streamsets-datacollector-webclient-impl-okhttp For OkHttp.
Includes:
  • Jira origin
  • Jira destination
  • Web Client origin
  • Web Client processor
  • Web Client destination
streamsets-datacollector-wholefile-transformer-lib Includes the Whole File Transformer processor.
streamsets-datacollector-windows-lib

For Windows. Includes the Windows Event Log origin.