Data stream

You can use the Data stream feature in IBM Process Mining to add a stream source to an existing project.

In IBM Process Mining, you can create a data stream connection by using the following services:

  • IBM Event Streams
  • Kafka
  • IBM MQ

Prerequisites for creating connection with IBM Event Streams and Kafka

It is mandatory to have an IBM Event Streams account in IBM Cloud before you create a data stream connection by using IBM Event Streams. For more information, see IBM Event Streams.

Before you create a data stream connection, do the following steps:

  1. Create an Event Streams account in IBM Cloud. For more information, see Configure Event Streams.
  2. Create a topic to store events in IBM Event Streams on the Topics page of the IBM Event Streams account. For more information, see Creating a Kafka topic.
  3. Use the Connect to this service menu to get the configuration properties on the home page.
  4. Collect the API Key from the Service credentials page.

To secure connections to IBM Event Streams and Kafka, the security.protocol property of the applications is set to SASL_SSL and the sasl.mechanism property is set to PLAIN.

Prerequisites for creating connection with IBM MQ

It is mandatory to have an IBM MQ Connector account in IBM Cloud before you create a data stream connection by using IBM MQ. For more information, Getting started on IBM MQ.

Before you create a data stream connection, do the following steps:

  1. Create an IBM MQ account in IBM Cloud. For more information, see Getting started on IBM MQ.
  2. Download the connection information in JSON text format from the IBM MQ account in IBM Cloud.

To view the list of available queue names, do the following steps:

  1. In the Queue managers tab of IBM MQ, click the account for which you need to view the available list of queue names.

  2. In the Administration tab of the selected IBM MQ account, click Launch MQ Console.

  3. In the MQ Console page, the Queues tab displays the available queues.

To collect the username and password for IBM MQ connection, do one of the following steps:

  1. To view the username for IBM MQ connection, navigate to the Application credentials tab.
  2. To view the password for the IBM MQ connection, click the more options button for the selected username, and then click Add new API key.

Creating a data stream connection

To add a data stream connection to your process mining project, do the following steps:

  1. Go to Data & Settings > Data > Data stream.
  2. On the Data stream page, choose if you want to use Event log or Object tables to create a connection.
    You can have one data stream connection per project, if you created a data stream by using an event log. If you created the data stream by using object tables, you can create more than one data stream.
  3. Click Add data stream.
  4. To start creating a new connection, choose one of the following options:
  5. Map your data stream connection. For more information, see Mapping a data stream connection.
  6. Schedule your data stream connection. For more information, see Scheduling a data stream connection.
  7. Create the data stream.

Creating a data stream connection by using IBM Event Streams

Before you begin creating a connection, make sure that you meet all prerequisites. For more information, see Prerequisites for creating connection with IBM Event Streams and Kafka.

On the Connection page of the Integrate data stream wizard, do the following steps:

  1. Click IBM Event Streams.
  2. In the Name field, type a name for the data stream.
  3. Optional: In the Summary field, type a description for the connection.
  4. In the Topic field, type the name of the channel that you use to receive the messages in IBM Event Streams.
  5. Optional: In the Group ID field, type the username of the connection that you use in IBM Event Streams.
    Group ID specifies the group that the user belongs to. In IBM Process Mining, you can choose to leave the field blank. In such cases, the application considers the selected project name as the value for Group ID.
  6. In the IBM Event Streams configuration properties field, type the configuration properties of IBM Event Streams for a secure connection.
  7. In the API key field, type the API Key that is used in Service credentials in IBM Event Streams.
  8. Click Verify connection to validate the accuracy of the information typed in all the fields.
  9. Go to the next step. For more information, see Mapping a data stream connection.

Creating a data stream connection by using Kafka

Before you begin creating a connection, make sure that you meet all prerequisites. For more information, see Prerequisites for creating connection with IBM Event Streams and Kafka.

In the Connection page of the Integrate data stream wizard, do the following steps:

  1. Click Kafka.
  2. In the Name field, type a name for the data stream.
  3. Optional: In the Summary field, type a description for the connection.
  4. In the Topic field, type the name of the channel that you use to receive the messages.
  5. Optional: In the Group ID field, type a unique name.
    The value in the Group ID field is a string that uniquely identifies the user's process group to which the user belongs.
  6. In the Host field, type the host and the port number.
  7. In the Kafka properties field, type the configuration properties of Kafka for a secure connection.
  8. In the Password field, type the password for Kafka.
  9. Click Verify connection to validate the accuracy of the information typed in all the fields.
  10. Go to the next step. For more information, see Mapping a data stream connection.

Creating a data stream connection by using IBM MQ

Before you begin creating a connection, make sure that you meet all prerequisites. For more information, see Prerequisites for creating connection with IBM MQ.

In the Connection page of the Integrate data stream wizard, do the following steps:

  1. Click IBM MQ.
  2. In the Name field, type a name for the data stream.
  3. Optional: In the Summary field, type a description for the data stream.
  4. Click Drag and drop file here or click to load connection info from JSON and go to the required folder to select the connection information file. Open the file.
  5. In the Queue name field, type the queue name for the IBM MQ account.
  6. In the Username field, type the username of the IBM MQ connection.
  7. In the Password field, type the API key for the IBM MQ connection.
  8. Click Verify connection to validate the connection.
  9. Go to the next step. For more information, see Mapping a data stream connection.

Mapping a data stream connection

In this step, you need to map the values ​​in your data stream. Depending on whether you create your data stream by using event log or object tables, mapping the data is a mandatory step during the creation process.

Mapping an event log data

On the Mapping page, do the following steps:

  1. Optional: In the Paste your sample message field, type the sample JSON message that you receive from a cloud provider. This helps you to map the fields to the correct JSON selector.
    When you include a sample JSON message, the JSON selectors that are entered in the Required mapped pairs section are validated with the sample JSON message.
  2. In the Required mapped pairs section, map the data source columns to the corresponding JSON selector.
    Map Process ID, Activity, and Start time values.
  3. Go to the next step. For more information, see Scheduling a data stream connection.

Mapping an object tables data

On the Mapping page, do the following steps:

  1. Select a table from the dropdown.
    Select a free table corresponding to your project.
  2. Optional: In the Paste your sample message field, type the sample JSON message that you receive from a cloud provider. This helps you to map the fields to the correct JSON selector.
    When you include a sample JSON message, the JSON selectors that are entered in the Required mapped pairs section are validated with the sample JSON message.
  3. Click Add mapped pair to map other data source fields to the corresponding fields. Map all mandatory fields that you configured in your object table.
    You need at least one mapped pair.
  4. Go to the next step. For more information, see Scheduling a data stream connection.

Scheduling a data stream connection

The data stream that is created in IBM Process Mining fetches data at the scheduled interval. In the Scheduling page, enter the time frame for running the data stream, and then click Create. The created connection is displayed on the Data stream page.

For example, if you set the schedule to every hour, the process starts on the next full hour. For that reason, set the minimum values.