Custom process apps
Overview
For any business enterprise, customized processes and data structures play a huge role in their day-to-day operations. Therefore, you require customized tools and planned processes to extract data from the enterprise system and transform and load them into the process mining applications. The custom process app feature in IBM Process Mining supports the creation of the custom tools that help to automate the ETL process for IBM Process Mining. The custom process apps provide the flexibility to edit and configure the ETL tool package per your business process requirement. In addition, you can share the process app with the stakeholders of the business process for continuous use. You can use the data output of the process apps as a data source for creating a project in IBM Process Mining.
The custom process app feature is compatible with all enterprise applications. You can edit and update the scripts and configuration in a custom process app per the data structures and requirements of the enterprise system.
To learn more about the use case of the process app in a Salesforce environment, refer to the Process App use case document.
To learn more about the tasks that you can perform in the custom process app feature, see the following topics:
- Understanding types of users and permissions in custom process app
- Creating a custom process app
- Editing a custom process app
- Importing a custom process app
- Exporting a custom process app
- Deleting a custom process app
- Generating a process by using the custom process app
- Example: Creating a custom process app for Salesforce opportunity management
Understanding types of users and permissions in custom process app
Understanding the permissions that you can give to the custom process app users helps you to manage and control the use of the custom process app in your enterprise system. The custom process app feature has the following types of users:
-
Creator
Creators are the users who create a custom process app](#creating-a-custom-process-app). The creator has the privilege to set up the permissions for general users and importers. To learn more about the permissions of a creator, see the Permissions of creator topic.
-
General user
General user uses the custom process app that is created by a creator to generate a process in IBM Process Mining. The creator defines the permissions of the general users. To learn more about the permissions of a general user, see the Permissions of general user topic.
-
Importer
Importer is the user who imports a custom process app from an external or same IBM Process Mining instance. The permissions of the importer depend on the permission configuration set up by the creator while exporting a custom process app. To learn more about the permissions of an importer, see the Permissions of importer topic.
Permissions of creator
The creators have the full privilege on the custom process app that they create. 'Table 1. Creator permissions in a custom process app' defines the different permissions of the creator.
Permission type | Action permission |
Closed source |
|
Open source |
ⓘ Tip: See the Permissions of importer topic to learn more about the permissions of an importer. |
Table 1. Creator permissions in custom process app
ⓘ Tip: When a creator exports a custom process app and then imports it in an IBM Process Mining instance, the creator's role is changed to importer of the custom process app that is imported to the IBM Process Mining instance.
Permissions of general user
The permissions of a general user are defined by the Closed source and Open source tags given by the creator while configuring the permissions for the custom process app. 'Table 2. General user permissions in a custom process app' defines the permissions of a general user.
Permission type | Action permission |
Closed source |
|
Open source |
|
Table 2. General user permissions in custom process app
Permissions of importer
The permissions of an importer are defined by the permissions that are configured by the creator before exporting a custom process app. 'Table 3. Importer permissions in a custom process app' defines the permissions of an importer.
Permission type | Edit status | Action permission |
Closed source | Editable |
|
Uneditable |
|
|
Open source |
|
Table 3. Importer permissions in custom process app
Creating a custom process app
The creation of a custom process app includes the following activities:
- Accessing Custom process app wizard
- Creating a Python logic file
- Creating a Summary card for a process app
- Uploading your process app logic
- Defining the user inputs
- Providing the user guidance for the process app
- Setting up a process for the process app
Accessing Custom process app wizard
To create a custom process app, you must access the "Custom process app" wizard. Do the following steps to go to the "Custom process app" wizard:
-
In the Processes dashboard, click the Process App tile to go to the Process App dashboard.
-
In the Process Apps dashboard, click the Create process app button on the upper right of the window to see the "Custom Process App" wizard.
Creating a Python logic file
While configuring a custom process app, you must write and upload a Python logic file. The Python logic file defines and transforms the data from your enterprise application, which becomes the source of data for your process mining project. You must have the following key structures in your Python logic file:
- Entry points
- Configurable variables
- Output data format
- Supported Python libraries
- Handling exceptions in the python code
Entry points
The entry point or the entry function is a dedicated function in the logic file that serves as the entry to the data transformation process.
def execute(context):
ⓘ Tips: Without a defined entry point, the data transformation fails even before the process begins.
Configurable variables
The logic file requires configurable variables to compute the data transformation process. The configurable variables include dynamic variables such as token key
, server
or systemURL
, username
,
and password
. You can set the configurable variables in the following ways:
-
Getting configurable variable as environment variables
You can access and retrieve the configurable variables from an environment variable within the logic file. In other words, you must set each configurable variable from an environment variable.
For example, if you want to configure
username
as a configurable variable within the logic file, you must use the following command:username = os.environ["username"]
ⓘ Tips: The configurable variable is case-sensitive. Therefore, you must ensure consistency of letter cases while writing the Python file. For example, if you define
username
in lowercase, you must use the same letter case,username
, elsewhere in the logic file.
-
Getting the configuration variables from the context object
The configuration variables are also available with a Python dictionary that you can retrieve by using the
context.config
syntax. In the following example, you define theexecute
function as acontext
parameter:def execute(context): config = context[“config”] # example retrieving an input property named ‘URL’ url = config[“URL”]
ⓘ Tips:
-
In the custom process app, you must configure the input variables that is given in the logic file to establish connection between the logic file and the process data. For more information about defining user input variable, see the Define the user inputs topic.
-
If the raw data source is in
.CSV
format, you can configure the Python logic file to transform the data to be the source for your process mining project. Before uploading the data, the process app user must transform the file from.CSV
format to.zip
format. The Python logic file accesses the.zip
file to perform the necessary data transformation in the same directory location. The logic file can also retrieve the name of the uploaded.zip
file either from an environment variable or from the contextJSON
that is passed into theexecute
function of the logic file. The name of the key for the uploaded logic file name isfileUploadName
and the value is the name of the uploaded.zip
file.For example, if the name of the uploaded
.zip
file is “Input_sample.zip
”, the logic file can retrieve the name of the.zip
file by using any one of the following commands:myFileUploadName = os.environ["fileUploadName"]
Or
def execute(context): # Example retrieving the name of an uploaded zip file myFileUploadName = context[“fileUploadName”]
For more information about uploading the raw data, see the Define the user inputs topic.
Output data format
The entry point, which you define in the logic file, returns the output of the data transformation process. The output format must be a python DataFrame
object, which is a data structure to manipulate and store data in Python.
The output of the data transformation process in the logic file must be either Pandas or Polars, which are the two DataFrame
modules. In the following example, you configure the DataFrame
object:
def output(event_list):
return pd.DataFrame(event_list)
def execute(context):
extract_and_transform(context)
return output(event_list)
ⓘ Remember: In the example, you need to change the functions per your requirements for the logic file.
Supported Python libraries
In a custom process app, you can use the Python libraries in the logic file to transform the data:
Python libraries | Version |
---|---|
jsonschema |
4.17.3 |
numpy |
1.24.1 |
pandas |
1.5.3 |
polars |
0.16.8 |
PyGithub |
1.57 |
python-dateutil |
2.8.2 |
python-dotenv |
0.21.1 |
pythonhub |
0.3.1 |
requests |
2.28.2 |
types-requests |
2.28.11.8 |
types-urllib3 |
1.26.25.4 |
typing_extensions |
4.4.0 |
urllib3 |
1.26.14 |
ibm-cos-sdk |
2.12.2 |
ibm-cos-sdk-core |
2.12.2 |
ibm-cos-sdk-s3transfer |
2.12.2 |
oracledb |
1.2.2 |
gitdb |
4.0.9 |
GitPython |
3.1.34 |
confluent-kafka |
2.0.2 |
pymongo |
4.3.3 |
boto3 |
1.26.100 |
impyla |
0.18.0 |
elasticsearch |
8.6.2 |
sqlalchemy |
2.0.7 |
google-cloud-bigquery |
3.8.0 |
redis |
4.5.4 |
salesforce-api |
0.1.45 |
servicenow |
2.1.0 |
servicenow-api |
0.12.0 |
simple-salesforce |
1.12.3 |
SOAPpy |
0.12.22 |
splunk-sdk |
1.7.3 |
hbase |
1.0.0 |
neo4j |
5.6.0 |
couchbase |
4.1.3 |
pyexasol |
0.25.2 |
pysolr |
3.9.0 |
pymemcache |
4.0.0 |
pysnc |
1.1.4 |
Handling exceptions in the python code
To inform the process app user about invalid credentials or lost connection to an external system, you can configure the python code to send a dedicated exception named ProcessAppException
in the process app user interface.
The following code snippet gives an example of ProcessAppException
configuration:
from process_app import ProcessAppException
def execute(context):
# Example raising a ProcessAppException
raise ProcessAppException("cannot connect to the system")
Creating a Summary card for a process app
In the Summary card step, do the following steps:
-
Enter the name of the process app in the Name field.
ⓘ Tip: You cannot use the same name of an existing process app for the new process app.
-
Enter a description about the process app in the Brief description field.
ⓘ Tip: The maximum number of characters that you can type in the Brief description field is 120.
-
Switch the toggle to Closed source or Open source to set the permissions for the process app users. You can choose one of the following options per your requirement:
-
Switch the toggle to Open source if you want to give the Use, View, Export, and Duplicate permissions to the process app users.
-
Switch the toggle to Closed source if you want to give only the Use and Export permissions to the process app users.
-
-
If you want to customize the icon and add feature highlights and documentation link on the summary card, expand the Customization section.
ⓘ Tip: The Customization section is optional. Therefore, if you want to skip customizing the icon and adding the feature highlights and documentation, you can click the Next button.
-
You can change the summary card icon by selecting an icon from the pack of icons in the Icon section. You can also upload your own icon by clicking the Add icon (
).
ⓘ Tips:
-
The maximum file size that you can upload is
200 KB
. -
You can upload only the SVG, JPG, JPEG, or PNG format.
-
-
If you want to display the feature highlights of your process app on the summary card, add them in the Feature highlights field in the Feature highlights and documentation section.
ⓘ Tip: You can add maximum three feature highlights for your process app.
-
If you want to display the documentation URL link for your process app on the summary card, add them in the Link to external documentation field in the Documentation section.
ⓘ Tip: You can add only one documentation URL link for your process app in the summary card.
-
Click the Next button to go to the Upload your Process App logic step.
Uploading your Process App logic
By uploading a process app logic in the "Custom Process App" wizard, you define the task of the process app. The process app logic files are customized Python scripts that connect your data sets and process logic to extract and transform the required data from the source application. You can use the data output from the execution of the Python script to generate a process in IBM Process Mining. In addition, you can reuse the process app logic file and apply it in multiple processes.
In the Upload your Process App logic step, click the Drag and drop file here to click to upload link to select and upload the python script from your system. After uploading the Python script, click the Next button to go to the Define user inputs step in the "Custom process app" wizard. (See Video 1. Uploading your Process App logic)
To learn more about structuring your Python script, you can download the template document that is provided in the Upload your Process App logic step by clicking the Download template link.
ⓘ Important: You must validate and verify the Python script for accuracy before uploading the Python file. If there is any exception or error message while running the Process App logic, you can view the error status in the Generating your process stage.
Video 1. Uploading your Process App logic
Defining the user inputs
In the "Custom process app" wizard, you can configure the options for the users to input data that the Python script must transform for process generation. It helps you to provide the process app users the flexibility to select the data that they want to analyze in the event logs of the business process. You can configure the input options in the following ways:
-
Allow the users to upload a data file to transform into a valid data source for process mining.
-
Allow the users to define the individual parameters for the Python script to generate a data source based on their business requirement.
ⓘ Remember: The configuration that you set up in the Define user inputs step determines the availability of the Load your data and Configure log events steps in the process
app during the process generation. For example, if you set the File upload required switch to Required
, the Load your data step is made visible for the process app users during
the process generation. Similarly, if you add the Custom input field, the Configure log events step is made visible for the process app users during the process generation.
If you want to allow the users to define the user input, do the following steps in the Define user inputs (optional) step:
-
In the File upload section, set the File upload required switch to
Required
if you want to allow the users of the process app to upload a file with raw data that needs to be transformed.When you set the File upload required switch to
Required
, the Load your data step is made visible for the process app users during the process generation.If the file upload is not mandatory for the process app user, set the File upload required switch to
Not required
. When you set the File upload required switch toNot required
, the Load your data step is hidden for the process app users during the process generation.
ⓘ Tip: If you want to configure the Python logic file to transform data as the source for your process mining project, you must compress the raw data to create a
.zip
file before you upload. For more information, see the Tips in the Configurable variables topic.
-
In the Custom input fields section, expand User input 1 to see the fields to define the user inputs.
-
Set the input required switch to
Required
if the input field is mandatory for the users. When you set the Custom input field toRequired
, the fields in the Configure log events step is made mandatory for the process app users during the process generation.If the input field is not mandatory for the users, set the input required switch to
Not required
. When you set the Custom input field toNot required
, the Configure log events step is optional for the process app users during the process generation. -
Define the following input fields for the users of the process app:
-
Label field
The field to define the label on the user interface for the input field.
-
Variable name field
The field to define the name of the variable that the user must enter based on the logic of the Python script. For example,
Enter the Password
-
Input type
The field to define the input type of the input field, which can be a String, Dropdown, or Password.
ⓘ Tip:
-
If you want to include multiple input fields for the process app users, click the Add custom input button to add more custom input fields.
-
If you add the Custom input field, the Configure log events step is made visible for the process app users during the process generation. If you do not add the Custom input field, the Configure log events step is hidden for the process app users during the process generation.
-
-
-
Click the Next button to go to the Provide user guidance step.
Providing the user guidance for the process app
In the "Custom process app" wizard, you can create user guides for the custom process app. The Provide user guidance step is optional.
ⓘ Remember: The configuration that you set up in the Provide user guidance step determines the availability of the Documentation step in the process app during the process generation. For example, if you do not set up the Provide user guidance step with the documentation information, the Documentaion step is hidden for the process app users during the process generation.
To create the user guide for the process app, do the following steps in the Provide user guidance step:
-
In the Guidance text section, enter the heading of the text in the Text heading field.
-
Enter the main body of the user guidance text in the User guidance body field (maximum 2000 characters).
ⓘ Tip: You can add multiple description block by clicking the Add description block link.
-
In the Documentation and other links section, you can add the documentation URL link in the URL field and the label name of the link in the Link label field.
ⓘ Tip: You can add multiple documentation link by clicking Add another link.
-
Click the Next button to go to the Process settings step.
Setting up a process for the process app
In the Process setting step, you can upload a process back up file in the .idp
format or duplicate settings from an existing process in IBM Process Mining. The process app uses the mapped data columns, the process
settings, filters, and dashboards from the process backup file to generate a new process.
To upload a process back file, do the following steps:
-
Select the Upload process backup file option in the Process setting step.
-
Click the Drag and drop file here or click to upload link to upload the process backup file.
-
Click the Save button.
To duplicate the settings of an existing process, do the following steps:
-
Select the Duplicate settings from a process option in the Process setting step.
-
Select a process from the Find a process dropdown.
-
Click the Create button.
Editing a custom process app
To edit a process app, do the following steps:
-
Click the menu icon on the upper right of the Process App tile to view the menu options.
-
Click the Edit Process App option to view the "Edit Process App" wizard.
ⓘ Tip: Only the process app creators can edit their process app in the same IBM Process Mining instance.
-
In the Edit process app wizard, follow the steps in the Creating a custom process app topic to complete your edits.
Importing a custom process app
In your IBM Process Mining instance, you can import a custom process app from another IBM Process Mining instance. Importing of a process app helps you to reuse and apply an existing process app logic to your process. Before you import a custom
process app, you must ensure that you have already downloaded the process app file in the .pma
format from the source IBM Process Mining instance. To import a custom process app, do the following steps:
-
In the Process Apps page, click the Import process app button to see the Import Process App from file dialog.
-
In the Import Process App from file dialog, drag or upload the file in the Upload files section.
ⓘ Tip:
-
You can import only the process app file in the
.pma
format. -
When you import a process app, the name of the new process app gets the suffix,
Copy-
<timestamp-id>
, along with the original process app name. -
If you want to import a process app that is created in IBM Process Mining 1.14.0 to IBM Process Mining 1.14.1, you must do the following steps:
-
Upgrade the IBM Process Mining instance, from which you want to export the process app file, to version 1.14.1.
-
Export the process app file from the upgraded IBM Process Mining instance.
-
Import the process app file in the target IBM Process Mining instance of version 1.14.1.
-
-
For more information about permissions of the importer per different permission types that are configured for the process app, see the Permissions of importer topic.
-
Exporting a custom process app
You can export a process app to save, reuse, and share the process app configuration. To export a process app, do the following steps:
-
Click the menu icon on the upper right of the Process App tile to view the menu options.
-
Click the Export option to run one of the following actions per your existing configuration:
-
If the process app is Open source, the process app is downloaded in the
.pma
format. -
If you are the creator of a Closed source process app, you see the How would you like to export the process app? dialog. On the How would you like to export the process app? dialog, switch the toggle to Editable or Uneditable per your requirement and click the Export button to download the process app.
ⓘ Tips:
-
If you switch the toggle to Editable, the importers get the Use, Edit, Export, and Delete permissions.
-
If you switch the toggle to Uneditable, the importers get the Use, Edit name, Export, and Delete permissions. However, the importers cannot change the Python script for extracting and transforming the data.
-
-
If you are a general user of a Closed source process app, you see the Warning dialog. On the Warning dialog, click the Yes button to download the process app.
-
ⓘ Tips:
-
You can export a process app file only if all the configurations in the Creating a custom process app are complete.
-
If you want to export a process app file that is built in IBM Process Mining 1.14.0 to IBM Process Mining 1.14.1, you must first upgrade IBM Process Mining version, from which you need to export, from 1.14.0 to 1.14.1.
-
For more information about permissions in process app, see the Understanding types of users and permissions in custom process app topic.
Duplicating a custom process app
To duplicate a process app, do the following steps:
-
Click the menu icon on the upper right of the Process App tile to view the menu options.
-
Click the Duplicate option to duplicate the process app.
ⓘ Tip:
-
When you duplicate a process app, the name of the new process app gets the suffix,
Copy-
<timestamp-id>
, along with the existing process app name. -
You can duplicate a process app file only if all the configurations in the Creating a custom process app are complete.
-
The permission to duplicate the process app depends on the user type. For more information about permissions in process app, see the Understanding types of users and permissions in custom process app topic.
-
Deleting a custom process app
To delete a process app, do the following steps:
-
Click the menu icon on the upper right of the Process App tile to view the menu options.
-
Click the Delete option to view the Confirm delete dialog box.
-
In the Confirm delete dialog box, enter the process app name in the Delete confirmation field.
-
Click the Delete button.
ⓘ Tip: Only creators and importers can delete their process app. For more information about permissions in process app, see the Understanding types of users and permissions in custom process app topic.
Generating a process by using the custom process app
You can generate a process in IBM Process Mining by using the custom process app. The custom process applies all the process logic and configurations that you set up in the process app during the creation of the process app. To generate a process by using the custom process app, you do the following steps in the process app:
- Setting up the process details in process apps
- View the documentation
- Configuring event logs
- Loading your data
- Generating your process
Setting up the process details in Process Apps
To set up the process details, do the following steps:
-
Click Get Started on the Process App tile in the Process app page to see the Process details step.
-
In the Process details step, enter the name of your process in the Name field.
-
You can choose to link or not to link your process to an organization. If you do not want to link your process to an organization, click the Next button to go to the Documentation step. If you want to link your process to an organization, use one of the following options:
-
Link to an existing organization: If you want to link your process to an existing organization, select the Use existing organization option and select an organization name from the list in the Organization name field.
-
Link to a new organization: If you want to link your process to a new organization, select the Create new organization option and enter the organization name in the Organization name field.
-
-
Click Next to go to the Documentation step.
Viewing the documentation
The availability of the Documentation step depends on the configuration that is set up in the Provide user guidance step in the "Create process app" wizard. For more information on configuring the Provide user guidance step, see Providing the user guidance for the process app topic. In this step, you can view the documentation for generating a process by using the process app. You can click the Next button to go to the Configuring event logs step.
Configuring event logs
The availability of the Configure event logs step depends on the configuration that is set up in the Define user input step in the "Create process app" wizard. For more information on configuring the Define user input step, see Defining the user input topic. In the Configure event logs step, you input the fields based on the configuration in the process app. Depending on the process app configuration, you must input the fields that are marked mandatory and click the Next button to go to the Load your data step.
Loading your data
The availability of the Load your data step depends on the configuration that you set up in the Define user input step in the "Create process app" wizard. For more information on configuring the Define user input step, see Defining the user input topic. To load the extracted data from your enterprise application to IBM Process Mining, do the following steps:
-
In the Load your data step, drag or upload the data file in the
.zip
format in the Add files by using drag and drop section. The process app then validates and uploads the data.
Tip:
-
The maximum file size that you can upload is 125 MB.
-
You can upload only
.zip
file.
-
-
Click the Next button to go to the Generate your process step.
Generating your process
The process app starts generating the process when you click the Next button in the Load your data step. You can see the progress of process generation by viewing the status, namely PENDING
, IN PROGRESS
,
COMPLETED
, and ERROR
, of the following steps in process generation:
- Transforming your data
- Generating event log
- Applying default analysis
- Finalizing the process
You can click the Finish button to go to the IBM Process Mining Processes dashboard while process generation happens in the background. When the process app completes the process generation, you can view the new process in the process list on the Processes dashboard.