Setting up Data Flow Automation

Edit online

Data Flow Automation is a collection of data services and pipelines that you can use to automate the flow of data into and out of IBM® ESG Suite.

A data service represents a connection point for exchanging data into and out of the platform. A data pipeline determines how data moves between data services to meet different use cases such as file loading and data export.

For example, you might configure an SFTP data service to receive files from an external system by using SFTP as the file transfer protocol. Then, you configure a data pipeline to poll the data service for file names that match one of the universal data loading templates that are provided in Envizi ESG Suite. If there is a match, the file is queued for processing and loading.

If you purchased a Connector AddOn, you might instead configure the data pipeline to check for file names in your custom format, and if found, to send the file to IBM App Connect for transformation. The file is transformed into one of the universal data loading templates so that the file can be processed and loaded in Envizi ESG Suite.

Table 1 shows a summary of automated connections into and out of Envizi ESG Suite.

Table 1. Data flow automation use cases
Use case Purpose Data Source (typical) Destination Data services Module or AddOn (in addition to Data Management Base)
Data file loading Load account or meter data files that are formatted to one of the standard data loading templates of Envizi ESG Suite. Account records or meter readings. Manually populated templates or custom report extracts from a client, supplier, or partner system. N/A File Loading:
  • Cloud Storage
  • SFTP, or
  • Mailbox
 None
Data file transformation and loading Transform and load account or meter data files that are formatted in a client or supplier specific format. Account records or meter readings. Standard report extracts from a client, supplier, or partner system. To and from IBM App Connect for transformation. File loading:
  • Cloud Storage,
  • SFTP, or
  • Mailbox
Transformation:
  • App Connect Data ETL
 Connector AddOn
Data loading using custom API connector Retrieve account or meter data from a client or supplier API and transform the data for loading into Envizi ESG Suite. Account records or meter readings. Direct API access to a client, supplier, or partner system. To or from IBM App Connect for API data retrieval and file transformation. For file loading
  • Cloud Storage,
  • SFTP, or
  • Mailbox

For transformation

  • App Connect Data ETL
 Connector AddOn and Custom API Integration via an IBM Expert Labs SOW
Bulk loading attachments Load attachments in bulk, for example, linking invoices to utility account records as supporting evidence. Attachments. Scanned physical invoices or client, supplier, or partner file repositories. N/A File Loading:
  • Cloud Storage
  • SFTP, or
  • Mailbox
 None
Scope 3 categorization using AI Assist Use the AI Assist (NLP) service to identify the most appropriate spend category for your Scope 3, Category 1 purchased goods and services spend data. Transactional spend-based records. Manually populated templates or custom report extracts from a client, supplier, or partner system. To or from a Natural Language Processing (NLP) service Transformation
  • NLP Service Data ETL
 Scope 3 GHG Accounting + Reporting module
Export Envizi ESG Suite data model Offline analysis of the Envizi ESG Suite data model. Envizi ESG Suite monthly dataset. Envizi ESG Suite To a Data Export service Data Export
  • Cloud Storage, or
  • SFTP
 Data Export AddOn
Planning Analytics integration Integrate the monthly dataset with Planning Analytics for planning and forecasting. Envizi ESG Suite monthly dataset. Envizi ESG Suite To IBM Planning Analytics Integration
  • Planning Analytics
 Planning Analytics AddOn
Supply Chain Intelligence integration Load Scope 3 Category 1 PCF data from Supply Chain Intelligence Scope 3 category 1 PCF data from Supply Chain Intelligence Supply Chain Intelligence From Supply Chain Intelligence

File loading

  • Cloud Storage
 Supply Chain Intelligence module

The key benefits of data flow automation include the following:

  • Data services are dedicated for your users and suppliers and configured with unique credentials, improving the security of your file loading, data export and integration pipelines.
  • You can control how and when data is loaded into your environment by provisioning and managing your own data services and data pipelines,
  • You have the option of transforming your files into one of our standard connector formats through a fully managed file transformation service.
  • You have end-to-end visibility of data pipelines, allowing you to view file loading status, download original and transformed files, and access error logs, all from the Envizi ESG Suite user interface.