Setting up Data Flow Automation

Edit online

Data Flow Automation is a collection of data services and pipelines that you can use to automate the flow of data into and out of IBM® Envizi ESG Suite.

A data service represents a connection point for exchanging data into and out of the platform. A data pipeline determines how data moves between data services to meet different use cases such as file loading and data export.

System pipelines are automatically created and connected to all enabled data services to process files that don't require transformation, for example, files that use the universal bulk upload templates. Custom pipelines are required when you are using a connector to transform files through IBM App Connect, or for special handling of the file such as renaming or delaying the file. For more information, see Provisioning data services and Provisioning data pipelines.

Note: The default size limit for loading data files and attachments through data flow automation is 50 MB. However, data files can be limited by other factors such as the Excel maximum row count. Whenever possible, use smaller file sizes for data loading. For large files, split the files into smaller files for faster processing and easier troubleshooting.

Table 1 shows a summary of automated connections into and out of Envizi ESG Suite.

Table 1. Data Flow Automation use cases
Use case	Purpose	Data	Source (typical)	Destination	Data services	Module or AddOn (in addition to Data Management Base)
Data file loading	Load account or meter data files that are formatted to one of the standard data loading templates of Envizi ESG Suite.	Account records or meter readings.	Manually populated templates or custom report extracts from a client, supplier, or partner system.	N/A	File Loading: Cloud Storage SFTP, or Mailbox	None
Data file transformation and loading	Transform and load account or meter data files that are formatted in a client or supplier specific format.	Account records or meter readings.	Standard report extracts from a client, supplier, or partner system.	To and from IBM App Connect for transformation.	File loading: Cloud Storage, SFTP, or Mailbox Transformation: App Connect Data ETL	Connector AddOn
Data loading using custom API connector	Retrieve account or meter data from a client or supplier API and transform the data for loading into Envizi ESG Suite.	Account records or meter readings.	Direct API access to a client, supplier, or partner system.	To or from IBM App Connect for API data retrieval and file transformation.	For file loading Cloud Storage, SFTP, or Mailbox For transformation App Connect Data ETL	Connector AddOn and Custom API Integration via an IBM Expert Labs SOW
Bulk loading attachments	Load attachments in bulk, for example, linking invoices to utility account records as supporting evidence.	Attachments.	Scanned physical invoices or client, supplier, or partner file repositories.	N/A	File Loading: Cloud Storage SFTP, or Mailbox	None
Scope 3 categorization using AI Assist	Use the AI Assist (NLP) service to identify the most appropriate spend category for your Scope 3, Category 1 purchased goods and services spend data.	Transactional spend-based records.	Manually populated templates or custom report extracts from a client, supplier, or partner system.	To or from a Natural Language Processing (NLP) service	Transformation NLP Service Data ETL	Scope 3 GHG Accounting + Reporting module
Export Envizi ESG Suite data model	Offline analysis of the Envizi ESG Suite data model.	Envizi ESG Suite monthly dataset.	Envizi ESG Suite	To a Data Export service	Data Export Cloud Storage, or SFTP	Data Export AddOn
Planning Analytics integration	Integrate the monthly dataset with Planning Analytics for planning and forecasting.	Envizi ESG Suite monthly dataset.	Envizi ESG Suite	To IBM Planning Analytics	Integration Planning Analytics	Envizi Planning Analytics AddOn
Envizi Supply Chain Intelligence integration	Load Scope 3 Category 1 PCF data from Envizi Supply Chain Intelligence	Scope 3 category 1 PCF data from Envizi Supply Chain Intelligence	Envizi Supply Chain Intelligence	From Envizi Supply Chain Intelligence	File loading Cloud Storage	Envizi Supply Chain Intelligence module

The key benefits of data flow automation include the following:

Data services are dedicated for your users and suppliers and configured with unique credentials, improving the security of your file loading, data export and integration pipelines.
You can control how and when data is loaded into your environment by provisioning and managing your own data services and data pipelines,
You have the option of transforming your files into one of our standard connector formats through a fully managed file transformation service.
You have end-to-end visibility of data pipelines, allowing you to view file loading status, download original and transformed files, and access error logs, all from the Envizi ESG Suite user interface.