Importing data into the IBM Counter Fraud Management system

IBM® Counter Fraud Management uses the data from your environment and systems as the object for analysis, scoring, and subsequent action. The data that you choose to manage and how you map it into ICFM is a key factor in the effectiveness of your Counter Fraud application.

The right data is a crucial part of a Counter Fraud solution. The onboarding of that data into the ICFM system is critical for not only analytical reasons but also for performance and scalability reasons. The ICFM system provides multiple ways to import data.

Incoming business data that is imported into the solution is captured in the fact store of Counter Fraud database. This fact store is provided by the schema CFFACT. The fact store is constructed as a set of domain-neutral tables that are designed to adequately capture the business data of any domain. It is not expected that the fact store is heavily customized either per business domain or per customer.

Before you determine the import techniques to use, first determine whether data will be imported, or if it will be federated. Importing data means that data is copied into the ICFM fact store. With a federation approach, the data stays in place and the ICFM system references it as if it was contained in the ICFM fact store directly. See the section on Data Federation in the product documentation.

Multiple methods exist to map and import the data from your system into ICFM:

ICFM Data Import tool

ICFM provides a data import tool that can use exported CSV files to import data, or import directly from a JDBC database. The data import tool is useful in the following scenarios:

Initial data load: If you are loading data from your own system into the Counter Fraud database in moderate size batches, you can use the data import tool. You create a mapping file between your data and the structure of the Counter Fraud database, and then run the import based on the mapping. Data Import is appropriate for test or prototype systems, or for data import instances that require smaller volume and velocity. If you want to do a large initial data load, that is, more than a few million records, consider using a separate ETL (extract, transform, and load) tool to help manage the import.
Ongoing data load: You can also use the Data Import tool to do a regular import of data. You can schedule cron jobs for a refresh of data by using the command line version of the tool.

ICFM REST services

Data importion that is based on the ICFM REST API is the most flexible method. However, it is also the slowest and has the most processor usage. REST Services data importion is most likely to be used in a scenario where interactive importion and analysis response is required. This importion method can have side effects, such as sending party information to IBM InfoSphere® Identity Insight.

Publish and subscription framework

The subscription framework is an event-driven data push from your system to ICFM. The framework uses messages to communicate the availability of data to import. When the data is imported, an analysis is initiated based on the rules that govern the framework. Use this method to capture events as they occur.

Extract, transform, and load (ETL) tool

For large data volumes, you can use an ETL tool like IBM InfoSphere DataStage®, which is provided with ICFM, to import data into the ICFM system.

How you import data depends on your goals for the operation. The following table shows the approaches for data importion and scenarios for which you might choose each approach:

Table 1. Decision criteria for a data importion method
Scenario	Type of Ingestion	ICFM Data Import	ICFM REST Services	ICFM Publish and Subscription Framework	ETL Tool
Batch load of data with multiple records loaded	Batch	Yes	Possible	No	Yes
Client application with interactive communication with ICFM	Interactive	No	Yes	No	No
Integration with investigation providers - fraud investigations that are coming in to ICFM	Event driven	No	Possible	Yes	No
Integration with data handlers - data that is going out from ICFM	Event driven	No	Possible	Yes	No
Nightly load of transactions - Low volume and velocity of data imported	Batch	Yes	Possible	Not applicable	Possible
Nightly load of transactions - Medium volume and velocity of data imported	Batch	Yes	Possible	Not applicable	Possible
Nightly load of transactions - High volume and velocity of data imported	Batch	Possible	No	Not applicable	Yes
Interactive load during online transactions - Low volume and velocity of data imported	Interactive	No	Yes	Yes	No
Interactive load during online transactions - Medium volume and velocity of data imported	Interactive	No	Yes	Yes	No
Interactive load during online transactions - High volume and velocity of data imported	Interactive	No	Yes	Possible	No
Event-driven - Low-Medium volume and velocity of incoming investigations	Event driven	Not applicable	Possible	Yes	Not applicable
Event-driven - Low-Medium volume and velocity of data outbound	Event driven	Not applicable	Not applicable	Yes	Not applicable