Contents


Develop an IBM i2 Analyze data access on-demand connector

Integrate investigation data from external data sources

Comments

IBM i2® Analyze is an extensible, scalable, and service-oriented analytical environment that is designed to provide organizations with access to intelligence when and where they need it, so that they can make faster and more informed decisions. It facilitates analysis of large volumes of data through an extensible, service-oriented environment that is designed to integrate into your existing enterprise infrastructure.

The product provides intuitive visual analysis tools to gain insight from the data that is collected from various sources. Acquiring data and transforming it into "analysis-ready" information is key to the success of a project. Data Access on Demand (DAOD) connectors extend the functionality of i2 by enabling users to visualize and investigate data from third-party providers. This approach is needed when data is provided by a web service (either REST or SOAP) or when the source of information is too large (a data warehouse, for instance).

IBM i2 Analyze models data in terms of entities, links, and properties, where:

  • Entities represent a real-world object.
  • Links associate two entities with each other.
  • Properties store a value that characterizes an entity or a link.

The data model states what entities and links can appear in the data, what properties those items can have, and the relationships that can exist between items of different types. Developing an i2 connector consists of deciding the search criteria; acquiring the input information to make up the request to the external data source; retrieve the response; and convert it from its original format into entities, links, and properties to allow visual investigation. When data is provided to i2 Analyze through a data connector, the external data source is not modified and remains complete and unchanged.

This tutorial will show you a step-by-step approach to creating an IBM i2 Analyze connector.

Setting up the development environment

To set up the development environment, the first thing you need to do is install the Developer Essentials development toolkit by following the GitHub instructions here.

To create the skeleton of a new connector for development:

  1. Create a project directory as a peer to the examples by copying a similar, existing example project directory. See Configuring the "data access on-demand" example project for instructions on how to deploy the example projects.
  2. Rename the new directory with the name of the project. Avoid spaces and special characters.
  3. Delete any temporary directories in the new project that were copied from the old project (for example, bin and build).
  4. In the new project directory, find and replace the old project name with the new project name across all directory/file names/contents, such as .settings\org.eclipse.wst.common.component and .project.
  5. If the new project does not depend on a resource that the old project did depend on, remove all references to the dependency across all file contents, for example, .settings\org.eclipse.wst.common.component and .classpath.
  6. Add the new project name to the end of master\settings.gradle.
  7. If the project depends on another resource, such as JAR files (even if it's the same as the old project's dependency), create a project dependencies block for the new project in master\build.gradle, immediately after the existing similar blocks.
  8. Run the commands/follow set up instructions that were necessary for the old project (for DAOD, addDaodDataSource, set topology flags and deploy), and modify the project name in the instructions where necessary.
  9. Test that the new project works in the same way as the example on which it was modeled.
  10. If the project's directory structure changes, update the Eclipse deployment assembly.
  11. Add a dependency.
    project(":$daSubsetRestConnector") {
        dependencies {
            compile fileTree(dir: "$sdkDir/sdk-projects/onyx-da-subset-mydaod/fragment/WEB-INF/lib", includes: ['**/*.jar'])
        }
    }
  12. Add JAR files to the WEB-INF/lib folder to ensure that they are included within the connector's WAR file.

The above instructions also apply when you port a connector's code to a new release. When the project skeleton is ready, you can start customizing your model and code.

Deploying a custom model

On a typical project, you will probably create your own IBM i2 custom model. An IBM i2 model is an entity-link-property model, and it will contain the entity and links that will be fed by the DAOD connector.

Follow the instructions below to deploy your own model within Developer Essentials. In the example instructions below, model.xml is the model file and model-ChartingSchemes.xml is the charting scheme file. As a good practice, remove spaces from the XML file names. Deploy your model:

  1. If you accept all of the default settings when you install Developer Essentials, the deployment toolkit is installed to C:\IBM\i2analyze.
    As an effect of running the command build –t deployExample, gradle generates a build folder into the path C:\IBM\i2analyze\SDK\sdk-projects\master. Copy the XML model files (for example, model.xml and model-ChartingSchemes.xml) into C:\IBM\i2analyze\SDK\sdk-projects\master\build\toolkit\configuration\fragments\common\WEB-INF\classes.
  2. Edit the file ApolloServerSettingsMandatory.properties to update the references to the XML model files: SchemaResource=model.xml and ChartingSchemesResource=model-ChartingSchemes.xml.
  3. Using a Windows command prompt with administrative privileges, run the deploy command. From C:\IBM\i2analyze\SDK\sdk-projects\master, run:
    build -t clearData --all -s onyx-server
    build -t deploy -s onyx-server
  4. Start the i2 Analyze server and log in to the IBM i2 Intelligence Portal to verify that the model reflects your changes.

Customizing the code

1

Decide which methods need to be implemented

Depending on the customer's use cases, you will need to decide which methods you want to implement. The key interfaces that need to be implemented are:

  • IExternalDataAdapter, which allows searches to an external source of data, by using the following methods:
    • dumbbellSearch(), which searches the data source for the presence of dumbbells (pairs of linked entities that match particular criteria).
    • expand(), which must not be implemented (this is only present for legacy).
    • fullTextSearch(), which searches all entities based on a keyword.
    • networkSearch(), which provides visual queries behaviors.
    • search(), which searches the external data source for information that meets a range of criteria (it is a property search).
  • IExternalDataItemRetriever, which allows searches to find the data that are linked to a specific entity:
    • getContext(), which finds the context of an entity.
    • getLatestItems(), which retrieves the most recent data.

The IExternalDataItemEditor that's provided in the Developer Essentials examples can often be used without modification, but you might need to edit it to change how card time stamps are generated.

A typical customer scenario will require the implementation of a search (full text search or property search). Once the results are displayed on the Intelligence Portal, you might want to send them to an IBM i2 Analyst's Notebook Chart. If you choose Add selection to chart, the getLatestItems method is called. If you chose Add result list to chart, the fullTextSearch method is invoked again.

When the selected items are on the chart, typically the investigation continues with subsequent expansions, so you will probably need to implement the getContext method as well. Therefore, the minimum set of methods to implement is search, fullTextSearch, getContext, and getLatestItems.

Set the functionalities you will implement in the topology file. For instance, edit the file topology.xml under directory C:\IBM\i2analyze\SDK\sdk-projects\master\build\toolkit\configuration\environment\ as follows:

<i2-data-source ar="false" id="daod5">
      <DataSource ScsSearchSupported="true" 
                  EdrsGetContextSupported="true" 
                  ScsNetworkSearchSupported="false" 
                  EdrsGetLatestItemsSupported="true" 
                  Version="0" 
                  ScsPresent="true" 
                  ScsFilteredSearchSupported="true" 
                  ScsBrowseSupported="false" 
                  Id="" 
                  ScsDumbbellSearchSupported="false" 
                  SesPresent="true" 
                  EdrsPresent="true">
        <Shape>DAOD</Shape>
        <Name>myDAOD</Name>
      </DataSource> 
</i2-data-source>

Edit the name tag with a meaningful label, as it is the name that will appear in the IBM i2 Intelligence Portal to identify your data source. Edit file ApolloServerSettingsDaodMandatory.properties under WEB-INF\classes to reflect the class names that implement the interfaces (properties IExternalDataItemRetriever and IExternalDataAdapter). Check also that ExternalDataCreateStandardEDRSRetrievalBlock=true in file ApolloServerSettingsDaod.properties under WEB-INF\classes so that your getContext method is correctly invoked.

2

Generate the mapping JAR with the new schema

You now need to create a JAR file that maps your custom IBM i2 schema so it becomes consumable by the IBM i2 Analyze server.

Generate the JAR file by running build -t generateMappingJar –x <model-file> -o <jar-file> from the master folder C:\IBM\i2analyze\SDK\sdk-projects\master:

build -t generateMappingJar -x C:\IBM\i2analyze\SDK\sdk-projects\onyx-da-subset-mydaod\common-
fragment\classes\model.xml -o C:\IBM\i2analyze\SDK\sdk-
projects\master\build\toolkit\configuration\fragments\onyx-da-subset-mydaod\WEB-INF\lib model_schema.jar

The command creates a file named model_schema.jar (use your own meaningful name in the command), that contains four XSD files (named schema1.xsd, schema2.xsd, schema3.xsd, and schema4.xsd), which define the IBM i2 data model as XML.

Make sure to generate or copy this JAR file into the lib folder of your connector:

C:\IBM\i2analyze\SDK\sdk-projects\master\build\toolkit\configuration\fragments\onyx-da-subset-mydaod\WEB-INF\lib
3

Generate JAXB classes starting from the new schema's XSD

The XSD files are useful, since you can use them to obtain Java™ classes representing the IBM i2 objects in an easy way. Here's how:

  1. Copy the XSD files inside the connector folder, into the data directory (for example: C:\IBM\i2analyze\SDK\sdk-projects\onyx-da-subset-mydaod\src\data).
  2. In Eclipse, right-click schema4.xsd, then choose Generate JAXB Classes. Following the wizard panel, you have to define in which Java package you want to save the generated classes.
  3. Troubleshooting of JAXB generation:
    • If you get an IllegalArgumentException: Illegal pattern character 'g' ... (SimpleDateFromat.compile), try adding -Duser.language=en -Duser.country=US to the default VM arguments of the default JRE (Preferences > Java > Installed JREs > default JRE).
    • If the generation fails with Could not find or load main class com.sun.tools.internal.xjc.XJCFacade, you need to set the Workspace default JRE as the JRE System library (Java build path > Libraries > JRE System library).
  4. Add to each generated class that will be used by your connector the methods equals() and hashCode(). For example, inside a Vehicle class, you will add the following:
    @Override 
    public int hashCode()
        { final int prime = 31; 
        int result = 1; 
        result = prime * result + ((itemId == null) ? 0 : itemId.hashCode()); 
        result = prime * result + (int) (itemVersion ^ (itemVersion >>> 32));
        return result; 
    } 
    
    @Override     
        public boolean equals(Object obj) {         
        if (this == obj)           
            return true;         
        if (obj == null)             
            return false;
        if (getClass() != obj.getClass())             
            return false;         
        Vehicle other = (Vehicle) obj;         
        if (itemId== null) {             
            if (other.itemId != null)               
                return false;         
        } else if (!itemId.equals(other.itemId))   
            return false;         
        if (itemVersion != other.itemVersion)             
            return false;         
        return true;
    }
  5. Open the QueryResult class and add @XmlRootElement(name="QueryResult") before @XmlAccessorType(XmlAccessType.FIELD).
    If you do not add this annotation, the following error will occur: "Unable to marshal type "com.connector.jaxb.QueryResult" as an element because it is missing an @XmlRootElement annotation."
    It is important to fix all issues that are obtained from the automatic generation of the corresponding JAXB class to avoid errors when you display the results on the chart.
4

Query the data source

If the data source you want to connect to is a SOAP web service, you can use its WSDL file to generate Java classes for each object that is retrieved from the web service. Use Eclipse tools to generate the classes by using the Generate serializable JAXB classes command from the Web Services/Generate > Java Bean Skeleton menu.

Using these generated classes, you can map the web service objects into IBM i2 domain-specific objects and build responses for each query.

If your web service uses SSL, you might need to add the web service certificate to a keystore in the application server.

If the data source is a database, you need to configure the application server with the corresponding data source. We suggest that you create a utility class to manage the JDBC connection. The data source should be added to the server.xml file, and as a resource-ref tag in file web.xml, as in this following sample code:

Listing 1. Example: server.xml
<dataSource id="Test" jndiName="ds/Test">
    <jdbcDriver libraryRef="DB2JCC4Lib"/>
    <properties.db2.jcc currentSchema="DBSCHEMA" databaseName="DBNAME" 
password="myPwd" portNumber="50000" serverName="localhost" user="db2admin"/>
</dataSource>
<library id="DB2JCC4Lib">
    <fileset dir="C:\IBM\SQLLIB\java" includes="db2jcc4.jar
db2jcc_license_cu.jar"/>
</library>
Listing 2. Example: web.xml
<resource-ref>
    <description>DataSource</description>
    <res-ref-name>ds/Test</res-ref-name>
    <res-type>javax.sql.DataSource</res-type>
    <res-auth>Container</res-auth>
</resource-ref>

A utility class to manage the connection can be useful in the case of REST web services.

We also suggest putting the connector's specific property files under WEB-INF and by using the following declaration to access them:

private static final String PROPERTY_FILE_NAME = "WEB-INF/myProperty.properties";

5

Build a response

Each time that you query a data source in Java, you obtain a result set that can be mapped to Java objects. These objects need to be converted into IBM i2 objects to build a response for the IBM i2 charts (adding them to QueryResult).

We found it useful to follow the Builder GoF design pattern, so we defined a QueryResultBuilder class that handles response creation and XML serialization. This class exposes the following methods:

  • addMyObject(), which allows adding an object (an entity or a link) starting from the result set by using the following algorithm:
    • Retrieve from the QueryResult instance a collection of entities or links.
    • If the QueryResult instance is null (it is the first invocation of the method), create an object set of entity/link type — new Set<MyObject> (to make it possible to de-duplicate elements natively).
    • If the collection's size is not null, copy the content into Set<MyObject>.
    • Iterate the result set to build entity/link instances.
    • If the element was built without errors (itemId field is not null), add it to Set<MyObject>.
    • Invoke on the collection the methods clear() and addAll().
    • Set the new collection into a QueryResult instance.
  • startBuilding(), which:
    • Creates an instance of QueryResult.
    • Creates a Set<ContextResultMapping>, which allows the graphical representation of the expansion results (it is an IBM i2 Java object that maps a QueryResult  to an IBM i2 context).
    • Saves ProvenanceGroupIds (a ProvenanceGroup is an IBM i2 object that contains metadata representing provenance data source).
  • stopBuildingForSearch(), which executes the marshalling of a QueryResult object into a StreamSource (a specific format that IBM i2 can understand).
  • stopBuildingForExpand(), which:
    • Executes the marshalling of QueryResult into a StreamSource
    • Iterates on ProvenanceGroupIds to create ContextResultMappings.

The effective building of elements (entities or links) is delegated to the QueryResultBuilderHelper class to keep high cohesion and low coupling.

6

Cache data to avoid multiple identical calls

The web service you invoke might not be free of charge— for every call you make you might need to pay a toll. Normally, there is only one call to the web service to retrieve all data. However, typically an IBM i2 search will need the retrieval of one entity only. If you need to expand the context of that entity, make a new call to the web service.

Furthermore, we pointed out that a new fullTextSearch is invoked when we add result lists to our chart. To avoid multiple calls to the web service when they are not needed, it can be useful to consider caching data in a database for subsequent expansions (that is, get context). A new call to the web service will be made only for new searches or when you need the latest data (for instance, you can set an expiration time for the cached data).

7

Manage multiple connectors

IBM i2 allows managing multiple external connections. If the customization you are developing requires connecting to more than one data source (when it requires more than one connector), and the data model contains contact points (items that are common to more data sources), you can manage multiple connections with these simple steps:

  1. Change the generated JAXB classes for IBM i2 objects by deleting namespaces references from classes CardProvenance.RetrievalBlocks, RetrievalBlock.RetrievalData, and RetrievalDatum.
  2. Inside the QueryResultBuilderHelper methods that are related to the contact points among the data sources, create one or more RetrievalBlocks, where the value of property i2.EDRS.DS is the dataSourceGUID of the other data source from which the element can be retrieved. We suggest that you use a property file to save the GUIDs, as they change from one environment to the other, and must be set with the correct value when you are deploying in production.

With these changes, it is possible to expand an entity from inside the notebook, by choosing which data source to query (right-click on the entity), or to expand an entity among all data sources (Show context). However, this approach could break the compatibility with the Intelligence Portal.

To remove namespaces inside the RetrievalBlock class, replace

@XmlElement(name = "RetrievalBlock", namespace = "http://www.i2group.com/Schemas/2013-10-03/ModelData/Provenance")

with @XmlElement(name = "RetrievalBlock")

For the CardProvenance class, please leave the following line:

@XmlType(name = "CardProvenance", namespace =
"http://www.i2group.com/Schemas/2013-10-03/ModelData/Provenance", propOrder = 
{})

For the RetrievalDatum class, replace

@XmlType(name = "RetrievalDatum", namespace = "http://www.i2group.com/Schemas/2013-10-03/ModelData/Provenance", propOrder = { "value" })

with @XmlType(name = "RetrievalDatum", propOrder = { "value" })

For RetrievalBlock class, replace

@XmlElement(name = "RetrievalDatum", namespace = "http://www.i2group.com/Schemas/2013-10-03/ModelData/Provenance")

with @XmlElement(name = "RetrievalDatum")

Note: Please check that property ValidateXml is set to false in the connector's property file C:\IBM\i2analyze\SDK\sdk-projects\onyx-da-subset-mydaod\fragment\WEB-INF\classes\ApolloServerSettingsDaodMandatory.properties

You can find the data source's GUID in the file:

C:\IBM\i2analyze\SDK\sdk-projects\master\build\toolkit\scripts\gradle\build\autogenFragments\onyx-da-subset-mydaod\WEB-INF\classes\DataSourceDaod.properties or by checking the property files under directory C:\IBM\i2analyze\SDK\sdk-projects\master\build\toolkit\configuration\environment\dsid.

Deploying the custom connector to a production environment

Use the following instructions to deploy your connector to a production's environment.

Copy the connector's sub-directory (such as onyx-da-subset-mydaod) of the development's machine, from directory C:\IBM\i2analyze\SDK\sdk-projects\master\build\toolkit\configuration\fragments.

If you have a data source set into your application server, you also have to copy the web.xml descriptor file under C:\IBM\i2analyze\SDK\sdk-projects\onyx-da-subset-mydaod\fragment\WEB-INF. We suggest you use this directory for property files as well, since all files contained into this directory are built into the WAR file to be deployed into the Liberty server.

Check also that ExternalDataCreateStandardEDRSRetrievalBlock is set to true in file ApolloServerSettingsDaod.properties under WEB-INF\classes.

1

Create the DAOD connector's structure

You can now start to deploy the connector in production.

Create the structure that represents your DAOD connector. To do this, run the following command from C:\IBM\i2analyze\toolkit\scripts:

setup -dn <connector-dir> -t addDaodDataSource

For example:
C:\IBM\i2analyze\toolkit\scripts>setup -dn onyx-da-subset-mydaod -t addDaodDataSource

2

Copy the connector's fragment

You now need to paste the folder you copied from your dev machine into C:\IBM\i2analyze\toolkit\configuration\fragments.
For example, copy:
onyx-da-subset-mydaod from
C:\IBM\i2analyze\SDK\sdk-projects\master\build\toolkit\configuration\fragments
of the dev machine under
C:\IBM\i2analyze\toolkit\configuration\fragments.

3

Modify the i2 topology

Edit the topology.xml file by changing the DataSource tag according to the features implemented by your connector. You can choose the connector's name to be displayed in the portal. For example:

<DataSource
ScsSearchSupported="true" EdrsGetContextSupported="true" 
ScsNetworkSearchSupported="false" EdrsGetLatestItemsSupported="true" 
Version="0" ScsPresent="true" ScsFilteredSearchSupported="true"
ScsBrowseSupported="false" Id="" ScsDumbbellSearchSupported="false"
SesPresent="true" EdrsPresent="true">
    <Shape>DAOD</Shape>
    <Name>MyConnector</Name>
</DataSource>

Note: If you have more than one connector to deploy, repeat steps 1-3 in order for the other connectors after they are completed for the previous one.

4

Stop the application server

Stop the Liberty server to allow the deployments, by running: setup -t stopLiberty -s onyx-server.

5

Add data sources to the Liberty configuration file

If you have data sources set, you have to add them manually into server.xml. For example:

<dataSource id="Test" jndiName="ds/Test">
    <jdbcDriver libraryRef="DB2JCC4Lib"/>
    <properties.db2.jcc currentSchema="DBSCHEMA" databaseName="DBNAME" 
password="myPwd" portNumber="50000" serverName="localhost" user="db2admin"/>
</dataSource>
<library  id="DB2JCC4Lib">
    <fileset  dir="C:\IBM\SQLLIB\java" includes="db2jcc4.jar
db2jcc_license_cu.jar"/>
</library>

Note: Check that web.xml can be found under C:\IBM\i2analyze\toolkit\configuration\fragments\onyx-da-subset-mydaod \WEB-INF.

Customize your connector's property files if needed.

6

Deploy and restart the application server

Run the following commands:

  1. setup -t deploy -s onyx-server
  2. setup -t startLiberty -s onyx-server

Troubleshooting and debugging

The following are a few suggestions regarding the custom connector's troubleshooting, based on our experience.

  1. Change the log level for your connector's logs.
    • This assumption is based on the fact that you followed good development practices to enhance serviceability and write log traces. If you choose to use log4j, then to change the log level, edit the file log4j.properties, under C:\IBM\i2analyze\deploy\wlp\usr\servers\i2analyze\apps\onyx-da-subset-mydaod.war\WEB- INF\classes.
    • Append log4j.logger.yourpackage=DEBUG, APOLLOLOG, where yourpackage is the Java package containing the source code of your connector.
    • Restart the Liberty server.
    • You can find the log files with traces at the DEBUG level under C:\IBM\i2analyze\deploy-dev\wlp\usr\servers\i2analyze\logs\onyx-da-subset-mydaod.war
  2. If you perform a search, but no results are rendered into the portal you could check the following:
    • That a QueryResult object is contained into the XML result.
    • That you have correctly set the security tags for your items, according to your security schema, and they allow users to view the items.
    • That the jar file that maps your custom IBM i2 schema is up to date.
    • That the secondary search properties ExternalDataSourceSearchMode and ExternalDataSourceFilteredSearchMode into the DAOD property file ApolloServerSettingsDaod.properties are set to 1. If they are set to 0, then a second search is performed on the search results, in order to highlight the search string in the result set. In case you have not returned the search string itself in the results, then the second search will not find any results, and no items will be displayed.
  3. If you notice that the “show context” feature is disabled on the portal, check if the ExternalDataCreateStandardEDRSRetrievalBlock property in the ApolloServerSettingsDaod.properties file is set to true. Check the same property if the “show context” feature is enabled but you cannot see the expected results. After you have changed the value of this property, you need to perform again a search so that the retrieval blocks are correctly set.
  4. When invoking a web service via SSL, you could get a NullPointerException during connect, or a WebServiceException caused by a SSLHandshakeException and a CertificateException. In this case, you need to add the web service certificate to the application server’s keystore.

Conclusion

Connectors' development extend the functionality of i2 Analyze, which is a product of the Security family. The product enables users to perform visual investigations of their data, and through the connectors it is possible to integrate several products of the Security family.

In conclusion, i2 Analyze allows you to make smart, informed business decisions by providing visual analysis tools to gain insight from your data. By developing i2 connectors, you can enrich your investigation with information from third-party data sources.


Downloadable resources


Related topics


Comments

Sign in or register to add and subscribe to comments.

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Security
ArticleID=1050999
ArticleTitle=Develop an IBM i2 Analyze data access on-demand connector
publish-date=11032017