December 13, 2013 | Written by: Vipin Chandran
Share this post:
When I moved from business to business (B2B) to analytics earlier this year, the first thing that struck me was the one major common aspect between the two domains: the need for data transformation. Both B2B and analytics need to transform data to make it conform to certain specifications. The raw input data is never close to the final format that they can use.
My primary experience has been on IBM B2B on cloud products, including IBM Sterling Gentran, IBM Sterling Integrator on SaaS platform and IBM WebSphere Transformation Extender. The IBM Sterling products have an intuitive data transformation tool called Application Integrator or Map Editor. This tool is used to transform or translate the raw application data from an enterprise resource planning (ERP) tool to globally recognized formats like American National Standards Institute (ANSI) and Electronic Data Interchange for Administration, Commerce and Transport (EDIFACT). The tool has a coding language of its own, which is used to map or link the specific input values to output values based on the functional requirement.
The user interface of Map Editor is very user friendly and easy to learn. The flow of data from the input (left side) to the output format (right side) can be visualized using the links in the tool. Here is a screenshot of Map Editor with the links:
When I recently moved to the IBM retail analytics product on cloud, IBM DemandTec, I found that the data transformation is a need in this domain, too. It also needs to follow certain specifications that are generally based on the database structure and the application requirements. DemandTec needs the customers to send the data files in a specific format which can then be imported into IBM DB2 database software. But many times the customer’s capability to generate the data in the right format may be limited and in such cases IBM transforms the data to the required format.
IBM DemandTec is one of the best products in the retail analytics domain, with many leading retail giants around the world as its customers. For background data transformation it uses Perl and shell scripts which have been written in a very robust, foolproof and professional manner. Coming from a B2B domain, it is my opinion that a data transformation tool like IBM Sterling Map Editor is a very good alternative to Perl or shell scripts. This tool is very intuitive, has excellent visibility for the mapping of data and is really easy to maintain as well. The IBM Sterling Map Editor is so good that all you need to learn and get going is a decent aptitude and some visualization skills. A scripting language is a more specialized skill and may require extensive training to get going. The scripts written to transform the data are generally of high to very high complexity and usually require experienced hands for maintenance.
I have not worked on too many analytics products but if the transformation methodology is more or less the same across all products then analytics can take a cue from B2B in this area.
This was meant to be an ideation blog. If you have any questions or comments on this article, please connect with me on Twitter or comment below.