How to configure and interpret WebSphere Transformation Extender Data Audit

Resolving The Problem

Use the Data Audit within the WTX Design Studio. This will generate diagnostic information in the map audit log that indicates where input data does not conform to the type tree.

WTX provides example maps if you select the "Custom" install option during the installation of the Design Studio. To describe the use of Data Audit Settings, the example map source for states.mms in the examples\general\states subdirectory is used below to explain how to configure and interpret the Data Audit.

Start by opening the states.mms map source with the WTX Design Studio and enable the Data Audit.

To enable this:

1. Configure the Map Settings:
1. From the Menu Bar, select Map > Map Settings
2. Right click on MapAudit and select "Expand all Subtypes"
3. Set Switch to ON
4. Set BurstAudit - Data to Always
5. Set AuditLocation to File
6. It is recommended to set FileName to Unique in production environments to ensure the map will be multi-threaded
7. Click on OK to accept these values
2. Configure Data Audit Settings
1. From the Menu Bar, select Map > Data Audit Settings. This will bring the focus to the Data Audit Settings tab at the bottom of the Design Studio.
2. Drag down, to an empty row in the Data Audit Settings, the highest level of data you want to Audit. If you want to audit all the data, select the root item that is displayed for a particular input card.
3. Set the appropriate Track, Details and Item Data values for the item you want to audit. Typically you want to audit items that report an error only, so it is suggested to set these values to Error from the pull down list
3. Rebuild the map for the appropriate platform and deploy it. This process will build a map named, master.mmc.

This completes the steps necessary for the map to generate Data Audit information.

To understand how this example states.mms map source is going to be used, you first need to understand the input data structure as defined by the input type tree of states.mtt. The Design Studio displays input cards 1 structure as:

Think of an indentation in the tree format as a different level, so the items at different levels would be:
Level 0: USData
Level 1: Region
Level 2: RegionID and States
Level 3: StateID, Human and Weather
Level 4: Population Statistic, Income Statistic, Summer Statistic, Winter Statistic and Rainfall Statistic

By default, the map source for states.mms will not generate any errors so you will need to edit the input data to force an error. To do this open the sts.txt data file which has content of:

Midwest:OH/257,3738/74,31,37:IL/,4285/76,26,33:IN/142,3687/75,29,39:MI/154,3994/75,27,31:WI/80,3632/69,19,30

Mountain:MT/5,3130/68,19,11:ID/8,2953/75,29,11:WY/3,3353/70,26,15:CO/21,3604/73,29,15:NM/8,2897/79,36,10:AZ/15,3372/90,50,7:UT/13,2997/77,28,14:NV/4,4458/71,28,8
Plains:MN/48,3635/73,12,25:IA/,3549/74,20,33:MO/67,3458/78,32,35:ND/9,3012/72,10,15:SD/9,3027/75,17,15:KS/27,3488/81,32,28


To understand how the data is formatted against the type tree and using the first record in this input data as an example, a Region would be:
Midwest:OH/257,3738/74,31,37:IL/,4285/76,26,33:IN/142,3687/75,29,39:MI/154,3994/75,27,31:WI/80,3632/69,19,30

A State would be OH, IL, IN or MI. A Human group would be: 257,3738 which is composed of a Population Statistic of 257 and a Income Statistic of 3738. A Weather group would be: 76,26,33 with a Summer Statistic of 75, a Winter Statistic of 26 and a Rainfall Statistic of 33.


To introduce an error you can add several additional valid input lines and an invalid record, resulting in a new data file of:
Midwest:OH/257,3738/74,31,37:IL/,4285/76,26,33:IN/142,3687/75,29,39:MI/154,3994/75,27,31:WI/80,3632/69,19,30
Mountain:MT/5,3130/68,19,11:ID/8,2953/75,29,11:WY/3,3353/70,26,15:CO/21,3604/73,29,15:NM/8,2897/79,36,10:AZ/15,3372/90,50,7:UT/13,2997/77,28,14:NV/4,4458/71,28,8
Mountain:MT/5,3130/68,19,11:ID/8,2953/75,29,11:WY/3,3353/70,26,15:CO/21,3604/73,29,15:NM/8,2897/79,36,10:AZ/15,3372/90,50,7:UT/13,2997/77,28,14:NV/4,4458/71,28,8
Plains:MN/48,3635/73,12,25:IA/,3549/74,20,33:MO/67,3458/78,32,35:ND/9,3012/72,10,15:SD/9,3027/7x,17,15:KS/27,3488/81,32,28
Mountain:MT/5,3130/68,19,11:ID/8,2953/75,29,11:WY/3,3353/70,26,15:CO/21,3604/73,29,15:NM/8,2897/79,36,10:AZ/15,3372/90,50,7:UT/13,2997/77,28,14:NV/4,4458/71,28,8


To show how Data Audit works, the error introduced in the data will be explained below as the Data Audit information provided in the audit log are described.

When you execute, master.mmc, the resulting audit log will contain a section with a tag of <DataLog>. Using the results from the master.mmc map and the invalid sts.txt data file, this section of the audit log displays:
<DataLog>

<input card="1">
<object index="1" level="0" size="725" status="E07">USData</object>
<object index="4" level="1" size="122" status="E07">Region</object>
<object index="5" level="2" size="18" status="E07">State</object>
<object index="1" level="3" size="8" status="E07">Weather</object>
<object index="1" level="4" size="2" status="E02">Summer Statistic</object>
<Text>7x</Text>
</input>
</DataLog>


What does this mean? If you follow this audit log it walks you through the data/tree structure where the error occurred. Since USData is the entire input, it will be index 1 and it is the top level (level 0) in the type tree and the size represents all of USData which happens to be 725 bytes.

The next entry in the audit log is Index 4 of Level 1. Level 1 is a Region entry and it is the 4th occurrence of Region that was encountered in the data stream. If you review the data, this would be the Plains data. (The Midwest and two Mountain entries were the first 3 indexes of Region) This entire Plains (Region) data is 122 bytes in length:
Plains:MN/48,3635/73,12,25:IA/,3549/74,20,33:MO/67,3458/78,32,35:ND/9,30
12/72,10,15:SD/9,3027/7x,17,15:KS/27,3488/81,32,28

Next, is Level 2, index 5. This would be the State data. Index 5 of the Plains > State data is the SD state entry (MN, IA, MO and ND are the first 4 indexes for this level). The length of this 5th index data is 18 bytes
SD/9,3027/7x,17,15

Next is Level 3 which is Weather. The first occurrence (index 1) of Weather for this state is in error and the Weather data is 8 bytes in length:
7x,17,15

Finally you come to Level 4 which indicates the first component (index 1) of Weather, which happens to be Summer Statistic is invalid and it tells you what the invalid value is. In this case 7x with a size of 2 bytes. This data has an error because the states.mtt type tree has defined the Summer Statistic to be a number (integer) and 7x is not a valid number integer value.