Check EDI compliance with WebSphere DataPower, Part 1
Artifacts and procedures to get you started
This content is part # of # in the series: Check EDI compliance with WebSphere DataPower, Part 1
This content is part of the series:Check EDI compliance with WebSphere DataPower, Part 1
Stay tuned for additional content in this series.
Electronic Data Interchange (EDI) compliance checking validates that incoming and outgoing X12 and EDIFACT documents comply with the formats that are published by the EDI standards development organization (SDO). The WebSphere Transformation Extender Pack for EDI – X12 and EDIFACT contains an EDI Compliance Check framework that checks incoming documents for compliance before they are transformed. You can use the framework to check newly created documents to ensure that they conform to the standard. Compliant documents should not create any issues with trading partners.
The X12 and EDIFACT Compliance Check framework can validate any EDI X12 or EDIFACT message version that the Pack supports. The framework checks the document for envelope validity.
For the X12 standard, the framework can create a TA1 acknowledgement and 997 functional acknowledgements, and also can log errors in the data.
The EDI Compliance Check framework creates acknowledgements that are generic and partner-neutral. The acknowledgements are not created by using "rules" associated with a particular partner, and they do not represent accurate control number sequencing.
You can generate partner-specific acknowledgements by extending the WebSphere DataPower policy. "Check EDI compliance with WebSphere DataPower, Part 2: Generate partner-specific acknowledgements" describes this process in detail.
Verifying the prerequisites
Ensure that your installation meets the requirements of both products.
WebSphere Transformation Extender:
- WebSphere Transformation Extender Design Studio, Version 22.214.171.124 or later
- WebSphere Transformation Extender Pack for EDI (X12 or EDIFACT), Version 126.96.36.199
The EDI Compliance Check framework is part of the WebSphere Transformation Extender Packs for EDI – X12 and EDIFACT.
- DataPower Models XI50, XI50, XB60, or XB62
- DataPower firmware Level 5.0 or later
- The WebSphere DataPower exported domain that is provided with this article as a sample-code file (see Downloadable resources)
- Free space on the DataPower appliance:
- For maps
- 2.6 MB for one EDI Standards version type
- 2 MB for each additional EDI Standards version record type
- For control files
- 4-7 MB for the support files that run the validation map
- For maps
Preparing the WebSphere DataPower appliance
This section describes the steps to enable the WebSphere DataPower (DataPower) appliance to perform EDI compliance checks. The Compliance Check framework sample-code file provided with this article was tested with both XB60/62 and XI50/52 appliances.
The sample-code file (see Downloadable resources) contains the objects that you need to prepare the DataPower appliance to perform EDI compliance checking. You also need to build and deploy WebSphere Transformation Extender maps for the DataPower runtime platform.
The sample code file contains two Multi-Protocol Gateways that you can tailor:
This gateway accepts HTTP requests and forwards the compliance-check output to an HTTP back end. It is configured to echo all of the compliance-check output back to the HTTP client as an installation verification test. After you complete the installation verification test, you can customize this gateway for your own use.
This gateway is an example of how to accept HTTP requests and forward the compliance-check output to WebSphere MQ queues. You must make configuration changes and setup MQ to use this gateway.
For the objects that you change to configure these gateways for your own use, see Imported DataPower object descriptions.
Note: Although this article focuses on the EDI X12 standard, the steps are equally applicable to EDIFACT messages.
Installing the EDI Compliance Check framework on the DataPower appliance
These sections explain how to install the sample-code file and configure the Multi-Protocol Gateway port.
Importing the sample-code file
To install the sample-code file onto the DataPower appliance:
- Copy the provided sample-code compressed file to a directory in your local workstation. Do not uncompress this file. It is in the required format to import to the DataPower appliance.
- Log in to the DataPower appliance.
- Create a new domain to contain the EDI Compliance Check framework. For the purpose of this example, use the name "X12_Compliance" as your domain name. Do not load the compliance-checking objects into the default domain.
- Switch to the newly created domain.
- Go to Administration > Configuration > Import Configuration.
- Click Choose File to locate the sample-code zip file on your local workstation.
- Click Next and then Import.
The import process automatically creates the required objects and their dependencies. Continue with the next section to complete the installation and verification steps. After you verify the installation, continue reading about the details of the imported objects and files that are contained in the sample-code file in Imported DataPower object descriptions.
Configuring the Front Side Handler to use an alternative port (optional)
The objects in the sample-code file provide a starting point to establish compliance checking on the DataPower appliance. The Front Side Handler in the sample code is configured to use port 43060 to test the installation.
- When port 43060 is not in use, the Op-State of the Multi-Protocol is set to 'up,' and you can continue by Preparing the Compliance Check framework by using Design Studio.
- If port 43060 is in use, use the following steps to change to an available port number.
To configure the Front Side Handler to use another port:
- On the DataPower appliance, go to Services > Multi-Protocol Gateway > Edit Multi-Protocol Gateway.
- Click X12_Compliance_using_HTTP Multi-Protocol Gateway to open it.
- Click and edit the Front Side Protocol X12_Compliance_using_HTTP_FSH and change the Port Number value to an available port number.
- If you made any changes to the Front Side Handler or the Multi-Protocol Gateway, remember to click Apply and save the domain configuration before you proceed.
- Go to Services > Multi-Protocol Gateway > Multi-Protocol Gateway and verify that the Op-state of the Multi-Protocol Gateway is 'up.'
Now you have set up the EDI Compliance Framework objects in the DataPower appliance. The next step is to build and deploy the WebSphere Transformation Extender maps.
Preparing the Compliance Check framework by using Design Studio
The following steps prepare the Compliance Check framework to run on the DataPower appliance.
- Configure WebSphere Transformation Extender Design Studio to connect to your DataPower appliance
- Import maps and trees from the installed directory into a project
- Build compliance-check maps for the DataPower runtime environment
- Upload framework maps and control files to the DataPower appliance
Each of these steps is described in detail in the sections that follow.
Configuring Design Studio to connect to your DataPower Device
The DataPower appliance has a built-in XML Management Interface that you can use to transfer files directly from WebSphere Transformation Extender Design Studio. Contact your DataPower administrator to make sure that the XML Management Interface is enabled. Proceed with the WebSphere Transformation Extender Design Studio configuration steps:
- Open the WebSphere Transformation Extender Design Studio and specify a new workspace (for example, C:\workspaces\DataPowerEDI).
- In the menu, click Window > Preferences.
- In the Preferences window, expand the Transformation Extender section, and then expand the Map section. Click DataPower.
- In the Host field of the Connectivity box on the right side of the window, enter the fully qualified host name or IP address of the DataPower appliance.
- Select the XML Management Interface Page tab.
- In the Host field of the Connectivity box, enter the fully qualified host name or IP address of the DataPower appliance.
- In the User field, enter the User name of the DataPower User that was provided to you by the DataPower Administrator.
- In the Password field, enter the password of the user.
- Use port 5550, the default port for the DataPower XML Management Interface. Your DataPower Administrator can confirm that the XML Management Interface is enabled and uses the default port number.
- In the Domain field, enter the domain name for the EDI compliance-checking framework, for example "X12_Compliance." You created this domain name when you imported the sample code.
- In the File Options > Default file field,
Importing maps and type trees into your workspace
These steps import the artifacts from the Pack's installation directory into your workspace, build the maps to run in the DataPower runtime environment, and upload the maps and control files to the DataPower appliance.
- Create an Extender project in your WebSphere Transformation Extender Design Studio workspace. Right-click anywhere in the Extender Navigator view and select New > Extender Project.
- In the Project name field, enter the name of your project (for example, "DataPowerPrep") and click Finish.
- Right click the new project name and select Import > File System.
- In the From directory field, navigate to the 'x12' directory in the WebSphere Transformation Extender Pack for X12 installation location and click OK.
- Expand the X12 directory. Select the compliance directory and the trees directory. (The examples directory is not required.)
- Click Finish to import the required Compliance Check framework artifacts into your project.
Building compliance-checking maps to run on the DataPower runtime environment
The maps in the Compliance Check framework are distributed as map source files. You must build the maps into executable maps before you can use them at run time. You change the map's runtime settings to indicate that the map is to run on the DataPower appliance.
To build the compliance check maps for DataPower:
- Right-click the project name and click Properties.
- In the Properties window, select Transformation Extender.
- In the MapRuntime field, select WebSphere DataPower and click Update. This changes all of the map source files in your project to the new runtime setting. Click OK.
- Double-click to open the ccx12.mms map source file found in the
compliance/mapsandschemas directory under Map Files in your project.
- In the Composition view, expand ccx12 to display the three maps:
Note: The putdatax and putmvsx maps are not used in the DataPower implementation.
- Right-click on the ccx12 map and click Build.
- In the Composition view, expand ccx12 to display the three maps:
- In the Map Files folder, open the x12segment.mms map source file. From the Map menu select Build All. A Building window displays for each map as it builds. When the Building window closes, the map-building process is complete for that map source file.
- In the Map files folder, open the x12ttval.mms map source file. From the Map menu, select Build All.
You now have built all of the maps that are necessary to run compliance checking on the DataPower appliance.
Modifying the configuration file for DataPower
The DataPower appliance uses Card Output as the default output type for the compliance-check maps. The default configuration file that is distributed with the Pack uses File Output. You must modify the configuration file to change the output type to Card Output.
- Under XML Files, right click X12DefaultConfig.xml and select Open With > Text Editor.
- Replace the three lines that start with <FileOutput> and end with </FileOutput> with one line that reads <CardOutput/>.
- Listing 1 shows the new X12DefaultConfig.xml file:
Listing 1. Modified X12DefaultConfig.xml file
<?xml version="1.0" encoding="UTF-8"> <Configuration> <RuntimeInfo> <CardOutput/> </RuntimeInfo> </Configuration>
- Save and close the X12DefaultConfig.xml file.
You will upload this file at the end of the next step.
Uploading the EDI compliance-checking artifacts to the DataPower appliance
The map-building process produces a number of .dpa files, which are binary maps that run on the DataPower appliance. You deploy the .dpa files to the DataPower appliance. The WebSphere Transformation Extender Design Studio has a built-in function to deploy maps to the DataPower appliance. To support the Compliance Check framework, there are additional non-map files that you need to deploy or upload.
- From the DataPower appliance, log in and switch to the EDI Compliance Check domain.
- On the Control Panel, open File Management.
- Expand "local:" folder and then expand the "compliance" folder.
- On the same line as "compliance," click Actions and select Create Subdirectory. Create a new folder and name it mapsandschemas.
- Navigate back to the compliance directory. Click the Actions link next to the mapsandschemas folder and select Upload Files.
- Click Choose File and navigate to your Design Studio project directory. This directory is contained within the workspace directory that you created when you configured Design Studio to connect to the DataPower appliance (for example, C:\workspaces\DataPowerEDI).
- Navigate to the compliance/mapsandschemas directory in your project workspace.
- Select ccx12.dpa and click Open.
- Click Add.
- Click Choose File > Add for the following
Note: The remainder of the segNNNN.dpa and ttvNNNN.dpa maps (where NNNN is the EDI version) provide compliance-checking support for other versions of the EDI standard. The 5010 maps are initially uploaded in this step to test the installation of the Compliance Check framework.
- When you see the files listed in the File to Upload window, click Upload. All of the selected files are uploaded to the DataPower appliance and stored in the mapsandschemas directory under the 'local:' directory.
- Add the supporting data files to a new folder under the compliance directory:
- To create the directory, click the Actions button on the same line as compliance, and select Create Subdirectory. Name the directory 'data'.
- Expand the mapsandschemas directory. On the same line of the data directory, click the Actions link and select Upload Files.
- Click Choose File and navigate to the WebSphere Transformation Extender Design Studio project directory (for example, C:\workspaces\DataPowerEDI). Navigate to the compliance/data directory.
- Select X12DefaultConfig.xml and click Open.
- When the X12DefaultConfig.xml file displays in the upload field, click Upload.
All of the artifacts that you need to run the compliance check are now uploaded to the DataPower appliance. Now you can run an installation verification test.
Verifying your Compliance Check framework installation
In this step, you use a standalone installation verification test to make sure that the compliance-checking sample-code file imported correctly, and that the maps that you built in the Design Studio deployed correctly. You can use the open source cURL tool to perform this test. See the Related topics for a link to the website where you can download cURL. To perform the test, you also need the EDI test files that are provided with the EDI Pack.
Install the cURL utility in the C:\CURL\ directory on your workstation and use the EDI X12 files that are provided with the Pack. You use cURL to do an HTTP POST of a valid EDI X12 test file to the X12_Compliance_using_HTTP Multi-Protocol Gateway. The output from the compliance check returns to the screen.
Change directory to the location of your EDI test files (shown as
C:\Data in this example), and run the following command:
C:\Data> \CURL\curl -–data-binary @x12_valid.txt
http://mydatapower.com:43060 -X POST
Where mydatapower.com is the fully qualified host name of the DataPower appliance, and 43060 is the port number of the X12_Compliance_using_HTTP_FSH Front Side Handler. Change the port number if you have modified the Front Side Handler port number.
Note that the CURL command is a single command line that is wrapped to accommodate
the width of the example. There are two dashes before the
data parameter, and one dash after.
When the Compliance Check framework is installed properly, the output displays each of the output cards wrapped in XML. The valid EDI file is contained in the <ValidData> element, and the default generated 997 response is in the <AckOut> element. Some of the outputs from the framework are empty, but they are shown for demonstration purposes.
Note: The references are an artifact of wrapping the data in XML for display purposes.
Figure 1. Example cURL tool output
If your output is the same as described above, you successfully installed the EDI Compliance Check framework on your DataPower appliance. You can continue by reading about the objects that are installed on the DataPower appliance in Imported DataPower object descriptions.
If the output contains any errors, go back and check the steps in Installing the EDI Compliance Check framework on the DataPower appliance and Preparing the Compliance Check framework using WebSphere Transformation Extender Design Studio to make sure that the framework is installed correctly on the DataPower appliance. Enabling the DataPower Multi-Protocol Gateway probe can also help you to diagnose errors.
Imported DataPower object descriptions
This section describes the objects that are created by the imported Compliance Check framework sample-code file. You can customize them for your environment.
In this section, take a look at the Multi-Protocol Gateways (MPGWs) that are created by the imported domain. Two MPGWs are created: one to service an HTTP front-end to HTTP back end, and the other to service an HTTP front-end to an MQ back end. Except for the back-end protocols and destinations, the compliance-checking functionality of both MPGWs are identical.
Each MPGW is delivered with a Front Side Handler (FSH)—one for an HTTP back end and one for an MQ back end. You can modify these handlers to suit your needs or create additional handlers. For example, you might want to use the FTP protocol to send EDI messages to the appliance. In this case, you can create an FTP FSH. Or you might want to send the EDI files securely, by enabling a secure socket layer (SSL) policy.
X12_XML_Manager is assigned as the XML Manager (as described in Dependent objects), and specifies the maximum message size that the framework can accept.
Figure 2 shows the General tab of the X12_Compliance_using_HTTP MPGW.
Figure 2. DataPower Multi-Protocol Gateway - General tab
The X12_Compliance_using_HTTP MPGW contains a processing policy called X12_Compliance_using_HTTP, and the X12_Compliance_using_MQ MPGWs contains a processing policy called X12_Compliance_using_MQ. Each policy contains a main processing rule, which consists of the following Action steps:
- Matching rule: By default, the URL-based matching rule listens for all posts by using a wild card. You can customize this rule to add a relative URI for your environment
- Filter: Because the MPGW has a dynamic back end, this step
instructs the rule to skip the back end by setting the
var://service/skip-backsidevariable to 1, so that the results actions can specify back-end URLs.
- Series of Fetch actions: This step fetches the supporting files
that are needed to run the map from the
local:directory of the appliance. The supporting files are X12 structures, X12 codes, EDI error codes, and an initialization file.
- Binary Transformation: This step is the heart of the compliance-checking. It invokes the Websphere Transformation Extender ccx12.dpa map file to validate the compliance of the incoming EDI file. The compliance-checking map has 5 named inputs and 8 named outputs that correspond to the input and output cards.
- Results action: The Results action (not shown in the sample) corresponds to the back-end transport, where the output of the compliance-checking map is delivered. In the provided HTTP MPGW, the output data is transformed into readable text for demonstration purposes.
You need to modify the actions after the Binary Transformation (step 4) to send the compliance-check output to destinations in your environment.
The sample-code file imports the following dependent objects that are referenced by the MPGWs:
- XML Manager
The import creates a XML Manager object called X12_XML_Manager, which customizes the accepted message size.
- MQ Queue Manager
The import creates a Queue Manager called QMDP, which has a default host name, port, channel, and other attributes. You must change the MQ host name to a host name and port that is valid in your environment. If you decide not to use MQ queues for the back end to receive validated X12 messages, you can disable this object in DataPower.
Customize the imported objects for your environment
You can customize the default configuration values that are provided in the framework to match your environment.
Important: Change only the objects listed below to match your environment. Changing any other objects might impact the function of the framework. If you want to perform more extensive customization, use the provided objects as models, and create your own MPGWs, processing policies, or rules based on the provided objects.
Table 1. Provided objects with descriptions and default values
|Object||Default Value(s)||Required Customization|
|URL Matching Rule|| Rule Name: MatchAll.|
URL Match: *
|Change it to specify a relative URI that is specific to your environment (for example, /edi-x12-compliance)|
|Front Side Handlers||One HTTP Front Side Handler each provided for HTTP and MQ MPGWs:||Change the port numbers as necessary. To use a different Front Side Handler (such as FTP or SFTP), create a new Front Side Handler under each MPGW.|
| HTTP Back-end URLs|
(* for X12_Compliance_using_HTTP MPGW only)
|None||In the Framework's Processing Policy, enter the actual HTTP or HTTPS URLs for your back end under the individual Results step. The back end can be a Business Process Manager V7.5, a WebSphere Process Server, or a WebSphere Enterprise Service Bus, for example.|
| MQ Queue Manager|
(* for X12_Compliance_using_MQ MPGW only)
| Queue Manager: QMDP|
Server connection channel: DP.CHANNEL
|Change the host name to your MQ host. You can also change the queue manager name, port, server connection channel, user ID, and other attributes to correspond to your environment. If you are not using an MQ back end, you can disable this object.|
| MQ back-end objects|
(* for X12_Compliance_using_MQ MPGW only)
|If you create queues with these names in your environment, no customization is necessary. Otherwise, change these to the queue names for your back-end MQ destinations in the provided XML configuration file. The WebSphere Transformation Extender Pack for X12 includes deployment scripts in the \compliance\data directory (create_x12_queues.mqsc)|
|XML Manager|| Name: X12_XML_Manager|
Maximum Node Size: 33554432
|The default node size is sufficient for file sizes up to 32 MB. Increase the value if you expect to pass larger messages through the DataPower appliance.|
Customizing the EDI Compliance Framework
After you determine which objects you want to customize for your environment, use the following procedures to make the changes.
URL matching rule
To restrict the incoming EDI HTTP traffic to a specific URI in your environment, add a new matching rule.
- Open Processing Policy X12_Compliance_using_HTTP from either of the provided Multi-Protocol Gateways.
- Double-click the Matching Rule Action step.
- Click the add button (+) and specify the details for the new matching rule.
Front Side Handler
- Go to Services > Multi-Protocol Gateway > Edit Multi-Protocol Gateway.
- Click on X12_Compliance_using_HTTP MPGW to open it.
- Click and edit Front Side Protocol X12_Compliance_using_HTTP_FSH. Change the Port Number value to the port number for your environment.
You can review and change any other front-side HTTP protocol attributes as needed. Repeat this step for X12_Compliance_using_MQ_FSH front side handler.
HTTP back-end URLs
This step applies only to X12_Compliance_using_HTTP MPGW. Skip this step if you use only the MQ back end.
- Open the Processing Policy X12_Compliance_using_HTTP from either of the MPGWs.
- Double-click each Results action to change the Destination field to match the back-end URL in your environment. The back end consumes the EDI message and other outputs (such as invalid, error, log, Ack, and TA1 messages).
Make sure you repeat this step for each provided Results action to ensure that the validation messages are delivered to the correct back-end destination.
Figure 3. DataPower Configure Results Action screen
MQ Queue Manager
This step applies only to X12_Compliance_using_MQ MPGW. Skip this step if you use only the HTTP back end.
This object must be customized to match the MQ host name and port in your environment.
- Open Objects > Network Settings and click MQ Queue Manager. Alternatively, search MQ Queue Manager from the Search field.
- Open the QMDP queue manager object and change the host name, port, queue manager name, channel name, and user name as necessary to match your environment.
If you decide not to use the MQ back end to deliver EDI messages, disable the QMDP object by clicking Disable > Apply.
MQ back-end objects
This step applies only to the X12_Compliance_using_MQ MPGW. Skip this step if you are using only the HTTP back end.
Depending on the incoming EDI message, the Compliance Check framework generates several outputs, such as valid EDI message, invalid EDI message, acknowledgement, TA1 (if applicable), errors, and a log file in XML format. If an ouput is generated, by default it is delivered to its corresponding individual queue.
As mentioned earlier, you can use the default queue names when you create these queues in your environment. To change the queue names, edit the XML configuration file that contains the queue names:
- Go to Administration > Main > File Management.
- Expand local: > compliance > data.
- Locate the EDITargetRoute.xml file that is provided with the compliance framework sample-code file. Click Edit.
In Listing 1, the XML file lists the queue names that are used as MQ destinations. The X12_Compliance_using_MQ MPGW uses the XML file to dynamically determine back-end queues. Change the queue names as necessary and Submit.
Listing 2. EDITargetRoute.xml file contents
<?xml version="1.0" encoding="UTF-8"?> <route> <wmqTarget id="dev"> <target id="valid">dpmq://QMDP/?RequestQueue=X12VALID</target> <target id="invalid">dpmq://QMDP/?RequestQueue=X12INVALID</target> <target id="error">dpmq://QMDP/?RequestQueue=X12ERROR</target> <target id="ackin">dpmq://QMDP/?RequestQueue=X12ACKIN</target> <target id="ackout">dpmq://QMDP/?RequestQueue=X12ACKOUT</target> <target id="ta1in">dpmq://QMDP/?RequestQueue=X12TA1IN</target> <target id="ta1out">dpmq://QMDP/?RequestQueue=X12TA1OUT</target> <target id="framework">dpmq://QMDP/?RequestQueue=X12LOG</target> <target id="none">dpmq://QMDP/?RequestQueue=X12VALID</target> </wmqTarget> <!-- Add more WMQ targets for other environments, ex. QA, as needed --> </route>
With additional customization of the framework, you can use the XML file to add more environments that have a separate set of queues, such as a QA environment.
XML Manager (optional)
The default XML maximum node size should be sufficient for most processing needs; however, you can change it as follows:
- Open Objects > XML Processing.
- Under Table, click XML Manager. Alternatively, search for XML Manager from the Search field.
- Click X12_XML_Manager to open it.
- In the XML Parser tab, increase or decrease the XML Maximum Node Size value as necessary.
Setting up your back end to process the validated EDI Messages
Modify the provided Multi-Protocol Gateways to forward the output of the Compliance Check framework to your back end for processing.
You are well on your way to providing the EDI Compliance Check framework as a new DataPower service. There are numerous ways to integrate the EDI Compliance Check framework into your EDI process flow. The DataPower objects that you imported into your domain are just a sample of the many ways to implement various source and target protocols.
See Related topics for more information on the EDI Compliance Check framework and configuring the DataPower appliance.
- "Check EDI compliance with WebSphere DataPower, Part 2: Generate partner-specific acknowledgements" (developerWorks, Nov 2012) builds upon the procedures you learned in this article. Learn how to add a WebSphere DataPower policy and WebSphere Transformation Extender maps to generate partner-specific EDI acknowledgements.
- ASC X12: Chartered by the American National Standards Institute (ANSI) in 1979, ASC X12 develops electronic data interchange (EDI) standards for national and global markets.
- WebSphere DataPower, Version 5 information center: Find information about the appliances in the DataPower SOA Appliances product family.
- WebSphere Transformation Extender information center: Read reference information for WebSphere Transformation Extender and related product components.
- EDI Compliance Checking - Read the Compliance Check documentation that is installed by the Pack in WTX_install_dir/docs/en/edi/1155.pdf
- EDI Compliance Checking online help - Read the latest documentation on the web.
- Download the cURL tool.