Introduction

IBM® WebSphere® Integration Developer (hereafter known as Integration Developer) provides adapters for integrating applications with a number of systems or infrastructures, and one of these adapters is the Flat File Adapter. It is designed to integrate applications with the file system by either reading or writing file data. Any format of data can be read or written through the use of data bindings, which transform the data between the native file format and the Service Data Objects format used by WebSphere Process Server (hereafter known as Process Server).

If your flat file content is serialized business objects, then you can read or write these without any custom data binding. For other data formats such as delimited or name value pairs, sample data bindings are provided. Creating your own custom data binding is beyond the scope of this article, but the steps included provide all of the information necessary for you to make use of any such binding, should you choose or need to create one. The sample bindings can also be used as a reference when creating your own, and they are introduced as part of the article. In this article you will use one of the sample data bindings to read in files with delimited data.

Integration Developer provides a common wizard for using any of the adapters. You will use the wizard to create an adapter component that can easily be connected with any other aspect of an integration solution by using Service Component Architecture (SCA) wiring on an assembly diagram. The adapter component performs the necessary work to read and write individual files, while the previously mentioned data bindings convert data from the native format to business objects running in Process Server.

Software requirements

You will need certain software installed and configured to be able to follow the steps in this article. This section describes how to prepare your own system.

WebSphere Integration Developer

This article uses WebSphere Integration Developer V6.0.2.2 with the latest fix at the time of writing, Interim Fix 1. You may be able to complete the steps on earlier fix packs although some steps may vary a little. If possible use the same level of maintenance, or later. This level can be reached by first installing WebSphere Integration Developer V6.0.2, then apply Fix Pack 2 to get to V6.0.2.2 using the Rational Product Updater. Finally use the same technique to install Interim Fix 1 for V6.0.2.2.

Rational Product Updater can locate, download, and install these fixes for you. However, if you do not have a fast reliable network connection, then you may wish to download the fixes manually first from the WebSphere Integration Developer support page, see the Resources section. If you choose to do this, then details are provided with each fix download that explain how to install from the local download.

Important: You must install the optional test environment if you wish to perform the final testing steps at the end of the article.

You don't need any programming experience to follow the steps in this article, but it will aid comprehension if you are already familiar with WebSphere Integration Developer and WebSphere Process Server.

WebSphere Process Server

You do not need to use a WebSphere Process Server installation to complete this article, but you will need to obtain the sample data bindings that are included in the file system under a Process Server installation.

Step-by-step guide to using the Flat File Adapter

This is the main section of the article and describes all of the steps that you need to create an application that uses the Flat File Adapter with a custom data binding. It is broken out into sections that describe each of the main steps needed to complete the article.

Adding the Flat File Adapter to your workspace

In this section, you will import the Flat File Adapter into your workspace so that you can later make us of the wizard to define your integration with the file system.

1. Start Integration Developer and open a new workspace somewhere in your file system when prompted. Short paths are recommended. Make a note of the location you choose as you will need to refer to it later.
2. Create a new module by selecting File => New => Module from the menus.
3. Provide any name for your module, for example MyModule, and click Finish.
4. From the Business Integration view, right-click on your new module and select New => Enterprise Service Discovery.
5. At this point, you are presented with a list of adapters to work with, two of which are available automatically for all workspaces. Other adapters need to be added on a per-workspace basis. We want to add the Flat File Adapter, so click Import Resource Adapter.
6. Browse to the flat file adapter resource adapter file at: <WID_HOME>\Resource Adapters\FlatFile\deploy\CWYFF_FlatFile.rar where <WID_HOME> is the base location of your Integration Developer installation on your machine.
7. Specify the name of the Connector project. The connector project name is automatically pre-filled, but you can specify a friendlier name if you like. This article uses the default, CWFF_FlatFile, as shown in Figure 1.
   
   Figure 1. Importing the Flat File Adapter into the workspace
   
   
   
8. Click Finish. Click No if prompted to switch to the J2EE perspective. At this point, the IBM WebSphere Adapter for Flat Files should now appear in your list of adapters as shown in Figure 2.
   
   Figure 2. Completed import of the Flat File Adapter
   
   
   
9. Click Cancel.
   The adapter is successfully imported into your workspace. However, more work needs to be done outside the wizard before you can proceed with the remaining enterprise service discovery wizard steps. What you just did shows you how to get started with the wizard and also how to import the resource adapter. This need only be done once per workspace. If you wanted, you could alternatively have imported the adapter directly using File => Import and selecting RAR File, rather than from within the enterprise service discovery wizard as you just did. Either way, the net result is that you now have the connector project in your workspace so you are ready to start building applications that use the adapter.

Working with data bindings

Before performing the next sequence of steps, it is useful to understand data bindings in more detail. Many of the adapters provided with Integration Developer are capable of introspecting the data source that they connect to and automatically building data binding code for you. They do this by relying on standard data formats and schemas used by systems such as databases, CICS®, or IMS® to describe their data in a known format.

Files are different because they can contain data in any format and no standard may exist for what that data looks like. It is therefore necessary to perform some additional steps with this adapter.

First, you may need to provide code that transforms between your file's data format and the business object representation of data, as used by Process Server. This code is known as a custom data binding. It is called custom because there is a standard data binding known as text/xml, that can transform between business objects and XML serialized business objects, a specific file data format. The standard data binding may be useful when you want to write out your data and read it back in again later. If you want to interface with other systems that use the file data, then it is more likely that a custom format is needed, and therefore a custom data binding. In this article you will use a sample custom data binding, so you will not need to actually write any of the data transformation code.

Second, you will need to define the business objects manually so that the data binding can successfully populate instances of them at runtime, using data from your files. You will see how to do this in this article.

Exploring the sample custom data binding for delimited data

Before using the sample custom data binding for delimited data, it is worth understanding it in some more detail. The sample that you will use is for delimited data, for example comma separated data, but that data must be structured in a certain way that the sample data binding expects. The following information describes what that structure must be.

The delimited data binding requires that the data be structured in a way that follows a certain pattern. The basic sequence must be:

BusinessObjectName, value_1, value_2, value_3 …. value_n

where BusinessObjectName is the name of the business object to be populated and value_1 through value_n are the attributes for that business object, in the order defined by the business object definition. Note that the delimiter need not be a comma, but that is what I show throughout this article.

For example, the following data could be read:

, 100, Tim Jones, 111 Swan Close, Springfield, MA

and transformed by the delimited data binding into a business object named Order, with attributes CustomerName, CustomerAddress, CustomerCity, CustomerState. Note that only the business object name need appear in the file and that the attributes can have any name, but must be defined in the correct sequence within the Order business object. An example of such a business object definition for Order is shown in Figure 3.

This basic pattern can then be extended for any subobjects or arrays of subobjects that belong to the part business object. Arrays must have a number present in the file that indicates the size of the array. The structure of the data must therefore look like this when arrays are included:

BusinessObjectName, attribute_1, attribute_2…attribute_n, ArraySize, 
ArraryBusinessObjectName, arrayobject1attribute_1, 
arrayobject_1attribute_2…arrayobject_1attribute_n, arrayobject_2attribute_1, 
arrayobject_2attribute_2…arrayobject_2attribute_n…….arrayobject_n_attribute_n

For example, the following data could be read:

, 100, Tim Jones, 111 Swan Close, Springfield, MA, 2,, book, $25, camera, $300

and transformed into business objects with definitions as shown in Figure 4:

Finally, the data binding sample allows for files that may contain multiple records that are split by line breaks as well.

Creating business object definitions

This article will read in data that looks like the example shown in Listing 1. It is slightly more complicated than the examples just shown so will require more complex business objects. It contains three separate lines each beginning with the word Order. There should be no other line breaks or word wrapping, although you may see breaks when viewing the data in this article because of the size of the page.

Order,1001,Lampen-Markt GmbHm,54 Puffy St.,Pumpkinspiceville,CA,65936,
info@lampenmarkt.de,standard,2,item,1400-300,SunFun / 1200 cm3,20,item,1400-100,
Headlight Deluxe,10,false Order,1002,Omega Soft-Hardware Markt,425 Happy-Jacks Hill,
Badger,KY,90455,info@omegasoft.de,preferred,2,item,AM2-GT,SAPSOTA FUN DRIVE 2000GT,
10,item,AM2-GTI,SAPSOTA FUN DRIVE 2000GTI,2,true Order,1000,Becker Berlin,
2417-A Fluffy St.,St. Anthony,IL,13467,becker@berlin.com,standard,2,item,100-100,
garden hose,100,item,P-100,pumpe,100,true

Looking at the data structure reveals that there are two business objects called Order and item. Each has a number of attributes that must be added to the business objects. The following steps describe how to create the business object definitions that match this structure.

1. Copy the example text from Listing 1 into a file called inputdata.txt, and save the contents somewhere on your file system. Notice that you can download the listing with this article instead of copied if you prefer (see the Resources section). Be sure that there are two line breaks so that each of the three lines begins with the word Order, No other line breaks or word wraps should be in the file. This file will be the input file that you will read in at the end of the article.
2. From the Integration Developer, right-click on your module and select New => Business Object.
3. Name the object item. Notice that the name, including case, must exactly match the name found in the file data, so it must be called item. The package and namespace do not matter, so it is fine to take the defaults for these values. However, you must use the same namespace later for the Order object if you change it here.
   
   Figure 5. Creating the item business object
   
   
   
4. Click Finish.
5. Right-click on the newly created item object in the business object editor, and choose Add attribute.
6. Name the attribute ItemNumber.
7. Repeat the previous two steps to add ItemDescription and Quantity attributes.
8. Change the type of the Quantity attribute to int. Notice that the type of each field is important, for example if you pick int then be sure that the incoming data will always be an int. You can change the type by clicking the current type and selecting a new one from the drop down list. The completed item business object should be as shown in Figure 6.
   
   
   Figure 6. Completed item business object
   
   
   
9. Save and close the item business object.
10. Create a new business object, this time called Order, by following the same steps as before. Be sure to use the same namespace that you used for the item business object.
11. Add the attributes to match the flat file format until your Order business object looks like that shown in Figure 7. Be careful that your definition matches the one shown exactly including the types for each attribute. Notice that the Items attribute should be set to type item and you should check the array box in the properties panel below the editor. The [] symbol confirms that you have selected the array option correctly.
    
    
    Figure 7. Completed Order business object
    
    
    

Adding the sample custom data binding to the Enterprise Service Discovery Wizard

Now that the business objects are ready, it is time to look towards running the Enterprise Service Discovery Wizard. But first you must enable the sample data binding within the wizard because it does not know about the sample by default. The following steps explain how to do this.

1. Switch to the J2EE perspective in Integration Developer.
2. Expand Connector Projects => CWYFF_FlatFile => connectorModule.
3. Right-click on FlatFileASI.xsd and select Open With => Text Editor as shown in Figure 8.
   
   
   Figure 8. Opening the FlatFileASI.xsd file with the test editor
   
   
   
4. Scroll down to the end of the file and insert the text from Listing 2 just before the closing </schema> tag.
   
   Listing 2. Additions to the FlatFileASI.xsd file
   

   <complexType name="DelimitedDataBindingMetadata"> <sequence minOccurs="1" maxOccurs="1"> <element name="ContentType" type="string" fixed="text/delimited1" /> <element name="DataBindingClass" type="string" fixed="com.ibm.websphere.samples.databinding.DelimitedEISDataBinding" /> <element name="TargetNamespace" type="string" /> <element name="Delimiter" type="string" /> </sequence> </complexType> <complexType name="FixedWidth1DataBindingMetadata"> <sequence minOccurs="1" maxOccurs="1"> <element name="ContentType" type="string" fixed="text/fixedwidth" /> <element name="DataBindingClass" type="string" fixed="com.ibm.websphere.samples.databinding.FixedWidthEISDataBinding" /> <element name="TargetNamespace" type="string" /> <element name="FieldWidth" type="string" /> <element name="PadCharacter" type="string" /> </sequence> </complexType> <complexType name="NameValueDataBindingMetadata"> <sequence minOccurs="1" maxOccurs="1"> <element name="ContentType" type="string" fixed="text/namevalue" /> <element name="DataBindingClass" type="string" fixed="com.ibm.websphere.samples.databinding.NameValueEISDataBinding" /> <element name="TargetNamespace" type="string" /> <element name="FieldWidth" type="string" /> </sequence> </complexType>

   
   
5. These entries add options to the wizard for each of the sample custom data binding types, text/delimited1, text/fixedwitdth and text/namevalue. They also specify the name of each of the respective sample custom data binding Java™ classes, for example com.ibm.websphere.samples.databinding.DelimitedEISDataBinding.
6. Close and save the file.

Importing the sample custom data bindings into the workspace

You must import the sample custom data bindings into your workspace before using them so that they are available to your application. The sample data bindings are provided with a Process Server installation (not Integration Developer) so you will first need to obtain them from a runtime installation. They can be found under the samples directory in the CustomEISDataBinding directory.

1. From the Integration Developer, switch back to the Business Integration perspective.
2. Right-click on your module and select File => Import.
3. Select File System and click Next.
4. Browse to the location of the sample custom data binding classes and select the top level src directory and click OK.
5. In the dialog, fully expand the tree and then check the box next to the com folder. You can also click on the word databinding at the bottom of the tree to see the code files; these are the files you should review if you wish to create your own custom data binding. Be sure that your dialog looks exactly as shown in Figure 9 except that the from directory and module names may be different. The module name should be pre-filled correctly if you right-clicked on your module to bring up the import dialog. If not, then use your browser to locate it manually.
   
   
   Figure 9. Importing the sample custom data bindings
   
   
   
6. Click Finish. At this point you will have many errors in the Problems view. Do not worry as these will go away when you run the enterprise service discovery wizard. They exist because the module that you imported the data bindings into does not have the connector project on its build path. This will be automatically corrected when you run the wizard, or you may choose to add the connector project to your module's Java build path now, as a quick fix.

Run the Enterprise Service Discovery Wizard

In this section, you will use the enterprise service discovery wizard to create an adapter component that uses your sample custom data binding and business objects to read a flat file.

1. Right-click on your module and select New => Enterprise Service Discovery.
2. Select the Flat File adapter and click Next.
3. The first item you are asked for is the Folder Name. This folder must contain the business objects you just defined, but you will need to browse to them in the file system, not the workspace directly. This means you need to know where your workspace is in your file system (you should have chosen this when you first started Integration Developer). Browse to this location and then select the folder that is named the same as your module (where you created the business objects). For example as shown in Figure 10 where my workspace directory was C:\MyDocuments\Work\workspaces\WID602\FFA_IC\ and my module was called MyModule.
4. Select UTF-8 for the Character Set.
5. Select text/delimited1 as the content type (recall that this option was added to the wizard when you edited the FlatFileASI.xsd earlier).
6. Specify the TargetNamespace to match the namespace you used for your business object definitions. If you accepted the default, then this will be http://<ModuleName >, for example http://MyModule
7. Specify the comma symbol , as the Delimiter.
8. Optionally expand Show Advanced >> and choose to change the log location and level if you wish, but be sure that the log directory exists.
   
   Figure 10. Enterprise Service Discovery Wizard Discovery Agent
   
   
   
9. Click Next.
10. Click Execute Query.
11. You should see the business objects you created. If this doesn't work, then you probably specified the Folder Name incorrectly on the previous options. Go back and correct this if needed.
12. Select the top level business object you want to work with. In this case, this is Order. You do not need to select the item child object; it will be automatically included.
13. Click Add to import list and check that your dialog looks like that shown in Figure 11.
    
    
    Figure 11. Querying the business objects using the enterprise service discovery wizard
    
    
    
14. Click Next.
15. The file is coming "Inbound" (reading a file, not writing one) so select Inbound as the Service Type.
16. The "BO location" field should be left blank. This is because when the wizard completes it will update the business objects you created earlier with additional information specific to the adapter. But if you specify a new location in this field, then the wizard will create new objects in this new location, rather than update the old objects. This will cause the new and old objects to clash with each other. For this reason, leave the field blank as shown in Figure 12.
    
    
    Figure 12. Configure Objects
    
    
    
17. Click Next.
18. The next page is the final page of the wizard, but it has many options. Only a few of these are required but others are important. Provide the following values:
    
    Table 1. Wizard options
    
    

    Event Directory	The place you want the adapter to pick up the file(s) from (must exist).
    Archive Directory	The place you want the adapter to send finished files to (must exist).
    Datasource JNDI name	Any JNDI name that points to a database datasource that you want the Flat File Adapter to use for its internal processing. The example reuses the default datasource that is part of the WebSphere Integration Developer WebSphere Process Server test environment: jdbc/BPEDB.
    Event Table Name	The name of a table within that database that will be used by the adapter to store its information. It uses the same name as the module. The table does not have to pre-exist.
    Split Criteria	How to split up the contents within each file (if at all). This is for multi-record files, for example, like the one in this article that has multiple Orders within the file, separated by new lines. New line is represented by \n so enter that value.
    Splitting Function Class Name	The name of a class that can split the data found within the file. Two are provided by IBM, and this field should be used in conjunction with the split criteria. For example, the split criteria here is by delimiter (\n) so the class name is com.ibm.j2ca.utils.filesplit.SplitByDelimiter If it was to split by size, the split criteria would be a number (of bytes) and the class name would be com.ibm.j2ca.utils.filesplit.SplitBySize
    Event Content Type	Must be set to "text/delimited1" to match the value from earlier.

    
    
    

    The relevant section of the completed dialog is shown in Figure 13.

    
    
    Figure 13. Completed wizard
    
    
    
19. Click Finish. The errors in your workspace should go away.

    Tip: If you have an error left about the namespace of the include element in Order.xsd, then you can fix this manually. Right-click on the Order business object in the Data Types section of your module and select Open With => Text Editor. Remove the line where the error is marked and save the file.

20. Open the Assembly Diagram by double clicking on it. You should see the flat file export icon on the module assembly diagram; this is your adapter component. If you select it and go to properties, as shown in Figure 14, you will see binding information for the flat file, much of which you specified in the wizard, but can change here if you wish.
    
    Figure 14. Completed adapter export with Flat File Binding
    
    
    

Important: Notice that you will have one warning about how the export is not wired to anything. Typically you must wire it to something or you will get an exception at runtime but what you wire it to is beyond the scope of this article. However, in the section below you will see a testing technique where it is initially wired to an unimplemented component for test purposes.

Testing

In this section, you unit test the adapter component using the built-in test environment and testing facilities for WebSphere Integration Developer.

1. Expand Interfaces in your module and select the FlatFileInboundInterface. Drag and drop it onto the assembly diagram canvas.
2. Select Component with no Implementation Type and click OK.
3. Right-click on the FlatFileInboundInterface export in the assembly diagram and choose wire to existing.
4. Save the assembly diagram, which should look like that shown in Figure 15.
   
   
   Figure 15. Completed Assembly Diagram
   
   
   
5. Start your test environment from the Servers view by right-clicking on it and choosing Start.
6. After it finishes starting, right-click on the server again and select Add and remove projects.
7. Select your module, and add it to your server by clicking Add.
8. Click Finish.
9. When it finishes adding to the server, right-click on your module in the Business Integration view and select Test => Attach. This allows the test component to intercept the incoming flat file contents.
10. Click Continue.
11. Expand WebSphere Process Servers, specify WebSphere Process Server 6.0, and then click Finish.
12. Drop the flat file inputdata.txt that you created earlier into the event directory that you specified earlier in the wizard. You might want to start with a file that only has one Order, not three, as you will get triple the errors (or success) with three! When the file is picked up by the adapter, the export will run the data binding and hopefully generate the business objects correctly and pass them to your Component1. Component1 has no implementation, however, the test attach uses an emulator by default for that component, so in fact the test client will intercept the call to the component1 and allow you to see what values came in and then emulate the empty component. When it intercepts, it will stop on this emulator at which point you can select the Request and see if the input data was correctly read in as shown in Figure 16. If you had three Order records in your file, then you should see all three as shown and can select each one of the Requests in turn.
    
    
    Figure 16. Successful test of the flat file adapter component
    
    
    
13. Check that the items arrays are correctly populated.
14. Select each of the Emulate steps, and click Continue to continue the test. After all records have been read, the test should end successfully and the file will have been removed from the event directory. Congratulations!

Conclusion

In this article, you learned how to use WebSphere Integration Developer and the Flat File Adapter to create applications that read in flat files for integration with other service components. You imported the Flat File Adapter and the sample custom data bindings to a new workspace. Then you created business objects for your data before running the enterprise service discovery wizard to create an adapter binding component to read input data from a file. Finally you tested your adapter component using the built-in testing capabilities of Integration Developer.

Back to top

Download

Information about download methods

Resources

• WebSphere Integration Developer support page
  
  
• WebSphere Process Integration Information Center (covers all products)
  
  
• WebSphere business integration zone
  
  
• WebSphere Process Server and WebSphere Integration Developer resource page
  
  

About the author

Richard Johnson is a Certified Consulting IT Specialist working in the WebSphere Business Process Management development organization supporting customer projects. He previously worked in CICS development, managed beta programs, and spent many years in IBM Software Services for WebSphere in the United Kingdom and the United States. You can reach Richard at rj1@us.ibm.com.

Rate this article

Comments

Back to top

Table of contents

• Introduction
• Software requirements
• Step-by-step guide to using the Flat File Adapter
• Run the Enterprise Service Discovery Wizard
• Testing
• Conclusion
• Download
• Resources
• About the author
• Comments

Next steps from IBM

• Try: WebSphere Integration Developer (in the SOA Sandbox: Connect)
• Demo: Process automation, adoption pattern
• Briefing: WebSphere flexible business processes
• Buy: WebSphere Integration Developer (in the SOA Sandbox: Connect)

Dig deeper into WebSphere on developerWorks

Try IBM PureSystems. No charge.

Experience today!

---

Get started with the cloud-based trial and pattern development kit.

Special offers