WebSphere Adapter for Flat Files - a technical deep dive, Part 1: How to configure for inbound processing

This 2-part article series describes a customer solution that extensively used the inbound and outbound processing of WebSphere® Adapter for Flat Files. Part 1 explains how to sequence and order the files and the different possibilities to configure the polling mechanism.

Susann Fritzsche (susann.fritzsche@de.ibm.com), IT Specialist , IBM

Photo of Susann FritzscheSusann Fritzsche is an IT Specialist on the IBM Software Services for WebSphere (ISSW) team in IBM Germany. Her focus is on Business Process Management and WebSphere Enterprise Service Bus.



Katrin Hagemann (katrin.hagemann@de.ibm.com), IT Specialist, IBM

Photo of Katrin HagemannKatrin Hagemann is an IT Specialist for IBM Software Services for WebSphere (ISSW). Before joining the ISSW Germany team, she was a Software Engineer in IBM's Development Lab in Boeblingen, Germany. Her current focus is WebSphere Business Integration with WebSphere Enterprise Service Bus. Katrin has worked successfully in several customer engagements.



15 January 2014

Introduction

IBM® Integration Designer includes various resource adapters to integrate several systems with each other. In our customer project, we used the inbound and outbound process of IBM WebSphere Adapter for Flat Files (hereafter called Flat File adapter). The Flat File adapter allows IBM Business Process Manager or WebSphere Enterprise Service Bus modules to interact with the files in the local file system without the need for custom file interaction coding.

In this 2-part series, we will show some of the challenges and how we solved them. We will also provide step-by-step examples for the solution.

In Part 1, the following problems are addressed:

To fully benefit from this article, you need a basic understanding of WebSphere Adapter, especially WebSphere Adapter for Flat Files. Your technical experience should also include good integration implementation skills with WebSphere Enterprise Service Bus (ESB).

Prerequisites

You need the following software to use the instructions in this article:

  • IBM Integration Designer V8.0.1
  • IBM Business Process Manager Advanced V8.0 or IBM WebSphere Enterprise Service Bus V8.0
  • IBM WebSphere Adapter for Flat Files V7.5.0.3

Sequencing the inbound file processing over the whole processing flow

Figure 1 shows the challenge of sequencing the inbound file processing over the whole processing flow, including the inbound FlatFile adapter and subsequent mediation flow. The "FlatFileExport" export is triggered when files are available in the event directory that is configured. In this case, the files are processed separately and successively. This means the next flat file is propagated to the mediation flow only when the processing of the predecessor file is finished by the mediation flow.

Figure 1. Sequence inbound file processing - scenario
Sequence inbound file processing - scenario

Resolution

To resolve this challenge, three configurations for "FlatFileExport" are necessary:

  • Type of delivery
  • Interaction style
  • JoinTransaction

When the "Type of delivery" configuration option is set to the value of "ORDERED", just a single thread is used to deliver the events to the mediation flow component (MFC). In comparison, the "UNORDERED" value causes the adapter to deliver the events simultaneously to the MFC by using multiple threads.

The second configuration, "Interaction style", determines how the export and MFC interact with each other. When you set the value to "sync", the transaction is spawned over both export and MFC, and the next event can only be delivered if the transaction before is committed. When you set the value to "async", the transaction of the export is committed immediately after the event is delivered to MFC.

Note: Set the "JoinTransaction" qualifier of the MFC interface to "true" when you set the "Interaction style" option to "sync". This ensures that the MFC joins the spawned transaction of the adapter.

Tutorial

  1. Import the project interchange file, Sequence_Example_1_PI.zip, which is provided in the Download section of this article.
  2. The project interchange file contains the "CWYFF_FlatFile" adapter connector module and the "Sequence_Example" mediation module.
    In the mediation module, there are two files: "FlatFileExport" export and "Sequence_Example" MFC.

    The MFC (Figure 2) contains just a CustomMediation primitive that waits 5 seconds and prints the current time and file name before and after the wait.The export is configured with the following options:

    • Type of delivery: UNORDERED
    • Interaction style: async
    • Event directory: C:\flatFileEventDir

    Note: You have to create the event directory or adjust the directory path.

    Figure 2. MFC Sequence_Example
    MFC Sequence_Example
  3. Deploy and test the solution on your local environment by copying two text files (file1.txt and file2.txt) into the event directory. Remember the statements that the CustomMediation primitive prints to SystemOut.log (Configuration: Type of delivery = UNORDERED; Interaction style = async).
  4. Set the Type of delivery to ORDERED:
    1. Select the export on the assembly diagram.
    2. Open in the Properties > Binding > End-Point Configuration.
    3. Click Advanced >>.
    4. In the "Event delivery configuration" section, set the "Type of delivery" option to ORDERED (Figure 3).
    5. Save the changes.
    Figure 3. Type of delivery
    Type of delivery

    Click to see larger image

    Figure 3. Type of delivery

    Type of delivery
  5. Test the solution by copying two text files (file1.txt and file2.txt) into the event directory. Remember the statements that the CustomMediation primitive prints to SystemOut.log (Configuration: Type of delivery = ORDERED; Interaction style = async).
  6. Set the Interaction style of the export to sync:
    1. Select the export on the assembly diagram.
    2. Open in the Properties > Binding > Performance Attributes.
    3. Set the "Interaction style" option to sync (Figure 4).
    4. Save the changes.
    Figure 4. Interaction style
    Interaction style
  7. Test the solution by copying two text files (file1.txt and file2.txt) into the event directory. Remember the statements that the CustomMediation primitive prints to SystemOut.log (Configuration: Type of delivery = ORDERED; Interaction style = sync).
  8. Compare the following log statements in Listings 1 to 3.
    Listing 1. Unordered + async
    Current time before wait: Fri Jun 21 14:22:40 CEST 2013; filename:file1.txt
    Current time before wait:Fri Jun 21 14:22:40 CEST 2013; filename:file2.txt
    Current time after wait:Fri Jun 21 14:22:45 CEST 2013; filename:file1.txt
    Current time after wait:Fri Jun 21 14:22:45 CEST 2013; filename:file2.txt

    Result: Multiple threads are used to deliver the events. Files will be processed concurrently by the MFC.

    Listing 2. Ordered + async
    Current time before wait:Fri Jun 21 14:25:03 CEST 2013; filename:file1.txt
    Current time before wait:Fri Jun 21 14:25:05 CEST 2013; filename:file2.txt
    Current time after wait:Fri Jun 21 14:25:08 CEST 2013; filename:file1.txt
    Current time after wait:Fri Jun 21 14:25:10 CEST 2013; filename:file2.txt

    Result: A single thread is used to deliver the events. The second event is immediately delivered after the first event. Files will be processed simultaneously by the MFC.

    Listing 3. Ordered + sync
    Current time before wait:Fri Jun 21 14:26:41 CEST 2013; filename:file1.txt
    Current time after wait:Fri Jun 21 14:26:46 CEST 2013; filename:file1.txt
    Current time before wait:Fri Jun 21 14:26:46 CEST 2013; filename:file2.txt
    Current time after wait:Fri Jun 21 14:26:51 CEST 2013; filename:file2.txt

    Result: A single thread is used to deliver the events. The second event is only delivered after the first event is also processed by the MFC. Files will be processed in sequence by the MFC.


Configuring the Flat File adapter to process the inbound files in a sorted order

Figure 5 shows the scenario for the challenge of configuring the Flat File adapter to process the inbound files in a sorted order. The "FlatFileExport" export is triggered when files are available in the event directory that is configured. In this case, the files are processed in a sorted order, either by timestamp or by file name.

Figure 5. Sorted inbound file processing scenario
Sorted inbound file processing scenario

Resolution

To define the sorting method for "FlatFileExport", you need to edit the value of the "Retrieve files in sorted order" property. The possible values are:

  • File name: Sort in ascending order on file name.
  • Time stamp: Sort in ascending order from the last modification time stamp.
  • No Sort: Not sorted.

Note: See the technote on Flat File adapter does not accurately sort by timestamp with timestamps are within the one second of one another.

Tutorial

  1. Import the project interchange file, Sort_Example_1_PI.zip, from the Download section of this article.
  2. The project interchange file contains the "CWYFF_FlatFile" adapter connector module and the "Sort_Example" mediation module.
    In the mediation module, there are two files: "FlatFileExport" export and "Sort_Example" MFC. The MFC (Figure 6) contains just a CustomMediation primitive that prints the file name. The export is configured with the following options:
    • Retrieve files in sorted order: No Sort
    • Event directory: C:\flatFileEventDir
    Note: You have to create the event directory or adjust the directory path.
    Figure 6. MFC Sort_Example
    MFS Sort_Example
  3. Deploy and test the solution on your local environment by copying three text files (file1.txt, file2.txt, and file3.txt) into the event directory. Remember the statements that the CustomMediation primitve prints to SystemOut.log (Configuration: Retrieve files in sorted order = No Sort).
  4. Set the Retrieve files in sorted order to File Name:
    1. Select the export on the assembly diagram.
    2. Open in the Properties > Binding > End-Point Configuration.
    3. Click Advanced >>.
    4. Set the "Retrieve files in sorted order" option to File name (Figure 7).
    5. Save the changes.
    Figure 7. Retrieve files in sorted order
    Retrieve files in sorted order
  5. Deploy and test the solution on your local environment by copying three text files (file1.txt, file2.txt, and file3.txt) into the event directory. Remember that the statements that the CustomMediation primitive prints to SystemOut.log
    (Configuration: Retrieve files in sorted order = File name).
  6. Set the Retrieve files in sorted order to time stamp:
    1. Select the export on the assembly diagram.
    2. Open in the Properties > Binding > End-Point Configuration.
    3. Click Advanced >>.
    4. Set the "Retrieve files in sorted order" option to Time stamp.
    5. Save the changes.
  7. Deploy and test the solution on your local environment by copying three text files (file1.txt, file2.txt, and file3.txt) into the event directory. Remember the statements that the CustomMediation primitive prints to SystemOut.log. (Configuration: Retrieve files in sorted order = Time stamp).
  8. Compare the following log statements in Listings 4 to 6.
    Listing 4. No Sort
    [02.07.13 14:39:22:117 CEST] SystemOut     O filename:file2.txt
    [02.07.13 14:39:22:120 CEST] SystemOut     O filename:file3.txt
    [02.07.13 14:39:22:120 CEST] SystemOut     O filename:file1.txt

    Result: The Flat File adapter retrieves files as they occur in the event directory.

    Listing 5. File name
    [02.07.13 14:41:19:091 CEST] SystemOut     O filename:file1.txt
    [02.07.13 14:41:20:838 CEST] SystemOut     O filename:file2.txt
    [02.07.13 14:41:22:850 CEST] SystemOut     O filename:file3.txt

    Result: The Flat File adapter retrieves files sorted by name in ascending order.

    Listing 6. Time stamp
    [02.07.13 14:45:54:814 CEST] SystemOut     O filename:file3.txt
    [02.07.13 14:45:56:568 CEST] SystemOut     O filename:file1.txt
    [02.07.13 14:45:58:561 CEST] SystemOut     O filename:file2.txt

    Result: The Flat File adapter retrieves files sorted by the last modification date in ascending order. Figure 8 shows the last modification dates of the test files.

    Figure 8. Last modification dates of test files
    Last modification dates of test files

Configuring the polling interval by using a Business Calendar

The "Interval between polling periods" property of the Flat File adapter allows you to define continued intervals in which the adapter polls for new files. For example, the adapter polls every two seconds. When you have the need to define non-continuous poll intervals, such as workdays between 8:00am and 5:00pm every 5 minutes, the "Polling based on business calendar" option is a better choice.

We faced this issue during the project because the upload or copy of files took some time because of the file size. With the business calendar option, we can define such granular and proper intervals so that the upload and the file processing are executed in separate time frames.

Note: Another solution to prevent the adapter from reading in incomplete files is to use the "Time interval for polling unchanged files (in milliseconds)" property. This property is by default set to 0. When you set a value, such as 10000, this means that the last modification date of the file must be older than 10000 milliseconds before the adapter processes this file.

Figure 9 shows the scenario of configuring the polling interval by using a Business Calendar. The "FlatFileExport" export is triggered when files are available in the event directory that is configured. In our case, the adapter polls for new files for workdays between 8:00am and 5:00pm for 2 minutes every 10 minutes. This means the adapter polls:

  • Monday, 08:00am – 08:02am
  • Monday, 08:10am – 08:12am
  • Monday, 08:20am – 08:22am
  • Friday, 04:40pm – 04:42pm
  • Friday, 04:50pm – 04:52pm
Figure 9. Inbound file processing based on business calendar - scenario
Inbound file processing based on business calendar - scenario

Resolution

By using a business calendar for polling intervals you can define non-continuous intervals, such as "Workdays between 8:00am and 5:00pm for 2 minutes every 10 minutes".

To realize this interval, you have to define a business calendar with an interval that runs every workday between 8:00am and 5:00pm. Furthermore, you have to define an exception so that the adapter does not poll for 8 minutes every 10 minutes.

Figure 10 shows the time frames of the interval in green, the time frames the adapter does not poll in red, and the next time frames the adapter polls in blue.

Figure 10. Execution time frames of the business calendar
Execution time frames of the business calendar

Click to see larger image

Figure 10. Execution time frames of the business calendar

Execution time frames of the business calendar

More information is found in the Information topic of Working with business calendar.

Tutorial

  1. Import the project interchange file, BusinessCalendar_Example_1_PI.zip, from the Download section of this article.
  2. The project interchange file contains the "CWYFF_FlatFile" adapter connector module and the "BusinessCalendar_Example" mediation module.
    In the mediation module, there are two files: the export "FlatFileExport" and the MFC "BusinessCalendar_Example". To show that the adapter polls just in the defined interval, the adapter and the MFC are configured as follows:
    • Type of delivery: ORDERED
    • Interaction style: sync
    • Event directory: C:\flatFileEventDir
    • Maximum events in polling period: 1
    • Interval between polling periods (in milliseconds): 2000

    The CustomMediation primitive contained in the MFC (Figure 11) blocks the processing of the file for 119 seconds and prints the current time and file name before and after the wait.
    In the tests, you copy three files to the event directory, but just one file is processed per polling interval. The other files are processed in subsequent intervals.

    Note: You have to create the event directory or adjust the directory path.

    Figure 11. MFC BusinessCalendar_Example
    MFC BusinessCalendar_Example
  3. Deploy and test the solution on your local environment by copying three text files (file1.txt, file2.txt, and file3.txt) into the event directory. Remember the statements that the CustomMediation primitve prints to SystemOut.log. (Configuration: Type of delivery = ORDERED, Interaction style = sync, Maximum events in polling period = 1, Intervall between polling = 2000).
  4. Create a business calendar:
    1. Select File > New > Module.
    2. Define BusinessCalendar for the module name.
    3. Click Finish.
    4. Right-click the module BusinessCalendar and select New > BusinessCalendar.
    5. Define FlatFileAdapterBusinessCalendar for the name.
    6. Click Finish.
  5. Configure the business calendar by adding an interval and an exception:
    1. In the Business Calendar Editor, click create an interval manually (Figure 12).
      Figure 12. Business Calendar Editor
      Business Calendar Editor
    2. Configure the interval as follows (Figure 13):
      1. Interval name: Weekdays between 8:00am and 5:00pm
      2. Schedule: Weekly
      3. Begins: <Date>08:00:00 AM; Ends: <Date> 05:00:00 PM
      4. Repeat weekly every 1 week.
      5. Select the Repeat on the selected weekdays checkbox and the following checkboxes: Mo, Tue, Wed, Thu, Fri.
      6. Select the Repeat forever option.
      Figure 13. Configuration for the “Weekdays between 8:00 am and 5:00 pm” interval
      Configuration for the
    3. Add an exception:
      1. Click Add Exception.
      2. Select the 1 minute every 10 minutes sample (Figure 14).
      Figure 14. Business Calendar samples
      Business Calendar samples
    4. Configure the exception as follows (Figure 15):
      1. Interval name: For 8 minutes every 10 minutes
      2. Schedule: By Minute
      3. Begins: <Date>08:02:00 AM; Ends: <Date> 08:10:00 PM
      4. Repeat by minute: Every 10 minutes
      5. Select the Repeat forever option.
      Figure 15. Configuration for exception “For 8 minutes every 10 minutes"
      Configuration for exception
  6. Define the business calendar for the Flat File adapter:
    1. Select FlatFileExport on the assembly diagram.
    2. Go to Properties > Binding > Endpoint configuration.
    3. Click Advanced>>.
    4. Click Browse for the Polling based on business calendar configuration option and select FlatFileAdapterBusinessCalendar (Figure 16).
    5. Save all changes.
    Figure 16. Polling based on business calendar
    Polling based on business calendar
  7. Deploy and test the solution on your local environment by copying three text files (file1.txt, file2.txt, and file3.txt) into the event directory. Remember the statements that the CustomMediation primitive prints to SystemOut.log.(Configuration: Type of delivery = ORDERED, Interaction style = sync, Maximum events in polling period = 1, Polling based on business calendar = FlatFileAdapterBusinessCalendar).
  8. Compare the following log statements in Listings 7 to 8.
    Listing 7. Interval between polling periods = 2000 milliseconds
    Current time before wait:Thu Jul 04 15:56:30 CEST 2013; filename:file1.txt
    Current time after wait:Thu Jul 04 15:58:29 CEST 2013; filename:file1.txt
    Current time before wait:Thu Jul 04 15:58:29 CEST 2013; filename:file2.txt
    Current time after wait:Thu Jul 04 16:00:28 CEST 2013; filename:file2.txt
    Current time before wait:Thu Jul 04 16:00:29 CEST 2013; filename:file3.txt
    Current time after wait:Thu Jul 04 16:02:28 CEST 2013; filename:file3.txt

    Result: The adapter polls in continuous intervals for new files and processes them.

    Listing 8. Polling based on business calendar
    Current time before wait:Thu Jul 04 14:50:02 CEST 2013; filename:file1.txt
    Current time after wait:Thu Jul 04 14:52:01 CEST 2013; filename:file1.txt
    Current time before wait:Thu Jul 04 15:00:01 CEST 2013; filename:file2.txt
    Current time after wait:Thu Jul 04 15:02:00 CEST 2013; filename:file2.txt
    Current time before wait:Thu Jul 04 15:10:01 CEST 2013; filename:file3.txt
    Current time after wait:Thu Jul 04 15:12:00 CEST 2013; filename:file3.txt

    Result: The adapter polls for the new files for workdays between 8:00am and 5:00pm for 2 minutes every 10 minutes. Files that cannot be processed during the polling interval are processed in the subsequent intervals. Between the two polling intervals configured on the business calendar, no files were processed.

Download files

Table 1 gives an overview about project interchange (PI) files that you can download. For each challenge the article addresses, there is a PI file you can start with to create the solution on your own. There is also a PI file with the solution already created.

Table 1. Downloadable files
File Description
Sequence_Example_1_PI.zip Start project interchange file for sequencing the inbound file processing over the whole processing flow, including the inbound FlatFile adapter and subsequent mediation flow.
Sequence_Example_2_PI.zip Solution project interchange file for sequencing the inbound file processing over the whole processing flow, including the inbound Flat File adapter and subsequent mediation flow.
Sort_Example_1_PI.zip Start the project interchange file for configuring the Flat File adapter to process the inbound files in a sorted order.
Sort_Example_2_PI.zip Solution project interchange file for configuring the Flat File adapter to process the inbound files in a sorted order, sorted by file name.
Sort_Example_3_PI.zip Solution project interchange file for configuring the Flat File adapter to process the inbound files in a sorted order, sorted by time stamp.
BusinessCalendar_Example_1_PI.zip Start project interchange file for configuring the polling interval by using a Business Calendar.
BusinessCalendar_Example_2_PI.zip Solution project interchange file for configuring the polling interval by using a Business Calendar.

Conclusion

Part 1 of this series described how to configure the WebSphere Adaper for Flat Files to achieve sequencing or ordering and how a Business Calendar can help you adjust the polling interval. In Part 2, you learn how to use outbound processing to write to one single file and configuration options to archive during inbound processing.


Downloads

DescriptionNameSize
Code sampleSequence_Example_1_PI.zip2141KB
Code sampleSequence_Example_2_PI.zip2141KB
Code sampleSort_Example_1_PI.zip11B
Code sampleSort_Example_2_PI.zip11KB
Code sampleSort_Example_3_PI.zip11KB
Code sampleBusinessCalendar_Example_1_PI.zip2141KB
Code sampleBusinessCalendar_Example_2_PI.zip2147KB

Resources

Learn

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into WebSphere on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=WebSphere
ArticleID=960213
ArticleTitle=WebSphere Adapter for Flat Files - a technical deep dive, Part 1: How to configure for inbound processing
publish-date=01152014