Topic
  • 5 replies
  • Latest Post - ‏2013-01-24T21:46:13Z by brian1305
brian1305
brian1305
10 Posts

Pinned topic WTX Crashing when processing large numbers of files

‏2012-12-31T21:28:13Z |
We have an issue where WTX is crashing when large numbers of files are received at once. The service has to be restarted and the files fed back through at a lesser rate. Does anyone have any suggestions as to what can be done to alleviate this? Thanks in advance.
Updated on 2013-01-24T21:46:13Z at 2013-01-24T21:46:13Z by brian1305
  • SystemAdmin
    SystemAdmin
    6184 Posts

    Re: WTX Crashing when processing large numbers of files

    ‏2013-01-03T16:28:20Z  
    What do you mean WTX is "crashing"? Be specific on what is happening befor the "crash" occurs. How are the files "received"? What is the maximum size of a file? How are you executing a single map or one map per file? To which WTX "service" are you referring? What release of WTX? On what OS?
  • brian1305
    brian1305
    10 Posts

    Re: WTX Crashing when processing large numbers of files

    ‏2013-01-03T21:21:30Z  
    What do you mean WTX is "crashing"? Be specific on what is happening befor the "crash" occurs. How are the files "received"? What is the maximum size of a file? How are you executing a single map or one map per file? To which WTX "service" are you referring? What release of WTX? On what OS?
    The files are brought in via FTP into a folder, thousands at a time. They are transferred to another folder, where WTX starts processing them via our 'move_inbound' msl service. Each file is a single transaction, somewhere in the 1K range, size-wise. These files are all processed via a single map. The service is called 'dtx81_launcher', which points to eventserverservice.exe in the WTX directory. We are running version 8.1 on Windows 2003.
  • SystemAdmin
    SystemAdmin
    6184 Posts

    Re: WTX Crashing when processing large numbers of files

    ‏2013-01-04T15:51:22Z  
    • brian1305
    • ‏2013-01-03T21:21:30Z
    The files are brought in via FTP into a folder, thousands at a time. They are transferred to another folder, where WTX starts processing them via our 'move_inbound' msl service. Each file is a single transaction, somewhere in the 1K range, size-wise. These files are all processed via a single map. The service is called 'dtx81_launcher', which points to eventserverservice.exe in the WTX directory. We are running version 8.1 on Windows 2003.
    OK. That helps. It sounds like Launcher is firing off too many instances of the map. I just realized I haven't worked with Launcher for 4 years but hopefully I recall enough. Also, let's hope someone else jumps in to help.

    To single thread the map, that is, process one file at a time, (which I assume you don't want to do and I don't think will work with the number of files you have):
    Option 1: In Map Settings, set MapAudit->AuditLocation=File->FileName=Custom->Action=Create->Value=some file name
    Option 2: In Map Settings, set WorkSpace->WorkFilePrefix=MapName

    Either of these will force the map to write to a single file so subsequent maps will go into Retry mode when they find the map files "locked". You could leave the above settings at default settings and set Map Settings: Retry->Swithch=ON, MaxAttempts=some number but with thousands of input files, this isn't practical. You'd end up with maps timing out.

    To multi-thread the map:
    Check Help screens for:
    "Running multiple concurrent instances of maps"

    In DTX.ini are a number of parameters that relate to maximum number of map instances. Unfortunately, I can't find the documentation in the Help screens. Some of these settings can be overwritten from within the Integration Flow Designer.

    Check the Performance Recommendations section of the Help screens.

    You have the option to design multiple map instances in the IFD that all watch the same directory but I don't believe this would help your situation. Multiple instances of the Launcher on multiple systems would help but multiple licenses would be required.

    Unfortunately I can't come up with any specific answers for you even though I know I have solved similar issues in the past. Hopefully it will come to me or someone will jump in with more specific suggestions.
  • SystemAdmin
    SystemAdmin
    6184 Posts

    Re: WTX Crashing when processing large numbers of files

    ‏2013-01-04T16:03:55Z  
    OK. That helps. It sounds like Launcher is firing off too many instances of the map. I just realized I haven't worked with Launcher for 4 years but hopefully I recall enough. Also, let's hope someone else jumps in to help.

    To single thread the map, that is, process one file at a time, (which I assume you don't want to do and I don't think will work with the number of files you have):
    Option 1: In Map Settings, set MapAudit->AuditLocation=File->FileName=Custom->Action=Create->Value=some file name
    Option 2: In Map Settings, set WorkSpace->WorkFilePrefix=MapName

    Either of these will force the map to write to a single file so subsequent maps will go into Retry mode when they find the map files "locked". You could leave the above settings at default settings and set Map Settings: Retry->Swithch=ON, MaxAttempts=some number but with thousands of input files, this isn't practical. You'd end up with maps timing out.

    To multi-thread the map:
    Check Help screens for:
    "Running multiple concurrent instances of maps"

    In DTX.ini are a number of parameters that relate to maximum number of map instances. Unfortunately, I can't find the documentation in the Help screens. Some of these settings can be overwritten from within the Integration Flow Designer.

    Check the Performance Recommendations section of the Help screens.

    You have the option to design multiple map instances in the IFD that all watch the same directory but I don't believe this would help your situation. Multiple instances of the Launcher on multiple systems would help but multiple licenses would be required.

    Unfortunately I can't come up with any specific answers for you even though I know I have solved similar issues in the past. Hopefully it will come to me or someone will jump in with more specific suggestions.
    See "Verifying that the map meets basic thread-safe checks" in the Help screens.
  • brian1305
    brian1305
    10 Posts

    Re: WTX Crashing when processing large numbers of files

    ‏2013-01-24T21:46:13Z  
    See "Verifying that the map meets basic thread-safe checks" in the Help screens.
    Thank you for your suggestions. I'll give these a try.