Topic
IC4NOTICE: developerWorks Community will be offline May 29-30, 2015 while we upgrade to the latest version of IBM Connections. For more information, read our upgrade FAQ.
11 replies Latest Post - ‏2012-12-15T00:33:10Z by sspavan
SystemAdmin
SystemAdmin
783 Posts
ACCEPTED ANSWER

Pinned topic Have you tried this - Recursion loop for Job steps based on the return code

‏2012-12-04T21:29:17Z |
Hi everyone

I have the following questions on the Modern Batch Framework

1. Can we repeat the same job step again and again based on the return code. I know there is <<step-scheduling> and <<returncode-expression>, but here there is only one step and I need to repeat it.

2. Can we create multiple Batch Data Streams programatically during runtime instead of static xJCL fixed number of BDSs.

Thanks
Updated on 2012-12-15T00:33:10Z at 2012-12-15T00:33:10Z by sspavan
  • sspavan
    sspavan
    60 Posts
    ACCEPTED ANSWER

    Re: Have you tried this - Recursion loop for Job steps based on the return code

    ‏2012-12-05T03:37:42Z  in response to SystemAdmin
    Hi,

    We can't create BDS at runtime without defining them in xjcl.
  • planetjones
    planetjones
    7 Posts
    ACCEPTED ANSWER

    Re: Have you tried this - Recursion loop for Job steps based on the return code

    ‏2012-12-05T08:20:52Z  in response to SystemAdmin
    Hi,

    AFAIK there is no way to repeat the same step in an XJCL, without copying and pasting the step definition X number of times and giving it a different name each time.

    If you want this kind of logic then I think you would need to do it programatically i.e. create one step and don't complete it until you're processed every single record. There's no reason a single step can't reopen cursors - in one of my jobs I have a queue of Work Items which it runs through - each work item opens the cursor again.

    Cheers.
    • SystemAdmin
      SystemAdmin
      783 Posts
      ACCEPTED ANSWER

      Re: Have you tried this - Recursion loop for Job steps based on the return code

      ‏2012-12-05T16:10:48Z  in response to planetjones
      Hi

      Thanks for your replies

      1. The reason I want to create BDS at runtime is, the filepaths are available from database and there can be 'n' number of files. so is there any way we can create the BDS on the fly instead of something fixed like xJCL. If we cannot do this, looks like it is a major disadvantage, nobody wants to manually fill in the file paths every time that too for unknown number of files.

      2. I also need the recursion here because I want to handle the 'n' number of BDS in the same fashion and individually, I mean the same business logic. Again here, I cannot repeat the same job step again and again in xJCL as I am not sure of the number of files.

      I am not sure if you have faced similar situations. For me this looks like a common problem when we are handling multiple files. Another disadvantage here is all the BDS are intialized before any job step can start and we cannot create initialize or create any BDS during the process.
      • sspavan
        sspavan
        60 Posts
        ACCEPTED ANSWER

        Re: Have you tried this - Recursion loop for Job steps based on the return code

        ‏2012-12-05T16:46:21Z  in response to SystemAdmin
        Hi,

        I have the same requirement as you reg No.1 where my no. of files dynamically change for each run and unfortunately the current product doesn't support it. In Springbatch we have a multi resource reader where it can read multiple files based on a pattern in a specific directory location. I already approached IBM Compute Grid team on this and they are planning to implement in future release.
        As a work around I did this in 2 steps, 1st step is a CI impl where I have java code to read multiple files and create one single file, in 2nd step I read the single file and process the data.

        Compute grid allocates all resources before executing the step, so the I/O has to be known. I asked IBM if we can dynamically set the filename in processor and they are working it for a possible solution. If you want to execute the same step with different input files you can try exploring/implementing Parallel Job mechanizm where you can parameterize each run.

        Thanks,
        Pavan
        • planetjones
          planetjones
          7 Posts
          ACCEPTED ANSWER

          Re: Have you tried this - Recursion loop for Job steps based on the return code

          ‏2012-12-06T13:40:40Z  in response to sspavan
          Is one option here not to have the stream create a Queue of all files to process i.e. in its open method it finds all files which need to be processed. You set the current file to be the head of the Queue.
          You then have a readRecord method which gives the step one record at a time.
          When the step runs out of records in the first file, it changes the current file to be the next file name on the Queue.
          It then starts giving the step records from the 2nd file.
          Same for the 3rd file, 4th file, etc.
          You have to think a little about restarts e.g. store current filename and record key in the restart info.
          • sspavan
            sspavan
            60 Posts
            ACCEPTED ANSWER

            Re: Have you tried this - Recursion loop for Job steps based on the return code

            ‏2012-12-06T15:13:14Z  in response to planetjones
            Hi,

            How do you queue the files ?
            Do you mean to say that we need specify all files as below, I tried it and it doesn't work.
            <prop name="FILENAME" value="c:\javabatch-testfiles\file-input.txt,c:\javabatch-testfiles\file-input1.txt" />
            • planetjones
              planetjones
              7 Posts
              ACCEPTED ANSWER

              Re: Have you tried this - Recursion loop for Job steps based on the return code

              ‏2012-12-06T15:41:21Z  in response to sspavan
              I thought your file input list was dynamic. I only really use the Z/OS file access, but I would have thought your open method could just use java.io to list all files in c:\javabatch-testfiles and create a queue of Files. Then process them one by one. I don't think you need to specify the filenames in the xJCL... not if you open the files yourself in the streams i.e. just roll your own stream by implementing http://pic.dhe.ibm.com/infocenter/wxdinfo/v6r1m1/index.jsp?topic=%2Fcom.ibm.websphere.xd.javadoc.doc%2Finfo%2Fapidocs%2Fcom%2Fibm%2Fwebsphere%2Fbatch%2FBatchDataStream.html.
              The xJCL could just give the directory containing the files.
              • SystemAdmin
                SystemAdmin
                783 Posts
                ACCEPTED ANSWER

                Re: Have you tried this - Recursion loop for Job steps based on the return code

                ‏2012-12-06T21:02:47Z  in response to planetjones
                Thanks Pavan and planet jones for the responses.

                1. First, we cannot give partial paths in xJCL for BDS file reader pattern, it will not work. This should be complete till the name of the file. Please let me know if it works for you.
                2. Second, I do not know the file names or number of files that will be processed till the first step of the job(where I will access DB to get the list). Even if I want to start the process of reading in second step, how can we create the BDS input readers at runtime or programatically.

                Pavan
                I do not want to club the files to avoid losing the identity of the source of the records and also a performance issue when I have multiple files and huge data.
                And also the PJM looks like altogether different concept, it does not help in repeating a single job step n number of times. please let me know if you are able to achieve this. And also it is a compute grid bundle feature and not of modern batch.

                I am thinking of maintaing a dynamic properties files(will be updated as per DB) to store the file full path. This properties file will be submitted along with xJCL to the WSGrid.sh scheduler. To repeat the job step, I am thinking of repeating the Job itself from the scheduler for a different file. Again there is some manual work here....
                • sspavan
                  sspavan
                  60 Posts
                  ACCEPTED ANSWER

                  Re: Have you tried this - Recursion loop for Job steps based on the return code

                  ‏2012-12-06T21:53:41Z  in response to SystemAdmin
                  Hi Sree,

                  If you are using Mobern batch and not compute grid bundle then you can't have a job with multiple implementation steps (ex: 1st step being a CI impl, 2nd step being a Batch impl) or PJM.
                  I did execute PJM where I had single step being run n number of times with different input files, I know how many files I read so I can define my no. of concurrent subjobs, by that configuration the step is run in multiple instances one for each file.
                  Here is my sample code, my file prefix is txtinput.txt.split. I am running the job as 3 subjobs and I have 3 files txtinput.txt.split.1, txtinput.txt.split.2, txtinput.txt.split.3

                  If you don't know the number of files you will be reading then how can you schedule jobs.

                  Also while processing records why do you want to identify the source ? If that is really a requirement
                  then while combining files and writing to single file you can add an additional column that has the source information. You need not worry about huge data processing, Compute Grid can handle any amount of data and if you specify ideal checkpoint values your process would run without issues.
                  Thanks,
                  Pavan
                  • SystemAdmin
                    SystemAdmin
                    783 Posts
                    ACCEPTED ANSWER

                    Re: Have you tried this - Recursion loop for Job steps based on the return code

                    ‏2012-12-14T22:06:39Z  in response to sspavan
                    Thanks Pavan

                    I have only Modern batch in my local system, but I can try this in our dev server

                    Let me clarify on not knowing the number of files statement.

                    I need to process the files from a particular location. The batch input files will be dropped at this location. There can be any number of files at this location at a particular time. All the file names might have a common prefix (not sure of this point). They should be executed by the same job step without repeating or editing the xJCL manually. I know you already said this is not possible in compute grid, but I am looking for something like this.

                    The reason we should have source information is, if there are validation or functional errors found while processing a job step, we should be able to track down to the source of the record. My thought of peformance issue here is not on the number of records that can be processed by compute grid but the extra step we need to do to merge all the files and add a source field for each of these records which we don't do generally.
                    • sspavan
                      sspavan
                      60 Posts
                      ACCEPTED ANSWER

                      Re: Have you tried this - Recursion loop for Job steps based on the return code

                      ‏2012-12-15T00:33:10Z  in response to SystemAdmin
                      Hi Sree,

                      If you don't want to add a source field then you can add coding in the merge process to report for any errors in the source file. With limitations in Compute Grid unfortunately we may have to live with some alternatives.
                      IBM is planning on a new Batch API and it is somewhat similar to Springbatch and I hope the feature will be available at that point.

                      Thanks,
                      Pavan