Topic
  • 3 replies
  • Latest Post - ‏2018-04-27T04:56:48Z by JoyceALynch
Allie_N
Allie_N
2 Posts

Pinned topic Exporting more than 1000 records using the Integration Object

‏2017-12-01T14:54:02Z | integration integration_object

I have an Integration Object that uses a static query to export records to a flat file. It works great when I click on the Execute action on the Integration Object. It can export more than 27,000 records. However, I only want to export a subset of those records so I am executing it from a Custom Task as described in https://www.ibm.com/support/knowledgecenter/en/SSHEB3_3.5.1/com.ibm.tap.doc/con_intobj/r_io_execution_custom_tasks.html.

Executing from Custom Task runs as expected if there are 1000 records or less to export. The workflow throws a NPE if there are 1001 records or more. How can I get it to export more than 1000 records?

We are currently on Platform 3.5.1.3.

 

Here is the stack trace

com.tririga.platform.error.PlatformRuntimeException: java.lang.NullPointerException

        at com.tririga.platform.workflow.runtime.taskhandler.WFTaskHandlerBase.handleTaskException(WFTaskHandlerBase.java:286)

        at com.tririga.platform.workflow.runtime.taskhandler.WFTaskHandlerBase.handleTaskExceptionWithInstanceInfo(WFTaskHandlerBase.java:249)

        at com.tririga.platform.workflow.runtime.taskhandler.CustomTaskHandler.executeCustomTask(CustomTaskHandler.java:232)

        at com.tririga.platform.workflow.runtime.taskhandler.CustomTaskHandler.processTask(CustomTaskHandler.java:133)

        at com.tririga.platform.workflow.runtime.taskhandler.WFTaskHandlerBase.execute(WFTaskHandlerBase.java:100)

        at com.tririga.platform.workflow.runtime.debugger.WFDBaseHook.executeStep(WFDBaseHook.java:26)

        at com.tririga.platform.workflow.runtime.WFProcessor.runFromNode(WFProcessor.java:228)

        at com.tririga.platform.workflow.runtime.WFProcessor.run(WFProcessor.java:149)

        at com.tririga.platform.workflow.runtime.WFRuntimeManagerImpl.internalRunUsingContext(WFRuntimeManagerImpl.java:578)

        at com.tririga.platform.workflow.runtime.WFRuntimeManagerImpl.internalRun(WFRuntimeManagerImpl.java:543)

        at com.tririga.platform.workflow.runtime.WFRuntimeManagerImpl.runFromAgent(WFRuntimeManagerImpl.java:223)

        at com.tririga.platform.workflow.runtime.WFRuntimeManagerImpl$$FastClassBySpringCGLIB$$ef0cbe4d.invoke(<generated>)

        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)

        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:708)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)

        at com.tririga.platform.context.execution.ExcecutionTypeContextAdvisor$1.execute(ExcecutionTypeContextAdvisor.java:51)

        at com.tririga.platform.context.execution.ExecutionTypeContextImpl.doInExecutionTypeContext(ExecutionTypeContextImpl.java:50)

        at com.tririga.platform.context.execution.ExecutionTypeContextImpl$$FastClassBySpringCGLIB$$a8da7f6e.invoke(<generated>)

        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)

        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:708)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)

        at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)

        at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:644)

        at com.tririga.platform.context.execution.ExecutionTypeContextImpl$$EnhancerBySpringCGLIB$$cf418e73.doInExecutionTypeContext(<generated>)

        at com.tririga.platform.context.execution.ExcecutionTypeContextAdvisor.invoke(ExcecutionTypeContextAdvisor.java:64)

        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:644)

        at com.tririga.platform.workflow.runtime.WFRuntimeManagerImpl$$EnhancerBySpringCGLIB$$b142edc6.runFromAgent(<generated>)

        at com.tririga.platform.workflow.agent.WFAgent$AgentRunThread.runOrResume(WFAgent.java:731)

        at com.tririga.platform.workflow.agent.WFAgent$AgentRunThread.runWf(WFAgent.java:680)

        at com.tririga.platform.workflow.agent.WFAgent$AgentRunThread.run(WFAgent.java:644)

Caused by: java.lang.NullPointerException

        at com.tririga.platform.workflow.runtime.taskhandler.WFTaskHandlerBase.getWFStepInstanceFromWFStepInfo(WFTaskHandlerBase.java:454)

        at com.tririga.platform.workflow.runtime.taskhandler.CustomTaskHandler.getWFParametersFromReturnMap(CustomTaskHandler.java:391)

        at com.tririga.platform.workflow.runtime.taskhandler.CustomTaskHandler.invokeCustomTaskWithParams(CustomTaskHandler.java:343)

        at com.tririga.platform.workflow.runtime.taskhandler.CustomTaskHandler.invokeCustomTaskWithParams(CustomTaskHandler.java:289)

        at com.tririga.platform.workflow.runtime.taskhandler.CustomTaskHandler.processCustomTaskWithParams(CustomTaskHandler.java:281)

        at com.tririga.platform.workflow.runtime.taskhandler.CustomTaskHandler.executeCustomTask(CustomTaskHandler.java:203)

        ... 30 more

  • AK88
    AK88
    105 Posts

    Re: Exporting more than 1000 records using the Integration Object

    ‏2017-12-01T19:56:20Z  

    Could you show us your workflow and the details of the custom task?

    With IO, you can also use a web call to get query results. This method will return the first 1000 rows and you can use the returned continuation token to get the rest of the data.

     

    https://<TRIRIGAURL>/<CONTEXT>/html/en/default/rest/Integration?USERNAME=<user>&PASSWORD=<password>&action=query&module=<MODULE>&bo=<BO>&query=<QUERY NAME>&f=pjson

     

    the format can be json, pjson, xml, pxml, tab

     

     

  • sgarcia0033
    sgarcia0033
    27 Posts

    Re: Exporting more than 1000 records using the Integration Object

    ‏2017-12-04T17:26:16Z  

    I cannot answer your question on how to limit your query results, but the exception you are seeing is an issue in the code.  Can you please submit a PMR with the details of your custom task and your process?

    This might help you debug your custom task though - The workflow logic is throwing the exception because it is expecting something different in the returned record ids than what it is getting from the custom task.  The returned record ids must be a record id or a Long value.  I cannot tell you what you are returning but it is not a record id or a Long.  You might want to check that out.  You have found an issue in the error handling.

    Please mention my name and RTC 292382 in the PMR and this discussion so we can get it fixed in the code.  Thank you!

    Updated on 2017-12-05T01:01:45Z at 2017-12-05T01:01:45Z by sgarcia0033
  • JoyceALynch
    JoyceALynch
    24 Posts

    Re: Exporting more than 1000 records using the Integration Object

    ‏2018-04-27T04:56:48Z  

    Allie, thank you for getting the ticket raised with Support on this matter.

    I am updating this thread for the benefit of anyone who might have been following this thread and was curious as to why you were running in to this issue with more than 1000 records.

    The reason why executing from the Custom Task only worked as expected when there were less than 1000 records was due to a known limitation that the 3.5.1 Connector Guide had attempted to document but actually misstated.  This is what the statement in the document should have said:

    Outbound integrations Custom tasks can process a maximum of 1000 records at one time. If there are more than 1000 records to be exported, create your workflow logic to run the integration until all records are processed.

    Adding workflow logic to process the records in less-than-1000-in-size batches should assist you in getting past this.

     

    In the meantime,

       Internal Defect 292382 was raised regarding the Null Pointer Exception you encountered

    and

      APAR IJ05928 was raised to correct the statement in the Connector Guide

    Both issues are targeted to be addressed in the 1H 2018 release that is targeted to become available mid-2018.

     

    Thank you for working with the TRIRIGA Support team on this!

    Updated on 2018-04-27T04:58:35Z at 2018-04-27T04:58:35Z by JoyceALynch