Topic
  • 6 replies
  • Latest Post - ‏2013-10-11T13:22:15Z by kenhygh
vardhan128tech
vardhan128tech
9 Posts

Pinned topic Is it safe to increase max size of XML Parser under Datapower XML Manager , max suggested size?

‏2013-06-08T20:41:06Z |

Hi Team/ Herman,

We are trying to expose Data as a service via Datapower for Oracle as a backend.

We have a situation where we have to retrive 100,000 records from Oracle and send back to client as an XML.

The default size mentioned under the XML Manager is not enought for us and hence we are trying to increase the size of XML Bytes Scanned under XML Manager.

default is 4194304 bytes and we need approximately .50 times and hence 50*4194304 = 209715200 Bytes.

1) Can we increase the size to 50*4194304 = 209715200 Bytes ?

2) Is it safe and allowable under Datapower XI50B?

3) If is ok, do we need to take care iof any additional implementation under that Service?

4) Is there any alternaives to vercome this issues ? like streaming service will be helpfull in this situation?

Appreciate your suggestions here.....

 

Regards,

Vardhan

Updated on 2013-06-08T20:42:49Z at 2013-06-08T20:42:49Z by vardhan128tech
  • HermannSW
    HermannSW
    6019 Posts
    ACCEPTED ANSWER

    Re: Is it safe to increase max size of XML Parser under Datapower XML Manager , max suggested size?

    ‏2013-06-10T00:50:55Z  

    Hi Ken,

    Yes while testing we got out of memory error under datapower even though I increased to 500 times*4194304.

    Would it be sucessfull if we implement as a streamed MPGW service?

     

    Regards,

    Vardhan

    > Yes while testing we got out of memory error under datapower even though I increased to 500 times*4194304.
    >
    Please think about what you did -- 500 x 4MB is 2GB, please do "show mem" before you execute a transaction.
     

    > Would it be sucessfull if we implement as a streamed MPGW service?
    >
    Definitely, but there are many restrictions on what you can do with streaming.

    You can process arbitrarily large files with streaming if your policy is streamable.

    Please see slides 7-10 on streaming in this WSTE webcast:
    http://www-01.ibm.com/support/docview.wss?uid=swg27019118



    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

  • kenhygh
    kenhygh
    2006 Posts

    Re: Is it safe to increase max size of XML Parser under Datapower XML Manager , max suggested size?

    ‏2013-06-09T12:15:53Z  

    Vardhan,

    200MB is within DataPower's capability, but you'll want to be careful about running out of memory. You might want to set an SLM policy for this service to limit concurrent transactions to something reasonable.

  • vardhan128tech
    vardhan128tech
    9 Posts

    Re: Is it safe to increase max size of XML Parser under Datapower XML Manager , max suggested size?

    ‏2013-06-09T21:43:22Z  
    • kenhygh
    • ‏2013-06-09T12:15:53Z

    Vardhan,

    200MB is within DataPower's capability, but you'll want to be careful about running out of memory. You might want to set an SLM policy for this service to limit concurrent transactions to something reasonable.

    Hi Ken,

    Yes while testing we got out of memory error under datapower even though I increased to 500 times*4194304.

    Would it be sucessfull if we implement as a streamed MPGW service?

     

    Regards,

    Vardhan

  • HermannSW
    HermannSW
    6019 Posts

    Re: Is it safe to increase max size of XML Parser under Datapower XML Manager , max suggested size?

    ‏2013-06-10T00:50:55Z  

    Hi Ken,

    Yes while testing we got out of memory error under datapower even though I increased to 500 times*4194304.

    Would it be sucessfull if we implement as a streamed MPGW service?

     

    Regards,

    Vardhan

    > Yes while testing we got out of memory error under datapower even though I increased to 500 times*4194304.
    >
    Please think about what you did -- 500 x 4MB is 2GB, please do "show mem" before you execute a transaction.
     

    > Would it be sucessfull if we implement as a streamed MPGW service?
    >
    Definitely, but there are many restrictions on what you can do with streaming.

    You can process arbitrarily large files with streaming if your policy is streamable.

    Please see slides 7-10 on streaming in this WSTE webcast:
    http://www-01.ibm.com/support/docview.wss?uid=swg27019118



    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

  • vardhan128tech
    vardhan128tech
    9 Posts

    Re: Is it safe to increase max size of XML Parser under Datapower XML Manager , max suggested size?

    ‏2013-06-10T21:30:42Z  
    • HermannSW
    • ‏2013-06-10T00:50:55Z

    > Yes while testing we got out of memory error under datapower even though I increased to 500 times*4194304.
    >
    Please think about what you did -- 500 x 4MB is 2GB, please do "show mem" before you execute a transaction.
     

    > Would it be sucessfull if we implement as a streamed MPGW service?
    >
    Definitely, but there are many restrictions on what you can do with streaming.

    You can process arbitrarily large files with streaming if your policy is streamable.

    Please see slides 7-10 on streaming in this WSTE webcast:
    http://www-01.ibm.com/support/docview.wss?uid=swg27019118



    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

    Thanks Ken and Hermann for the suggestions!

  • samanderson
    samanderson
    172 Posts

    Re: Is it safe to increase max size of XML Parser under Datapower XML Manager , max suggested size?

    ‏2013-10-11T11:02:47Z  

    Hi Ken/Herman -  I have similar requirement to increase size by 100MB. What can be ideal value for SLM?

  • kenhygh
    kenhygh
    2006 Posts

    Re: Is it safe to increase max size of XML Parser under Datapower XML Manager , max suggested size?

    ‏2013-10-11T13:22:15Z  

    Hi Ken/Herman -  I have similar requirement to increase size by 100MB. What can be ideal value for SLM?

    You will have to find out this answer during load tests. There's no generic answer.