Topic
  • 13 replies
  • Latest Post - ‏2014-06-18T07:20:29Z by HermannSW
SystemAdmin
SystemAdmin
6772 Posts

Pinned topic Upper Limit of Maximum Node Size value in XML Parser

‏2010-06-02T08:52:35Z |
Hi all,

Is there any upper limit of Maximum Node Size value in XML Parser(MPGW/XMLFW)?

Thanks in advance.
XI50 (3.8.0.4)
Updated on 2013-03-14T14:48:25Z at 2013-03-14T14:48:25Z by SystemAdmin
  • mandeepk
    mandeepk
    5 Posts

    Re: Upper Limit of Maximum Node Size value in XML Parser

    ‏2010-12-08T06:37:25Z  
    The default size is given in XMLManager-->XMLParser
    XML Maximum Node Size is default(33554432bytes) i.e.,32MB

    regards,
    Sandeep M.
  • SystemAdmin
    SystemAdmin
    6772 Posts

    Re: Upper Limit of Maximum Node Size value in XML Parser

    ‏2013-03-12T20:30:36Z  
    Technically there is no limit in Datapower. In the XML Manager you can set XML Bytes Scanned and XML Maximum Node size to 0 which would allow files to come through of any size. However, keep in mind that the reason the XML Manager is here is to protect against XML threats and to manage what size transactions you send to the backend. For security reasons you will likely want to limit this to whatever your system requires. Additionally, Proxy limits override any similar limit set by the XML manager. You can set "Maximum Message Size" to 0 to allow any size files through. You can also play with your "Proxy parser limits" and "Attachment Byte Count Limit." Hope this helps.
  • HermannSW
    HermannSW
    6138 Posts

    Re: Upper Limit of Maximum Node Size value in XML Parser

    ‏2013-03-13T14:38:24Z  
    Technically there is no limit in Datapower. In the XML Manager you can set XML Bytes Scanned and XML Maximum Node size to 0 which would allow files to come through of any size. However, keep in mind that the reason the XML Manager is here is to protect against XML threats and to manage what size transactions you send to the backend. For security reasons you will likely want to limit this to whatever your system requires. Additionally, Proxy limits override any similar limit set by the XML manager. You can set "Maximum Message Size" to 0 to allow any size files through. You can also play with your "Proxy parser limits" and "Attachment Byte Count Limit." Hope this helps.
    > ... Technically there is no limit in Datapower. ...
    >
    There is, but its quite high.
    It is definitely below 78.51GB (total memory, see below) on XI52 models ;-)
    But I once did process a 25GB XML file with a non-streaming stylesheet, so its above 25GB.

    Version: XI52.5.0.0.5 build 223327 on 2013/01/15 14:47:52
    Serial number: 6801610
     
    xi52# show mem
     
        Memory Usage: 3 %
        Total Memory: 82333842 kilobytes
         Used Memory: 2951717 kilobytes
         Free Memory: 79382125 kilobytes
    Requested Memory: 3568000 kilobytes
         Hold Memory: 616283 kilobytes
     Reserved Memory: 16863558 kilobytes
    Installed Memory: 99197400 kilobytes
     
     
    xi52#
    


    Hermann <myXsltBlog/> <myXsltTweets/> <myCE/>
    Updated on 2014-03-25T02:37:58Z at 2014-03-25T02:37:58Z by iron-man
  • SystemAdmin
    SystemAdmin
    6772 Posts

    Re: Upper Limit of Maximum Node Size value in XML Parser

    ‏2013-03-13T14:49:12Z  
    • HermannSW
    • ‏2013-03-13T14:38:24Z
    > ... Technically there is no limit in Datapower. ...
    >
    There is, but its quite high.
    It is definitely below 78.51GB (total memory, see below) on XI52 models ;-)
    But I once did process a 25GB XML file with a non-streaming stylesheet, so its above 25GB.

    <pre class="java dw" data-editor-lang="java" data-pbcklang="java" dir="ltr">Version: XI52.5.0.0.5 build 223327 on 2013/01/15 14:47:52 Serial number: 6801610 xi52# show mem Memory Usage: 3 % Total Memory: 82333842 kilobytes Used Memory: 2951717 kilobytes Free Memory: 79382125 kilobytes Requested Memory: 3568000 kilobytes Hold Memory: 616283 kilobytes Reserved Memory: 16863558 kilobytes Installed Memory: 99197400 kilobytes xi52# </pre>

    Hermann <myXsltBlog/> <myXsltTweets/> <myCE/>
    Hermann,
    That limit is not the same incase of XI50 right? I believe its 4GB

    Regards,
    Kumar
  • HermannSW
    HermannSW
    6138 Posts

    Re: Upper Limit of Maximum Node Size value in XML Parser

    ‏2013-03-13T15:01:28Z  
    Hermann,
    That limit is not the same incase of XI50 right? I believe its 4GB

    Regards,
    Kumar
    Hi Kumar,

    > That limit is not the same incase of XI50 right? I believe its 4GB
    >
    yes, 4GB:
    Version: XI50.4.0.2.9kn build 224171kn on 2013/02/05 21:39:44
    Serial number: 68A3089
     
    xi50# show mem
     
        Memory Usage: 13 %
        Total Memory: 4194304 kilobytes
         Used Memory: 566103 kilobytes
         Free Memory: 3628201 kilobytes
    Requested Memory: 1474688 kilobytes
         Hold Memory: 908585 kilobytes
     
     
    xi50#
    

    Hermann <myXsltBlog/> <myXsltTweets/> <myCE/>
    Updated on 2014-03-25T02:37:53Z at 2014-03-25T02:37:53Z by iron-man
  • SystemAdmin
    SystemAdmin
    6772 Posts

    Re: Upper Limit of Maximum Node Size value in XML Parser

    ‏2013-03-13T15:12:28Z  
    • HermannSW
    • ‏2013-03-13T15:01:28Z
    Hi Kumar,

    > That limit is not the same incase of XI50 right? I believe its 4GB
    >
    yes, 4GB:
    <pre class="java dw" data-editor-lang="java" data-pbcklang="java" dir="ltr">Version: XI50.4.0.2.9kn build 224171kn on 2013/02/05 21:39:44 Serial number: 68A3089 xi50# show mem Memory Usage: 13 % Total Memory: 4194304 kilobytes Used Memory: 566103 kilobytes Free Memory: 3628201 kilobytes Requested Memory: 1474688 kilobytes Hold Memory: 908585 kilobytes xi50# </pre>
    Hermann <myXsltBlog/> <myXsltTweets/> <myCE/>
    Hi Hermann,
    So if the appliance is XI50 we can't parse more than 4GB if even XML manager parsing limit is set zero right?
  • HermannSW
    HermannSW
    6138 Posts

    Re: Upper Limit of Maximum Node Size value in XML Parser

    ‏2013-03-13T15:19:53Z  
    Hi Hermann,
    So if the appliance is XI50 we can't parse more than 4GB if even XML manager parsing limit is set zero right?
    Hi,

    > So if the appliance is XI50 we can't parse more than 4GB if even XML manager parsing limit is set zero right?
    >
    that is only true for non-streaming processing.

    I did successfully process 10GB XML files in a streaming service on XI50, there should be no upper limit.

     
    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>
  • SystemAdmin
    SystemAdmin
    6772 Posts

    Re: Upper Limit of Maximum Node Size value in XML Parser

    ‏2013-03-13T15:26:57Z  
    • HermannSW
    • ‏2013-03-13T15:19:53Z
    Hi,

    > So if the appliance is XI50 we can't parse more than 4GB if even XML manager parsing limit is set zero right?
    >
    that is only true for non-streaming processing.

    I did successfully process 10GB XML files in a streaming service on XI50, there should be no upper limit.

     
    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>
    Hi Hermann,

    > So if the appliance is XI50 we can't parse more than 4GB if even XML manager parsing limit is set zero right?
    >
    >that is only true for non-streaming processing.

    >I did successfully process 10GB XML files in a streaming service on XI50, there should be no upper limit.

    Can you please explain more about non-steaming and streaming processing?
    Regards,
    Kumar
  • HermannSW
    HermannSW
    6138 Posts

    Re: Upper Limit of Maximum Node Size value in XML Parser

    ‏2013-03-14T07:01:19Z  
    Hi Hermann,

    > So if the appliance is XI50 we can't parse more than 4GB if even XML manager parsing limit is set zero right?
    >
    >that is only true for non-streaming processing.

    >I did successfully process 10GB XML files in a streaming service on XI50, there should be no upper limit.

    Can you please explain more about non-steaming and streaming processing?
    Regards,
    Kumar
    Hi,

    > >I did successfully process 10GB XML files in a streaming service on XI50, there should be no upper limit.
    >
    > Can you please explain more about non-steaming and streaming processing?
    please look at slides 7-10 and hear into audio recording of this WSTE webcast:
    http://www-01.ibm.com/support/docview.wss?uid=swg27019118

    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>
  • SystemAdmin
    SystemAdmin
    6772 Posts

    Re: Upper Limit of Maximum Node Size value in XML Parser

    ‏2013-03-14T14:48:25Z  
    • HermannSW
    • ‏2013-03-14T07:01:19Z
    Hi,

    > >I did successfully process 10GB XML files in a streaming service on XI50, there should be no upper limit.
    >
    > Can you please explain more about non-steaming and streaming processing?
    please look at slides 7-10 and hear into audio recording of this WSTE webcast:
    http://www-01.ibm.com/support/docview.wss?uid=swg27019118

    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>
    Thanks Hermann.

    Regards,
    Kumar
  • lenisha
    lenisha
    24 Posts

    Re: Upper Limit of Maximum Node Size value in XML Parser

    ‏2014-06-17T23:08:19Z  
    • HermannSW
    • ‏2013-03-13T14:38:24Z
    > ... Technically there is no limit in Datapower. ...
    >
    There is, but its quite high.
    It is definitely below 78.51GB (total memory, see below) on XI52 models ;-)
    But I once did process a 25GB XML file with a non-streaming stylesheet, so its above 25GB.

    <pre class="java dw" data-editor-lang="java" data-pbcklang="java" dir="ltr">Version: XI52.5.0.0.5 build 223327 on 2013/01/15 14:47:52 Serial number: 6801610 xi52# show mem Memory Usage: 3 % Total Memory: 82333842 kilobytes Used Memory: 2951717 kilobytes Free Memory: 79382125 kilobytes Requested Memory: 3568000 kilobytes Hold Memory: 616283 kilobytes Reserved Memory: 16863558 kilobytes Installed Memory: 99197400 kilobytes xi52# </pre>

    Hermann <myXsltBlog/> <myXsltTweets/> <myCE/>

    >> ... Technically there is no limit in Datapower. ...
    >>
    > There is, but its quite high.
    > It is definitely below 78.51GB (total memory, see below) on XI52 models ;-)
    > But I once did process a 25GB XML file with a non-streaming stylesheet, so its above 25GB.
     

    Herman,

    would "xml bytes scanned" setting limit message size even in streaming mode, or it's applicable to non-streaming only?

     

     

     



     

  • HermannSW
    HermannSW
    6138 Posts

    Re: Upper Limit of Maximum Node Size value in XML Parser

    ‏2014-06-18T07:19:45Z  
    • lenisha
    • ‏2014-06-17T23:08:19Z

    >> ... Technically there is no limit in Datapower. ...
    >>
    > There is, but its quite high.
    > It is definitely below 78.51GB (total memory, see below) on XI52 models ;-)
    > But I once did process a 25GB XML file with a non-streaming stylesheet, so its above 25GB.
     

    Herman,

    would "xml bytes scanned" setting limit message size even in streaming mode, or it's applicable to non-streaming only?

     

     

     



     

    Regardless of processing mode XML thread protection has to work -- and it does in DataPower.


    Hermann <myBlog/> <myTweets/> | <GraphvizFiddle/> | <xqib/> | <myCE/> <myFrameless/>

  • HermannSW
    HermannSW
    6138 Posts

    Re: Upper Limit of Maximum Node Size value in XML Parser

    ‏2014-06-18T07:20:29Z  
    • lenisha
    • ‏2014-06-17T23:08:19Z

    >> ... Technically there is no limit in Datapower. ...
    >>
    > There is, but its quite high.
    > It is definitely below 78.51GB (total memory, see below) on XI52 models ;-)
    > But I once did process a 25GB XML file with a non-streaming stylesheet, so its above 25GB.
     

    Herman,

    would "xml bytes scanned" setting limit message size even in streaming mode, or it's applicable to non-streaming only?

     

     

     



     

    Regardless of processing mode XML threat protection has to work -- and it does in DataPower.


    Hermann <myBlog/> <myTweets/> | <GraphvizFiddle/> | <xqib/> | <myCE/> <myFrameless/>

    Updated on 2014-06-18T07:20:51Z at 2014-06-18T07:20:51Z by HermannSW