Topic
  • 23 replies
  • Latest Post - ‏2013-03-26T19:49:09Z by HermannSW
terza
terza
99 Posts

Pinned topic Best practices for limiting request size?

‏2011-07-05T07:18:33Z |
Hi!
I have a need to limit the maximum request size for the incoming soap messages. I was wondering what kind of methods are being used for this task? I have to take in to account that there might be malicious users only trying to disturb the service by sending gigantic requests and then again there valid users who just might be sending a requests that are too big for the backend.

First thing that came in to my mind was the XML threat protection, but it seems a bit difficult because with XML threat protectin related errors its impossible to enter a error rule and send back a nice formatted error message. Then again if I would check the request size within a XSLT then the entire request will be read in to memory?

So what would be the best strategy for handling this case? Maximum request size will be 50 Mb so we are talking fairly large requests.

I came up with 2 different solutions:
option 1: Make two limitations, one with xml threat prot and one with xslt. Xml threat protection limit will always be higher than the limit inside a stylesheet.

option 2: Insert a new XML firewall infront of my current WSP service. Set it to streaming mode so there wont be much performance hit. Configure the xml threat protection to the WSP:s xml manager and allow this new XMLFW to handle the formatting of the error messages in case when the WSP returns an error for too large request.
Any other ideas? Anything im missing?

Thanks!
Updated on 2013-03-26T19:49:09Z at 2013-03-26T19:49:09Z by HermannSW
  • JeffreyG
    JeffreyG
    1 Post

    Re: Best practices for limiting request size?

    ‏2011-07-06T03:44:15Z  
    At least with MPGW, I am certain that XML threat detection as well as XML parser limits do hit the error rule for processing. We have an error rule in place to produce application specific output for the related error codes. You can run with a probe enabled and send a request that is just a small amount over the XML threat size to verify that the error rule is being called (don’t forget to lower the max size to a smaller value that the probe can handle).

    The issue you are seeing has more to do with what happens when DataPower encounters a processing error while the client is still sending data. Another example of this, other than XML threat detection, would be an XML parse error (e.g. mismatched tag) early in an XML document. In these types of cases, the client usually notices a write exception before the client is able to read the application formatted response sent from DataPower. Since I have never looked at a packet capture of this interaction, I’m not certain DataPower actually sends the response, but the result is that the client would see the write failure before DataPower replied anyway.

    As far as mitigation goes…

    I think Option 2 will open you up to more issues. For starters, the proxying XMLFW will at times suffer from the same results as the end client. In some cases, the XMLFW will see the WSP message too large / element too large message. In other cases, the XMLFW will see a connection closed exception. The latter exception could be a result of any number processing events (e.g. throttling, system exception, etc.).

    Option 1 could be a possibility, assuming you can come up with reasonable sizes. One thing to watch out for is if your WSP is streaming, the request-size variable may not be available for reading:

    The var://service/mpgw/request-size variable is a read-only variable. It gets the size of a request message. The value 0 indicates that the size cannot be determined, perhaps temporarily, due to message streaming or some other processing issue.

    Unfortunately, no matter what you do, there will always be some cases where the client will not be able to read application formatted response. Because of this, we ended up going with basic XML threat detection.
  • HermannSW
    HermannSW
    4903 Posts

    Re: Best practices for limiting request size?

    ‏2011-07-06T11:01:40Z  
    Hi,

    > Hi!
    > I have a need to limit the maximum request size for the incoming soap messages. I was wondering what kind of methods are being used for this task? I have to take in to account that there
    > might be malicious users only trying to disturb the service by sending gigantic requests and then again there valid users who just might be sending a requests that are too big for the backend.
    >
    > First thing that came in to my mind was the XML threat protection, but it seems a bit difficult because with XML threat protectin related errors its impossible to enter a error rule and send
    > back a nice formatted error message. Then again if I would check the request size within a XSLT then the entire request will be read in to memory?
    >
    not true, see attached XML FW export.

    As Jeff wrote the error rule gets hit for parser limit violation.
    Just match for "error code" in Error rule and select it from list, 0x00030003 is for XML parser limits exceeded.

    Instead of 50MB in your case attached XML FW limits input to 50 bytes (does a "store:///identity.xsl" transform only):
    
    $ cat 50B.xml <data>012345678901234567890123456789012345</data> $ $ cat 51B.xml <data>0123456789012345678901234567890123456</data> $ $ curl --data-binary @50B.xml http:
    //dp3-l3:5050; echo <?xml version=
    "1.0" encoding=
    "UTF-8"?> <data>012345678901234567890123456789012345</data> $ $ curl --data-binary @51B.xml http:
    //dp3-l3:5050; echo XML parser limits exceeded $ $ cat 0x00030003.xsl <xsl:stylesheet version=
    "1.0" xmlns:xsl=
    "http://www.w3.org/1999/XSL/Transform" xmlns:dp=
    "http://www.datapower.com/extensions" extension-element-prefixes=
    "dp" > <xsl:output omit-xml-declaration=
    "yes" /> <xsl:template match=
    "/"> <xsl:text>XML parser limits exceeded</xsl:text> </xsl:template> </xsl:stylesheet> $
    


    Please rely on XML thread protection because this is what DataPower does really good.
    You should not "just parse" yourself.

    Also keep in mind that the real input file size can be much smaller and still trigger the 50MB limit ...
    https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14555059#14555059

     
    Hermann<myXsltBlog/>

    Attachments

  • HermannSW
    HermannSW
    4903 Posts

    Re: Best practices for limiting request size?

    ‏2011-07-06T11:02:35Z  
    • HermannSW
    • ‏2011-07-06T11:01:40Z
    Hi,

    > Hi!
    > I have a need to limit the maximum request size for the incoming soap messages. I was wondering what kind of methods are being used for this task? I have to take in to account that there
    > might be malicious users only trying to disturb the service by sending gigantic requests and then again there valid users who just might be sending a requests that are too big for the backend.
    >
    > First thing that came in to my mind was the XML threat protection, but it seems a bit difficult because with XML threat protectin related errors its impossible to enter a error rule and send
    > back a nice formatted error message. Then again if I would check the request size within a XSLT then the entire request will be read in to memory?
    >
    not true, see attached XML FW export.

    As Jeff wrote the error rule gets hit for parser limit violation.
    Just match for "error code" in Error rule and select it from list, 0x00030003 is for XML parser limits exceeded.

    Instead of 50MB in your case attached XML FW limits input to 50 bytes (does a "store:///identity.xsl" transform only):
    <pre class="jive-pre"> $ cat 50B.xml <data>012345678901234567890123456789012345</data> $ $ cat 51B.xml <data>0123456789012345678901234567890123456</data> $ $ curl --data-binary @50B.xml http: //dp3-l3:5050; echo <?xml version= "1.0" encoding= "UTF-8"?> <data>012345678901234567890123456789012345</data> $ $ curl --data-binary @51B.xml http: //dp3-l3:5050; echo XML parser limits exceeded $ $ cat 0x00030003.xsl <xsl:stylesheet version= "1.0" xmlns:xsl= "http://www.w3.org/1999/XSL/Transform" xmlns:dp= "http://www.datapower.com/extensions" extension-element-prefixes= "dp" > <xsl:output omit-xml-declaration= "yes" /> <xsl:template match= "/"> <xsl:text>XML parser limits exceeded</xsl:text> </xsl:template> </xsl:stylesheet> $ </pre>

    Please rely on XML thread protection because this is what DataPower does really good.
    You should not "just parse" yourself.

    Also keep in mind that the real input file size can be much smaller and still trigger the 50MB limit ...
    https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14555059#14555059

     
    Hermann<myXsltBlog/>
    50 byte input file

    Attachments

  • HermannSW
    HermannSW
    4903 Posts

    Re: Best practices for limiting request size?

    ‏2011-07-06T11:03:11Z  
    • HermannSW
    • ‏2011-07-06T11:02:35Z
    50 byte input file
    51 bytes input file

    Attachments

  • terza
    terza
    99 Posts

    Re: Best practices for limiting request size?

    ‏2011-07-07T05:36:14Z  
    • HermannSW
    • ‏2011-07-06T11:03:11Z
    51 bytes input file
    Thank you both! My error rule was all wrong so no wonder it wasn't catching these errors.

    One more question about the XML parser configurations. Could some one explain the difference between "XML Bytes scanned" and "XML Maximum node size"? I there a difference if I want to use these configurations to limit the maximum request size?
  • terza
    terza
    99 Posts

    Re: Best practices for limiting request size?

    ‏2011-07-07T06:44:06Z  
    • terza
    • ‏2011-07-07T05:36:14Z
    Thank you both! My error rule was all wrong so no wonder it wasn't catching these errors.

    One more question about the XML parser configurations. Could some one explain the difference between "XML Bytes scanned" and "XML Maximum node size"? I there a difference if I want to use these configurations to limit the maximum request size?
    And one more question: my service contains 2 operations, lets call them "upload" and "download" for example. 50 Mb request limit related to these "upload" operations. But now automatically I will have the same size limit for the response messages for my "download" operations. Is there any way to get past this expect to divide the service in to different WSP:s? Response message size over 50 Mb is valid scenario for my download operation.

    Thanks again!
  • HermannSW
    HermannSW
    4903 Posts

    Re: Best practices for limiting request size?

    ‏2011-07-07T09:17:33Z  
    • terza
    • ‏2011-07-07T06:44:06Z
    And one more question: my service contains 2 operations, lets call them "upload" and "download" for example. 50 Mb request limit related to these "upload" operations. But now automatically I will have the same size limit for the response messages for my "download" operations. Is there any way to get past this expect to divide the service in to different WSP:s? Response message size over 50 Mb is valid scenario for my download operation.

    Thanks again!
    > ...
    > One more question about the XML parser configurations. Could some one explain the difference between "XML Bytes scanned" and "XML Maximum node size"?
    > I there a difference if I want to use these configurations to limit the maximum request size?
    >
    If a document contains eg. many <product>s, then "XML Bytes scanned" restricts the complete document length.
    "XML Maximum node size" restricts the size of each node, so a <product> node as all other nodes might not exceed that size.

    > And one more question: my service contains 2 operations, lets call them "upload" and "download" for example. 50 Mb request limit related to these "upload" operations.
    > But now automatically I will have the same size limit for the response messages for my "download" operations.
    > Is there any way to get past this expect to divide the service in to different WSP:s? Response message size over 50 Mb is valid scenario for my download operation.
    >
    Since the parser limits are bound to the XML Manager object, and a service has only one XML Manager, you need two services for your scenario.

     
    Hermann<myXsltBlog/>
  • terza
    terza
    99 Posts

    Re: Best practices for limiting request size?

    ‏2011-07-12T09:57:23Z  
    • HermannSW
    • ‏2011-07-07T09:17:33Z
    > ...
    > One more question about the XML parser configurations. Could some one explain the difference between "XML Bytes scanned" and "XML Maximum node size"?
    > I there a difference if I want to use these configurations to limit the maximum request size?
    >
    If a document contains eg. many <product>s, then "XML Bytes scanned" restricts the complete document length.
    "XML Maximum node size" restricts the size of each node, so a <product> node as all other nodes might not exceed that size.

    > And one more question: my service contains 2 operations, lets call them "upload" and "download" for example. 50 Mb request limit related to these "upload" operations.
    > But now automatically I will have the same size limit for the response messages for my "download" operations.
    > Is there any way to get past this expect to divide the service in to different WSP:s? Response message size over 50 Mb is valid scenario for my download operation.
    >
    Since the parser limits are bound to the XML Manager object, and a service has only one XML Manager, you need two services for your scenario.

     
    Hermann<myXsltBlog/>
    > HermannSW wrote:
    > If a document contains eg. many <product>s, then "XML Bytes scanned" restricts the complete document length.
    > "XML Maximum node size" restricts the size of each node, so a <product> node as all other nodes might not exceed that size.

    So why is the default configuration such that the "XML Bytes scanned" which is the size limit for the entire request is 4 MB and the size limit for a single node is 32MB, this second limit will never be used with 4 MB limitation...?
  • HermannSW
    HermannSW
    4903 Posts

    Re: Best practices for limiting request size?

    ‏2011-07-13T14:37:37Z  
    • terza
    • ‏2011-07-12T09:57:23Z
    > HermannSW wrote:
    > If a document contains eg. many <product>s, then "XML Bytes scanned" restricts the complete document length.
    > "XML Maximum node size" restricts the size of each node, so a <product> node as all other nodes might not exceed that size.

    So why is the default configuration such that the "XML Bytes scanned" which is the size limit for the entire request is 4 MB and the size limit for a single node is 32MB, this second limit will never be used with 4 MB limitation...?
    Hi,
    >
    > So why is the default configuration such that the "XML Bytes scanned" which is the size limit for the entire request
    > is 4 MB and the size limit for a single node is 32MB, this second limit will never be used with 4 MB limitation...?
    >
    you are totally right.
    I tried to come up with different ways on hitting the 32MB limit (like Billion laughs attack).
    But always document size exceeded comes first -- so the 32MB default value is maybe not a good one ...

     
    Hermann<myXsltBlog/>
  • terza
    terza
    99 Posts

    Re: Best practices for limiting request size?

    ‏2011-07-14T13:14:56Z  
    • HermannSW
    • ‏2011-07-13T14:37:37Z
    Hi,
    >
    > So why is the default configuration such that the "XML Bytes scanned" which is the size limit for the entire request
    > is 4 MB and the size limit for a single node is 32MB, this second limit will never be used with 4 MB limitation...?
    >
    you are totally right.
    I tried to come up with different ways on hitting the 32MB limit (like Billion laughs attack).
    But always document size exceeded comes first -- so the 32MB default value is maybe not a good one ...

     
    Hermann<myXsltBlog/>
    Once again, back to this question. Hermann are you sure that its possible to hit the error rule inside a Web Services proxy?

    I have an error rule that matches for all errors but I cant reach it when I exceed the "XML bytes scanned" limitation. This is what I get in my log:
    16:10:37 ws-proxy error 10027281 192.162.1.17 0x80e0029e source-http (myFSH): No WS-Proxy service endpoints match request, envelope is missing or malformed.

    16:10:37 multistep error 10027281 request 192.168.1.17 0x80c00008 xmlmgr (default): rule (default): Pipeline beginning at implied action Parse input as XML, attempt pipeline failed: document size limit of 17000000 bytes exceeded, aborting

    16:10:37 xmlparse error 10027281 request 192.168.1.17 0x80e003aa xmlmgr (default): document size limit of 17000000 bytes exceeded, aborting

    I have verified that I hit the error rule with other scenarios, for example when the request is not schema valid.
  • meaton78
    meaton78
    25 Posts

    Re: Best practices for limiting request size?

    ‏2011-07-14T13:37:41Z  
    • terza
    • ‏2011-07-14T13:14:56Z
    Once again, back to this question. Hermann are you sure that its possible to hit the error rule inside a Web Services proxy?

    I have an error rule that matches for all errors but I cant reach it when I exceed the "XML bytes scanned" limitation. This is what I get in my log:
    16:10:37 ws-proxy error 10027281 192.162.1.17 0x80e0029e source-http (myFSH): No WS-Proxy service endpoints match request, envelope is missing or malformed.

    16:10:37 multistep error 10027281 request 192.168.1.17 0x80c00008 xmlmgr (default): rule (default): Pipeline beginning at implied action Parse input as XML, attempt pipeline failed: document size limit of 17000000 bytes exceeded, aborting

    16:10:37 xmlparse error 10027281 request 192.168.1.17 0x80e003aa xmlmgr (default): document size limit of 17000000 bytes exceeded, aborting

    I have verified that I hit the error rule with other scenarios, for example when the request is not schema valid.
    Go to Objects > Service Configuration > Web Service Proxy. Click on yours. Click on the Parser Limits tab and set the Maximun Message Size.

    Maximum Message Size
    Specifies the maximum size of a SOAP or XML message in kilobytes. If this value is 0, no limit is enforced.
  • terza
    terza
    99 Posts

    Re: Best practices for limiting request size?

    ‏2011-07-15T07:01:05Z  
    • meaton78
    • ‏2011-07-14T13:37:41Z
    Go to Objects > Service Configuration > Web Service Proxy. Click on yours. Click on the Parser Limits tab and set the Maximun Message Size.

    Maximum Message Size
    Specifies the maximum size of a SOAP or XML message in kilobytes. If this value is 0, no limit is enforced.
    Hi meaton78, your suggestion does not change the behaviour of the error handling. I can not enter a error rule when the size limits are exceed.

    I tested this with a XML Firewall and got it working instantly (hit the error rule), so im starting to wonder if this is another "feature" of the web services proxy? There seem to be similiar limitations for example error handling in WSP when the operation name is wrong.

    I would like to hear a confirmation if anyone really can hit the error rule inside a WSP when sending too large messages.
  • meaton78
    meaton78
    25 Posts

    Re: Best practices for limiting request size?

    ‏2011-07-15T17:30:44Z  
    • terza
    • ‏2011-07-15T07:01:05Z
    Hi meaton78, your suggestion does not change the behaviour of the error handling. I can not enter a error rule when the size limits are exceed.

    I tested this with a XML Firewall and got it working instantly (hit the error rule), so im starting to wonder if this is another "feature" of the web services proxy? There seem to be similiar limitations for example error handling in WSP when the operation name is wrong.

    I would like to hear a confirmation if anyone really can hit the error rule inside a WSP when sending too large messages.
    Sorry, I misunderstood what you were asking. I played around with it and got the same results. Looking at the logs, it appears that in the XML Firewall, the Max Message Size is handled internally, but in the Web Service Proxy, it is handled by the xml manager. I turned on the probe for the Web Service Proxy, and the message never makes it that far.

    I did see that the message size is included in var://service/input-size. If you really needed to hit the error rule, maybe you could do your own internal check and route accordingly?
  • irazabal
    irazabal
    218 Posts

    Re: Best practices for limiting request size?

    ‏2011-07-21T04:13:29Z  
    • terza
    • ‏2011-07-14T13:14:56Z
    Once again, back to this question. Hermann are you sure that its possible to hit the error rule inside a Web Services proxy?

    I have an error rule that matches for all errors but I cant reach it when I exceed the "XML bytes scanned" limitation. This is what I get in my log:
    16:10:37 ws-proxy error 10027281 192.162.1.17 0x80e0029e source-http (myFSH): No WS-Proxy service endpoints match request, envelope is missing or malformed.

    16:10:37 multistep error 10027281 request 192.168.1.17 0x80c00008 xmlmgr (default): rule (default): Pipeline beginning at implied action Parse input as XML, attempt pipeline failed: document size limit of 17000000 bytes exceeded, aborting

    16:10:37 xmlparse error 10027281 request 192.168.1.17 0x80e003aa xmlmgr (default): document size limit of 17000000 bytes exceeded, aborting

    I have verified that I hit the error rule with other scenarios, for example when the request is not schema valid.
    XML Parsing is done OUTSIDE the processing Policy (by the XML Manager object) - so when an parsing exception is thrown is can not be caught by the PP's error rule. I vaguely recall from a similar problem that if the parsing error was in the response, it would be possible to catch it with an error action ...but don't quote me on that.
  • terza
    terza
    99 Posts

    Re: Best practices for limiting request size?

    ‏2011-07-21T11:39:01Z  
    • irazabal
    • ‏2011-07-21T04:13:29Z
    XML Parsing is done OUTSIDE the processing Policy (by the XML Manager object) - so when an parsing exception is thrown is can not be caught by the PP's error rule. I vaguely recall from a similar problem that if the parsing error was in the response, it would be possible to catch it with an error action ...but don't quote me on that.
    > irazabal wrote:
    > XML Parsing is done OUTSIDE the processing Policy (by the XML Manager object) - so when an parsing exception is thrown is can not be caught by the PP's error rule. I vaguely recall from a similar problem that if the parsing error was in the response, it would be possible to catch it with an error action ...but don't quote me on that.

    This seems to be the case for Web Services Proxy. With XML firewall you can catch the error in the processing policy. I guess it means that I will have to add an XMLFW infront of my WSP.

    I would like to hear confirmation from Hermann just to be sure that the first comments were a misunderstanding because i was talking about a Web Services Proxy all the time...
  • Bankrupt
    Bankrupt
    39 Posts

    Re: Best practices for limiting request size?

    ‏2013-01-04T17:35:34Z  
    • terza
    • ‏2011-07-21T11:39:01Z
    > irazabal wrote:
    > XML Parsing is done OUTSIDE the processing Policy (by the XML Manager object) - so when an parsing exception is thrown is can not be caught by the PP's error rule. I vaguely recall from a similar problem that if the parsing error was in the response, it would be possible to catch it with an error action ...but don't quote me on that.

    This seems to be the case for Web Services Proxy. With XML firewall you can catch the error in the processing policy. I guess it means that I will have to add an XMLFW infront of my WSP.

    I would like to hear confirmation from Hermann just to be sure that the first comments were a misunderstanding because i was talking about a Web Services Proxy all the time...
    Hermann -- Any updates on this ? I am coming across the same issue where the 'Size Exceeded' condition is not being caught by the Error Rule. Let me know if there is something else we need to do so that the WSP's Error Rule can catch this condition.
    Thanks
  • msiebler
    msiebler
    140 Posts

    Re: Best practices for limiting request size?

    ‏2013-01-04T21:10:52Z  
    • Bankrupt
    • ‏2013-01-04T17:35:34Z
    Hermann -- Any updates on this ? I am coming across the same issue where the 'Size Exceeded' condition is not being caught by the Error Rule. Let me know if there is something else we need to do so that the WSP's Error Rule can catch this condition.
    Thanks
    I can confirm what Alex said; the problem is ; you can have multiple web service proxies that use the same port. The device needs to parse the message first to decide which is the right proxy to use. If the message is too big this error will be caught during parsing. So this happens before the request rule is run. And therefore no error rule can be invoked.
    Putting a fw first is a good workaround.
  • mdindagur
    mdindagur
    41 Posts

    Re: Best practices for limiting request size?

    ‏2013-03-08T17:58:47Z  
    • msiebler
    • ‏2013-01-04T21:10:52Z
    I can confirm what Alex said; the problem is ; you can have multiple web service proxies that use the same port. The device needs to parse the message first to decide which is the right proxy to use. If the message is too big this error will be caught during parsing. So this happens before the request rule is run. And therefore no error rule can be invoked.
    Putting a fw first is a good workaround.
    Hi,

    I do have a multiprotocol gateway behaving similar to web service proxy. i.e. xml size errors are not captured in error rule. Here is a sample log for your reference

    11:29:31 network error 26758513 request 168.38.103.250 0x80e00041 mpgw (mpg_external_gw): url-open: Error parsing response from 'cid:///PartName%3DE49BECC3-54D2-563A-7CFC-C63EE1060656@mindreef.com?Encode=hexbin'

    11:29:31 xmlparse error 26758513 request 168.38.103.250 0x80e003aa mpgw (mpg_external_gw): document size limit of 4194304 bytes exceeded, aborting

    11:29:31 network error 26758513 request 168.38.103.250 0x80e00041 mpgw (mpg_external_gw): url-open: Error parsing response from 'cid:///PartName%3DE49BECC3-54D2-563A-7CFC-C63EE1060656@mindreef.com?Encode=hexbin'

    11:29:31 xmlparse error 26758513 request 168.38.103.250 0x80e003aa mpgw (mpg_external_gw): document size limit of 4194304 bytes exceeded, aborting

    11:29:31 network error 26758513 request 168.38.103.250 0x80e00041 mpgw (mpg_external_gw): url-open: Error parsing response from 'cid:///PartName%3DE49BECC3-54D2-563A-7CFC-C63EE1060656@mindreef.com?Encode=hexbin'

    11:29:31 xmlparse error 26758513 request 168.38.103.250 0x80e003aa mpgw (mpg_external_gw): document size limit of 4194304 bytes exceeded, aborting

    11:29:31 network error 26758513 request 168.38.103.250 0x80e00041 mpgw (mpg_external_gw): url-open: Error parsing response from 'cid:///PartName%3DE49BECC3-54D2-563A-7CFC-C63EE1060656@mindreef.com?Encode=hexbin'

    11:29:31 xmlparse error 26758513 request 168.38.103.250 0x80e003aa mpgw (mpg_external_gw): document size limit of 4194304 bytes exceeded, aborting

    But strange thing about this is that the processing rule (request) is completed, response is back to consumer application as if nothing happened. The xml manager has XML Bytes Scanned set by default to 4194304(4mb). This is not caught as XML threat as the Maximum Message Size is set to '0'. What is this error about 'aborting'?

    Appreciate inputs from experts in this regard.
    thnaks,
    -Mahesh
  • HermannSW
    HermannSW
    4903 Posts

    Re: Best practices for limiting request size?

    ‏2013-03-10T00:10:10Z  
    • mdindagur
    • ‏2013-03-08T17:58:47Z
    Hi,

    I do have a multiprotocol gateway behaving similar to web service proxy. i.e. xml size errors are not captured in error rule. Here is a sample log for your reference

    11:29:31 network error 26758513 request 168.38.103.250 0x80e00041 mpgw (mpg_external_gw): url-open: Error parsing response from 'cid:///PartName%3DE49BECC3-54D2-563A-7CFC-C63EE1060656@mindreef.com?Encode=hexbin'

    11:29:31 xmlparse error 26758513 request 168.38.103.250 0x80e003aa mpgw (mpg_external_gw): document size limit of 4194304 bytes exceeded, aborting

    11:29:31 network error 26758513 request 168.38.103.250 0x80e00041 mpgw (mpg_external_gw): url-open: Error parsing response from 'cid:///PartName%3DE49BECC3-54D2-563A-7CFC-C63EE1060656@mindreef.com?Encode=hexbin'

    11:29:31 xmlparse error 26758513 request 168.38.103.250 0x80e003aa mpgw (mpg_external_gw): document size limit of 4194304 bytes exceeded, aborting

    11:29:31 network error 26758513 request 168.38.103.250 0x80e00041 mpgw (mpg_external_gw): url-open: Error parsing response from 'cid:///PartName%3DE49BECC3-54D2-563A-7CFC-C63EE1060656@mindreef.com?Encode=hexbin'

    11:29:31 xmlparse error 26758513 request 168.38.103.250 0x80e003aa mpgw (mpg_external_gw): document size limit of 4194304 bytes exceeded, aborting

    11:29:31 network error 26758513 request 168.38.103.250 0x80e00041 mpgw (mpg_external_gw): url-open: Error parsing response from 'cid:///PartName%3DE49BECC3-54D2-563A-7CFC-C63EE1060656@mindreef.com?Encode=hexbin'

    11:29:31 xmlparse error 26758513 request 168.38.103.250 0x80e003aa mpgw (mpg_external_gw): document size limit of 4194304 bytes exceeded, aborting

    But strange thing about this is that the processing rule (request) is completed, response is back to consumer application as if nothing happened. The xml manager has XML Bytes Scanned set by default to 4194304(4mb). This is not caught as XML threat as the Maximum Message Size is set to '0'. What is this error about 'aborting'?

    Appreciate inputs from experts in this regard.
    thnaks,
    -Mahesh
    Hi Mahesh,

    I agree with what Alex said.
    XML Parser limits are "normally" checked before entering rule processing.

    What your log messages do show is that you try to access an attachment by dp:url-open(or FETCH action).
    And you ask the attachments to be hex encoded (...?Encode=hexbin).

    dp:url-open as well as FETCH action DO check XML parser limits.
    Is it possible that your attachments are bigger than 4194304 bytes?

    Also, the log message says "aborting" -- and that means aborting of the dp:url-open/FETCH.
    There is no reason why the processing rule should be aborted.
    If you want that, you can check the size of what you got back and <xsl:message terminate="true"> or dp:reject explicitely.

     
    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>
  • mdindagur
    mdindagur
    41 Posts

    Re: Best practices for limiting request size?

    ‏2013-03-11T14:12:27Z  
    • HermannSW
    • ‏2013-03-10T00:10:10Z
    Hi Mahesh,

    I agree with what Alex said.
    XML Parser limits are "normally" checked before entering rule processing.

    What your log messages do show is that you try to access an attachment by dp:url-open(or FETCH action).
    And you ask the attachments to be hex encoded (...?Encode=hexbin).

    dp:url-open as well as FETCH action DO check XML parser limits.
    Is it possible that your attachments are bigger than 4194304 bytes?

    Also, the log message says "aborting" -- and that means aborting of the dp:url-open/FETCH.
    There is no reason why the processing rule should be aborted.
    If you want that, you can check the size of what you got back and <xsl:message terminate="true"> or dp:reject explicitely.

     
    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>
    Hi Hermann,

    Appreciate your response and explanation. In the scenario where I sent you the logs, I am using dp:url-open to decode hex-data from request into binary data for the next step (i.e. scan for virus/malware). In this scenario, when the dp:url-open is aborted, but the process action continues, do you think the result from malware scan action is a valid one?

    In another use case, the data is sent to datapower gateway as an attachment directly. In both cases, the original file I used to reproduce the issue was around 2,666kb (and nowhere near the 4096 kb xml parser limit).

    I am also not clear on how to provide the right response to service consumers about 'message or attachment size too large', when parser limit or xml threat protection message size limit is reached. As it is, the consumer at best can get a response 'internal error', which is the default behavior. We dont have a way to handle the parser/threat protection exceptions. Some mention was made of using a xml firewall as a front to the gateway, but I believe that is not suggested to be the best practice.

    Or is it that the consumer shouldn't even know why the message is rejected?
    thanks,
    -Mahesh
  • HermannSW
    HermannSW
    4903 Posts

    Re: Best practices for limiting request size?

    ‏2013-03-13T15:25:17Z  
    • mdindagur
    • ‏2013-03-11T14:12:27Z
    Hi Hermann,

    Appreciate your response and explanation. In the scenario where I sent you the logs, I am using dp:url-open to decode hex-data from request into binary data for the next step (i.e. scan for virus/malware). In this scenario, when the dp:url-open is aborted, but the process action continues, do you think the result from malware scan action is a valid one?

    In another use case, the data is sent to datapower gateway as an attachment directly. In both cases, the original file I used to reproduce the issue was around 2,666kb (and nowhere near the 4096 kb xml parser limit).

    I am also not clear on how to provide the right response to service consumers about 'message or attachment size too large', when parser limit or xml threat protection message size limit is reached. As it is, the consumer at best can get a response 'internal error', which is the default behavior. We dont have a way to handle the parser/threat protection exceptions. Some mention was made of using a xml firewall as a front to the gateway, but I believe that is not suggested to be the best practice.

    Or is it that the consumer shouldn't even know why the message is rejected?
    thanks,
    -Mahesh
    Hi,

    > Appreciate your response and explanation. In the scenario where I sent you the logs, I am using dp:url-open to decode hex-data from request into binary data for the next step (i.e. scan for virus/malware).
    > In this scenario, when the dp:url-open is aborted, but the process action continues, do you think the result from malware scan action is a valid one?
    >
    you have to check the reponse of <dp:url-open>, and either do a <dp:reject> to abort rule processing after Transform action completed,
    or <xsl:message terminate="yes">..</xsl:message> to immediately "kill" processing.

    > In another use case, the data is sent to datapower gateway as an attachment directly.
    > In both cases, the original file I used to reproduce the issue was around 2,666kb (and nowhere near the 4096 kb xml parser limit).
    >
    Numbers sometimes are not accurate to the byte level, eg. when Probe is on.
    On the other hand normally it is exact to the byte level, see here:
    https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14637065&#14637065

    > I am also not clear on how to provide the right response to service consumers about 'message or attachment size too large', when parser limit or xml threat protection message size limit is reached.
    > As it is, the consumer at best can get a response 'internal error', which is the default behavior.
    >
    The reason is that DataPower is a security device.
    Any verbose error message you provide gives possible attackers a hint.

    > We dont have a way to handle the parser/threat protection exceptions.
    >
    Sorry, no solution here, you may need to raise an Enhancement Request if you want that, as said DataPower behaves so intentionally:
    http://www.ibm.com/developerworks/rfe/

    > Some mention was made of using a xml firewall as a front to the gateway, but I believe that is not suggested to be the best practice.
    >
    And you may open up to run into exactly the XML threat issues you want to catch.
    If you do not abort malicious XML processing at the front MPGW you already have lost.

    > Or is it that the consumer shouldn't even know why the message is rejected?
    >
    Totally yes, at least in case of XML Threat protection aborts.

    The only "solution" I can think of is to have the front MPGW configured with (slightly) greater XML Parser limits.
    When request is too big for that setting, abort without details.
    If it passes and gets aborted in next service, you can provide the message you like.
     
    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>
  • mdindagur
    mdindagur
    41 Posts

    Re: Best practices for limiting request size?

    ‏2013-03-26T14:23:39Z  
    • HermannSW
    • ‏2013-03-13T15:25:17Z
    Hi,

    > Appreciate your response and explanation. In the scenario where I sent you the logs, I am using dp:url-open to decode hex-data from request into binary data for the next step (i.e. scan for virus/malware).
    > In this scenario, when the dp:url-open is aborted, but the process action continues, do you think the result from malware scan action is a valid one?
    >
    you have to check the reponse of <dp:url-open>, and either do a <dp:reject> to abort rule processing after Transform action completed,
    or <xsl:message terminate="yes">..</xsl:message> to immediately "kill" processing.

    > In another use case, the data is sent to datapower gateway as an attachment directly.
    > In both cases, the original file I used to reproduce the issue was around 2,666kb (and nowhere near the 4096 kb xml parser limit).
    >
    Numbers sometimes are not accurate to the byte level, eg. when Probe is on.
    On the other hand normally it is exact to the byte level, see here:
    https://www.ibm.com/developerworks/forums/thread.jspa?messageID=14637065&#14637065

    > I am also not clear on how to provide the right response to service consumers about 'message or attachment size too large', when parser limit or xml threat protection message size limit is reached.
    > As it is, the consumer at best can get a response 'internal error', which is the default behavior.
    >
    The reason is that DataPower is a security device.
    Any verbose error message you provide gives possible attackers a hint.

    > We dont have a way to handle the parser/threat protection exceptions.
    >
    Sorry, no solution here, you may need to raise an Enhancement Request if you want that, as said DataPower behaves so intentionally:
    http://www.ibm.com/developerworks/rfe/

    > Some mention was made of using a xml firewall as a front to the gateway, but I believe that is not suggested to be the best practice.
    >
    And you may open up to run into exactly the XML threat issues you want to catch.
    If you do not abort malicious XML processing at the front MPGW you already have lost.

    > Or is it that the consumer shouldn't even know why the message is rejected?
    >
    Totally yes, at least in case of XML Threat protection aborts.

    The only "solution" I can think of is to have the front MPGW configured with (slightly) greater XML Parser limits.
    When request is too big for that setting, abort without details.
    If it passes and gets aborted in next service, you can provide the message you like.
     
    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>
    Hi Hermann,

    Appreciate your response and clarifications. The information provided helps a lot in figuring out how XML threat protection works. However, I have one concern.

    You mentioned that dp:url-open action can be controlled to exit the processing rule when an error related to XML parser size happens. I have not tried the solution you mentioned, yet. But, I noticed, in a scenario where the message has an attachment of size x, and I am not using dp:url-open exclusively, the following error shows up in the logs:
    document size limit of 47185920 bytes exceeded, aborting
    In another scenario where node size is exceeded, I got the following error on the logs
    Max node size exceeded.

    In each such case, the processing rule is continued, and exception rule not invoked. Looking carefully at the logs, there seems to be a call to url-open internally, which eventually gives the errors. How do we control such internal calls where we are not using url-open function explicitly?

    Appreciate your response in this regard.
    thanks,
    -Mahesh
  • HermannSW
    HermannSW
    4903 Posts

    Re: Best practices for limiting request size?

    ‏2013-03-26T19:49:09Z  
    • mdindagur
    • ‏2013-03-26T14:23:39Z
    Hi Hermann,

    Appreciate your response and clarifications. The information provided helps a lot in figuring out how XML threat protection works. However, I have one concern.

    You mentioned that dp:url-open action can be controlled to exit the processing rule when an error related to XML parser size happens. I have not tried the solution you mentioned, yet. But, I noticed, in a scenario where the message has an attachment of size x, and I am not using dp:url-open exclusively, the following error shows up in the logs:
    document size limit of 47185920 bytes exceeded, aborting
    In another scenario where node size is exceeded, I got the following error on the logs
    Max node size exceeded.

    In each such case, the processing rule is continued, and exception rule not invoked. Looking carefully at the logs, there seems to be a call to url-open internally, which eventually gives the errors. How do we control such internal calls where we are not using url-open function explicitly?

    Appreciate your response in this regard.
    thanks,
    -Mahesh
    Hi Manesh,

    please carify your concerns, perhaps you have a specific scenario in mind?

    Any XML threat for the original input is catched by the XML threat protection of the policy.
    Any dp:url-open related XML threat detection does only invalidate the dp:url-open call.
    What is wrong with that? For the policy, as well as for the dp:url-open, independent XML threat protection is done.

     
    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>