Topic
  • 6 replies
  • Latest Post - ‏2012-12-10T20:24:49Z by SystemAdmin
umav
umav
22 Posts

Pinned topic Storing a big request in a context variable

‏2012-05-30T13:05:58Z |
In a MPG, there is a processing rule for request and a processing rule for response. We are storing the original request in the request processing rule to a context variable to be used later in the processing rule for response.

The original request is kind of big (may be around 5-10K). Is there any performance penalty in storing this big original request in the context variable?
Updated on 2012-12-10T20:24:49Z at 2012-12-10T20:24:49Z by SystemAdmin
  • swlinn
    swlinn
    1347 Posts

    Re: Storing a big request in a context variable

    ‏2012-05-30T13:59:44Z  
    I'm used to seeing requests and responses placed into context variables for packaging in the response rule of an audit type of message, and generally the larger messages are responses, of which I've seen 10-50K sizes. From a performance point of view, the memory footprint of the transaction will obviously contain these context variables, but I have not seen messages of that size to be an issue unless the backend is very slow. The transaction footprint will hang around until the entire request and response rules are completed. The longer the backend latency, the more used memory you'll see on the appliance, which under load could reduce your throughput. You could of course mitigate this somewhat by setting a reasonable backside timeout instead of defaulting to 120 seconds. Performance and memory utilization would be something I'd monitor during a load tests to understand what the capacity of your DataPower application will be under varying backend conditions.

    As for the data itself, unless you're keeping the entire request for auditing purposes, I'd think keeping the entire request in a context variable for response processing is somewhat unusual. If you need only certain parts of the request, could you store just those parts in 1..n context variables instead of the entire request? No matter what the programming language or platform, minimizing the memory footprint is always a good thing.

    Best Regards,
    Steve
  • HermannSW
    HermannSW
    4640 Posts

    Re: Storing a big request in a context variable

    ‏2012-05-30T14:21:12Z  
    > In a MPG, there is a processing rule for request and a processing rule for response. We are storing the original
    > request in the request processing rule to a context variable to be used later in the processing rule for response.
    >
    > The original request is kind of big (may be around 5-10K). Is there any performance penalty in storing this big original request in the context variable?
    >
    Yes, there is, but it is a really small penalty (less than 95 microseconds per request).

    A simple method to do measurements on DataPower at sub millisecond level is Stylesheet Profiling:
    http://www-304.ibm.com/support/docview.wss?uid=swg27019118&aid=1#page=4.

    I did send 1000 reqests of size 7348 bytes each against below described service.
    As you can see the total time for request and response stylesheet processing is 95 microcseconds per single request.



    This is stylesheet sv.xsl used on request rule, which copies INPUT context to 'var://context/cpyINPUT':
    
    <xsl:stylesheet version=
    "1.0" xmlns:xsl=
    "http://www.w3.org/1999/XSL/Transform" xmlns:dp=
    "http://www.datapower.com/extensions" extension-element-prefixes=
    "dp" > <xsl:output omit-xml-declaration=
    "yes" /> <xsl:template match=
    "/"> <dp:set-variable name=
    "'var://context/cpyINPUT'" value=
    "*" /> </xsl:template> </xsl:stylesheet>
    

    And stylesheet v.xsl used on response rule just reads that copied context.
    
    <xsl:stylesheet version=
    "1.0" xmlns:xsl=
    "http://www.w3.org/1999/XSL/Transform" xmlns:dp=
    "http://www.datapower.com/extensions" extension-element-prefixes=
    "dp" > <xsl:output omit-xml-declaration=
    "yes" /> <xsl:template match=
    "/"> <xsl:copy-of select=
    "dp:variable('var://context/cpyINPUT')" /> <xsl:copy-of select=
    "." /> </xsl:template> </xsl:stylesheet>
    


     
    Hermann<myXsltBlog/> <myXsltTweets/>
  • zachahuy83
    zachahuy83
    45 Posts

    Re: Storing a big request in a context variable

    ‏2012-09-18T22:19:22Z  
    • HermannSW
    • ‏2012-05-30T14:21:12Z
    > In a MPG, there is a processing rule for request and a processing rule for response. We are storing the original
    > request in the request processing rule to a context variable to be used later in the processing rule for response.
    >
    > The original request is kind of big (may be around 5-10K). Is there any performance penalty in storing this big original request in the context variable?
    >
    Yes, there is, but it is a really small penalty (less than 95 microseconds per request).

    A simple method to do measurements on DataPower at sub millisecond level is Stylesheet Profiling:
    http://www-304.ibm.com/support/docview.wss?uid=swg27019118&aid=1#page=4.

    I did send 1000 reqests of size 7348 bytes each against below described service.
    As you can see the total time for request and response stylesheet processing is 95 microcseconds per single request.



    This is stylesheet sv.xsl used on request rule, which copies INPUT context to 'var://context/cpyINPUT':
    <pre class="jive-pre"> <xsl:stylesheet version= "1.0" xmlns:xsl= "http://www.w3.org/1999/XSL/Transform" xmlns:dp= "http://www.datapower.com/extensions" extension-element-prefixes= "dp" > <xsl:output omit-xml-declaration= "yes" /> <xsl:template match= "/"> <dp:set-variable name= "'var://context/cpyINPUT'" value= "*" /> </xsl:template> </xsl:stylesheet> </pre>
    And stylesheet v.xsl used on response rule just reads that copied context.
    <pre class="jive-pre"> <xsl:stylesheet version= "1.0" xmlns:xsl= "http://www.w3.org/1999/XSL/Transform" xmlns:dp= "http://www.datapower.com/extensions" extension-element-prefixes= "dp" > <xsl:output omit-xml-declaration= "yes" /> <xsl:template match= "/"> <xsl:copy-of select= "dp:variable('var://context/cpyINPUT')" /> <xsl:copy-of select= "." /> </xsl:template> </xsl:stylesheet> </pre>

     
    Hermann<myXsltBlog/> <myXsltTweets/>
    Hi,

    I have been following the same method. We store the entire request in a variable then place it in the log in combination with error or response during server-to-client rule.

    However, on a WS-Proxy during a schema validation error the request rule does not even get executed so our storing action did not run. In this case, how would we retrieve the original request message even though it failed schema validation?
  • HermannSW
    HermannSW
    4640 Posts

    Re: Storing a big request in a context variable

    ‏2012-09-19T00:50:52Z  
    Hi,

    I have been following the same method. We store the entire request in a variable then place it in the log in combination with error or response during server-to-client rule.

    However, on a WS-Proxy during a schema validation error the request rule does not even get executed so our storing action did not run. In this case, how would we retrieve the original request message even though it failed schema validation?
    Hi,

    > ...
    > However, on a WS-Proxy during a schema validation error the request rule does not even get executed so our storing action did not run.
    > In this case, how would we retrieve the original request message even though it failed schema validation?
    >
    as you said, WSP rule processing does not get started and therefore WSP cannot do that for you.

    If you really need to store requests which do not pass schema validation, you may chain a MPGW before the WSP just storing
    the (XML?) request in a context variable and pass the input unmodified to the WSP, response rule could be pass-thru for this.

     
    Hermann<myXsltBlog/> <myXsltTweets/>
  • Saakitm
    Saakitm
    2 Posts

    Re: Storing a big request in a context variable

    ‏2012-09-19T08:24:32Z  
    Hi,

    I have been following the same method. We store the entire request in a variable then place it in the log in combination with error or response during server-to-client rule.

    However, on a WS-Proxy during a schema validation error the request rule does not even get executed so our storing action did not run. In this case, how would we retrieve the original request message even though it failed schema validation?

    ***********************

    Hi,

    You can capture the incoming request in Error rule of WSP. Usually in Error rule whatever requests goes into request rule will be passed into the error rule if any error is encountered and unless we customize the error rule.

    Even if you want to display you can display the request which failed in faultmessage

    Thanks
    Saaki
  • SystemAdmin
    SystemAdmin
    6772 Posts

    Re: Storing a big request in a context variable

    ‏2012-12-10T20:24:49Z  
    • Saakitm
    • ‏2012-09-19T08:24:32Z
    Hi,

    I have been following the same method. We store the entire request in a variable then place it in the log in combination with error or response during server-to-client rule.

    However, on a WS-Proxy during a schema validation error the request rule does not even get executed so our storing action did not run. In this case, how would we retrieve the original request message even though it failed schema validation?

    ***********************

    Hi,

    You can capture the incoming request in Error rule of WSP. Usually in Error rule whatever requests goes into request rule will be passed into the error rule if any error is encountered and unless we customize the error rule.

    Even if you want to display you can display the request which failed in faultmessage

    Thanks
    Saaki
    1) If we make the side calls, like wrapping the messages into soap and send to the required system to process them. Even then also we run into this issue ? Do we need to pay the price for this i.e. hurting the perfomance ...

    2) If we make the async calls can we overcome this issue ...

    3) Does ITCAM for SOA gives us this option ? Do we've the flexiblity to store them for the required amount of time in terms of days, weeks, months ... can we be the drivers if we use this tool rather ... do not want to boggle down with another issue ...

    Please let me know.

    this is ( all the 3 ) wrt to request and response ( payload logging ).

    Thanks! -- Salla