Topic
6 replies Latest Post - ‏2012-12-10T20:24:49Z by SystemAdmin
umav
umav
22 Posts
ACCEPTED ANSWER

Pinned topic Storing a big request in a context variable

‏2012-05-30T13:05:58Z |
In a MPG, there is a processing rule for request and a processing rule for response. We are storing the original request in the request processing rule to a context variable to be used later in the processing rule for response.

The original request is kind of big (may be around 5-10K). Is there any performance penalty in storing this big original request in the context variable?
Updated on 2012-12-10T20:24:49Z at 2012-12-10T20:24:49Z by SystemAdmin
  • swlinn
    swlinn
    1344 Posts
    ACCEPTED ANSWER

    Re: Storing a big request in a context variable

    ‏2012-05-30T13:59:44Z  in response to umav
    I'm used to seeing requests and responses placed into context variables for packaging in the response rule of an audit type of message, and generally the larger messages are responses, of which I've seen 10-50K sizes. From a performance point of view, the memory footprint of the transaction will obviously contain these context variables, but I have not seen messages of that size to be an issue unless the backend is very slow. The transaction footprint will hang around until the entire request and response rules are completed. The longer the backend latency, the more used memory you'll see on the appliance, which under load could reduce your throughput. You could of course mitigate this somewhat by setting a reasonable backside timeout instead of defaulting to 120 seconds. Performance and memory utilization would be something I'd monitor during a load tests to understand what the capacity of your DataPower application will be under varying backend conditions.

    As for the data itself, unless you're keeping the entire request for auditing purposes, I'd think keeping the entire request in a context variable for response processing is somewhat unusual. If you need only certain parts of the request, could you store just those parts in 1..n context variables instead of the entire request? No matter what the programming language or platform, minimizing the memory footprint is always a good thing.

    Best Regards,
    Steve
  • HermannSW
    HermannSW
    4357 Posts
    ACCEPTED ANSWER

    Re: Storing a big request in a context variable

    ‏2012-05-30T14:21:12Z  in response to umav
    > In a MPG, there is a processing rule for request and a processing rule for response. We are storing the original
    > request in the request processing rule to a context variable to be used later in the processing rule for response.
    >
    > The original request is kind of big (may be around 5-10K). Is there any performance penalty in storing this big original request in the context variable?
    >
    Yes, there is, but it is a really small penalty (less than 95 microseconds per request).

    A simple method to do measurements on DataPower at sub millisecond level is Stylesheet Profiling:
    http://www-304.ibm.com/support/docview.wss?uid=swg27019118&aid=1#page=4.

    I did send 1000 reqests of size 7348 bytes each against below described service.
    As you can see the total time for request and response stylesheet processing is 95 microcseconds per single request.



    This is stylesheet sv.xsl used on request rule, which copies INPUT context to 'var://context/cpyINPUT':
    
    <xsl:stylesheet version=
    "1.0" xmlns:xsl=
    "http://www.w3.org/1999/XSL/Transform" xmlns:dp=
    "http://www.datapower.com/extensions" extension-element-prefixes=
    "dp" > <xsl:output omit-xml-declaration=
    "yes" /> <xsl:template match=
    "/"> <dp:set-variable name=
    "'var://context/cpyINPUT'" value=
    "*" /> </xsl:template> </xsl:stylesheet>
    

    And stylesheet v.xsl used on response rule just reads that copied context.
    
    <xsl:stylesheet version=
    "1.0" xmlns:xsl=
    "http://www.w3.org/1999/XSL/Transform" xmlns:dp=
    "http://www.datapower.com/extensions" extension-element-prefixes=
    "dp" > <xsl:output omit-xml-declaration=
    "yes" /> <xsl:template match=
    "/"> <xsl:copy-of select=
    "dp:variable('var://context/cpyINPUT')" /> <xsl:copy-of select=
    "." /> </xsl:template> </xsl:stylesheet>
    


     
    Hermann<myXsltBlog/> <myXsltTweets/>
    • zachahuy83
      zachahuy83
      45 Posts
      ACCEPTED ANSWER

      Re: Storing a big request in a context variable

      ‏2012-09-18T22:19:22Z  in response to HermannSW
      Hi,

      I have been following the same method. We store the entire request in a variable then place it in the log in combination with error or response during server-to-client rule.

      However, on a WS-Proxy during a schema validation error the request rule does not even get executed so our storing action did not run. In this case, how would we retrieve the original request message even though it failed schema validation?
      • HermannSW
        HermannSW
        4357 Posts
        ACCEPTED ANSWER

        Re: Storing a big request in a context variable

        ‏2012-09-19T00:50:52Z  in response to zachahuy83
        Hi,

        > ...
        > However, on a WS-Proxy during a schema validation error the request rule does not even get executed so our storing action did not run.
        > In this case, how would we retrieve the original request message even though it failed schema validation?
        >
        as you said, WSP rule processing does not get started and therefore WSP cannot do that for you.

        If you really need to store requests which do not pass schema validation, you may chain a MPGW before the WSP just storing
        the (XML?) request in a context variable and pass the input unmodified to the WSP, response rule could be pass-thru for this.

         
        Hermann<myXsltBlog/> <myXsltTweets/>
  • Saakitm
    Saakitm
    2 Posts
    ACCEPTED ANSWER

    Re: Storing a big request in a context variable

    ‏2012-09-19T08:24:32Z  in response to umav
    Hi,

    I have been following the same method. We store the entire request in a variable then place it in the log in combination with error or response during server-to-client rule.

    However, on a WS-Proxy during a schema validation error the request rule does not even get executed so our storing action did not run. In this case, how would we retrieve the original request message even though it failed schema validation?

    ***********************

    Hi,

    You can capture the incoming request in Error rule of WSP. Usually in Error rule whatever requests goes into request rule will be passed into the error rule if any error is encountered and unless we customize the error rule.

    Even if you want to display you can display the request which failed in faultmessage

    Thanks
    Saaki
    • SystemAdmin
      SystemAdmin
      6772 Posts
      ACCEPTED ANSWER

      Re: Storing a big request in a context variable

      ‏2012-12-10T20:24:49Z  in response to Saakitm
      1) If we make the side calls, like wrapping the messages into soap and send to the required system to process them. Even then also we run into this issue ? Do we need to pay the price for this i.e. hurting the perfomance ...

      2) If we make the async calls can we overcome this issue ...

      3) Does ITCAM for SOA gives us this option ? Do we've the flexiblity to store them for the required amount of time in terms of days, weeks, months ... can we be the drivers if we use this tool rather ... do not want to boggle down with another issue ...

      Please let me know.

      this is ( all the 3 ) wrt to request and response ( payload logging ).

      Thanks! -- Salla