Topic
14 replies Latest Post - ‏2013-06-12T14:34:27Z by sgb90
sgb90
sgb90
95 Posts
ACCEPTED ANSWER

Pinned topic Streamed Transactions in DataPower as an FTP Server

‏2013-05-15T21:21:39Z |

Hi everyone,

Here I am, again with a question about Datapower as an FTP Server.

Up to this date, my team and I only worked with Buffer Messages in our Stream Output to Back.

When switched this to Stream Messages, and tried to upload a file of 207 megabytes (which worked OK with Buffer Messages), the transaction aborts and the default log shows a message that says "Out of Memory".

Trying to resolve this problem, we created a Compile Options Policy that tries to stream the transformed document before DataPower finishes running the stylesheet. After configuring this, the log message change to "<xslname>.xsd <linenumber>: stylesheet not streamable" where the line number is one that has a <dp:set-variable> XSL function.

Is there something we're missing? It seems our problem is in DataPower and not in our real FTP Server.

 

Regards,

Sebastian

  • HermannSW
    HermannSW
    3145 Posts
    ACCEPTED ANSWER

    Re: Streamed Transactions in DataPower as an FTP Server

    ‏2013-05-16T07:40:50Z  in response to sgb90

    Hi Sebastian,

    DataPower can only stream if all parts can stream (front side, back side, processing).
    As the log message states the <dp:set-variable> statement is not streamable.

    There is a whole chapter "Optimizing through Streaming" in InfoCenter:
    http://pic.dhe.ibm.com/infocenter/wsdatap/v5r0m0/index.jsp?topic=%2Fcom.ibm.dp.xa.doc%2Foptimizingthroughstreaming.htm

    You may also find slides 7-10 of this WSTE Webcast interesting (audio recording for details) useful:
    http://www-01.ibm.com/support/docview.wss?uid=swg27019119


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

    • sgb90
      sgb90
      95 Posts
      ACCEPTED ANSWER

      Re: Streamed Transactions in DataPower as an FTP Server

      ‏2013-05-16T17:26:25Z  in response to HermannSW

      Hi Hermann,

      I have two other questions to ask after taking a look at the information you gave me:

      1- After reading the slides and listening to the presentation, I realized that I never stated that my 207 megabytes file was a binary file. So, the flat file definition would allow me to stream my transaction without any concern about the binary file, am I right? If this statement is ok, I need to ask you: Are there any workarounds to enable us to stream our stylesheet, even though at first sight it is not streamable? The thing is we're using the xsl to route the requests depending on the client IP and the file type (XML or Binary), so we can't get rid of it.

      2- Is that "Out of Memory" error message related to streaming the output to the backend sever?

       

      Thank you for replying, it's been really useful!

      Sebastian.

      • HermannSW
        HermannSW
        3145 Posts
        ACCEPTED ANSWER

        Re: Streamed Transactions in DataPower as an FTP Server

        ‏2013-05-17T09:17:58Z  in response to sgb90

        Hi Sebastian,

        > 2- Is that "Out of Memory" error message related to streaming the output to the backend sever?
        >
        I would expect the OoM is because the transaction did not stream and therefore everything got buffered on DataPower box.

        207MB should be processable on a clean box without traffic even in buffered mode, so I suspect that you had other traffic at that time.


        > Are there any workarounds to enable us to stream our stylesheet, even though at first sight it is not streamable?
        >
        What is input and output context of your stylesheet?
        Since you only want to process header variables input and output context NULL should be possible and may fix your problem.


        > The thing is we're using the xsl to route the requests depending on the client IP and the file type (XML or Binary), so we can't get rid of it.
        >
        I know that a Techsales person did streaming routing to different FTP servers based on header values like client IP.
        (with  NULL--Transform Action--NULL, not Transform Binary Action)
         

        What is definitely impossible with streaming is "consuming" any part of the input message twice.

        So if you look only into the very first bytes of the message to determine whether it is XML or Non-XML this would break streaming.

        Having two different endpoints, one for XML and one for Non-XML, should be doable.


        Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

        • sgb90
          sgb90
          95 Posts
          ACCEPTED ANSWER

          Re: Streamed Transactions in DataPower as an FTP Server

          ‏2013-05-17T14:12:08Z  in response to HermannSW

          Thank you again, Hermann.

          As you'll see in the stylesheet I attached, the thing about a NULL context is that we're routing ALWAYS to two different locations.

          One of the destinations is a folder that depends on the client who sends the file and the file type which is being sent. The other destination is always the same, we're using it as a "backup directory".

          We decided to use a context variable (/context/test/urls) so we're able to route to both destinations. Do you think there's a way to avoid this and still keep our routing layout even when using a null context?

           

          Regards,

          Sebastian

          Updated on 2013-05-17T14:12:40Z at 2013-05-17T14:12:40Z by sgb90
          • HermannSW
            HermannSW
            3145 Posts
            ACCEPTED ANSWER

            Re: Streamed Transactions in DataPower as an FTP Server

            ‏2013-05-17T19:55:04Z  in response to sgb90

            Your stylesheet uses dp:input-mapping and dp:output-mapping.

            As said before any of those breaks streaming.

            Since you do not seem to use the input at all just remove both and change the "Transform Binary" to a "Transform" action.

             

            Streaming means that DP starts sending data to backend before full request has been seen.

            That means that no reuse of information is possible.

            At the end of your stylesheet I see that you want input be sent to two outputs.

            That breaks streaming, sending to more than one output in streaming mode is not possible, sorry.

             


            Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

            • sgb90
              sgb90
              95 Posts
              ACCEPTED ANSWER

              Re: Streamed Transactions in DataPower as an FTP Server

              ‏2013-05-21T13:49:45Z  in response to HermannSW

              Hermman,

              Your tips have been really useful and helped us get closer to our goals. We managed to make streaming work in some cases (pass-thru or without transformations).

              Even though we achieved all of this, we haven't been able to stream our routing stylesheet, even when using NULL context as Input and Output of our Transform action (Not Binary). Do you have more information about that Techsales example? It would be really helpful.

               

              Regards,

              Sebastian

               

              PS: Sorry for keeping this thread for so long, but it should help us get a better picture of this functionality.

              • HermannSW
                HermannSW
                3145 Posts
                ACCEPTED ANSWER

                Re: Streamed Transactions in DataPower as an FTP Server

                ‏2013-05-22T15:28:52Z  in response to sgb90

                OK,

                so now I started a FTP service on my laptop and used below stylesheet in this service:

                • MPGW
                • request type Non-XML
                • response type Non-XML
                • HTTP FSH
                • dynamic backend
                • policy with
                  • match all
                  • NULL--Transform--NULL
                  • INPUT--RESULT--OUTPUT


                Sending request to that service just stores file on my ftp server with the filename being the request URI.
                Next I added a "Streaming Rule" to XML manager's Compile option policy to enforce streaming.
                Then I got the same "stylesheet is not streamable" error you see.

                I know that there is a solution, but I do not know it.

                I did try to contact the person having claimed to having done that for a customer in the past.

                 

                <xsl:stylesheet version="1.0"
                  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                  xmlns:dp="http://www.datapower.com/extensions"
                  extension-element-prefixes="dp"
                >
                  <xsl:output omit-xml-declaration="yes" />
                    
                  <xsl:template match="/">
                    <dp:set-variable name="'var://service/routing-url'"
                      value="concat('ftp://stammw:thatIsWhatYouWantToKnow@9.152.201.79/ftp-out',dp:variable('var://service/URI'))"
                    />
                  </xsl:template>
                 
                </xsl:stylesheet>

                 


                Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

                • sgb90
                  sgb90
                  95 Posts
                  ACCEPTED ANSWER

                  Re: Streamed Transactions in DataPower as an FTP Server

                  ‏2013-05-22T19:22:14Z  in response to HermannSW

                  Hermann,

                  Thank you for your help, we now have things a lot more clear. It's good to know that we're on the same scenario now. Let me know when you have a response from this persone to continue.

                   

                  Regards,

                  Sebastian

                  • agrohman
                    agrohman
                    60 Posts
                    ACCEPTED ANSWER

                    Re: Streamed Transactions in DataPower as an FTP Server

                    ‏2013-05-23T19:05:43Z  in response to sgb90

                    Hi Sebastian,

                    So, I'm the one that used dynamic routing with streaming binary files.  I haven't looked at your stylesheet, but from just reading through the thread, I don't think what you want is possible.  Specifically, you can't create two copies of input in streaming mode.  It's meant for simple 1:1 input/output use cases.  You can set the destination dynamically by, as you discovered, executing a stylesheet with NULL input that sets the var://service/routing-url variable.  But you can not have more than one destination -- that will require buffering the message, which disables the streaming.

                    An alternative might be to transfer the file twice... Once to the backup directory, and then poll the backup directory to send it to the real destination, using a rename rule instead of a delete rule to consider the file from the backup directory as 'processed,' and avoiding processing it repeatedly.

                    Just a thought -- no idea if it'd work, as you might lose some of the original input data required for the dynamic destination, anyway.  But, maybe renaming the files dynamically can help you get it all worked out.

                    Good luck,

                    Andy

                    • sgb90
                      sgb90
                      95 Posts
                      ACCEPTED ANSWER

                      Re: Streamed Transactions in DataPower as an FTP Server

                      ‏2013-05-24T20:25:35Z  in response to agrohman

                      Hi Andy!

                      Thank you for replying, nice to have you as reference.

                      The last test we ran had a single destination routing, using a stylesheet with a <dp:set-variable> sentence which changed the value of var://service/routing-url variable between NULL contexts. Even though we did this, it didn't work and kept taking the stylesheet as not streamable.

                      So, we already stopped trying to route to multiple destinations (for e.g. 1:2 input/output) and are still having the same result. Any idea why? Please, let me know if you need anything about our work so you can take a closer look.

                      Regards,

                      Sebastián

                       

                      • agrohman
                        agrohman
                        60 Posts
                        ACCEPTED ANSWER

                        Re: Streamed Transactions in DataPower as an FTP Server

                        ‏2013-05-29T20:17:19Z  in response to sgb90

                        I honestly can't recall.  I am pretty sure I just had everything set to Stream and processing set to Non-XML, and this worked.

                        • sgb90
                          sgb90
                          95 Posts
                          ACCEPTED ANSWER

                          Re: Streamed Transactions in DataPower as an FTP Server

                          ‏2013-06-12T14:16:37Z  in response to agrohman

                          Hi Andy,

                          We're still working on this issue, but still have the same result.

                          Can I send you an export of our actual MPGW so you can take a look? Maybe there's something we're missing. We want to rule out any problems in our set-up.

                          Thank you and again, sorry for keeping this post for so long.

                          Regards,

                          Sebastian

                           

                          • agrohman
                            agrohman
                            60 Posts
                            ACCEPTED ANSWER

                            Re: Streamed Transactions in DataPower as an FTP Server

                            ‏2013-06-12T14:28:29Z  in response to sgb90

                            Hi Sebastian,

                             

                            I've taken a new role with IBM, so I won't be able to dig into your export and validate like I used to.  I encourage you to reach out to your local Technical Sales Specialist for this type of troubleshooting assistance.  You can connect him/her to me, and we can chat internally about it.

                             

                            Andy

                            • sgb90
                              sgb90
                              95 Posts
                              ACCEPTED ANSWER

                              Re: Streamed Transactions in DataPower as an FTP Server

                              ‏2013-06-12T14:34:27Z  in response to agrohman

                              Thank you for the quick response Andy, I will certainly do that. He will return this Friday, so I guess we'll be getting in touch around that day.

                               

                              Regards,

                              Sebastian