Topic
  • 14 replies
  • Latest Post - ‏2013-06-12T14:34:27Z by sgb90
sgb90
sgb90
114 Posts

Pinned topic Streamed Transactions in DataPower as an FTP Server

‏2013-05-15T21:21:39Z |

Hi everyone,

Here I am, again with a question about Datapower as an FTP Server.

Up to this date, my team and I only worked with Buffer Messages in our Stream Output to Back.

When switched this to Stream Messages, and tried to upload a file of 207 megabytes (which worked OK with Buffer Messages), the transaction aborts and the default log shows a message that says "Out of Memory".

Trying to resolve this problem, we created a Compile Options Policy that tries to stream the transformed document before DataPower finishes running the stylesheet. After configuring this, the log message change to "<xslname>.xsd <linenumber>: stylesheet not streamable" where the line number is one that has a <dp:set-variable> XSL function.

Is there something we're missing? It seems our problem is in DataPower and not in our real FTP Server.

 

Regards,

Sebastian

  • agrohman
    agrohman
    60 Posts
    ACCEPTED ANSWER

    Re: Streamed Transactions in DataPower as an FTP Server

    ‏2013-06-12T14:28:29Z  
    • sgb90
    • ‏2013-06-12T14:16:37Z

    Hi Andy,

    We're still working on this issue, but still have the same result.

    Can I send you an export of our actual MPGW so you can take a look? Maybe there's something we're missing. We want to rule out any problems in our set-up.

    Thank you and again, sorry for keeping this post for so long.

    Regards,

    Sebastian

     

    Hi Sebastian,

     

    I've taken a new role with IBM, so I won't be able to dig into your export and validate like I used to.  I encourage you to reach out to your local Technical Sales Specialist for this type of troubleshooting assistance.  You can connect him/her to me, and we can chat internally about it.

     

    Andy

  • HermannSW
    HermannSW
    4735 Posts

    Re: Streamed Transactions in DataPower as an FTP Server

    ‏2013-05-16T07:40:50Z  

    Hi Sebastian,

    DataPower can only stream if all parts can stream (front side, back side, processing).
    As the log message states the <dp:set-variable> statement is not streamable.

    There is a whole chapter "Optimizing through Streaming" in InfoCenter:
    http://pic.dhe.ibm.com/infocenter/wsdatap/v5r0m0/index.jsp?topic=%2Fcom.ibm.dp.xa.doc%2Foptimizingthroughstreaming.htm

    You may also find slides 7-10 of this WSTE Webcast interesting (audio recording for details) useful:
    http://www-01.ibm.com/support/docview.wss?uid=swg27019119


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

  • sgb90
    sgb90
    114 Posts

    Re: Streamed Transactions in DataPower as an FTP Server

    ‏2013-05-16T17:26:25Z  
    • HermannSW
    • ‏2013-05-16T07:40:50Z

    Hi Sebastian,

    DataPower can only stream if all parts can stream (front side, back side, processing).
    As the log message states the <dp:set-variable> statement is not streamable.

    There is a whole chapter "Optimizing through Streaming" in InfoCenter:
    http://pic.dhe.ibm.com/infocenter/wsdatap/v5r0m0/index.jsp?topic=%2Fcom.ibm.dp.xa.doc%2Foptimizingthroughstreaming.htm

    You may also find slides 7-10 of this WSTE Webcast interesting (audio recording for details) useful:
    http://www-01.ibm.com/support/docview.wss?uid=swg27019119


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

    Hi Hermann,

    I have two other questions to ask after taking a look at the information you gave me:

    1- After reading the slides and listening to the presentation, I realized that I never stated that my 207 megabytes file was a binary file. So, the flat file definition would allow me to stream my transaction without any concern about the binary file, am I right? If this statement is ok, I need to ask you: Are there any workarounds to enable us to stream our stylesheet, even though at first sight it is not streamable? The thing is we're using the xsl to route the requests depending on the client IP and the file type (XML or Binary), so we can't get rid of it.

    2- Is that "Out of Memory" error message related to streaming the output to the backend sever?

     

    Thank you for replying, it's been really useful!

    Sebastian.

  • HermannSW
    HermannSW
    4735 Posts

    Re: Streamed Transactions in DataPower as an FTP Server

    ‏2013-05-17T09:17:58Z  
    • sgb90
    • ‏2013-05-16T17:26:25Z

    Hi Hermann,

    I have two other questions to ask after taking a look at the information you gave me:

    1- After reading the slides and listening to the presentation, I realized that I never stated that my 207 megabytes file was a binary file. So, the flat file definition would allow me to stream my transaction without any concern about the binary file, am I right? If this statement is ok, I need to ask you: Are there any workarounds to enable us to stream our stylesheet, even though at first sight it is not streamable? The thing is we're using the xsl to route the requests depending on the client IP and the file type (XML or Binary), so we can't get rid of it.

    2- Is that "Out of Memory" error message related to streaming the output to the backend sever?

     

    Thank you for replying, it's been really useful!

    Sebastian.

    Hi Sebastian,

    > 2- Is that "Out of Memory" error message related to streaming the output to the backend sever?
    >
    I would expect the OoM is because the transaction did not stream and therefore everything got buffered on DataPower box.

    207MB should be processable on a clean box without traffic even in buffered mode, so I suspect that you had other traffic at that time.


    > Are there any workarounds to enable us to stream our stylesheet, even though at first sight it is not streamable?
    >
    What is input and output context of your stylesheet?
    Since you only want to process header variables input and output context NULL should be possible and may fix your problem.


    > The thing is we're using the xsl to route the requests depending on the client IP and the file type (XML or Binary), so we can't get rid of it.
    >
    I know that a Techsales person did streaming routing to different FTP servers based on header values like client IP.
    (with  NULL--Transform Action--NULL, not Transform Binary Action)
     

    What is definitely impossible with streaming is "consuming" any part of the input message twice.

    So if you look only into the very first bytes of the message to determine whether it is XML or Non-XML this would break streaming.

    Having two different endpoints, one for XML and one for Non-XML, should be doable.


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

  • sgb90
    sgb90
    114 Posts

    Re: Streamed Transactions in DataPower as an FTP Server

    ‏2013-05-17T14:12:08Z  
    • HermannSW
    • ‏2013-05-17T09:17:58Z

    Hi Sebastian,

    > 2- Is that "Out of Memory" error message related to streaming the output to the backend sever?
    >
    I would expect the OoM is because the transaction did not stream and therefore everything got buffered on DataPower box.

    207MB should be processable on a clean box without traffic even in buffered mode, so I suspect that you had other traffic at that time.


    > Are there any workarounds to enable us to stream our stylesheet, even though at first sight it is not streamable?
    >
    What is input and output context of your stylesheet?
    Since you only want to process header variables input and output context NULL should be possible and may fix your problem.


    > The thing is we're using the xsl to route the requests depending on the client IP and the file type (XML or Binary), so we can't get rid of it.
    >
    I know that a Techsales person did streaming routing to different FTP servers based on header values like client IP.
    (with  NULL--Transform Action--NULL, not Transform Binary Action)
     

    What is definitely impossible with streaming is "consuming" any part of the input message twice.

    So if you look only into the very first bytes of the message to determine whether it is XML or Non-XML this would break streaming.

    Having two different endpoints, one for XML and one for Non-XML, should be doable.


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

    Thank you again, Hermann.

    As you'll see in the stylesheet I attached, the thing about a NULL context is that we're routing ALWAYS to two different locations.

    One of the destinations is a folder that depends on the client who sends the file and the file type which is being sent. The other destination is always the same, we're using it as a "backup directory".

    We decided to use a context variable (/context/test/urls) so we're able to route to both destinations. Do you think there's a way to avoid this and still keep our routing layout even when using a null context?

     

    Regards,

    Sebastian

    Updated on 2013-05-17T14:12:40Z at 2013-05-17T14:12:40Z by sgb90
  • HermannSW
    HermannSW
    4735 Posts

    Re: Streamed Transactions in DataPower as an FTP Server

    ‏2013-05-17T19:55:04Z  
    • sgb90
    • ‏2013-05-17T14:12:08Z

    Thank you again, Hermann.

    As you'll see in the stylesheet I attached, the thing about a NULL context is that we're routing ALWAYS to two different locations.

    One of the destinations is a folder that depends on the client who sends the file and the file type which is being sent. The other destination is always the same, we're using it as a "backup directory".

    We decided to use a context variable (/context/test/urls) so we're able to route to both destinations. Do you think there's a way to avoid this and still keep our routing layout even when using a null context?

     

    Regards,

    Sebastian

    Your stylesheet uses dp:input-mapping and dp:output-mapping.

    As said before any of those breaks streaming.

    Since you do not seem to use the input at all just remove both and change the "Transform Binary" to a "Transform" action.

     

    Streaming means that DP starts sending data to backend before full request has been seen.

    That means that no reuse of information is possible.

    At the end of your stylesheet I see that you want input be sent to two outputs.

    That breaks streaming, sending to more than one output in streaming mode is not possible, sorry.

     


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

  • sgb90
    sgb90
    114 Posts

    Re: Streamed Transactions in DataPower as an FTP Server

    ‏2013-05-21T13:49:45Z  
    • HermannSW
    • ‏2013-05-17T19:55:04Z

    Your stylesheet uses dp:input-mapping and dp:output-mapping.

    As said before any of those breaks streaming.

    Since you do not seem to use the input at all just remove both and change the "Transform Binary" to a "Transform" action.

     

    Streaming means that DP starts sending data to backend before full request has been seen.

    That means that no reuse of information is possible.

    At the end of your stylesheet I see that you want input be sent to two outputs.

    That breaks streaming, sending to more than one output in streaming mode is not possible, sorry.

     


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

    Hermman,

    Your tips have been really useful and helped us get closer to our goals. We managed to make streaming work in some cases (pass-thru or without transformations).

    Even though we achieved all of this, we haven't been able to stream our routing stylesheet, even when using NULL context as Input and Output of our Transform action (Not Binary). Do you have more information about that Techsales example? It would be really helpful.

     

    Regards,

    Sebastian

     

    PS: Sorry for keeping this thread for so long, but it should help us get a better picture of this functionality.

  • HermannSW
    HermannSW
    4735 Posts

    Re: Streamed Transactions in DataPower as an FTP Server

    ‏2013-05-22T15:28:52Z  
    • sgb90
    • ‏2013-05-21T13:49:45Z

    Hermman,

    Your tips have been really useful and helped us get closer to our goals. We managed to make streaming work in some cases (pass-thru or without transformations).

    Even though we achieved all of this, we haven't been able to stream our routing stylesheet, even when using NULL context as Input and Output of our Transform action (Not Binary). Do you have more information about that Techsales example? It would be really helpful.

     

    Regards,

    Sebastian

     

    PS: Sorry for keeping this thread for so long, but it should help us get a better picture of this functionality.

    OK,

    so now I started a FTP service on my laptop and used below stylesheet in this service:

    • MPGW
    • request type Non-XML
    • response type Non-XML
    • HTTP FSH
    • dynamic backend
    • policy with
      • match all
      • NULL--Transform--NULL
      • INPUT--RESULT--OUTPUT


    Sending request to that service just stores file on my ftp server with the filename being the request URI.
    Next I added a "Streaming Rule" to XML manager's Compile option policy to enforce streaming.
    Then I got the same "stylesheet is not streamable" error you see.

    I know that there is a solution, but I do not know it.

    I did try to contact the person having claimed to having done that for a customer in the past.

     

    <xsl:stylesheet version="1.0"
      xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
      xmlns:dp="http://www.datapower.com/extensions"
      extension-element-prefixes="dp"
    >
      <xsl:output omit-xml-declaration="yes" />
        
      <xsl:template match="/">
        <dp:set-variable name="'var://service/routing-url'"
          value="concat('ftp://stammw:thatIsWhatYouWantToKnow@9.152.201.79/ftp-out',dp:variable('var://service/URI'))"
        />
      </xsl:template>
     
    </xsl:stylesheet>

     


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

  • sgb90
    sgb90
    114 Posts

    Re: Streamed Transactions in DataPower as an FTP Server

    ‏2013-05-22T19:22:14Z  
    • HermannSW
    • ‏2013-05-22T15:28:52Z

    OK,

    so now I started a FTP service on my laptop and used below stylesheet in this service:

    • MPGW
    • request type Non-XML
    • response type Non-XML
    • HTTP FSH
    • dynamic backend
    • policy with
      • match all
      • NULL--Transform--NULL
      • INPUT--RESULT--OUTPUT


    Sending request to that service just stores file on my ftp server with the filename being the request URI.
    Next I added a "Streaming Rule" to XML manager's Compile option policy to enforce streaming.
    Then I got the same "stylesheet is not streamable" error you see.

    I know that there is a solution, but I do not know it.

    I did try to contact the person having claimed to having done that for a customer in the past.

     

    <xsl:stylesheet version="1.0"
      xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
      xmlns:dp="http://www.datapower.com/extensions"
      extension-element-prefixes="dp"
    >
      <xsl:output omit-xml-declaration="yes" />
        
      <xsl:template match="/">
        <dp:set-variable name="'var://service/routing-url'"
          value="concat('ftp://stammw:thatIsWhatYouWantToKnow@9.152.201.79/ftp-out',dp:variable('var://service/URI'))"
        />
      </xsl:template>
     
    </xsl:stylesheet>

     


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

    Hermann,

    Thank you for your help, we now have things a lot more clear. It's good to know that we're on the same scenario now. Let me know when you have a response from this persone to continue.

     

    Regards,

    Sebastian

  • agrohman
    agrohman
    60 Posts

    Re: Streamed Transactions in DataPower as an FTP Server

    ‏2013-05-23T19:05:43Z  
    • sgb90
    • ‏2013-05-22T19:22:14Z

    Hermann,

    Thank you for your help, we now have things a lot more clear. It's good to know that we're on the same scenario now. Let me know when you have a response from this persone to continue.

     

    Regards,

    Sebastian

    Hi Sebastian,

    So, I'm the one that used dynamic routing with streaming binary files.  I haven't looked at your stylesheet, but from just reading through the thread, I don't think what you want is possible.  Specifically, you can't create two copies of input in streaming mode.  It's meant for simple 1:1 input/output use cases.  You can set the destination dynamically by, as you discovered, executing a stylesheet with NULL input that sets the var://service/routing-url variable.  But you can not have more than one destination -- that will require buffering the message, which disables the streaming.

    An alternative might be to transfer the file twice... Once to the backup directory, and then poll the backup directory to send it to the real destination, using a rename rule instead of a delete rule to consider the file from the backup directory as 'processed,' and avoiding processing it repeatedly.

    Just a thought -- no idea if it'd work, as you might lose some of the original input data required for the dynamic destination, anyway.  But, maybe renaming the files dynamically can help you get it all worked out.

    Good luck,

    Andy

  • sgb90
    sgb90
    114 Posts

    Re: Streamed Transactions in DataPower as an FTP Server

    ‏2013-05-24T20:25:35Z  
    • agrohman
    • ‏2013-05-23T19:05:43Z

    Hi Sebastian,

    So, I'm the one that used dynamic routing with streaming binary files.  I haven't looked at your stylesheet, but from just reading through the thread, I don't think what you want is possible.  Specifically, you can't create two copies of input in streaming mode.  It's meant for simple 1:1 input/output use cases.  You can set the destination dynamically by, as you discovered, executing a stylesheet with NULL input that sets the var://service/routing-url variable.  But you can not have more than one destination -- that will require buffering the message, which disables the streaming.

    An alternative might be to transfer the file twice... Once to the backup directory, and then poll the backup directory to send it to the real destination, using a rename rule instead of a delete rule to consider the file from the backup directory as 'processed,' and avoiding processing it repeatedly.

    Just a thought -- no idea if it'd work, as you might lose some of the original input data required for the dynamic destination, anyway.  But, maybe renaming the files dynamically can help you get it all worked out.

    Good luck,

    Andy

    Hi Andy!

    Thank you for replying, nice to have you as reference.

    The last test we ran had a single destination routing, using a stylesheet with a <dp:set-variable> sentence which changed the value of var://service/routing-url variable between NULL contexts. Even though we did this, it didn't work and kept taking the stylesheet as not streamable.

    So, we already stopped trying to route to multiple destinations (for e.g. 1:2 input/output) and are still having the same result. Any idea why? Please, let me know if you need anything about our work so you can take a closer look.

    Regards,

    Sebastián

     

  • agrohman
    agrohman
    60 Posts

    Re: Streamed Transactions in DataPower as an FTP Server

    ‏2013-05-29T20:17:19Z  
    • sgb90
    • ‏2013-05-24T20:25:35Z

    Hi Andy!

    Thank you for replying, nice to have you as reference.

    The last test we ran had a single destination routing, using a stylesheet with a <dp:set-variable> sentence which changed the value of var://service/routing-url variable between NULL contexts. Even though we did this, it didn't work and kept taking the stylesheet as not streamable.

    So, we already stopped trying to route to multiple destinations (for e.g. 1:2 input/output) and are still having the same result. Any idea why? Please, let me know if you need anything about our work so you can take a closer look.

    Regards,

    Sebastián

     

    I honestly can't recall.  I am pretty sure I just had everything set to Stream and processing set to Non-XML, and this worked.

  • sgb90
    sgb90
    114 Posts

    Re: Streamed Transactions in DataPower as an FTP Server

    ‏2013-06-12T14:16:37Z  
    • agrohman
    • ‏2013-05-29T20:17:19Z

    I honestly can't recall.  I am pretty sure I just had everything set to Stream and processing set to Non-XML, and this worked.

    Hi Andy,

    We're still working on this issue, but still have the same result.

    Can I send you an export of our actual MPGW so you can take a look? Maybe there's something we're missing. We want to rule out any problems in our set-up.

    Thank you and again, sorry for keeping this post for so long.

    Regards,

    Sebastian

     

  • agrohman
    agrohman
    60 Posts

    Re: Streamed Transactions in DataPower as an FTP Server

    ‏2013-06-12T14:28:29Z  
    • sgb90
    • ‏2013-06-12T14:16:37Z

    Hi Andy,

    We're still working on this issue, but still have the same result.

    Can I send you an export of our actual MPGW so you can take a look? Maybe there's something we're missing. We want to rule out any problems in our set-up.

    Thank you and again, sorry for keeping this post for so long.

    Regards,

    Sebastian

     

    Hi Sebastian,

     

    I've taken a new role with IBM, so I won't be able to dig into your export and validate like I used to.  I encourage you to reach out to your local Technical Sales Specialist for this type of troubleshooting assistance.  You can connect him/her to me, and we can chat internally about it.

     

    Andy

  • sgb90
    sgb90
    114 Posts

    Re: Streamed Transactions in DataPower as an FTP Server

    ‏2013-06-12T14:34:27Z  
    • agrohman
    • ‏2013-06-12T14:28:29Z

    Hi Sebastian,

     

    I've taken a new role with IBM, so I won't be able to dig into your export and validate like I used to.  I encourage you to reach out to your local Technical Sales Specialist for this type of troubleshooting assistance.  You can connect him/her to me, and we can chat internally about it.

     

    Andy

    Thank you for the quick response Andy, I will certainly do that. He will return this Friday, so I guess we'll be getting in touch around that day.

     

    Regards,

    Sebastian