Topic
  • 28 replies
  • Latest Post - ‏2015-01-14T21:54:45Z by PPotkay
Ravi_Kiran_Reddy
Ravi_Kiran_Reddy
12 Posts

Pinned topic How to write incoming messages to a local File

‏2013-10-11T05:29:29Z |

Hi

I have a requirement to store incoming messages from MQ server to a local file in DP.

One solution is to use dump nodes. But this will write the content to Temp directory.

Is there any way to store the messages to a file in local or store directory?

or can i store them in system variable which always persist, and can i use the sytem variable in my transactions? is this a valid scenario?

Thanks.

  • Sudarshan Hebbar
    Sudarshan Hebbar
    57 Posts
    ACCEPTED ANSWER

    Re: How to write incoming messages to a local File

    ‏2013-10-11T06:02:28Z  

    There are 2 options here.

    1. Using System variables. I personally like this for your scenario. In other words, it will achieve cacheing after the initial transaction.

    2. Second one is to construct the xml and send a SOMA request to DP by your service. This will store the messages as xml in your local directory.

    Hope this, helps your requirement.

     

  • HermannSW
    HermannSW
    6019 Posts
    ACCEPTED ANSWER

    Re: How to write incoming messages to a local File

    ‏2013-10-15T10:20:12Z  

    Hi Ken,

       The scenario is more or less the same as you explained. But only 1 change. I have a set of account numbers with me. I will compare them with the account numbers i receive between 5.30AM and 6AM. If it matches, i will forward the matched account number further.

    Everyday i will override the account numbers stored with the new account numbers i receive for that day. So, overriding is one option to get rid of old account numbers.

     

     

    Hi Ravi,

    so it seems that you need to write the file in local file system once a day only.
    That is good because writing flash memory (eg. local: file system) for each transaction fill kill the flash quickly.

    In this posting you can see how to write files to "local:" from within a stylesheet:
    https://www.ibm.com/developerworks/community/blogs/HermannSW/entry/re_storing_file_at_run_time_in_local_disk15

    Because you need to do that once a day only, I would prefer to just use shell script "set-file" mentioned at end of that posting.
    It generates a SOMA set-file script and that does the actual write to local, no service on DataPower needed for that.


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

  • Sudarshan Hebbar
    Sudarshan Hebbar
    57 Posts

    Re: How to write incoming messages to a local File

    ‏2013-10-11T06:02:28Z  

    There are 2 options here.

    1. Using System variables. I personally like this for your scenario. In other words, it will achieve cacheing after the initial transaction.

    2. Second one is to construct the xml and send a SOMA request to DP by your service. This will store the messages as xml in your local directory.

    Hope this, helps your requirement.

     

  • Ravi_Kiran_Reddy
    Ravi_Kiran_Reddy
    12 Posts

    Re: How to write incoming messages to a local File

    ‏2013-10-11T06:20:44Z  

    There are 2 options here.

    1. Using System variables. I personally like this for your scenario. In other words, it will achieve cacheing after the initial transaction.

    2. Second one is to construct the xml and send a SOMA request to DP by your service. This will store the messages as xml in your local directory.

    Hope this, helps your requirement.

     

    Hi Sudarshan thanks for the reply.. Hope this helps me..

    But one question is, in my scenarion... i will be receiving 3000+ messages(Account Numbers) from MQ .. Can i keep on appending them to System variable.. is this possible? And what is the max size or limit for System variable?

    System variable should be updated then and there when the message is arrived from MQ. Can this be done?

     

  • kenhygh
    kenhygh
    2013 Posts

    Re: How to write incoming messages to a local File

    ‏2013-10-11T09:41:53Z  

    That's not a requirement, it's an implementation. What's the requirement?

  • Ravi_Kiran_Reddy
    Ravi_Kiran_Reddy
    12 Posts

    Re: How to write incoming messages to a local File

    ‏2013-10-11T11:11:43Z  
    • kenhygh
    • ‏2013-10-11T09:41:53Z

    That's not a requirement, it's an implementation. What's the requirement?

    Hi Ken,

    I will be receiving Messages from MQ Server. I will be receiving more than 3000 messages in a day between(5.30 AM to 6 AM IST).

    I already have a account number with me. Now i have to compare the account number( which i have ) with those i receive from MQ server ( 3000+ Messages). If account number matches, then i will send the matched account number to Target System. Thanks in advance.

  • kenhygh
    kenhygh
    2013 Posts

    Re: How to write incoming messages to a local File

    ‏2013-10-11T13:28:05Z  

    Hi Ken,

    I will be receiving Messages from MQ Server. I will be receiving more than 3000 messages in a day between(5.30 AM to 6 AM IST).

    I already have a account number with me. Now i have to compare the account number( which i have ) with those i receive from MQ server ( 3000+ Messages). If account number matches, then i will send the matched account number to Target System. Thanks in advance.

    Let me see if I have your scenario straight. You will receive 3000 messages between 5:30 and 6:00 AM every day, and each message contains an account number. Later during the day, you will receive different messages, and if the account numbers within these business transactions match an account number you received between 5:30-6:00, you forward the message. Otherwise you do something else.

    Is this your scenario? if so, it's really not a good fit for DataPower, which is designed to be stateless.

    I would take the 3000 messages taking account numbers and write them to a database, and retrieve them later during the business transactions.

    You also need to think about how to get rid of old account numbers.

     

  • Ravi_Kiran_Reddy
    Ravi_Kiran_Reddy
    12 Posts

    Re: How to write incoming messages to a local File

    ‏2013-10-15T07:25:16Z  
    • kenhygh
    • ‏2013-10-11T13:28:05Z

    Let me see if I have your scenario straight. You will receive 3000 messages between 5:30 and 6:00 AM every day, and each message contains an account number. Later during the day, you will receive different messages, and if the account numbers within these business transactions match an account number you received between 5:30-6:00, you forward the message. Otherwise you do something else.

    Is this your scenario? if so, it's really not a good fit for DataPower, which is designed to be stateless.

    I would take the 3000 messages taking account numbers and write them to a database, and retrieve them later during the business transactions.

    You also need to think about how to get rid of old account numbers.

     

    Hi Ken,

       The scenario is more or less the same as you explained. But only 1 change. I have a set of account numbers with me. I will compare them with the account numbers i receive between 5.30AM and 6AM. If it matches, i will forward the matched account number further.

    Everyday i will override the account numbers stored with the new account numbers i receive for that day. So, overriding is one option to get rid of old account numbers.

     

     

  • HermannSW
    HermannSW
    6019 Posts

    Re: How to write incoming messages to a local File

    ‏2013-10-15T10:20:12Z  

    Hi Ken,

       The scenario is more or less the same as you explained. But only 1 change. I have a set of account numbers with me. I will compare them with the account numbers i receive between 5.30AM and 6AM. If it matches, i will forward the matched account number further.

    Everyday i will override the account numbers stored with the new account numbers i receive for that day. So, overriding is one option to get rid of old account numbers.

     

     

    Hi Ravi,

    so it seems that you need to write the file in local file system once a day only.
    That is good because writing flash memory (eg. local: file system) for each transaction fill kill the flash quickly.

    In this posting you can see how to write files to "local:" from within a stylesheet:
    https://www.ibm.com/developerworks/community/blogs/HermannSW/entry/re_storing_file_at_run_time_in_local_disk15

    Because you need to do that once a day only, I would prefer to just use shell script "set-file" mentioned at end of that posting.
    It generates a SOMA set-file script and that does the actual write to local, no service on DataPower needed for that.


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

  • Ravi_Kiran_Reddy
    Ravi_Kiran_Reddy
    12 Posts

    Re: How to write incoming messages to a local File

    ‏2013-10-16T11:22:56Z  
    • HermannSW
    • ‏2013-10-15T10:20:12Z

    Hi Ravi,

    so it seems that you need to write the file in local file system once a day only.
    That is good because writing flash memory (eg. local: file system) for each transaction fill kill the flash quickly.

    In this posting you can see how to write files to "local:" from within a stylesheet:
    https://www.ibm.com/developerworks/community/blogs/HermannSW/entry/re_storing_file_at_run_time_in_local_disk15

    Because you need to do that once a day only, I would prefer to just use shell script "set-file" mentioned at end of that posting.
    It generates a SOMA set-file script and that does the actual write to local, no service on DataPower needed for that.


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

    Hey Hermann,

    Thanks for the reply. And i am using set-file/ using SOMA call (within a stylesheet) for my requirement. Your post on SOMA helped me alot :) . And the post was so informative. Thanks alot for that.

    1 question here. Does making SOMA call using dp:soap-call reduce the performance?. Because i will be doing this every time i receive a message.

     

  • HermannSW
    HermannSW
    6019 Posts

    Re: How to write incoming messages to a local File

    ‏2013-10-16T12:21:15Z  

    Hey Hermann,

    Thanks for the reply. And i am using set-file/ using SOMA call (within a stylesheet) for my requirement. Your post on SOMA helped me alot :) . And the post was so informative. Thanks alot for that.

    1 question here. Does making SOMA call using dp:soap-call reduce the performance?. Because i will be doing this every time i receive a message.

     

    Should net be a performance problem, even if you get 3000 messages each moring.

    I only found lifetime information for compact flash storage card:
    http://pic.dhe.ibm.com/infocenter/wsdatap/v5r0m0/index.jsp?topic=%2Fcom.ibm.dp.doc%2F9235installationusersguide64.htm

    From there:

    ... ...
    Heavy usage, new data is written to it daily Every 1 year


    Be prepared to replace the flash each year if you do write to flash daily.

    Better would be to write to "local:///ondisk" raid instead, or to an external storage.


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

     

     

  • Ravi_Kiran_Reddy
    Ravi_Kiran_Reddy
    12 Posts

    Re: How to write incoming messages to a local File

    ‏2013-10-16T13:46:16Z  
    • HermannSW
    • ‏2013-10-16T12:21:15Z

    Should net be a performance problem, even if you get 3000 messages each moring.

    I only found lifetime information for compact flash storage card:
    http://pic.dhe.ibm.com/infocenter/wsdatap/v5r0m0/index.jsp?topic=%2Fcom.ibm.dp.doc%2F9235installationusersguide64.htm

    From there:

    ... ...
    Heavy usage, new data is written to it daily Every 1 year


    Be prepared to replace the flash each year if you do write to flash daily.

    Better would be to write to "local:///ondisk" raid instead, or to an external storage.


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

     

     

    Almost 3000 soma calls a day will not be a perfomance problem then. Thanks Hermann.

    Updated on 2013-10-16T14:04:41Z at 2013-10-16T14:04:41Z by Ravi_Kiran_Reddy
  • kenhygh
    kenhygh
    2013 Posts

    Re: How to write incoming messages to a local File

    ‏2013-10-16T13:52:53Z  

    Almost 3000 soma calls a day will not be a perfomance problem then. Thanks Hermann.

    Um, where do you thing local:/// is?

    Flash.

  • Ravi_Kiran_Reddy
    Ravi_Kiran_Reddy
    12 Posts

    Re: How to write incoming messages to a local File

    ‏2013-10-16T14:04:27Z  
    • kenhygh
    • ‏2013-10-16T13:52:53Z

    Um, where do you thing local:/// is?

    Flash.

    Yup.. Just go this.. so i am writing to Flash Memory. I need to Check this now :)

  • HermannSW
    HermannSW
    6019 Posts

    Re: How to write incoming messages to a local File

    ‏2013-10-16T14:15:23Z  

    Yup.. Just go this.. so i am writing to Flash Memory. I need to Check this now :)

    Not sure if that might help, but for teh record:

    "temporary:" folder lives in ramdisk and even survives firmware reload -- it does not survive a device reboot, though.


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

  • XIntegration50
    XIntegration50
    96 Posts

    Re: How to write incoming messages to a local File

    ‏2013-10-16T19:35:32Z  
    • HermannSW
    • ‏2013-10-16T14:15:23Z

    Not sure if that might help, but for teh record:

    "temporary:" folder lives in ramdisk and even survives firmware reload -- it does not survive a device reboot, though.


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

    Hi,

    Do we have a link where I can know what directory runs on what kind of memory ? Like ....  local :// on flash and temp:// on RAM etc.

    Thanks,

    Venkat

  • HermannSW
    HermannSW
    6019 Posts

    Re: How to write incoming messages to a local File

    ‏2013-10-16T19:55:39Z  

    Hi,

    Do we have a link where I can know what directory runs on what kind of memory ? Like ....  local :// on flash and temp:// on RAM etc.

    Thanks,

    Venkat

    From technote "DataPower File system explained":
    http://www-01.ibm.com/support/docview.wss?uid=swg21496334

    Temporary and Internal Space
    The "temporary" and "internal" space are not persisted storage and the files will be lost if the appliance is rebooted or powered down but will not be cleared if use the "Reload" option.

    The temporary space includes these user accessible directories: temporary, image, and logtemp
    The internal space is not accessible by the user.

     

    So everything else is flash memory in case no raid drive (eg. 9325-4BX/4CX) or compact flash drive (eg. 9325-42X) has been initialized.

    If they are initialized then they get monted under these default names if you do not modify:

    local:///ondisk
    logstore:///ondisk

    or

    local:///cf0
    logstore:///cf0


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>
     

     

    Updated on 2013-10-16T19:55:53Z at 2013-10-16T19:55:53Z by HermannSW
  • XIntegration50
    XIntegration50
    96 Posts

    Re: How to write incoming messages to a local File

    ‏2013-10-21T16:19:20Z  
    • HermannSW
    • ‏2013-10-16T19:55:39Z

    From technote "DataPower File system explained":
    http://www-01.ibm.com/support/docview.wss?uid=swg21496334

    Temporary and Internal Space
    The "temporary" and "internal" space are not persisted storage and the files will be lost if the appliance is rebooted or powered down but will not be cleared if use the "Reload" option.

    The temporary space includes these user accessible directories: temporary, image, and logtemp
    The internal space is not accessible by the user.

     

    So everything else is flash memory in case no raid drive (eg. 9325-4BX/4CX) or compact flash drive (eg. 9325-42X) has been initialized.

    If they are initialized then they get monted under these default names if you do not modify:

    local:///ondisk
    logstore:///ondisk

    or

    local:///cf0
    logstore:///cf0


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>
     

     

    Thank you Herman.

  • Ravi_Kiran_Reddy
    Ravi_Kiran_Reddy
    12 Posts

    Re: How to write incoming messages to a local File

    ‏2013-12-10T05:31:12Z  
    • HermannSW
    • ‏2013-10-15T10:20:12Z

    Hi Ravi,

    so it seems that you need to write the file in local file system once a day only.
    That is good because writing flash memory (eg. local: file system) for each transaction fill kill the flash quickly.

    In this posting you can see how to write files to "local:" from within a stylesheet:
    https://www.ibm.com/developerworks/community/blogs/HermannSW/entry/re_storing_file_at_run_time_in_local_disk15

    Because you need to do that once a day only, I would prefer to just use shell script "set-file" mentioned at end of that posting.
    It generates a SOMA set-file script and that does the actual write to local, no service on DataPower needed for that.


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

    Hi Hermann,

    Need your suggetion here. Previously it was 3k messages. But now its 30k messages. All those 30 k message were put into Queue in 2 mins. Now its my turn to read them. I am able to pull them 1 by 1 and write them to local file using SOMA call. But it is taking hell lot of time to write all those 30k messages.

    To increase the performance, i increased the total connectivity Limiti in Queue Mangager to 150. This time i am able to pull the 30k messages from Queue very fastly. But the sad part is that, the 149 concurrent txns pulling the messages from Queue are not reaching DP, they are lost. Only 1 txn listening to queue is getting the messages from queue and posting to DP. Any solution here?

    Is it a good a way to make 30k SOMA calls daily?

  • HermannSW
    HermannSW
    6019 Posts

    Re: How to write incoming messages to a local File

    ‏2013-12-10T08:41:58Z  

    Hi Hermann,

    Need your suggetion here. Previously it was 3k messages. But now its 30k messages. All those 30 k message were put into Queue in 2 mins. Now its my turn to read them. I am able to pull them 1 by 1 and write them to local file using SOMA call. But it is taking hell lot of time to write all those 30k messages.

    To increase the performance, i increased the total connectivity Limiti in Queue Mangager to 150. This time i am able to pull the 30k messages from Queue very fastly. But the sad part is that, the 149 concurrent txns pulling the messages from Queue are not reaching DP, they are lost. Only 1 txn listening to queue is getting the messages from queue and posting to DP. Any solution here?

    Is it a good a way to make 30k SOMA calls daily?

    You started with 300 messages a day, now you are at 30k messages.

    Where do you write the files in local: filesystem?
    Do you have raid and write there?

    If not, what you are doing is a bad practice and you should reconsider your complete architecture.
    Otherwise you will wear out your flash memory and then the appliance is brokem.

    Others might help you on the MQ issue, but SOMA calls do take time and this might be another reason why you need a different design.


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/> <myFrameless/>

  • Ravi_Kiran_Reddy
    Ravi_Kiran_Reddy
    12 Posts

    Re: How to write incoming messages to a local File

    ‏2013-12-10T14:29:18Z  
    • HermannSW
    • ‏2013-12-10T08:41:58Z

    You started with 300 messages a day, now you are at 30k messages.

    Where do you write the files in local: filesystem?
    Do you have raid and write there?

    If not, what you are doing is a bad practice and you should reconsider your complete architecture.
    Otherwise you will wear out your flash memory and then the appliance is brokem.

    Others might help you on the MQ issue, but SOMA calls do take time and this might be another reason why you need a different design.


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/> <myFrameless/>

    Actually I am overriding the same file every day. For e.g, on the first day I recieved 30k messages and I wrote all of them to local:filesystem.. next day I again received 30k messages and I override the existing file so that it contains only 30k messages at most in a given day. The file size will not exceed 1MB given the kind of messages I am storing. Therefore, I want to know if there is any risk associated with the approach that I am following.
    Also how can we prevent concurrent writes to the same file?

  • HermannSW
    HermannSW
    6019 Posts

    Re: How to write incoming messages to a local File

    ‏2013-12-10T14:40:27Z  

    Actually I am overriding the same file every day. For e.g, on the first day I recieved 30k messages and I wrote all of them to local:filesystem.. next day I again received 30k messages and I override the existing file so that it contains only 30k messages at most in a given day. The file size will not exceed 1MB given the kind of messages I am storing. Therefore, I want to know if there is any risk associated with the approach that I am following.
    Also how can we prevent concurrent writes to the same file?

    You did not answer whether your appliance has a harddisk(raid).

    > Also how can we prevent concurrent writes to the same file?
    >
    Why can that happen for you?

    Why can't you store the 30k files on a nearby webserver?
    If you define a document cache then there will be no noticable difference to storage on local: flash space.

    > I want to know if there is any risk associated with the approach that I am following.
    >
    If you can make sure that you do not damage flash memory (which would happen by storing data on flash for each transaction, not once or few times a day), then there should be no risk.
     

    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/> <myFrameless/>

  • Ravi_Kiran_Reddy
    Ravi_Kiran_Reddy
    12 Posts

    Re: How to write incoming messages to a local File

    ‏2013-12-10T16:43:30Z  
    • HermannSW
    • ‏2013-12-10T14:40:27Z

    You did not answer whether your appliance has a harddisk(raid).

    > Also how can we prevent concurrent writes to the same file?
    >
    Why can that happen for you?

    Why can't you store the 30k files on a nearby webserver?
    If you define a document cache then there will be no noticable difference to storage on local: flash space.

    > I want to know if there is any risk associated with the approach that I am following.
    >
    If you can make sure that you do not damage flash memory (which would happen by storing data on flash for each transaction, not once or few times a day), then there should be no risk.
     

    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/> <myFrameless/>

    I am not sure whether my appliance has RAID. Can you please tell me how can i check it in my applaince.

     

  • HermannSW
    HermannSW
    6019 Posts

    Re: How to write incoming messages to a local File

    ‏2013-12-10T18:25:05Z  

    I am not sure whether my appliance has RAID. Can you please tell me how can i check it in my applaince.

     

    Login via CLI and execute "show features".

    If you see a string of the form "Raid*" then you have.

    My 9325 boxes with model type 4BX and 4CX have, the 42X ones do not.


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/> <myFrameless/>

  • PPotkay
    PPotkay
    133 Posts

    Re: How to write incoming messages to a local File

    ‏2014-12-31T17:46:05Z  
    • HermannSW
    • ‏2013-10-16T12:21:15Z

    Should net be a performance problem, even if you get 3000 messages each moring.

    I only found lifetime information for compact flash storage card:
    http://pic.dhe.ibm.com/infocenter/wsdatap/v5r0m0/index.jsp?topic=%2Fcom.ibm.dp.doc%2F9235installationusersguide64.htm

    From there:

    ... ...
    Heavy usage, new data is written to it daily Every 1 year


    Be prepared to replace the flash each year if you do write to flash daily.

    Better would be to write to "local:///ondisk" raid instead, or to an external storage.


    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

     

     

    Herman,

    Other than this thread and it's reference to the DataPower 5.0 InfoCenter, (plus a couple of other threads with posts by you) I see no mention by IBM about wearing out your DataPower flash drive. That seems, odd. The risk of using up an appliance's component and rendering it inoperable in a year or two seems like it would at least have a Technote published, and ideally a lot of warnings in various places like Redbooks, Knowledge Centers, developerWorks articles, etc. I just reviewed the student handbook for my IBM DataPower Admin class (Firmware 5.0 version of the class) and no mention of this topic either.

    Is this no longer a concern on the 7198 / 7198 models and/or Firmware 6.0 and newer?

     

    -Peter

  • HermannSW
    HermannSW
    6019 Posts

    Re: How to write incoming messages to a local File

    ‏2015-01-01T09:15:30Z  
    • PPotkay
    • ‏2014-12-31T17:46:05Z

    Herman,

    Other than this thread and it's reference to the DataPower 5.0 InfoCenter, (plus a couple of other threads with posts by you) I see no mention by IBM about wearing out your DataPower flash drive. That seems, odd. The risk of using up an appliance's component and rendering it inoperable in a year or two seems like it would at least have a Technote published, and ideally a lot of warnings in various places like Redbooks, Knowledge Centers, developerWorks articles, etc. I just reviewed the student handbook for my IBM DataPower Admin class (Firmware 5.0 version of the class) and no mention of this topic either.

    Is this no longer a concern on the 7198 / 7198 models and/or Firmware 6.0 and newer?

     

    -Peter

    Hi Peter,

    the problem is not a DataPower problem, but a problem inherent to Flash memory.
    Please see the details here:
    http://en.wikipedia.org/wiki/Flash_memory#Memory_wear


    Is this no longer a concern on the 7198 / 7198 models and/or Firmware 6.0 and newer?
    >
    It is still Flash memory, perhaps with an increased number of writes, but still limited.
     

     ... see no mention by IBM about wearing out your DataPower flash drive. That seems, odd.
    > The risk of using up an appliance's component and rendering it inoperable in a year or two
    > seems like it would at least have a Technote published, 

    >
    I disagree here, there is no standard API to write anything to flash memory.
    And the number of writes is more than enough for all standard Flash write operations like "save config".

    You have to do SOMA chin-ups in order to be able to write to Flash memory at all.
    Writing to "temporary:" folder is easy by "dp:dump-nodes()", but that is a RAM based filesystem and not Flash.


    Besides the inherent issue for transaction latency by using SOMA call on transactional basis, the Flash memory issue is not existent as long as you write in "local:///ondisk" raid file system.


    You may want to create a PMR asking for technote creation you mentioned.
     

    Hermann.

    Updated on 2015-01-01T09:18:14Z at 2015-01-01T09:18:14Z by HermannSW