Topic
22 replies Latest Post - ‏2013-12-10T18:25:05Z by HermannSW
Ravi_Kiran_Reddy
Ravi_Kiran_Reddy
12 Posts
ACCEPTED ANSWER

Pinned topic How to write incoming messages to a local File

‏2013-10-11T05:29:29Z |

Hi

I have a requirement to store incoming messages from MQ server to a local file in DP.

One solution is to use dump nodes. But this will write the content to Temp directory.

Is there any way to store the messages to a file in local or store directory?

or can i store them in system variable which always persist, and can i use the sytem variable in my transactions? is this a valid scenario?

Thanks.

  • Sudarshan Hebbar
    Sudarshan Hebbar
    57 Posts
    ACCEPTED ANSWER

    Re: How to write incoming messages to a local File

    ‏2013-10-11T06:02:28Z  in response to Ravi_Kiran_Reddy

    There are 2 options here.

    1. Using System variables. I personally like this for your scenario. In other words, it will achieve cacheing after the initial transaction.

    2. Second one is to construct the xml and send a SOMA request to DP by your service. This will store the messages as xml in your local directory.

    Hope this, helps your requirement.

     

    • Ravi_Kiran_Reddy
      Ravi_Kiran_Reddy
      12 Posts
      ACCEPTED ANSWER

      Re: How to write incoming messages to a local File

      ‏2013-10-11T06:20:44Z  in response to Sudarshan Hebbar

      Hi Sudarshan thanks for the reply.. Hope this helps me..

      But one question is, in my scenarion... i will be receiving 3000+ messages(Account Numbers) from MQ .. Can i keep on appending them to System variable.. is this possible? And what is the max size or limit for System variable?

      System variable should be updated then and there when the message is arrived from MQ. Can this be done?

       

  • kenhygh
    kenhygh
    873 Posts
    ACCEPTED ANSWER

    Re: How to write incoming messages to a local File

    ‏2013-10-11T09:41:53Z  in response to Ravi_Kiran_Reddy

    That's not a requirement, it's an implementation. What's the requirement?

    • Ravi_Kiran_Reddy
      Ravi_Kiran_Reddy
      12 Posts
      ACCEPTED ANSWER

      Re: How to write incoming messages to a local File

      ‏2013-10-11T11:11:43Z  in response to kenhygh

      Hi Ken,

      I will be receiving Messages from MQ Server. I will be receiving more than 3000 messages in a day between(5.30 AM to 6 AM IST).

      I already have a account number with me. Now i have to compare the account number( which i have ) with those i receive from MQ server ( 3000+ Messages). If account number matches, then i will send the matched account number to Target System. Thanks in advance.

      • kenhygh
        kenhygh
        873 Posts
        ACCEPTED ANSWER

        Re: How to write incoming messages to a local File

        ‏2013-10-11T13:28:05Z  in response to Ravi_Kiran_Reddy

        Let me see if I have your scenario straight. You will receive 3000 messages between 5:30 and 6:00 AM every day, and each message contains an account number. Later during the day, you will receive different messages, and if the account numbers within these business transactions match an account number you received between 5:30-6:00, you forward the message. Otherwise you do something else.

        Is this your scenario? if so, it's really not a good fit for DataPower, which is designed to be stateless.

        I would take the 3000 messages taking account numbers and write them to a database, and retrieve them later during the business transactions.

        You also need to think about how to get rid of old account numbers.

         

        • Ravi_Kiran_Reddy
          Ravi_Kiran_Reddy
          12 Posts
          ACCEPTED ANSWER

          Re: How to write incoming messages to a local File

          ‏2013-10-15T07:25:16Z  in response to kenhygh

          Hi Ken,

             The scenario is more or less the same as you explained. But only 1 change. I have a set of account numbers with me. I will compare them with the account numbers i receive between 5.30AM and 6AM. If it matches, i will forward the matched account number further.

          Everyday i will override the account numbers stored with the new account numbers i receive for that day. So, overriding is one option to get rid of old account numbers.

           

           

          • HermannSW
            HermannSW
            3148 Posts
            ACCEPTED ANSWER

            Re: How to write incoming messages to a local File

            ‏2013-10-15T10:20:12Z  in response to Ravi_Kiran_Reddy

            Hi Ravi,

            so it seems that you need to write the file in local file system once a day only.
            That is good because writing flash memory (eg. local: file system) for each transaction fill kill the flash quickly.

            In this posting you can see how to write files to "local:" from within a stylesheet:
            https://www.ibm.com/developerworks/community/blogs/HermannSW/entry/re_storing_file_at_run_time_in_local_disk15

            Because you need to do that once a day only, I would prefer to just use shell script "set-file" mentioned at end of that posting.
            It generates a SOMA set-file script and that does the actual write to local, no service on DataPower needed for that.


            Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

            • Ravi_Kiran_Reddy
              Ravi_Kiran_Reddy
              12 Posts
              ACCEPTED ANSWER

              Re: How to write incoming messages to a local File

              ‏2013-10-16T11:22:56Z  in response to HermannSW

              Hey Hermann,

              Thanks for the reply. And i am using set-file/ using SOMA call (within a stylesheet) for my requirement. Your post on SOMA helped me alot :) . And the post was so informative. Thanks alot for that.

              1 question here. Does making SOMA call using dp:soap-call reduce the performance?. Because i will be doing this every time i receive a message.

               

              • HermannSW
                HermannSW
                3148 Posts
                ACCEPTED ANSWER

                Re: How to write incoming messages to a local File

                ‏2013-10-16T12:21:15Z  in response to Ravi_Kiran_Reddy

                Should net be a performance problem, even if you get 3000 messages each moring.

                I only found lifetime information for compact flash storage card:
                http://pic.dhe.ibm.com/infocenter/wsdatap/v5r0m0/index.jsp?topic=%2Fcom.ibm.dp.doc%2F9235installationusersguide64.htm

                From there:

                ... ...
                Heavy usage, new data is written to it daily Every 1 year


                Be prepared to replace the flash each year if you do write to flash daily.

                Better would be to write to "local:///ondisk" raid instead, or to an external storage.


                Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

                 

                 

                • Ravi_Kiran_Reddy
                  Ravi_Kiran_Reddy
                  12 Posts
                  ACCEPTED ANSWER

                  Re: How to write incoming messages to a local File

                  ‏2013-10-16T13:46:16Z  in response to HermannSW

                  Almost 3000 soma calls a day will not be a perfomance problem then. Thanks Hermann.

                  Updated on 2013-10-16T14:04:41Z at 2013-10-16T14:04:41Z by Ravi_Kiran_Reddy
                  • kenhygh
                    kenhygh
                    873 Posts
                    ACCEPTED ANSWER

                    Re: How to write incoming messages to a local File

                    ‏2013-10-16T13:52:53Z  in response to Ravi_Kiran_Reddy

                    Um, where do you thing local:/// is?

                    Flash.

                    • Ravi_Kiran_Reddy
                      Ravi_Kiran_Reddy
                      12 Posts
                      ACCEPTED ANSWER

                      Re: How to write incoming messages to a local File

                      ‏2013-10-16T14:04:27Z  in response to kenhygh

                      Yup.. Just go this.. so i am writing to Flash Memory. I need to Check this now :)

                      • HermannSW
                        HermannSW
                        3148 Posts
                        ACCEPTED ANSWER

                        Re: How to write incoming messages to a local File

                        ‏2013-10-16T14:15:23Z  in response to Ravi_Kiran_Reddy

                        Not sure if that might help, but for teh record:

                        "temporary:" folder lives in ramdisk and even survives firmware reload -- it does not survive a device reboot, though.


                        Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>

                        • XIntegration50
                          XIntegration50
                          72 Posts
                          ACCEPTED ANSWER

                          Re: How to write incoming messages to a local File

                          ‏2013-10-16T19:35:32Z  in response to HermannSW

                          Hi,

                          Do we have a link where I can know what directory runs on what kind of memory ? Like ....  local :// on flash and temp:// on RAM etc.

                          Thanks,

                          Venkat

                          • HermannSW
                            HermannSW
                            3148 Posts
                            ACCEPTED ANSWER

                            Re: How to write incoming messages to a local File

                            ‏2013-10-16T19:55:39Z  in response to XIntegration50

                            From technote "DataPower File system explained":
                            http://www-01.ibm.com/support/docview.wss?uid=swg21496334

                            Temporary and Internal Space
                            The "temporary" and "internal" space are not persisted storage and the files will be lost if the appliance is rebooted or powered down but will not be cleared if use the "Reload" option.

                            The temporary space includes these user accessible directories: temporary, image, and logtemp
                            The internal space is not accessible by the user.

                             

                            So everything else is flash memory in case no raid drive (eg. 9325-4BX/4CX) or compact flash drive (eg. 9325-42X) has been initialized.

                            If they are initialized then they get monted under these default names if you do not modify:

                            local:///ondisk
                            logstore:///ondisk

                            or

                            local:///cf0
                            logstore:///cf0


                            Hermann<myXsltBlog/> <myXsltTweets/> <myCE/>
                             

                             

                            Updated on 2013-10-16T19:55:53Z at 2013-10-16T19:55:53Z by HermannSW
            • Ravi_Kiran_Reddy
              Ravi_Kiran_Reddy
              12 Posts
              ACCEPTED ANSWER

              Re: How to write incoming messages to a local File

              ‏2013-12-10T05:31:12Z  in response to HermannSW

              Hi Hermann,

              Need your suggetion here. Previously it was 3k messages. But now its 30k messages. All those 30 k message were put into Queue in 2 mins. Now its my turn to read them. I am able to pull them 1 by 1 and write them to local file using SOMA call. But it is taking hell lot of time to write all those 30k messages.

              To increase the performance, i increased the total connectivity Limiti in Queue Mangager to 150. This time i am able to pull the 30k messages from Queue very fastly. But the sad part is that, the 149 concurrent txns pulling the messages from Queue are not reaching DP, they are lost. Only 1 txn listening to queue is getting the messages from queue and posting to DP. Any solution here?

              Is it a good a way to make 30k SOMA calls daily?

              • HermannSW
                HermannSW
                3148 Posts
                ACCEPTED ANSWER

                Re: How to write incoming messages to a local File

                ‏2013-12-10T08:41:58Z  in response to Ravi_Kiran_Reddy

                You started with 300 messages a day, now you are at 30k messages.

                Where do you write the files in local: filesystem?
                Do you have raid and write there?

                If not, what you are doing is a bad practice and you should reconsider your complete architecture.
                Otherwise you will wear out your flash memory and then the appliance is brokem.

                Others might help you on the MQ issue, but SOMA calls do take time and this might be another reason why you need a different design.


                Hermann<myXsltBlog/> <myXsltTweets/> <myCE/> <myFrameless/>

                • Ravi_Kiran_Reddy
                  Ravi_Kiran_Reddy
                  12 Posts
                  ACCEPTED ANSWER

                  Re: How to write incoming messages to a local File

                  ‏2013-12-10T14:29:18Z  in response to HermannSW

                  Actually I am overriding the same file every day. For e.g, on the first day I recieved 30k messages and I wrote all of them to local:filesystem.. next day I again received 30k messages and I override the existing file so that it contains only 30k messages at most in a given day. The file size will not exceed 1MB given the kind of messages I am storing. Therefore, I want to know if there is any risk associated with the approach that I am following.
                  Also how can we prevent concurrent writes to the same file?

                  • HermannSW
                    HermannSW
                    3148 Posts
                    ACCEPTED ANSWER

                    Re: How to write incoming messages to a local File

                    ‏2013-12-10T14:40:27Z  in response to Ravi_Kiran_Reddy

                    You did not answer whether your appliance has a harddisk(raid).

                    > Also how can we prevent concurrent writes to the same file?
                    >
                    Why can that happen for you?

                    Why can't you store the 30k files on a nearby webserver?
                    If you define a document cache then there will be no noticable difference to storage on local: flash space.

                    > I want to know if there is any risk associated with the approach that I am following.
                    >
                    If you can make sure that you do not damage flash memory (which would happen by storing data on flash for each transaction, not once or few times a day), then there should be no risk.
                     

                    Hermann<myXsltBlog/> <myXsltTweets/> <myCE/> <myFrameless/>

                    • Ravi_Kiran_Reddy
                      Ravi_Kiran_Reddy
                      12 Posts
                      ACCEPTED ANSWER

                      Re: How to write incoming messages to a local File

                      ‏2013-12-10T16:43:30Z  in response to HermannSW

                      I am not sure whether my appliance has RAID. Can you please tell me how can i check it in my applaince.