Topic
  • 3 replies
  • Latest Post - ‏2013-10-11T14:49:54Z by kenhygh
samanderson
samanderson
172 Posts

Pinned topic large xml inline attachments max size, best practices without performance degradation

‏2013-10-11T03:06:44Z |

Hi IBM Experts -

             Can you please comment on below scenario's. 

We are planning an MPG with which can handle 100mb size message with volume of 200 msg/sec with MQ frnt end and backout roll back processing in case of error in backend or datapower for which we read 100mb message into DP context variable and use it in error rule. We even make message transformation after it.

Can you help me understand with below questions or best practices

1.) Is there any limitation volume or size of the messages? I dont see any reference?

2.) Can enabling streaming mode and unprocessed mode effect box performance?

3.) We don't want box restarts or degrade of performance if dP  as this is used by many other domains.  Is not an idle fit for msg size greater than 4MB (xml parse limits) for security reason. Should we not consider DP as idle candidate with this scenario or not?

4.)  Since we are planning to cache message on to context variable and use it in error rule to back out does it kills the memory ?  is Unit Of Work best for this scenario.

5.) Any limitation on message size stylesheet can process ?

 

  • kenhygh
    kenhygh
    2003 Posts

    Re: large xml inline attachments max size, best practices without performance degradation

    ‏2013-10-11T09:54:51Z  

    If I'm doing my math right, 200 100mb messages a second is 20GB/second. I seriously doubt your network can handle that. AND "is used by many other domains". So I don't think you can even horizontally scale to achieve this throughput.

    There's no limit on message size (assuming you've configured your XML Manager for your max message size), but you will run out of memory. If you figure an average of 5 seconds for your backend to respond when under load, and this is one massive load, that's 100GB of memory just for the transactions in flight. Your DataPower box doesn't have 100GB of memory. Here you could horizontally scale - multiple DP boxes in tier - to handle the load.

    You *might* be able to do streaming mode, depending on what all you're doing to the messages in the device. If you're doing unprocessed, then why flow the messages through DP at all? 

    If you don't want restarts, you'll have to carefully tune SLM to limit the number of messages that any one box is processing at any given time. Otherwise, as noted above, you'll consume all the memory of the box and it will restart.

    You probably want to set unit of work so DP will automatically fail the GET if there's a failure at the backend. You don't want to try being a transaction manager yourself.

    A stylesheet can, in theory, process any size message. You will run into memory issues before a stylesheet failure.

  • samanderson
    samanderson
    172 Posts

    Re: large xml inline attachments max size, best practices without performance degradation

    ‏2013-10-11T14:46:35Z  
    • kenhygh
    • ‏2013-10-11T09:54:51Z

    If I'm doing my math right, 200 100mb messages a second is 20GB/second. I seriously doubt your network can handle that. AND "is used by many other domains". So I don't think you can even horizontally scale to achieve this throughput.

    There's no limit on message size (assuming you've configured your XML Manager for your max message size), but you will run out of memory. If you figure an average of 5 seconds for your backend to respond when under load, and this is one massive load, that's 100GB of memory just for the transactions in flight. Your DataPower box doesn't have 100GB of memory. Here you could horizontally scale - multiple DP boxes in tier - to handle the load.

    You *might* be able to do streaming mode, depending on what all you're doing to the messages in the device. If you're doing unprocessed, then why flow the messages through DP at all? 

    If you don't want restarts, you'll have to carefully tune SLM to limit the number of messages that any one box is processing at any given time. Otherwise, as noted above, you'll consume all the memory of the box and it will restart.

    You probably want to set unit of work so DP will automatically fail the GET if there's a failure at the backend. You don't want to try being a transaction manager yourself.

    A stylesheet can, in theory, process any size message. You will run into memory issues before a stylesheet failure.

    Hi Ken - Thanks for response ..

    Can I make DP process one message at time? so its always 100MB message. We have MQ FSH for this not HTTP. Does SLM come into picture at all?

    We actually make one to one transform and change format of the message so we are using DP for message mediation.

  • kenhygh
    kenhygh
    2003 Posts

    Re: large xml inline attachments max size, best practices without performance degradation

    ‏2013-10-11T14:49:54Z  

    Hi Ken - Thanks for response ..

    Can I make DP process one message at time? so its always 100MB message. We have MQ FSH for this not HTTP. Does SLM come into picture at all?

    We actually make one to one transform and change format of the message so we are using DP for message mediation.

    sure, SLM of one concurrent transaction. DP can probably handle more than this, though. YMMV.