IC SunsetThe developerWorks Connections platform will be sunset on December 31, 2019. On January 1, 2020, this community and its apps will no longer be available. More details available on our FAQ.
Topic
  • 4 replies
  • Latest Post - ‏2014-04-02T16:05:27Z by AndrewPaier
DQ1D_Glenn_Smith
DQ1D_Glenn_Smith
100 Posts

Pinned topic How durable is a durable subscription?

‏2014-03-27T21:21:41Z | events message

How long is a message held waiting potential subscribers?

I did the following experiment  IBM BPM 8.0.1

I have a trivial process with 1 user task to enter some data and a send message event which sends that data.

I have another process with a user task to enter a correlation key and then a receive a message event.

I run the send process.  The message is sent and then delivered to any active listeners, as expected.  Several hours later, I start another instance of the listener process and give it the same correlation key.  The message is also deliverred to that process.

In normal JMS usage, a durable description only guarantees delivery to subscribers which are subscribed at the time the message was sent.  This ensures a finite number of subscribers, and the server can know eventually that all possible deliveries are complete.  The process instance did not exist when the message was sent, so it could not have been the subscriber.  This suggests that the UCA actually registers the subscription.  But the UCA cannot possibly know how many process instances may, at some time in the future, be started.  If this is the behavior, then somewhere, the message must be persisted indefinately for potential future delivery.

 Is there a time limit, analogous to the JMSExpiration header field?  Is it configurable?

What am I missing?

Thanks

Updated on 2014-03-27T21:55:43Z at 2014-03-27T21:55:43Z by DQ1D_Glenn_Smith
  • dogren@gmail.com
    dogren@gmail.com
    438 Posts
    ACCEPTED ANSWER

    Re: How durable is a durable subscription?

    ‏2014-04-01T19:29:52Z  

    David,

       Thanks for the quick response.  2 follow up questions:

    Is this documented anywhere?  I cannot find it.

    Are there scalability or performance concerns?  If we go with the OOTB message event/UCA approach the system which will have several thousand active subscriptions and potentially receive thousands of incoming events/day.  Manual maintenance of that many subscriptions does not seem practical.  What is the resource cost of durable subscriptions which are never going to deliver another message?

    Is this documented anywhere?  I cannot find it.

    It's briefly mentioned in both the section about IMEs ( http://bit.ly/1jwV1kx ) and in the section on the BPMDeleteDurableMessages command ( http://bit.ly/1pIdp9V ) . No real detail provided, basically just saying that they "accumulate" in the database and should be periodically purged using BPMDeleteDurableMessages. Note that this admin command is fairly recent and as far as I know it wasn't documented significantly before this command. The behavior was the same previously, but you had to manually purge the table. (Or just let it grow without bounds.)

    If we go with the OOTB message event/UCA approach the system which will have several thousand active subscriptions and potentially receive thousands of incoming events/day.  Manual maintenance of that many subscriptions does not seem practical.

    The BPMDeleteDurableMessages is entirely date based. It doesn't matter how many different UCAs you have, or how many IMEs, or how many messages. Essentially you just say "delete durable messages received more the X days ago" where X is an argument to the command. This does mean that you can't have different policies for different events, but given the minimal impact of durable messages (see below) this probably isn't an issue.

    What is the resource cost of durable subscriptions which are never going to deliver another message?

    I hate to make absolute declarations about performance impacts. Every case and every environment can be different. That said, the table is indexed well and I don't think you should have any problems. Thousands of durable messages a day isn't really that much compared to many other systems I've seen.

    Essentially every time you get to an IME it is going to make a lookup into that table. But since it is directly hitting the index the CPU/memory impact should be trivial. I suspect that a lot of users don't even know to run the BPMDeleteDurableMessages command (or purge the table manually) and still are operating just fine even with millions of rows in the table. The biggest resource cost is probably the disk space on the DB server. 

    YMMV, of course. But, I think the bigger pain point around durable messages is that it can make testing harder. I usually hear this question in association with "I ran my instance, everything worked fine. But then I deleted the instance and ran it again, and now it never stops at the IME." The answer being, of course, that they've marked the IME as durable and if they keep using the same correlation id, then the durable event will be retained even if you purge the instance via Process Inspector.

    David

     

  • dogren@gmail.com
    dogren@gmail.com
    438 Posts

    Re: How durable is a durable subscription?

    ‏2014-03-27T22:26:52Z  

    Durable events last until you delete them. (Sometime recently IBM added the BPMDeleteDurableMessages admin command. Before that you had to manually maintain the table.)

    This is very much by design. BPM events have a very different use case than JMS queues. Durable events were built very specifically for the case where the incoming event may have been received long before the instance ever is started.

    David

    Updated on 2014-03-27T22:29:46Z at 2014-03-27T22:29:46Z by dogren@gmail.com
  • DQ1D_Glenn_Smith
    DQ1D_Glenn_Smith
    100 Posts

    Re: How durable is a durable subscription?

    ‏2014-04-01T13:11:10Z  

    Durable events last until you delete them. (Sometime recently IBM added the BPMDeleteDurableMessages admin command. Before that you had to manually maintain the table.)

    This is very much by design. BPM events have a very different use case than JMS queues. Durable events were built very specifically for the case where the incoming event may have been received long before the instance ever is started.

    David

    David,

       Thanks for the quick response.  2 follow up questions:

    Is this documented anywhere?  I cannot find it.

    Are there scalability or performance concerns?  If we go with the OOTB message event/UCA approach the system which will have several thousand active subscriptions and potentially receive thousands of incoming events/day.  Manual maintenance of that many subscriptions does not seem practical.  What is the resource cost of durable subscriptions which are never going to deliver another message?

  • dogren@gmail.com
    dogren@gmail.com
    438 Posts

    Re: How durable is a durable subscription?

    ‏2014-04-01T19:29:52Z  

    David,

       Thanks for the quick response.  2 follow up questions:

    Is this documented anywhere?  I cannot find it.

    Are there scalability or performance concerns?  If we go with the OOTB message event/UCA approach the system which will have several thousand active subscriptions and potentially receive thousands of incoming events/day.  Manual maintenance of that many subscriptions does not seem practical.  What is the resource cost of durable subscriptions which are never going to deliver another message?

    Is this documented anywhere?  I cannot find it.

    It's briefly mentioned in both the section about IMEs ( http://bit.ly/1jwV1kx ) and in the section on the BPMDeleteDurableMessages command ( http://bit.ly/1pIdp9V ) . No real detail provided, basically just saying that they "accumulate" in the database and should be periodically purged using BPMDeleteDurableMessages. Note that this admin command is fairly recent and as far as I know it wasn't documented significantly before this command. The behavior was the same previously, but you had to manually purge the table. (Or just let it grow without bounds.)

    If we go with the OOTB message event/UCA approach the system which will have several thousand active subscriptions and potentially receive thousands of incoming events/day.  Manual maintenance of that many subscriptions does not seem practical.

    The BPMDeleteDurableMessages is entirely date based. It doesn't matter how many different UCAs you have, or how many IMEs, or how many messages. Essentially you just say "delete durable messages received more the X days ago" where X is an argument to the command. This does mean that you can't have different policies for different events, but given the minimal impact of durable messages (see below) this probably isn't an issue.

    What is the resource cost of durable subscriptions which are never going to deliver another message?

    I hate to make absolute declarations about performance impacts. Every case and every environment can be different. That said, the table is indexed well and I don't think you should have any problems. Thousands of durable messages a day isn't really that much compared to many other systems I've seen.

    Essentially every time you get to an IME it is going to make a lookup into that table. But since it is directly hitting the index the CPU/memory impact should be trivial. I suspect that a lot of users don't even know to run the BPMDeleteDurableMessages command (or purge the table manually) and still are operating just fine even with millions of rows in the table. The biggest resource cost is probably the disk space on the DB server. 

    YMMV, of course. But, I think the bigger pain point around durable messages is that it can make testing harder. I usually hear this question in association with "I ran my instance, everything worked fine. But then I deleted the instance and ran it again, and now it never stops at the IME." The answer being, of course, that they've marked the IME as durable and if they keep using the same correlation id, then the durable event will be retained even if you purge the instance via Process Inspector.

    David

     

  • AndrewPaier
    AndrewPaier
    1197 Posts

    Re: How durable is a durable subscription?

    ‏2014-04-02T16:05:27Z  

    One important note.  Many people I interview don't seem to really understand durability and consumption.  You only need to check the "durable" box if the following is true -

    1. The event we are listening for can happen before the token reaches the IME we are looking at.
    2. If the event happens before we reach the IME, we still want to behave as if if just happened.

    Unless those 2 things are true, you don't need durable.  Durable certainly added to your processing time when an IME is trying to determine if it can move forward, however how much is an open question.  It is going to be related to the number of UCA's with durable subscribers that have been triggered, but the relationship is complex as there is a difference on the performance if you have 1 UCA with a durable subscriber that has been triggered 1 million times vs. 100 UCAs each with durable subscribers, and each UCA has been called 10,000 times.  While the number of entires in the DB table that handles this will be the same, the performance impact is likely different.

    And the correlation key likely also figures in...

    ANDREW PAIER
    VICE PRESIDENT, LABS
    T: 512.600.3239 X 240 / apaier@bp-3.com / @apaier
    BP3 /// www.bp-3.com / Blogs / Twitter / LinkedIn