Topic
7 replies Latest Post - ‏2014-04-28T15:30:51Z by carlornd2
SystemAdmin
SystemAdmin
7615 Posts
ACCEPTED ANSWER

Pinned topic Transaction support in Lombardi

‏2012-11-13T15:21:46Z |
We have recently implemented a workflow in Lombardi 7.2 for one of our customers. One thing that surprised me is that we haven't used any transactions (UOW) for multiple system updates in the system lane. IBM recommended us not to use TXN support as we were told that its not reliable in Lombardi. We have used exception handling available in Lombardi (+our custom framework) to handle error scenarios. The following are my questions in this context.

1. What is the level of Transaction support available in Lombardi 7.2? Are there any enhancements in IBM BPM 8.0 (on the human-centric Lombardi side of things)?

2. The steer from IBM is to use system-centric transactions in WPS world (of advanced flavour). Is this the only option available for leveraging transaction support?
Updated on 2012-11-24T20:27:28Z at 2012-11-24T20:27:28Z by kolban
  • kolban
    kolban
    3314 Posts
    ACCEPTED ANSWER

    Re: Transaction support in Lombardi

    ‏2012-11-13T19:09:35Z  in response to SystemAdmin
    It is my understanding that in the BPMN diagrams in the BPMN side of the house, there is no transaction support. This means that as you move from one activity to another, any work performed by those activities is committed when the activity completes. Although I haven't "walked the walk", it may be possible for an individual activity to perform a number of non-blocking operations in a single UOW but that is completed at the end of the containing BPM activity.

    The product maintains internal consistency (it had better) so in the event of a lights-out, the system should restore itself to a consistent state but there is no roll-back to previously completed steps.

    With IBM BPM Advanced, you have a lot more options especially in the BPEL side of the house. From there you can perform multiple steps within the same logical unit of work or, if that is not possible because the services you are invoking don't support 2PC, you can define a "compensation" set of rules which will undo previously completed steps to restore transactional integrity.

    Can you paint a picture of an example of transactional integrity that you may need?

    Neil
  • SystemAdmin
    SystemAdmin
    7615 Posts
    ACCEPTED ANSWER

    Re: Transaction support in Lombardi

    ‏2012-11-13T20:09:18Z  in response to SystemAdmin
    The DB connectors available in WLE (and BPM Standard) do allow for a little bit of transaction support, but not much. Basically there are some connectors where you can issue multiple SQL statements against the same datasource and they will be done as a single transaction ending in a commit or roll back. The product team did not provide any connectors that would support transactions that span datasources.

    The DB connectors provided are simply just wrappers to JDBC calls. As such if you really wanted you could write your own DB connector that leveraged WebSphere to allow for DB transactions that span datasources. One of the reasons the Lombardi team never wrote those types of connectors is that the product (until the 7.0 release) supported multiple application servers and each had a slightly different take on how to provide such a transaction. Now that WLE/IBM BPM will only every support WebSphere it is much easier to write such a connector.

    As Neil mentions above (and I believe you have already been told) IBM BPM advanced, can support much more sophisticated transaction boundaries than those that are readily available in WLE/BPM Standard through the Integration Designer. If you do not wish to create your own java integration to handle your particular use case, then this is your other option. Which is right for you probably depends on a number of variables such as -

    • The technical team's comfort with Java.
    • The sophistication of the transaction(s) in question.
    • The team's comfort in using Integration Designer.
    • How many sophisticated transactions you need to support.
    • The cost differential between BPM Standard in BPM Advanced.

    For example if you only have 1 multi-data source transaction, are relatively comfortable in java/JDBC coding, and the cost delta is high, you might just write your own connector. If however you have hundreds of these things to write, are already comfortable with Integration Designer, and need to support very sophisticated transaction handling, then ID might be the right choice.

    Both options should be fully "supported" however IBM isn't going to help you with your Java code if you are having problems with your transaction interaction. I would assume however if Integration Designer is doing the wrong thing, that is the type of thing you could report to the support team and get addressed.

    Andrew Paier | Director of Special Operations | BP3 Global, Inc. www.bp-3.com
    • kolban
      kolban
      3314 Posts
      ACCEPTED ANSWER

      Re: Transaction support in Lombardi

      ‏2012-11-13T20:47:28Z  in response to SystemAdmin
      I tend to find that the transaction boundaries requested are much more than database. Steps in a business process are portrayed as activities where each activity is implemented as a logical service. Many of these services are remote to IBM BPM and accessed through Web Services, JMS, MQ, file adapters and other forms of communication beyond just DB. If we intersperse Human Service in the mix, we simply aren't allowed to hold the locks on a 2PC transaction for that length of time. Through the use of Advanced we can also choreograph Web Services through WS-Transaction (rarely seen) or through BPEL compensation. BPEL's compensation model seems to be the "best seen so far" solution for being able to transactionally orchestrate what would otherwise appear to be non-transactional resources. In addition, it allows for process steps other than those classically seen for transactions. For example, what is the transactional story to "un-send" an email? Or the transactional story to "un-cash a cheque". BPEL allows for alternate logic paths to be executed automatically to undo such steps.

      BPEL is without question a much more technical language than BPMN but I would still stress that no Java coding is required to achieve BPEL orchestrated processes. For sheer performance, BPEL continues to far out-strip BPMN execution and is the tool-of-choice for STP oriented processes. We also have the ability to create interesting hybrid solutions where, for example, a BPMN based process does not integrate directly with the target services but instead uses AIS to ask a BPEL process to perform those tasks on our behalf. We can tag the BPEL process as being stateful (aka long running) so that when it performs its work on behalf of the BPMN process, it remembers "what it did". This BPEL process can subsequently be asked to "roll back" some or all of the activities that it has performed on behalf of the BPMN process through compensation.

      I used to be a developer on IBM's CICS which is the perennial transaction processing system and when I came across the BPEL compensation story, I initially did a head scratch. Theoretical transaction processing has the concept of the "ACID" properties. I think a case could be made that the compensation model supports ACD :-) .... I don't see it supporting "Isolation". However, the loose definition of "If I request that the transaction be un-done, will the state of the environment be as though I had never started the transaction in the first place?" ... the answer is mostly yes.

      Neil
      • SystemAdmin
        SystemAdmin
        7615 Posts
        ACCEPTED ANSWER

        Re: Transaction support in Lombardi

        ‏2012-11-22T17:37:46Z  in response to kolban
        Neil, Andrew:

        Thank you very much for brain-storming this topic. I will put in my thoughts in the next couple of days after running through your views and digesting them. I will also mention the transaction needs in my situation. Been a bit busy these days with other server issues.

        Regards.
  • SystemAdmin
    SystemAdmin
    7615 Posts
    ACCEPTED ANSWER

    Re: Transaction support in Lombardi

    ‏2012-11-24T17:20:56Z  in response to SystemAdmin
    Neil: Please find my PoV against your points, in italics.

    Although I haven't "walked the walk", it may be possible for an individual activity to perform a number of non-blocking operations in a single UOW but that is completed at the end of the containing BPM activity. -- I'm not sure if this feature is available in 7.2/7.5/8.0 versions of the product

    The product maintains internal consistency (it had better) so in the event of a lights-out, the system should restore itself to a consistent state but there is no roll-back to previously completed steps. -- I agree with this point. We found this on many occasions that on resuming failed BPD instances from process inspector.

    With IBM BPM Advanced, you have a lot more options especially in the BPEL side of the house. From there you can perform multiple steps within the same logical unit of work or, if that is not possible because the services you are invoking don't support 2PC, you can define a "compensation" set of rules which will undo previously completed steps to restore transactional integrity. -- I agree with the compensation part, but on the 2PC i came across many situations where it was extremely difficult to configure all the particpating systems to take part in 2PC.

    Can you paint a picture of an example of transactional integrity that you may need? -- The transactional needs of my situation are relatively simple in the sense that our UOW spanned only one system activity that had a bunch of general system services interacting with same DB at different stages of the underlying microflow. Hence we were simply looking for having UOW across multiple operations on the same physical DB connection, but at different points in the execution logic.

    Andrew: Please find my PoV against your points, in italics.

    The DB connectors available in WLE (and BPM Standard) do allow for a little bit of transaction support, but not much. Basically there are some connectors where you can issue multiple SQL statements against the same datasource and they will be done as a single transaction ending in a commit or roll back. The product team did not provide any connectors that would support transactions that span datasources. -- We kind of used this feature in a custom manner though. We coded it by setting the connection's autocommit property to false and then committing it at the end of the SQL statements. I guess we can't span datasources across multiple services within the same activity.
    For example if you only have 1 multi-data source transaction, are relatively comfortable in java/JDBC coding, and the cost delta is high, you might just write your own connector. If however you have hundreds of these things to write, are already comfortable with Integration Designer, and need to support very sophisticated transaction handling, then ID might be the right choice. -- I agree with this point. At the same time as you already pointed out, skillset availability also matters when using IID. I worked in the WPS world and SCA Export/Import as the kind of user-friendly artifacts that are available to the user (pun intended :-)). This is understandable as IBM positions WPS as part of the advanced offering. The QoS qualifiers are many and it takes thorough understanding of the messaging and transactional concepts.

    Neil: Please find my PoV against your points, in italics.

    BPEL is without question a much more technical language than BPMN but I would still stress that no Java coding is required to achieve BPEL orchestrated processes. For sheer performance, BPEL continues to far out-strip BPMN execution and is the tool-of-choice for STP oriented processes. -- I'm curious to know on how this is achieved under the covers? We have lot of microflows currently coded in the Lombardi system lane activities and we have plans of moving them to BPEL world.

    I used to be a developer on IBM's CICS which is the perennial transaction processing system and when I came across the BPEL compensation story, I initially did a head scratch. Theoretical transaction processing has the concept of the "ACID" properties. I think a case could be made that the compensation model supports ACD :-) .... I don't see it supporting "Isolation". However, the loose definition of "If I request that the transaction be un-done, will the state of the environment be as though I had never started the transaction in the first place?" ... the answer is mostly yes. -- Well put Neil. Per my understanding, compensations are exception flows that are coded. What if the compensation flow itself fails? I look at this is an attempt of trying to mimic the undo activity.

    Finally, in the performance context, i'm curious to know if you guys have come across remote-messaging, remote-support deloyment topologies for IBM BPM 7.5.x production environments.
    Thanks for your time for providing some really cool KT.
    • kolban
      kolban
      3314 Posts
      ACCEPTED ANSWER

      Re: Transaction support in Lombardi

      ‏2012-11-24T20:27:28Z  in response to SystemAdmin
      Again, a great conversation. Each item here could easily be its own thread of discussion. I'll keep coming back to this thread as time permits and I have thoughts on it. The one I'll try and put some words to (low hanging fruit) is the notion of a compensation failure.

      In BPEL, we can define "compensating" BPEL activities which can either be explicitly or implicitly executed when a roll-back is required. These are themselves just "more" BPEL activities which, when executed, should result in the "un-doing" of what was done before. Now if these BPEL activities themselves should fail, one could have additional compensation handler for these activities. Remember, compensation is what you define it to be so a compensation failure could be defined as "Create a human task and suspend the process until the administrator is ready". IBM's BPEL provides an additional compensation feature ... that of the "failed process". Beyond compensation, we can flag a process to "stop" when an error is encountered. At this point, the process is pasivated. It can be woken up to continue by administrative functions (including a browser based tool).

      So it seems that if we have a BPEL sequence of steps and they all work ... all is good.
      If we have a BPEL sequence of steps and they fail, we can perform implicit or explicit compensation and then carry on ...
      If we have a BPEL sequence of steps and they fail, we can either perform compensation or "stop" the process and await an administrative decision on what we want our process to do next. If the compensation fails, we can again stop the process and again await an administrator.

      At the highest level, BPEL compensation gives us a set of capabilities and it is up to us to determine how we wish to use/leverage those.

      Neil