Topic
  • 24 replies
  • Latest Post - ‏2013-03-29T01:14:58Z by SystemAdmin
SystemAdmin
SystemAdmin
7615 Posts

Pinned topic MQ to start BPD

‏2013-03-22T00:50:13Z |
I see in Neil's video how to start a BPD from MQ using SCA in Integration Designer in 7.5. I was wondering if in 8.0.1 that is available in Process Designer alone? If not, why not, it seems logical.
Updated on 2013-03-29T01:14:58Z at 2013-03-29T01:14:58Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    7615 Posts

    Re: MQ to start BPD

    ‏2013-03-22T06:49:12Z  
    Hello!
    You have some ways to solve it without IID and SCA. For example:
    1. You can develop your own MDB (using Java), that will read messages from MQ queue and will send appropriate messages to IBM BPM Event Manager according to http://pic.dhe.ibm.com/infocenter/dmndhelp/v8r0m1/topic/com.ibm.wbpm.wle.editor.doc/modeling/topic/sending_message_events.html
    (it's the same for 7.5)
    http://pic.dhe.ibm.com/infocenter/dmndhelp/v7r5mx/topic/com.ibm.wbpm.wle.editor.doc/modeling/topic/sending_message_events.html

    2. You can develop service, that will start periodically (via UCA), read messages from MQ queue and invoke appropriate UCAs for them.

    In both of these cases you have to put message start event in you BPD diagram.

    WBR
    Yury
  • SystemAdmin
    SystemAdmin
    7615 Posts

    Re: MQ to start BPD

    ‏2013-03-25T05:24:58Z  
    Hello!
    You have some ways to solve it without IID and SCA. For example:
    1. You can develop your own MDB (using Java), that will read messages from MQ queue and will send appropriate messages to IBM BPM Event Manager according to http://pic.dhe.ibm.com/infocenter/dmndhelp/v8r0m1/topic/com.ibm.wbpm.wle.editor.doc/modeling/topic/sending_message_events.html
    (it's the same for 7.5)
    http://pic.dhe.ibm.com/infocenter/dmndhelp/v7r5mx/topic/com.ibm.wbpm.wle.editor.doc/modeling/topic/sending_message_events.html

    2. You can develop service, that will start periodically (via UCA), read messages from MQ queue and invoke appropriate UCAs for them.

    In both of these cases you have to put message start event in you BPD diagram.

    WBR
    Yury
    Ok I found the event manager and added a Queue there and it shows in the Process Designer. Does that Queue also need to be created in MQ Explorer with the JMS generated? Or does adding it to Event Manager do that?
  • SystemAdmin
    SystemAdmin
    7615 Posts

    Re: MQ to start BPD

    ‏2013-03-25T05:27:45Z  
    Ok I found the event manager and added a Queue there and it shows in the Process Designer. Does that Queue also need to be created in MQ Explorer with the JMS generated? Or does adding it to Event Manager do that?
    In a related question on reading the documentation I came across this note...

    Note: Ensure that the sender and receiver of the message both use the same UCA. For example, if the sender of the message is a message end event in another BPD, then select the same UCA for both the receiving start event and the sending message end event in the other BPD.

    Am I correct in saying that this applies only to multiple BPDs in one Application? Or can a UCA be shared between applications somehow?
  • SystemAdmin
    SystemAdmin
    7615 Posts

    Re: MQ to start BPD

    ‏2013-03-25T05:44:22Z  
    In a related question on reading the documentation I came across this note...

    Note: Ensure that the sender and receiver of the message both use the same UCA. For example, if the sender of the message is a message end event in another BPD, then select the same UCA for both the receiving start event and the sending message end event in the other BPD.

    Am I correct in saying that this applies only to multiple BPDs in one Application? Or can a UCA be shared between applications somehow?
    Also on a related question. If MQ/JMS is used in a request-response model how does the Message End Event correlate back to the original message. I assume its automatic.
  • SystemAdmin
    SystemAdmin
    7615 Posts

    Re: MQ to start BPD

    ‏2013-03-25T07:50:01Z  
    Ok I found the event manager and added a Queue there and it shows in the Process Designer. Does that Queue also need to be created in MQ Explorer with the JMS generated? Or does adding it to Event Manager do that?
    What queue do you mean? JMS queue for EM is already present, you have no need to create this one. May be you create sync queue?
    Usually you also haven't to use it. EM "queues" (sync and async) are not JMS or MQ queues - they are entities inside EM.

    WBR,
    Yury
  • SystemAdmin
    SystemAdmin
    7615 Posts

    Re: MQ to start BPD

    ‏2013-03-25T08:00:18Z  
    Also on a related question. If MQ/JMS is used in a request-response model how does the Message End Event correlate back to the original message. I assume its automatic.
    I'm not sure that I understand you correctly. What is "Message end event" in your case? If you mean BPD-to-BPD or any other UCA-based interaction, they have no relation to JMS/MQ directly. EM JMS queue is only the way to push event to EM from any external source.

    WBR,
    Yury
  • SystemAdmin
    SystemAdmin
    7615 Posts

    Re: MQ to start BPD

    ‏2013-03-25T20:18:23Z  
    I'm not sure that I understand you correctly. What is "Message end event" in your case? If you mean BPD-to-BPD or any other UCA-based interaction, they have no relation to JMS/MQ directly. EM JMS queue is only the way to push event to EM from any external source.

    WBR,
    Yury
    > {quote:title=Yurych wrote:}{quote}
    > I'm not sure that I understand you correctly. What is "Message end event" in your case? If you mean BPD-to-BPD or any other UCA-based interaction, they have no relation to JMS/MQ directly. EM JMS queue is only the way to push event to EM from any external source.
    >
    > WBR,
    > Yury

    In BPM BPD there is a Message End step. If an external event writes to a MQ I want my BPD to run and return a value. Specifically I want an HTTP/SOAP request that flows through a DataPower appliance and feeds an MQ with the SOAP payload, then has my BPD process the payload and return a response back through DP to the requesting application via the HTTP/SOAP Response.
  • SystemAdmin
    SystemAdmin
    7615 Posts

    Re: MQ to start BPD

    ‏2013-03-26T00:42:20Z  
    Hello!
    You have some ways to solve it without IID and SCA. For example:
    1. You can develop your own MDB (using Java), that will read messages from MQ queue and will send appropriate messages to IBM BPM Event Manager according to http://pic.dhe.ibm.com/infocenter/dmndhelp/v8r0m1/topic/com.ibm.wbpm.wle.editor.doc/modeling/topic/sending_message_events.html
    (it's the same for 7.5)
    http://pic.dhe.ibm.com/infocenter/dmndhelp/v7r5mx/topic/com.ibm.wbpm.wle.editor.doc/modeling/topic/sending_message_events.html

    2. You can develop service, that will start periodically (via UCA), read messages from MQ queue and invoke appropriate UCAs for them.

    In both of these cases you have to put message start event in you BPD diagram.

    WBR
    Yury
    > {quote:title=Yurych wrote:}{quote}
    > Hello!
    > You have some ways to solve it without IID and SCA. For example:
    > 1. You can develop your own MDB (using Java), that will read messages from MQ queue and will send appropriate messages to IBM BPM Event Manager according to http://pic.dhe.ibm.com/infocenter/dmndhelp/v8r0m1/topic/com.ibm.wbpm.wle.editor.doc/modeling/topic/sending_message_events.html
    > In both of these cases you have to put message start event in you BPD diagram.
    >
    > WBR
    > Yury

    OK I need to build a Message Driven Bean (MDB) That will run on WebSphere and read from MQ and send to BPM Event Manager (EM)

    I am still not quite connecting the dots....can the MDB interact with EM Synchronous Queues in a request-response model so I can do the following?

    HTTP/SOAP WSDL exposed via a DataPower Appliance that will feed the SOAP payload XML to an MQ, then the MDB consume that payload and feed the EM Synchronous Queue which the Integration Service UCA on my Message Start Event will pick up and initiate the flow, and upon reaching the Message End Event, return the Response to the EM Synchronous Queue and to the MQ via the MDB for response back to the HTTP/SOAP requester?

    Is there an example someplace for an MDB picking up from MQ and interacting with EM?

    I am beginning to think it easier to just use IID to create the Integration service and have it feed the EM if I can do the scenario above. I have IID so IID UCA seems easier than creating an MDB and deploying it on WebSphere.

    Ollie
  • SystemAdmin
    SystemAdmin
    7615 Posts

    Re: MQ to start BPD

    ‏2013-03-26T01:16:21Z  
    > {quote:title=Yurych wrote:}{quote}
    > Hello!
    > You have some ways to solve it without IID and SCA. For example:
    > 1. You can develop your own MDB (using Java), that will read messages from MQ queue and will send appropriate messages to IBM BPM Event Manager according to http://pic.dhe.ibm.com/infocenter/dmndhelp/v8r0m1/topic/com.ibm.wbpm.wle.editor.doc/modeling/topic/sending_message_events.html
    > In both of these cases you have to put message start event in you BPD diagram.
    >
    > WBR
    > Yury

    OK I need to build a Message Driven Bean (MDB) That will run on WebSphere and read from MQ and send to BPM Event Manager (EM)

    I am still not quite connecting the dots....can the MDB interact with EM Synchronous Queues in a request-response model so I can do the following?

    HTTP/SOAP WSDL exposed via a DataPower Appliance that will feed the SOAP payload XML to an MQ, then the MDB consume that payload and feed the EM Synchronous Queue which the Integration Service UCA on my Message Start Event will pick up and initiate the flow, and upon reaching the Message End Event, return the Response to the EM Synchronous Queue and to the MQ via the MDB for response back to the HTTP/SOAP requester?

    Is there an example someplace for an MDB picking up from MQ and interacting with EM?

    I am beginning to think it easier to just use IID to create the Integration service and have it feed the EM if I can do the scenario above. I have IID so IID UCA seems easier than creating an MDB and deploying it on WebSphere.

    Ollie
    You still have several issues.

    1. can the MDB interact with EM Synchronous Queues Probably the best way for an MDB to do this is to send a message to the JMS queue that acts as a bridge to the EM.

    2. in a request-response model Not really. And that really has nothing to do with the MQ or JMS queues. It has to do with the fact that BPMN is inherently asynchronous: there is no way to know whether the BPMN will complete in 3 seconds, 3 hours, or 3 years. Now you can add an message handler in an end message event that sends a return MQ message so it is theoretically possible to do request-response: but there is no automatically correlation and you'll have to account for the fact that many of the responses may be received long past when an HTTP request times out.

    3. Is there an example someplace for an MDB picking up from MQ and interacting with EM? One example might be the Sonic integration toolkit on the wiki. It's quite old, but the principles are the same.

    4. I am beginning to think it easier to just use IID to create the Integration service and have it feed the EM if I can do the scenario above. Perhaps. ID would let you skip the step of writing the MDB. But you'd still have to worry about correlation and trying to find a relatively high latency async model into a sync response architecture.

    5. Speaking of which, perhaps you are misunderstanding the sync queues. They aren't synchronous as in they will give synchronous responses. They are merely deliver responses in the order they were received. They have more latency because they serialize incoming requests.

    Because of the above you might consider using a BPEL flow rather than a BPMN flow and triggering a BPMN flow only when something async is needed.

    David
  • SystemAdmin
    SystemAdmin
    7615 Posts

    Re: MQ to start BPD

    ‏2013-03-26T02:07:06Z  
    You still have several issues.

    1. can the MDB interact with EM Synchronous Queues Probably the best way for an MDB to do this is to send a message to the JMS queue that acts as a bridge to the EM.

    2. in a request-response model Not really. And that really has nothing to do with the MQ or JMS queues. It has to do with the fact that BPMN is inherently asynchronous: there is no way to know whether the BPMN will complete in 3 seconds, 3 hours, or 3 years. Now you can add an message handler in an end message event that sends a return MQ message so it is theoretically possible to do request-response: but there is no automatically correlation and you'll have to account for the fact that many of the responses may be received long past when an HTTP request times out.

    3. Is there an example someplace for an MDB picking up from MQ and interacting with EM? One example might be the Sonic integration toolkit on the wiki. It's quite old, but the principles are the same.

    4. I am beginning to think it easier to just use IID to create the Integration service and have it feed the EM if I can do the scenario above. Perhaps. ID would let you skip the step of writing the MDB. But you'd still have to worry about correlation and trying to find a relatively high latency async model into a sync response architecture.

    5. Speaking of which, perhaps you are misunderstanding the sync queues. They aren't synchronous as in they will give synchronous responses. They are merely deliver responses in the order they were received. They have more latency because they serialize incoming requests.

    Because of the above you might consider using a BPEL flow rather than a BPMN flow and triggering a BPMN flow only when something async is needed.

    David
    Excellent info, and thanks.

    So given the scenario above you would recommend using IID / BPEL instead of Process Designer / BPMN, especially since no human intervention is needed and since we are talking about an HTTP/SOAP initiated process with a synchronous response?

    I guess I was coming to that same conclusion above.

    Ollie
  • kolban
    kolban
    3316 Posts

    Re: MQ to start BPD

    ‏2013-03-26T03:37:55Z  
    Excellent info, and thanks.

    So given the scenario above you would recommend using IID / BPEL instead of Process Designer / BPMN, especially since no human intervention is needed and since we are talking about an HTTP/SOAP initiated process with a synchronous response?

    I guess I was coming to that same conclusion above.

    Ollie
    Ollie,
    Let us back up a step. Without thinking about "how" we are going to achieve the task, can you describe "what" you want to achieve. Let us not talk in terms of implementation but rather in terms of data flows, logic and systems involved. Please be as descriptive as possible ... but don't immediately overlay that with IBM implementation details. Once we hear a logical description of what you want, I think that will aid us in providing some guidance.

    Neil
  • SystemAdmin
    SystemAdmin
    7615 Posts

    Re: MQ to start BPD

    ‏2013-03-26T05:08:10Z  
    • kolban
    • ‏2013-03-26T03:37:55Z
    Ollie,
    Let us back up a step. Without thinking about "how" we are going to achieve the task, can you describe "what" you want to achieve. Let us not talk in terms of implementation but rather in terms of data flows, logic and systems involved. Please be as descriptive as possible ... but don't immediately overlay that with IBM implementation details. Once we hear a logical description of what you want, I think that will aid us in providing some guidance.

    Neil
    Neil,

    Great thanks.

    The application is a transactional application that is a service that partner clients use to post transactions via an HTTP/SOAP interface.

    Elements beyond my control are:

    DataPower as the application firewall and Governance appliance. The HTTP/SOAP wsdl is existing and we are re-implementing behind the DataPower.

    We are evaluating several approaches on several similar services. The first one was done with WMB behind the datapower as a single monolithic application (done by a three letter contractor). We are evaluating BPM in an orchestration layer above a number of "Basic Services" that are all integrations with existing subsystems mostly Java API but a couple of older systems are TCP/IP. Message Broker with MQ interfaces are used for all the basic services that wrap the legacy services.

    In the current application, we want to use BPM because we wanted to reduce the need for programmers to make changes and that along with the potential for Operator intervention on errors/exceptions led us to look at Process Designer and BPMN flows.

    There are two scenarios to be implemented, one where the source is the HTTP/SOAP service on DataPower, the other the same data comes from a Cell Phone SMS message processor, both populate the same SOAP payload. There is an authentication step for the HTTP/SOAP inputs that isn't needed on the SMS inputs. The payload is validated in one basic service. Another executes some stored procedures to populate some fields in the message. Another looks up a value in another database to decide which path to follow. The if the flag is true we do an additional step and call another service to commit the transaction. If the value is false we loop through records in the message and commit the transactions, and if any fail we roll back all the transactions. After commit we format the response and send notifications in another basic service. Finally the SOAP response is formatted in another basic service before being returned to DataPowser and back to the requesting client.

    I cannot get away totally on the how because some of those decisions have already been made, so a free design isn't possible.
  • SystemAdmin
    SystemAdmin
    7615 Posts

    Re: MQ to start BPD

    ‏2013-03-26T11:07:17Z  
    Neil,

    Great thanks.

    The application is a transactional application that is a service that partner clients use to post transactions via an HTTP/SOAP interface.

    Elements beyond my control are:

    DataPower as the application firewall and Governance appliance. The HTTP/SOAP wsdl is existing and we are re-implementing behind the DataPower.

    We are evaluating several approaches on several similar services. The first one was done with WMB behind the datapower as a single monolithic application (done by a three letter contractor). We are evaluating BPM in an orchestration layer above a number of "Basic Services" that are all integrations with existing subsystems mostly Java API but a couple of older systems are TCP/IP. Message Broker with MQ interfaces are used for all the basic services that wrap the legacy services.

    In the current application, we want to use BPM because we wanted to reduce the need for programmers to make changes and that along with the potential for Operator intervention on errors/exceptions led us to look at Process Designer and BPMN flows.

    There are two scenarios to be implemented, one where the source is the HTTP/SOAP service on DataPower, the other the same data comes from a Cell Phone SMS message processor, both populate the same SOAP payload. There is an authentication step for the HTTP/SOAP inputs that isn't needed on the SMS inputs. The payload is validated in one basic service. Another executes some stored procedures to populate some fields in the message. Another looks up a value in another database to decide which path to follow. The if the flag is true we do an additional step and call another service to commit the transaction. If the value is false we loop through records in the message and commit the transactions, and if any fail we roll back all the transactions. After commit we format the response and send notifications in another basic service. Finally the SOAP response is formatted in another basic service before being returned to DataPowser and back to the requesting client.

    I cannot get away totally on the how because some of those decisions have already been made, so a free design isn't possible.
    Hello Ollie!
    It seems that BPEL ir right way for your task. But if you will use long-running BPEL processes, you will have the same problem as for BPMN processes - you can have HTTP timeout. Of course you can develop BPMN with AIS (advanced integration services) as implementation for automated steps. And in this case you can use two-way invocation of your BPD from any mediation flow/BPEL, developed in IID. It will save some time for you, because two-way SCA invocation of BPD is OOTB functionality for IBM BPM Advanced.
    Alternative ways decsribed in my previous messages are suitable for IBM BPM Standart or if you have no competence in IID (BPEL, mediation flows and so on)

    WBR,
    Yury
  • SystemAdmin
    SystemAdmin
    7615 Posts

    Re: MQ to start BPD

    ‏2013-03-26T14:16:01Z  
    Hello Ollie!
    It seems that BPEL ir right way for your task. But if you will use long-running BPEL processes, you will have the same problem as for BPMN processes - you can have HTTP timeout. Of course you can develop BPMN with AIS (advanced integration services) as implementation for automated steps. And in this case you can use two-way invocation of your BPD from any mediation flow/BPEL, developed in IID. It will save some time for you, because two-way SCA invocation of BPD is OOTB functionality for IBM BPM Advanced.
    Alternative ways decsribed in my previous messages are suitable for IBM BPM Standart or if you have no competence in IID (BPEL, mediation flows and so on)

    WBR,
    Yury
    What kind of changes do you anticipate that you want to avoid programmer intervention for? (The integration work? New system systems? UI of the exception handling?)

    I'm not sure what anything long-running buys you here. My guy instinct (which absent the full specs and environment can certainly be wrong) is to use stateless BPEL for the primary flow. It fits both the transactional nature and request/response the best. But if one of the situations that require manual intervention is reached, fire an event to start a BPMN process.

    David
  • SystemAdmin
    SystemAdmin
    7615 Posts

    Re: MQ to start BPD

    ‏2013-03-26T16:36:52Z  
    Neil,

    Great thanks.

    The application is a transactional application that is a service that partner clients use to post transactions via an HTTP/SOAP interface.

    Elements beyond my control are:

    DataPower as the application firewall and Governance appliance. The HTTP/SOAP wsdl is existing and we are re-implementing behind the DataPower.

    We are evaluating several approaches on several similar services. The first one was done with WMB behind the datapower as a single monolithic application (done by a three letter contractor). We are evaluating BPM in an orchestration layer above a number of "Basic Services" that are all integrations with existing subsystems mostly Java API but a couple of older systems are TCP/IP. Message Broker with MQ interfaces are used for all the basic services that wrap the legacy services.

    In the current application, we want to use BPM because we wanted to reduce the need for programmers to make changes and that along with the potential for Operator intervention on errors/exceptions led us to look at Process Designer and BPMN flows.

    There are two scenarios to be implemented, one where the source is the HTTP/SOAP service on DataPower, the other the same data comes from a Cell Phone SMS message processor, both populate the same SOAP payload. There is an authentication step for the HTTP/SOAP inputs that isn't needed on the SMS inputs. The payload is validated in one basic service. Another executes some stored procedures to populate some fields in the message. Another looks up a value in another database to decide which path to follow. The if the flag is true we do an additional step and call another service to commit the transaction. If the value is false we loop through records in the message and commit the transactions, and if any fail we roll back all the transactions. After commit we format the response and send notifications in another basic service. Finally the SOAP response is formatted in another basic service before being returned to DataPowser and back to the requesting client.

    I cannot get away totally on the how because some of those decisions have already been made, so a free design isn't possible.
    If we are starting with a web service call, why do you want to add MQ into the mix? Both BPM Standard and Advanced can provide web services that could be called from Datapower to initiate whatever it is you want to do in BPM rather than messing around with low level items like queues. Given a choice unless there is evidence of a significant performance problem I would much prefer to deal with the high level abstraction of a Web Service call. If nothing else it makes it easy to switch out underlying implementations at a later date if needed.

    Andrew Paier | Director of Special Operations | BP3 Global, Inc. www.bp-3.com
  • SystemAdmin
    SystemAdmin
    7615 Posts

    Re: MQ to start BPD

    ‏2013-03-26T19:58:23Z  
    If we are starting with a web service call, why do you want to add MQ into the mix? Both BPM Standard and Advanced can provide web services that could be called from Datapower to initiate whatever it is you want to do in BPM rather than messing around with low level items like queues. Given a choice unless there is evidence of a significant performance problem I would much prefer to deal with the high level abstraction of a Web Service call. If nothing else it makes it easy to switch out underlying implementations at a later date if needed.

    Andrew Paier | Director of Special Operations | BP3 Global, Inc. www.bp-3.com
    Andrew,

    Like I said, some aspects are beyond our control and we are tasked to evaluate alternatives. One such consideration is performance as this is a high volume application. MQ provides impedance mismatching and from our own testing at least 50% better performance than HTTP/SOAP.
  • SystemAdmin
    SystemAdmin
    7615 Posts

    Re: MQ to start BPD

    ‏2013-03-27T02:11:07Z  
    What kind of changes do you anticipate that you want to avoid programmer intervention for? (The integration work? New system systems? UI of the exception handling?)

    I'm not sure what anything long-running buys you here. My guy instinct (which absent the full specs and environment can certainly be wrong) is to use stateless BPEL for the primary flow. It fits both the transactional nature and request/response the best. But if one of the situations that require manual intervention is reached, fire an event to start a BPMN process.

    David
    Hi y'all, thanks for your insights.

    We are learning obviously. The most experience we have is with MessageBroker and the first modernization we did was to use a MessageBroker application to orchestrate another HTTP/SOAP application through DataPower and wrapping the legacy APIs and TCP/IP services.

    This next project is to evaluate BPM as the orchestration layer above a number of WMB Basic Services that will be shared across many applications.

    Although this particular application doesn't have any requirements for Human Intervention, we anticipate operations staff to intervene on errors or exceptions in the future.

    The ancillary benefit of BPMN is supposed to be to allow business analysts to change a process without requiring developer resources, and we want to explore that, which is why I was asking about opinions on that.

    This application which I described above is a fairly straignt through application as in there are no complex routing or decision trees and although the SOAP payload can come from two sources HTTP/SOAP & SMS Server, aside from some differences in authentication and validation the remaining flow is very straignt forward, runs some stored procedures and insert the results in the message and call a service and if that succeeds send a notice and a response, if it fails roll back, send a notice and response...that's pretty much it.

    Right now I am leaning toward IID and an Integration Solution with about 3 modules. One main module to handle the majority of the process orchestration and two pre processor modules to handle the different data sources that call the main module.

    The WMB Basic Services are all wrapped with MQ Request-response SOAP payloads

    My developers are most familiar with Java and most of the legacy apis are Java as sell.

    So what are the pitfalls for using IID Integration Solution with two pre processor modules for the two data sources each with MQ/JMS inbound and outbound and Java Components for each callout to the Basic Services and the main module?
  • kolban
    kolban
    3316 Posts

    Re: MQ to start BPD

    ‏2013-03-27T04:58:33Z  
    Hi y'all, thanks for your insights.

    We are learning obviously. The most experience we have is with MessageBroker and the first modernization we did was to use a MessageBroker application to orchestrate another HTTP/SOAP application through DataPower and wrapping the legacy APIs and TCP/IP services.

    This next project is to evaluate BPM as the orchestration layer above a number of WMB Basic Services that will be shared across many applications.

    Although this particular application doesn't have any requirements for Human Intervention, we anticipate operations staff to intervene on errors or exceptions in the future.

    The ancillary benefit of BPMN is supposed to be to allow business analysts to change a process without requiring developer resources, and we want to explore that, which is why I was asking about opinions on that.

    This application which I described above is a fairly straignt through application as in there are no complex routing or decision trees and although the SOAP payload can come from two sources HTTP/SOAP & SMS Server, aside from some differences in authentication and validation the remaining flow is very straignt forward, runs some stored procedures and insert the results in the message and call a service and if that succeeds send a notice and a response, if it fails roll back, send a notice and response...that's pretty much it.

    Right now I am leaning toward IID and an Integration Solution with about 3 modules. One main module to handle the majority of the process orchestration and two pre processor modules to handle the different data sources that call the main module.

    The WMB Basic Services are all wrapped with MQ Request-response SOAP payloads

    My developers are most familiar with Java and most of the legacy apis are Java as sell.

    So what are the pitfalls for using IID Integration Solution with two pre processor modules for the two data sources each with MQ/JMS inbound and outbound and Java Components for each callout to the Basic Services and the main module?
    As best as I can tell ... you have a very good match in IID to your needs. IID and its associated runtime, IBM BPM Advanced have a stellar track record for high-speed, low latency straight through processing including transactionality and error handling. The power of SCA will afford you the opportunity to loosely couple your services to the orchestration you wish to achieve. Although not necessary, you have the opportunity to drop down to Java as needed. You will find that as you investigate IID you will find places where you can either achieve the same function in IID or in Message Broker. The choice of which to use is yours. Where IID differs dramatically from Message Broker is in the stateful orchestration that IID provides through both BPEL and SCA. The SCA world provides first class integration with MQ, JMS, Web Services, REST and many, many more connection options.

    So far, all sounds good.

    Neil
  • SystemAdmin
    SystemAdmin
    7615 Posts

    Re: MQ to start BPD

    ‏2013-03-27T06:47:24Z  
    • kolban
    • ‏2013-03-27T04:58:33Z
    As best as I can tell ... you have a very good match in IID to your needs. IID and its associated runtime, IBM BPM Advanced have a stellar track record for high-speed, low latency straight through processing including transactionality and error handling. The power of SCA will afford you the opportunity to loosely couple your services to the orchestration you wish to achieve. Although not necessary, you have the opportunity to drop down to Java as needed. You will find that as you investigate IID you will find places where you can either achieve the same function in IID or in Message Broker. The choice of which to use is yours. Where IID differs dramatically from Message Broker is in the stateful orchestration that IID provides through both BPEL and SCA. The SCA world provides first class integration with MQ, JMS, Web Services, REST and many, many more connection options.

    So far, all sounds good.

    Neil
    Excellent Neil, Thanks. Is there a Tutorial on BPM/BPEL I have your book and vimeo videos but both are kind of light on BPEL. Correct me if I am wrong, but if we have 8 basic services to integrate and expose the BPEL Process the recommended way is to create 8 Integration Modules with unique interfaces?

    Ollie
  • bmruter
    bmruter
    40 Posts

    Re: MQ to start BPD

    ‏2013-03-27T12:07:30Z  
    If we are starting with a web service call, why do you want to add MQ into the mix? Both BPM Standard and Advanced can provide web services that could be called from Datapower to initiate whatever it is you want to do in BPM rather than messing around with low level items like queues. Given a choice unless there is evidence of a significant performance problem I would much prefer to deal with the high level abstraction of a Web Service call. If nothing else it makes it easy to switch out underlying implementations at a later date if needed.

    Andrew Paier | Director of Special Operations | BP3 Global, Inc. www.bp-3.com
    Andrew -

    When given the choice we will always use MQ as a best practice. This has saved us many times. If we need to take process server down for maintenance or DB2 has a maintenance window, customer requests stack up in the Queue and all process without error once the system is back on line. BPEL connectors make this very is to code for and I consider it an architectural best practice for high availability. You still get very loosely coupled systems this way. We have some systems where multiple channels drop data into the same queue to feed a single BPEL process. Also works great for high performance. No lost messages when the system get a sudden spike of input.
  • kolban
    kolban
    3316 Posts

    Re: MQ to start BPD

    ‏2013-03-27T14:54:00Z  
    Excellent Neil, Thanks. Is there a Tutorial on BPM/BPEL I have your book and vimeo videos but both are kind of light on BPEL. Correct me if I am wrong, but if we have 8 basic services to integrate and expose the BPEL Process the recommended way is to create 8 Integration Modules with unique interfaces?

    Ollie
    Hi Ollie,
    There are many patterns that can be used to create an SCA implementation. One pattern is to create an SCA module that contains only the BPEL process and for each partner that it will interact with, create a separate module. The module hosting BPEL can then be linked via an SCA Import/Export pair to the modules that provide the actual interaction with the back-end systems. The advantage of this is reusability and the ease of switching out one back-end for another without impact to your overall BPEL based process.

    However, for me personally, if I were new to BPEL and SCA, I'd start by building out the solution all in one module. It can always be refactored later.

    As for knowledge on BPEL and SCA ... the trick here is to realize that IBM once upon a time had a produce called "WebSphere Process Server" (WPS). This product first saw the light of day back in the mid-2000s. It is THAT product that when merged with the old Lombardi Teamworks became what we now know as IBM BPM Advanced. The reason I mention this is that there are about 9 years worth of redbooks, developerWorks articles and more of collateral beyond what is found in the InfoCenter. Go search IBM's developerworks and redbooks using the phrase "WebSphere Process Server" and you will find what you need. The development tooling that we call IID today used to be called "WebSphere Integration Developer" (WID) but is essentially the same thing.

    As far as I can tell, IBM will not be dropping support for BPEL in the foreseeable future. They have MANY consumers using it today and it still serves a perfect purpose for technical integration needs.

    Neil
  • SystemAdmin
    SystemAdmin
    7615 Posts

    Re: MQ to start BPD

    ‏2013-03-27T15:00:17Z  
    • bmruter
    • ‏2013-03-27T12:07:30Z
    Andrew -

    When given the choice we will always use MQ as a best practice. This has saved us many times. If we need to take process server down for maintenance or DB2 has a maintenance window, customer requests stack up in the Queue and all process without error once the system is back on line. BPEL connectors make this very is to code for and I consider it an architectural best practice for high availability. You still get very loosely coupled systems this way. We have some systems where multiple channels drop data into the same queue to feed a single BPEL process. Also works great for high performance. No lost messages when the system get a sudden spike of input.
    I agree with the benefits of messaging, especially for reliability.

    However I think the idea of doing it for performance reasons might be pre-mature optimization. Because in this case, using MQ is adding a lot more "hops". If you pick it up in a custom MDB that MDB would have to relay it via a JMS call to the EM JMS (which would add another JMS hop overhead) and if you picked it up in ID and then passed it into the BPMN that has the overhead of going from the BPEL engine to the BPMN (which I hear is roughly the equivalent of a web service call overhead).

    In the end, I think the best protocol (if we got to choose, which I guess we don't) will depend on what you end up implementing the "process" as. If you did it as a generic service, that would lead me down the past of a web service since we could be truly synchronous. If you did it as BPEL, direct MQ might be best. If you did it as BPMN, I'd probably opt for JMS since you could have Datapower translate the payload into the native JMS EM bridge format. But I guess this is moot because the original poster said that the protocol is pre-decided.

    P.S.

    I do despair that the original poster said that these things have been decided and can't be changed. Because the benefit of Datapower is supposed to be that these things can be flexible. And the point of that is so the external systems don't have to worry about how to access the internal systems and each side can use the transport that is best for them.
  • SystemAdmin
    SystemAdmin
    7615 Posts

    Re: MQ to start BPD

    ‏2013-03-27T15:15:27Z  
    Hi y'all, thanks for your insights.

    We are learning obviously. The most experience we have is with MessageBroker and the first modernization we did was to use a MessageBroker application to orchestrate another HTTP/SOAP application through DataPower and wrapping the legacy APIs and TCP/IP services.

    This next project is to evaluate BPM as the orchestration layer above a number of WMB Basic Services that will be shared across many applications.

    Although this particular application doesn't have any requirements for Human Intervention, we anticipate operations staff to intervene on errors or exceptions in the future.

    The ancillary benefit of BPMN is supposed to be to allow business analysts to change a process without requiring developer resources, and we want to explore that, which is why I was asking about opinions on that.

    This application which I described above is a fairly straignt through application as in there are no complex routing or decision trees and although the SOAP payload can come from two sources HTTP/SOAP & SMS Server, aside from some differences in authentication and validation the remaining flow is very straignt forward, runs some stored procedures and insert the results in the message and call a service and if that succeeds send a notice and a response, if it fails roll back, send a notice and response...that's pretty much it.

    Right now I am leaning toward IID and an Integration Solution with about 3 modules. One main module to handle the majority of the process orchestration and two pre processor modules to handle the different data sources that call the main module.

    The WMB Basic Services are all wrapped with MQ Request-response SOAP payloads

    My developers are most familiar with Java and most of the legacy apis are Java as sell.

    So what are the pitfalls for using IID Integration Solution with two pre processor modules for the two data sources each with MQ/JMS inbound and outbound and Java Components for each callout to the Basic Services and the main module?
    The ancillary benefit of BPMN is supposed to be to allow business analysts to change a process without requiring developer resources, and we want to explore that, which is why I was asking about opinions on that.

    Let me address this in a standalone response for a moment.

    I think you you need to be a little more specific on that desire, and that's why I asked that question about what flexibility you want to allow in an earlier post. I am a huge advocate of using BPM to allow control of BPM applications by business analysts and even end users. But BPM is not a magic bullet. It only allows you to surface the business architecture in a separate layer from the IT architecture.

    • EPVs and other techniques can allow end users to control pre-designated thresholds and other "knobs and dials" of the process without requiring changes to the process.

    • PD can allow business users to change aspects of screens, processes, rules, and flow without having to understand all of the details of the technical implementations.

    • PD can allow technical users to define service definitions in ways that are more configurable and understandable by business analysts.

    • ID can allow technical users to change aspects of the technical implementations via configuration rather than code.

    But, despite all of those things, it sounds like you have a flow that is entirely integration centric. If I'm right (and I might not be: we obviously don't have the whole picture from just a few posts), then this might not be a good process to evaluate the ability of business analysts to participate and change the system. No matter the tool, a business analyst isn't going to be able to change how you mediate an MQ message into your backend systems, or how to compensate for a failed transaction by themselves. Just because they aren't going to have the required background skills in transaction management or data architecture.

    Just my two cents. And that doesn't mean that BPEL/BPM isn't a good fit for this system. Just that the benefits might not be related to business analyst participation for this particular scenario.

    David
  • SystemAdmin
    SystemAdmin
    7615 Posts

    Re: MQ to start BPD

    ‏2013-03-29T01:14:58Z  
    The ancillary benefit of BPMN is supposed to be to allow business analysts to change a process without requiring developer resources, and we want to explore that, which is why I was asking about opinions on that.

    Let me address this in a standalone response for a moment.

    I think you you need to be a little more specific on that desire, and that's why I asked that question about what flexibility you want to allow in an earlier post. I am a huge advocate of using BPM to allow control of BPM applications by business analysts and even end users. But BPM is not a magic bullet. It only allows you to surface the business architecture in a separate layer from the IT architecture.

    • EPVs and other techniques can allow end users to control pre-designated thresholds and other "knobs and dials" of the process without requiring changes to the process.

    • PD can allow business users to change aspects of screens, processes, rules, and flow without having to understand all of the details of the technical implementations.

    • PD can allow technical users to define service definitions in ways that are more configurable and understandable by business analysts.

    • ID can allow technical users to change aspects of the technical implementations via configuration rather than code.

    But, despite all of those things, it sounds like you have a flow that is entirely integration centric. If I'm right (and I might not be: we obviously don't have the whole picture from just a few posts), then this might not be a good process to evaluate the ability of business analysts to participate and change the system. No matter the tool, a business analyst isn't going to be able to change how you mediate an MQ message into your backend systems, or how to compensate for a failed transaction by themselves. Just because they aren't going to have the required background skills in transaction management or data architecture.

    Just my two cents. And that doesn't mean that BPEL/BPM isn't a good fit for this system. Just that the benefits might not be related to business analyst participation for this particular scenario.

    David
    David,

    Yes you are on target. While I stated it was a goal to evaluate all BPM technology we are only starting with this application which is indeed a headless flow. We were upgrading two Mule applications to IBM SOA Technologies. The first one was done BY IBM and they used WMB for the whole thing....sadly they built one huge monolithic application in Broker instead of breaking each basic service out int separate reusable services.

    So for this second project we wanted to look at more not so much because they were needed on this project but for future consideration.

    In that we have settled now on the BPEL flow and wrapping the legacy services in WMB with MQ interfaces, we have learned what is best suited for what and have that knowledge in under our belt and this thread helped a great deal, saving us from Trial and error and extensive research.

    Thanks to all and I will mark this answered.