Topic
  • 8 replies
  • Latest Post - ‏2012-04-08T08:48:26Z by Erwin_Karbasi
KAN6_Sanjay_Nath
KAN6_Sanjay_Nath
9 Posts

Pinned topic A pub-sub kind of a model is required in WXS

‏2011-12-07T15:55:39Z |
Hello,
I need to implement a client that can retrieve all tuples (based upon a criteria) as of a particular point in time, say 930AM. While these tuples are being retrieved and for a good deal longer than that duration, tuples will be inserted by other clients. Now my question is : how to I retrieve all tuples that match certain criteria at that point in time AND also get me the tuples that get inserted over the next few minutes (say till 4pm) and meet the same criteria. Jared suggested I use QueryQueue object for this. This looks a little cumbersome, that is, to keep certain field to act like a flag that is reset to indicate that the tuple has already been looked at and then issue the query repeatedly etc. This is the same as polling in my opinion. Is there some other solution to this scenario? Secondly, the Queryquery object seems to work well with EntityManager. I do not have objects that are related to each other and hence can do well without the EntityManager. So, do I have to use EntityManager and related objects? Or I can use the ObjectMap to retrieve the tuples that I need? In that case, is there some sample code I can look at? Please help.

thanks

Sanjay
Updated on 2012-04-08T08:48:26Z at 2012-04-08T08:48:26Z by Erwin_Karbasi
  • SystemAdmin
    SystemAdmin
    1485 Posts

    Re: A pub-sub kind of a model is required in WXS

    ‏2011-12-07T21:20:47Z  
    You need to split this into two different pieces:
    1. A way to track changes to a Map that you are interested in and a means to obtain those changes.
    2. A way to get the initial set of items you are interested in.

    Obviously you need to have #1 completed and in place before you attempt #2 so you don't miss any changes. To accomplish #2, you either write an Agent or client code that grabs all the keys via the MapIndexPlugin.SYSTEM_KEY_INDEX_NAME and determine for each value, whether you are interested in the tuple.

    To implement #2, you just need an ObjectGridEventListener that is interested in the TransactionEvents. On transactionEnd, you can process the committed changes and determine which log elements (changes) you are interested or no longer interested in. You will then have to determine how you get these changes to your client.
  • KAN6_Sanjay_Nath
    KAN6_Sanjay_Nath
    9 Posts

    Re: A pub-sub kind of a model is required in WXS

    ‏2011-12-14T16:15:53Z  
    You need to split this into two different pieces:
    1. A way to track changes to a Map that you are interested in and a means to obtain those changes.
    2. A way to get the initial set of items you are interested in.

    Obviously you need to have #1 completed and in place before you attempt #2 so you don't miss any changes. To accomplish #2, you either write an Agent or client code that grabs all the keys via the MapIndexPlugin.SYSTEM_KEY_INDEX_NAME and determine for each value, whether you are interested in the tuple.

    To implement #2, you just need an ObjectGridEventListener that is interested in the TransactionEvents. On transactionEnd, you can process the committed changes and determine which log elements (changes) you are interested or no longer interested in. You will then have to determine how you get these changes to your client.
    I have an environment where the WXS is deployed on a machine (say B) and java clients on another machine (say A) on the same network.

    I'm using ObjectMap for storing the entries received from the clients. There's only one ObjectMap at this time. I'm not making use of the near cache as of now.

    From some clients, entries are continuously being inserted in WXS and one of the clients (say *retrieveEntries*) needs to receive notifications from the grid (machine B) as and when entries are inserted in the BackingMap(satisfying some criteria). The idea is to have the client retrieveEntries notified (pub-sub kind of a mechanism perhaps) and process these entries. I'm looking at the following different mechanisms:

    a) Query Queue set up on the WXS server side but not sure how to get the entries over to the client retrieveEntries
    b) JMSObjectGridEventListener. Would this be appropriate in this case? Any sample code that uses this? Looking at the API specs, it's not clear how to use this and under what conditions is it useful.

    If these are not appropriate, what would work in my scenario? Do I need to use the inbuilt JMS mechanism in WXS and setup my publish-subscribe model on top of that? Is there any sample code that achieves similar functionality?

    Awaiting your comments/suggestions.

    Sanjay
  • cheng1
    cheng1
    44 Posts

    Re: A pub-sub kind of a model is required in WXS

    ‏2011-12-14T22:21:12Z  
    I have an environment where the WXS is deployed on a machine (say B) and java clients on another machine (say A) on the same network.

    I'm using ObjectMap for storing the entries received from the clients. There's only one ObjectMap at this time. I'm not making use of the near cache as of now.

    From some clients, entries are continuously being inserted in WXS and one of the clients (say *retrieveEntries*) needs to receive notifications from the grid (machine B) as and when entries are inserted in the BackingMap(satisfying some criteria). The idea is to have the client retrieveEntries notified (pub-sub kind of a mechanism perhaps) and process these entries. I'm looking at the following different mechanisms:

    a) Query Queue set up on the WXS server side but not sure how to get the entries over to the client retrieveEntries
    b) JMSObjectGridEventListener. Would this be appropriate in this case? Any sample code that uses this? Looking at the API specs, it's not clear how to use this and under what conditions is it useful.

    If these are not appropriate, what would work in my scenario? Do I need to use the inbuilt JMS mechanism in WXS and setup my publish-subscribe model on top of that? Is there any sample code that achieves similar functionality?

    Awaiting your comments/suggestions.

    Sanjay
    My understanding is your clients want to be notified from server side grid when updates occur.
    The JMSObjectGridEventListener should be able to meet your requirement.

    1. You have to extend JMSObjectGridEventListener and override onMessage() method. For example, you can name it as ExtendedJMSObjectGridEventListener.
    In the onMessage() method, you can process LogSequence collection embedded on received JMS message. sample code is attached.
    2. Configute ObjectGrid to use ExtendedJMSObjectGridEventListener as ObjectGridEventListener with properties for client invalidation mechanism. You will need to set up your JMS service to be used by grid.

    sample:

    <objectGrids>
    <objectGrid name="AgentObjectGrid">
    <bean id="ObjectGridEventListener" className="com.ibm.ws.objectgrid.test.jms.ExtendedJMSObjectGridEventListener">
    <property name="invalidationModel" type="java.lang.String" value="CLIENT_SERVER_MODEL" description="" />
    <property name="invalidationStrategy" type="java.lang.String" value="PUSH" description="" />

    <!-- this property control which maps to publish, if now used, all maps will be included.
    <property name="mapsToPublish" type="java.lang.String" value="agent" description="" />
    -->

    <property name="jms_topicConnectionFactoryJndiName" type="java.lang.String" value="defaultTCF" description="" />
    <property name="jms_topicJndiName" type="java.lang.String" value="defaultTopic" description="" />
    <property name="jms_topicName" type="java.lang.String" value="defaultTopic" description="" />
    <property name="jms_userid" type="java.lang.String" value="" description="" />
    <property name="jms_password" type="java.lang.String" value="" description="" />
    <property name="jndi_properties" type="java.lang.String" value="java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory;java.naming.provider.url=tcp://localhost:61616;connectionFactoryNames=defaultTCF;topic.defaultTopic=defaultTopic" description="jndi properties" />
    </bean>

    <backingMap name="agent" readOnly="false" pluginCollectionRef="agent" preloadMode="false" lockStrategy="OPTIMISTIC" copyMode="COPY_ON_READ_AND_COMMIT" ttlEvictorType="LAST_ACCESS_TIME" timeToLive="10" />
    <backingMap name="profile" readOnly="false" pluginCollectionRef="profile" preloadMode="false" lockStrategy="PESSIMISTIC" copyMode="COPY_ON_READ_AND_COMMIT" ttlEvictorType="LAST_ACCESS_TIME" timeToLive="10" />
    <backingMap name="pessimisticMap" readOnly="false" pluginCollectionRef="pessimisticMap" preloadMode="false" lockStrategy="PESSIMISTIC" copyMode="COPY_ON_READ_AND_COMMIT" ttlEvictorType="LAST_ACCESS_TIME" timeToLive="2700" />
    <backingMap name="excludedMap1" readOnly="false" pluginCollectionRef="excludedMap1" preloadMode="false" lockStrategy="PESSIMISTIC" copyMode="COPY_ON_READ_AND_COMMIT" ttlEvictorType="LAST_ACCESS_TIME" timeToLive="2700" />
    <backingMap name="excludedMap2" readOnly="false" pluginCollectionRef="excludedMap2" preloadMode="false" lockStrategy="PESSIMISTIC" copyMode="COPY_ON_READ_AND_COMMIT" ttlEvictorType="LAST_ACCESS_TIME" timeToLive="2700" />
    </objectGrid>
    </objectGrids>
    3. If all clients need to be notified about server side change, no further action.
    If only few clients, says retrieveEntries, need to be notified, you have to use client override xml file for those non-*retrieveEntries* cliens to avoid unnecessary JMS message processing.
    The client override xml file for those non-*retrieveEntries* should remove ObjectGridEventListener plugin from ObjectGrid configuration.
  • KAN6_Sanjay_Nath
    KAN6_Sanjay_Nath
    9 Posts

    Re: A pub-sub kind of a model is required in WXS

    ‏2011-12-16T16:52:54Z  
    • cheng1
    • ‏2011-12-14T22:21:12Z
    My understanding is your clients want to be notified from server side grid when updates occur.
    The JMSObjectGridEventListener should be able to meet your requirement.

    1. You have to extend JMSObjectGridEventListener and override onMessage() method. For example, you can name it as ExtendedJMSObjectGridEventListener.
    In the onMessage() method, you can process LogSequence collection embedded on received JMS message. sample code is attached.
    2. Configute ObjectGrid to use ExtendedJMSObjectGridEventListener as ObjectGridEventListener with properties for client invalidation mechanism. You will need to set up your JMS service to be used by grid.

    sample:

    <objectGrids>
    <objectGrid name="AgentObjectGrid">
    <bean id="ObjectGridEventListener" className="com.ibm.ws.objectgrid.test.jms.ExtendedJMSObjectGridEventListener">
    <property name="invalidationModel" type="java.lang.String" value="CLIENT_SERVER_MODEL" description="" />
    <property name="invalidationStrategy" type="java.lang.String" value="PUSH" description="" />

    <!-- this property control which maps to publish, if now used, all maps will be included.
    <property name="mapsToPublish" type="java.lang.String" value="agent" description="" />
    -->

    <property name="jms_topicConnectionFactoryJndiName" type="java.lang.String" value="defaultTCF" description="" />
    <property name="jms_topicJndiName" type="java.lang.String" value="defaultTopic" description="" />
    <property name="jms_topicName" type="java.lang.String" value="defaultTopic" description="" />
    <property name="jms_userid" type="java.lang.String" value="" description="" />
    <property name="jms_password" type="java.lang.String" value="" description="" />
    <property name="jndi_properties" type="java.lang.String" value="java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory;java.naming.provider.url=tcp://localhost:61616;connectionFactoryNames=defaultTCF;topic.defaultTopic=defaultTopic" description="jndi properties" />
    </bean>

    <backingMap name="agent" readOnly="false" pluginCollectionRef="agent" preloadMode="false" lockStrategy="OPTIMISTIC" copyMode="COPY_ON_READ_AND_COMMIT" ttlEvictorType="LAST_ACCESS_TIME" timeToLive="10" />
    <backingMap name="profile" readOnly="false" pluginCollectionRef="profile" preloadMode="false" lockStrategy="PESSIMISTIC" copyMode="COPY_ON_READ_AND_COMMIT" ttlEvictorType="LAST_ACCESS_TIME" timeToLive="10" />
    <backingMap name="pessimisticMap" readOnly="false" pluginCollectionRef="pessimisticMap" preloadMode="false" lockStrategy="PESSIMISTIC" copyMode="COPY_ON_READ_AND_COMMIT" ttlEvictorType="LAST_ACCESS_TIME" timeToLive="2700" />
    <backingMap name="excludedMap1" readOnly="false" pluginCollectionRef="excludedMap1" preloadMode="false" lockStrategy="PESSIMISTIC" copyMode="COPY_ON_READ_AND_COMMIT" ttlEvictorType="LAST_ACCESS_TIME" timeToLive="2700" />
    <backingMap name="excludedMap2" readOnly="false" pluginCollectionRef="excludedMap2" preloadMode="false" lockStrategy="PESSIMISTIC" copyMode="COPY_ON_READ_AND_COMMIT" ttlEvictorType="LAST_ACCESS_TIME" timeToLive="2700" />
    </objectGrid>
    </objectGrids>
    3. If all clients need to be notified about server side change, no further action.
    If only few clients, says retrieveEntries, need to be notified, you have to use client override xml file for those non-*retrieveEntries* cliens to avoid unnecessary JMS message processing.
    The client override xml file for those non-*retrieveEntries* should remove ObjectGridEventListener plugin from ObjectGrid configuration.
    Thanks Cheng. Is there some document that explains the different options that can be used in the xml files?

    Sanjay
  • cheng1
    cheng1
    44 Posts

    Re: A pub-sub kind of a model is required in WXS

    ‏2011-12-16T19:16:11Z  
    The JMSObjectGridEventListener API doc should explain how to configure.
    http://publib.boulder.ibm.com/infocenter/wxsinfo/v7r1m1/topic/com.ibm.websphere.extremescale.javadoc.doc/topics/com/ibm/websphere/objectgrid/plugins/builtins/JMSObjectGridEventListener.html

    The following are related infomration for your reference.

    http://publib.boulder.ibm.com/infocenter/wxsinfo/v7r1m1/topic/com.ibm.websphere.extremescale.doc/cxsjmsevntls.html

    http://publib.boulder.ibm.com/infocenter/wxsinfo/v7r1m1/index.jsp?topic=%2Fcom.ibm.websphere.extremescale.doc%2Fcxsjmsevntls.html
  • KAN6_Sanjay_Nath
    KAN6_Sanjay_Nath
    9 Posts

    Re: A pub-sub kind of a model is required in WXS

    ‏2011-12-19T06:17:56Z  
    Thanks Cheng. Is there some document that explains the different options that can be used in the xml files?

    Sanjay
    Hi Cheng,

    In your reply, you have said that I need to setup the JMS service to be used by the grid. How's that to be done? Is it taken care of by the lines in the sample xml file with references to JMS? I'm referring to the XML file pasted in your reply. Or I need to do something else to get the JMS going?

    Sanjay
  • cheng1
    cheng1
    44 Posts

    Re: A pub-sub kind of a model is required in WXS

    ‏2011-12-19T16:06:40Z  
    In a standalone WXS environment, you can use any standalone JMS service. There are few open source project you can consider to use. Depens on how you setup JMS service, you will have JMS connnection and JNDI information that you have to fill in JMSObjectGridEventListener configruation. The JMSObjectGridEventListener need JNDI to JMS service information in order to connect to it. The JMS connection information is used to make JMS connection for topic.
  • Erwin_Karbasi
    Erwin_Karbasi
    186 Posts

    Re: A pub-sub kind of a model is required in WXS

    ‏2012-04-08T08:48:26Z  
    • cheng1
    • ‏2011-12-19T16:06:40Z
    In a standalone WXS environment, you can use any standalone JMS service. There are few open source project you can consider to use. Depens on how you setup JMS service, you will have JMS connnection and JNDI information that you have to fill in JMSObjectGridEventListener configruation. The JMSObjectGridEventListener need JNDI to JMS service information in order to connect to it. The JMS connection information is used to make JMS connection for topic.
    Hello,

    I have also the same Pub/Sub scenario.

    I'd appreciate if you could enlighten the following points:

    1. Could i use AMQP - RabbitMQ as a broker? If yes, i would appreciate to be provided with some samples.
    2. The WXSUtils documentation has a chapter on LISTS AND DIRTY SETS which explains how this can be leveraged. Can i use this feature from WXSUtils for Pub/Sub scenario?

    Your insight would be appreciated.

    Thanks in advance,
    Erwin