Contents


Integrating IBM Integration Bus with WebSphere Service Registry and Repository

Part 7: Configuring the WebSphere Service Registry and Repository cache in IBM Integration Bus

Comments

Content series:

This content is part # of # in the series: Integrating IBM Integration Bus with WebSphere Service Registry and Repository

Stay tuned for additional content in this series.

This content is part of the series:Integrating IBM Integration Bus with WebSphere Service Registry and Repository

Stay tuned for additional content in this series.

This article describes the behavior of a broker-managed cache referenced by WSRR integration look-up nodes available for inclusion within IBM Integration Bus message flows. The article then shows you how to configure the broker managed cache (hereafter referred to as the WSRR cache) to integrate WSRR and IBM Integration Bus, and how to validate its correct operation. Lastly, the article shows you how to manage the WSRR cache to decouple the two products to create a service window, so that WSRR can be upgraded without shutting down IBM Integration Bus or its runtime flows, and without requiring any downtime in the production environment. This procedure has significant business value, because naturally, an IBM Integration Bus runtime environment needs to see as little downtime as possible.

Characteristics

The WSRR integration nodes: Endpoint Lookup and Registry Lookup are designed to interact with WSRR, enabling the extraction of data gathered through queries issued against this registry system. Therefore these nodes allow information to be looked up from registry and injected into their out going message trees, for subsequent downstream processing within any flow they are embedded within.

Broker implements a local cache that, if enabled, stores the results of such calls against the query statement concerned. With caching enabled (default setting) the WSRR integration nodes may retrieve data directly from the local cache, if present due to an exact matching query being previously issued. Accessing locally cached query result sets, rather than re-issuing an identical query statement, improves message throughput and thus the performance of such flows which integrate with WSRR.

Note: As a rule, if message processing through put is your primary concern, then the recommendation is to enable caching. However, if your key goal is to process each message against the latest volatile content stored in WSRR, then caching should be disabled.

The first occurrence of each query is always sent to WSRR via an established HTTP based web services connection. By default this activity occurs when a WSRR node first issues a specific query, although it is possible to pre populate the cache when the broker is started (see Cache preloading).

The WSRR nodes can operate without the cache. If the cache is disabled, every query that is issued by the node is sent to WSRR, ensuring that the results of the query always reflect the current contents of the registry. This option will have an affect on performance.

The WSRR cache is provided on a per execution group basis and as such is a shared resource across message flows within a given execution group. The cache is locked when it is being queried (read from) or updated (written to), either by a message flow or by dynamic updates (see Cache notification).

WSRR cache operation

The message flows within IBM Integration Bus Execution Groups invoke the WSRR Endpoint Lookup or Registry Lookup integration nodes. Queries from such nodes are forwarded to the WSRR proxy, an IBM Integration Bus component that interfaces to the results of previous queries held in the cache.

If an existing entry is in the cache for a query, the corresponding data is returned directly to the message flow without referral to WSRR. If no such entry exists, a synchronous query is made to WSRR, and the results of that query are stored in the cache and then returned to the message flow.

All cache entries are expired and removed from the cache after the configured timeout interval. This means that if a query then follows, requesting the data that was held by an expired cache entry, a fresh lookup will need to be made to WSRR.

Note: At the point when a message enters a WSRR node in a message flow, the cache is locked across the entire execution group to preserve its state whilst accessed by that node. Should an update notification be received whilst the cache is locked, the update is processed after the message flow relinquishes its lock on the cache. As such, it is feasible that should a message and a notification arrive at the same time, the message will be populated with data from the cache which no longer reflects the data on the server.

Therefore notification updates can influence the WSRR cache content; which subsequently could be accessed by downstream WSRR nodes within a flow for a given message instance, being processed within it. That is for cache notification we can't guarantee that the contents of the cache is constant throughout the entire time a given flow instance processes a specific message.

In fact due to cache entry timeouts (see below) broker doesn't guarantee consistency in cache contents across WSRR nodes for a given flow and message instance, regardless of whether cache notification is enabled or not. Therefore, a scenario is therefore possible where further WSRR nodes in a flow, processing the same message instance, work against different values due to cache entries becoming refreshed after the lock is released upstream in the flow.

Although each broker execution group process maintains a copy of its own WSRR cache, the characteristics of all such caches are identical across a given broker instance. WSRR cache configuration parameter values are tied to the ServiceRegistries configurable service; you will need to recycle your broker to activate any changes made against this service. For more information, see the Resources section.

However, to drop (invalidate) a specific WSRR cache, you can recycle its associated execution group. Therefore, separate out any WSRR node specific flows into one or more of their own execution groups. Taking this approach forces separation, containing them and permitting other flows to remain active whilst a WSRR cache for a given group of flows is invalidated.

In addition to invalidating the entire cache, timeouts are applied to individual entries within the cache. When a query is issued to WSRR and a response is received, a timestamp is appended to the query statement and its associated result set. This means that entries timeout from the cache on a per entry basis.

The cache expiry timeout value is altered by setting the timeout parameter tied to the ServiceRegistries configurable service. Each cache entry (the cached results of a query) is discarded after the specified time (in milliseconds) elapses. The default cache expiry timeout value is 100,000,000 milliseconds or approximately 27.8 hours.

The next reference by a node to the timed out query forces the query to be resent to WSRR; the corresponding result replaces its stale entry in the cache and is associated with a new timestamp.

The number of result set entries available within the cache is unbounded. That is, there is no enforced maximum cache size.

The above capabilities of the WSRR cache are further enhanced by two additional capabilities which combine to promote increased performance in registry look-up and facilitate the provision of a service window for a predictable period of time. The relevant capabilities are described below.

Cache preloading

By default, no items are preloaded in the cache and the first occurrence of every query is sent to WSRR. You can specify predefined queries that are executed when the broker starts or when a message flow containing WSRR flows is first deployed. These populate the cache for use by subsequent WSRR nodes. By specifying predefined queries performance might be affected at startup, rather than on the first occurrence of a query at run time.

Cache notification

Cache notification is a more flexible method than expiry timeout for refreshing data in the cache. It allows individual WSRR entities to be invalidated in the cache at the time they are modified in WSRR, ensuring the cache never contains stale data in its stored entries.

For cache notification to be enabled the cache itself must also be. When this is the case, the cache subscribes to events occurring in WSRR and is notified when an object within it is either updated or deleted. In this scenario entries within the cache may be invalidated either directly through their own timeout expiring or indirectly through event subscription.

Subscription is achieved via a JMS based cache notifications connection, which is established in addition to the web services connection. As stated, the cache notification mechanism is used to ensure the contents of the cache are current and in-step with the contents of the WSRR server.

WSRR cache update notification

Suppose that the IBM Integration Bus WSRR cache is already loaded with data and a user of a governance WSRR system makes a change relating to a piece of data in the WSRR cache. WSRR publishes a message to the IBM Integration Bus Cache Update Listener component to notify of this change.

The Cache Update Listener receives this message and then acts upon the said data from the cache. The action taken is dependent on the type of change made in the WSRR server. If an item has been deleted, then just that item will be removed from the cache. Any other actions will result in the cache being expired to ensure the cache remains in a consistent state. By default, this means that the next time a message flow query is made requesting this data the request is referred back to WSRR. Note that the timeout mechanism also continues to function when Cache update notification is configured for use.

Configuring cache preloading

Cache preloading is concerned with enhancing flow performance (message throughput) in broker. Normally the cache is populated on demand, where a performance hit is accepted for the first occurrence of a query statement being issued by a WSRR node within a flow. Hence for the first message processed by such a node, on startup or following an elapsed timeout, there is an anticipated cost associated with collecting the query results directly from WSRR rather than from the local broker cache.

To improve on this, it is possible to eliminate this initial overhead at runtime. This is achieved by declaring up front a list of known query expressions that may be issued by broker; when defined the queries are executed on initial broker startup, thus preloading the cache with query results prior to the processing of any messages by the run time system.

The predefined list of WSRR XPath query expressions are exposed to broker via the ServiceRegistries configurable service's predefinedCacheQueries parameter. A list of expressions, separated by semicolons (each with an optional depth specification) can be defined through this parameter directly. For more information, see the Related topics section.

Alternatively these expressions maybe accessed indirectly by placing the expressions within a file pointed to via the parameter. You can gather the collection of expressions issued by a broker system by switching on user trace and exercising the runtime for the flows concerned. An examination of the user trace shows the WSRR XPath query expression generated by any deployed WSRR integration nodes.

Building the service query file

To implement broker isolation from WSRR, it is necessary to configure cache preloading within broker. This allows broker to pull all known service artifacts from WSRR after the execution group is started. Cache preloading will occur every time an execution group within broker is started, or when a message flow containing WSRR integration nodes is first deployed.

The cache implementation uses the service queries that are executed against WSRR as keys into the cache. These service queries are unique and, when configuring cache preloading, they need to be specified in exactly the same format as used by the Endpoint Lookup node or Registry Lookup node at run time. If these queries are unknown, the only way to determine them is to enable user trace within broker and execute the relevant flows. The queries can then be read from the broker trace files. There are two ways to start the user trace:

  • The first is to enable user trace within broker itself. The types of tracing are normal or debug. For this exercise the trace will need to be set at debug. This is accomplished by running the following command:
    mqsichangetrace <broker_name> -u –e <execution_group> -l debug
  • The second is to enable user trace from within MQ Explorer. This is accomplished by switching to WebSphere MQ Explorer. From the Navigator View, expand the Integration Nodes tree element and right click on the execution group on which to enable user trace. Click User Trace All Flows and then the level of trace. In this case Debug. The image shows the view:

When a user trace is enabled, you must execute all of the flows that might execute WSRR queries. During the execution of these, each time an Endpoint Lookup node or Registry Lookup node executes a query against WSRR, the query will be captured in the trace.

It will be these queries that are used to create a Query file. After all of the relevant queries are run, scan the trace files for the phrase ‘BIP3676I’. This is the message ID for the Endpoint lookup. It looks like the following in the trace file:

EndpointLookup query string user trace logged by broker
2013-12-06 17:01:56.414672    14395   UserTrace   BIP3676I: The query 
string from node ''EndpointLookup'' that will be used to query the 
WebSphere Service Registry and Repository (WSRR) is: 

'/WSRR/WSDLService/ports[binding/portType[@name='MathServerPortType' 
and @namespace='http://math.pot.ibm.com' 
and @version='1.0']
and exactlyClassifiedByAllOf(., 'http://www.ibm.com/xmlns/prod/
serviceregistry/lifecycle/v6r3/LifecycleDefinition#Online')]' 

This is the query string that has been generated to query the WSRR. 
No user action required.

The query string is everything in the single quotes. So, the previous trace has the following WSRR query:

/WSRR/WSDLService/ports[binding/portType[@name='MathServerPortType' 
and @namespace='http://math.pot.ibm.com' and @version='1.0']
and exactlyClassifiedByAllOf(., 'http://www.ibm.com/xmlns/prod/
serviceregistry/lifecycle/v6r3/LifecycleDefinition#Online')]

This query string must be modified to escape any single quotation marks. Each single quote must be replaced with &apos;. The new string query looks like this:

/WSRR/WSDLService/ports[binding/portType[@name=&apos;
MathServerPortType&apos; 
and @namespace=&apos;http://math.pot.ibm.com&apos; 
and @version=&apos;1.0&apos;]
and exactlyClassifiedByAllOf(., &apos;
http://www.ibm.com/xmlns/prod/serviceregistry/lifecycle/v6r3/
LifecycleDefinition#Online&apos;)]

Equally trace files will contain the phrase ‘BIP3685I’. This is the message id for the registry look-up. It looks like the following in the trace file:

RegistryLookup query string user trace logged by broker
2013-12-06 17:05:14.519078    10849   UserTrace   BIP3685I: The 
following details will be used to query WebSphere Service Registry 
and Repository (WSRR) from node 'RegistryLookup': query string=
'//*[@name='MathService' and @namespace='http://math.pot.ibm.com' 
and @version='1.0']', depth='1'

The Depth Policy that controls how the retrieved data will be displayed is MatchPlusImmediate. The specified query string and depth are sent to WSRR. The information returned from WSRR is added to the LocalEnvironment as specified by the DepthPolicy. If these are not the settings you expected, check whether the node properties are being overridden in the LocalEnvironment, being careful to check the spelling of the DepthPolicy override value if this is being used.

The set of queries that are executed by the broker message flows against WSRR should be stored in a separate text file. Each query string within the file must be separated by a single semi-colon and all of the query strings must appear on a single line, i.e. no new lines or line breaks can be appear anywhere in the text file. The number of XPATH queries that can be specified and preloaded is unbounded. Here is an An example of a query file:

/WSRR/WSDLService/ports[binding/portType[@name=&apos;
MathServerPortType&apos; 
and @namespace=&apos;http://math.pot.ibm.com&apos;
and @version=&apos;1.0&apos;]
and exactlyClassifiedByAllOf(., &apos;
http://www.ibm.com/xmlns/prod/serviceregistry/lifecycle/v6r3/
LifecycleDefinition#Online&apos;)]&apos; 
/WSRR/WSDLService/ports[binding/portType[@name=&apos;
MathServerPortType_V2&apos; 
and @namespace=&apos;http://math.pot.ibm.com/V2&apos;
and @version=&apos;2.0&apos;]
and exactlyClassifiedByAllOf(., &apos;
http://www.ibm.com/xmlns/prod/serviceregistry/lifecycle/v6r3/
LifecycleDefinition#Online&apos;)]&apos; 
//*[@name=&apos;MathService&apos; 
and @namespace=&apos;http://math.pot.ibm.com&apos; 
and @version=&apos;1.0&apos;]&apos;, 
depth=&apos;1&apos;

This text file needs to be saved on the server or in a network location that the server can access. At this point, the user trace can be disabled by setting the trace level back to None. As explained previously, this can be accomplished from either the command line or MQ Explorer.

Enabling cache preloading

After the query file is created and placed on the server, configure the broker to use the text file at execution group start up. The change properties command will be used to achieve this and is accomplished by running the following command:

Command to enable preloading of the WSRR cache from file
mqsichangeproperties <broker_name> -c ServiceRegistries -p 
"<wsrr_query_text_file>" -o DefaultWSRR -n predefinedCacheQueries

Replace <broker_name> with the name of the broker that is being used. Replace <wsrr_query_text_file> with the absolute file path location and name of the query file. The execution group should be restarted in order for the changes to the cache configuration to take effect. Rerun all of the test flows to validate everything is still working as expected.

This query text file will only be used at start up. The data in the cache will still be invalidated based on the value of the cache expiry timeout. Entries that have been invalidated will need to be fetched again from WSRR when the relevant flow within broker is executed.

For this reason, the cache expiry timeout needs to be set to a suitable value to create the required service window. An example of the broker command to use to report the timeout value is as follows:

Command to report the WSRR related properties set against a given broker system
mqsireportproperties <broker_name> -c ServiceRegistries 
–o DefaultWSRR -r

From the results, the timeout value is the length of time that a query result set in the cache will be considered current, that is, valid. The time out value is displayed in milliseconds. To change this setting, enter the following command:

Command to modify the WSRR cache timeout property set against a given broker system
mqsichangeproperties <broker_name> -c ServiceRegistries 
–o DefaultWSRR –n timeout –v 10000000

As explained previously, the value specified should be large enough to create a service window that can accommodate an upgrade to WSRR, WebSphere Application Server and the WSRR database instance. This timeout value is something that should be determined by the production team; based on the average time it takes them to typically perform an upgrade of the specified products.

Maintaining the query file

At the point a query file is first established, it should be regarded as a piece of source code and therefore placed into the production team's source code management system. As new Endpoint Lookup or Registry Lookup nodes are added to broker flows, their relevant queries also need to be appended to this text file. The process to establish new or modified queries remains unchanged, as described previously during the original construction of the query text file. Therefore, the construction and modification to this text file should follow the teams adopted Software Development Life Cycle (SDLC) processes.

Validating the cache

At some point (pre production), the production team will need to confirm the operational validity of their broker managed caches; specifically they will want to test that the isolation of their run time broker from its targeted WSRR environment is possible.

In doing so, they will validate that with an enabled and preloaded cache, the WSRR environment can be deactivated without interfering with the operation of the broker run time. Broker will continue to run without any downtime. This will become invaluable in the upgrade process of WSRR.

After all of the preloading configuration steps are completed and the cache is fully populated following initial broker startup, the WebSphere Application Server instance under which WSRR is running should be stopped. This can be achieved either through the command line or by using the WebSphere Administration Console.

After the WSRR server is stopped, all broker test cases (WSRR integration flows) will need to be re-executed by the production team in order for them to validate that their broker is still completely operational despite not having a running WSRR system to target. When the broker test cases are exercised and their operation is validated, the production team can restart their WSRR server instance.

Managing the cache and JVM heap size

One question that immediately comes to mind when making use of the WSRR cache, is whether the production team needs to be concerned about breaching JVM heap size limits. That is, will they need to raise the default JVM limits in broker, because of the use of the WSRR cache in an execution group? The answer to this is no, not necessarily.

In IBM Integration Bus you can monitor the JVM heap size and rate of garbage collection by enabling and using resource statistics published by broker. This capability will then give an indication of whether the production team needs to increase the JVM limits. The broker publishes resource statistics consumed on an individual execution group basis. Therefore, to narrow down the WSRR cache usage figures, the production team should isolate their WSRR flows into one or more specific execution groups to monitor. For more information, see the Related topics section.

Configuring cache notification

With cache notification enabled and on the start-up of an execution group containing one or more message flows, which include at least one Endpoint Lookup or Registry Lookup node, the broker will subscribe to the WSRR JMS publication topic: jms/SuccessTopic on a given WebSphere Application Server messaging engine. Any messages that are published against this topic by WebSphere Application Server (on behalf of WSRR) will be received by the broker and will be analyzed.

If the message is of event type: create, update, transition, make_governable or remove_governance, the entire cache is invalidated. If the message is of event type: delete, then the object to which the message refers, as identified by a unique WSRR Uniform Resource Identifier (bsrURI), is removed from the cache. These notifications are acted upon when they are received.

Should the cache notification connection fail, for whatever reason, the connection will be continuously retried indefinitely. An error message reporting a failed connection will be output on decreasing frequency, which will increase by one minute after each error output. After the connection is re-established the entire cache is invalidated to ensure it becomes refreshed over time.

During cache notification connection failure, cache entry timeouts will still continue to occur. As stated earlier, cache timeouts are on a per query basis, so it depends on when a specific query was first issued as to when its elapsed time starts and thus in turn when it expires.

If the JMS connection drops for cache notifications, the whole look up process is attempted to re-establish a subscription connection i.e. the JNDI server is re- accessed. No exception is thrown in the flow and the execution group will indefinitely re-attempt to connect.

Enabling JMS notification of the cache

When the WSRR cache is enabled, broker only updates a cache entry after it's requested by a look-up node and its timeout value is breached. Therefore if a service definition in WSRR is modified before this configured timeout has elapsed, then broker will not reflect the change, rather it will return a stale value contained in the local cache.

The cache within broker can be configured to receive JMS notifications from WSRR; these would be issued instantly whenever content is updated within the registry system.

To enable cache notification, the first step is to run the following command:

Command to enable the WSRR cache, on a broker system, to begin processing update notifications published by WSRR
mqsichangeproperties <broker_name> -c ServiceRegistries 
–o DefaultWSRR –n enableCacheNotification –v true

Next, set the location of the binding property (IIOP URI) with the following command.

Command to set the system end point where the WebSphere Application Server JMS provider JNDI bindings are located
mqsichangeproperties <broker_name> -c ServiceRegistries 
–o DefaultWSRR –n locationJNDIBinding –v iiop://<WSRR_host>:<PORT>

Change the –v parameter to be the location of the JNDI Binding on your system. The parameter represents a URL to the WebSphere Application Server JMS provider JNDI bindings, where the hostname <WSRR_host> is variable and represents a JNDI server on a network where the binding definition can be looked up. This setting is broker wide and picked up by all execution groups for a given broker. Each execution group JMS connection is established as part of its start-up processing.

Additionally, Internet InterORB Protocol (IIOP) is an implementation of the General Inter-ORB Protocol (GIOP) for use over the Internet. It provides a mapping between GIOP messages and the TCP/IP layer.

The WebSphere Application Server that WSRR is running in can have the following settings:

  • Security enabled
  • A high availability environment (A connection factory can stipulate multiple bootstrap servers.)

In this case, run the following command:

Command to set the JMS connection factory name; used to locate a messaging engine
mqsichangeproperties <broker_name> -n jms::DefaultWSRR@jms/SRConnectionFactory
 –u <userid> -p <password>

The -n token describes the connection factory that is used to connect to the WSRR service integration bus. After you run this command, restart the broker instance by running the following commands:

mqsistop <broker_name>
mqsistart <broker_name>

To confirm that JMS notification of the cache is now enabled and configured correctly, a previously valid and operational test case might need to be re-exercised in the modified broker environment. Here, for example, a service might need to be changed within the WSRR system; without restarting broker or any of its deployed run time artifacts, the same test might need to be re-executed. The purpose is to validate that the registry change was detected and that the broker cache has been refreshed, reflecting the applied change.

Cache JMS notification and high availability

At the time of writing this article, IBM Integration Bus has a known restriction in that it can only use a simple (single hostname) IIOP URI to look up the WebSphere Application Server JMS provider JNDI bindings in the WebSphere Application Server namespace.

Multiple alternate endpoints are not supported by broker and this setting is applied at the broker level. As stated, this single point of failure is used to acquire the JMS provider JNDI bindings which in turn are used to look up a connection factory; which in a high availability environment could stipulate multiple bootstrap servers through which a messaging engine is selected.

For more information about how to configure, under WebSphere Application Server, a connection to a non-default bootstrap server, see the Related topics section.

To make this JNDI server endpoint more available there are various system architectural topologies that may be deployed.

Virtual IP

For automatic high availability between members within a WebSphere Application Server cluster, the use of a network appliance solution is recommended. Here a Virtual IP (VIP) is used to monitor ports within the WebSphere Application Server cluster, routing requests to the active application server.

Under VIP an IP address is maintained by some intermediate system (usually a network device from a vendor like F5 or Cisco) that masks multiple back end systems. Such devices have lots of configuration options for workload balancing across the multiple systems hidden behind the VIP. One is usually to support the active/passive pattern between the systems, with some regular port monitoring to notice when the back-end system (a WebSphere Application Server cluster member in this case, which acts as the JNDI server) is down. This option, if available, is the easiest way to support the integration of a broker with a highly available registry system.

Locate the JNDI WebSphere Application Server server locally on the broker machine

The purpose here is to provide somewhere to put the JNDI object, which importantly in theory should always be available when the broker is available.

Provide a disaster recovery environment

To support this, you can point the IIOP URI at a host name that's dynamic. A dynamic hostname is similar to a VIP, but is provided through a Domain Name System (DNS) server so that the solution doesn't require a network device. However, someone needs to control the server's DNS records. One approach is a simple hosts file on the local operating system, which can be updated by using any script. Here the DNS only has one entry in it at any point, but another system would dynamically update it based on monitoring of the WebSphere Application Server side. Known monitoring technologies that support this include AIX PowerHA (HACMP).

Production maintenance

Service upgrade in a non-HA environment

The first upgrade step in such a WSRR environment will be to ensure that the WSRR caches managed by broker will not be invalidated whilst the singular targeted WSRR system is stopped. Because it is not possible to determine when entries in a managed cache were last invalidated, deploy each execution group to the production run time broker that is restarted, one at a time. Again, only execution groups that contain one or more message flows which include at least one Endpoint Lookup or Registry Lookup node need to be recycled.

By restarting each execution group in sequence, the whole broker environment will never be completely unavailable. When the restart is completed, each cache that is owned by an execution group is immediately populated with all of the required data from WSRR (assuming preloading configured) and will not be invalidated until the broker wide cache expiry timeout is reached.

This timeout value should be set by the production team; it should provide a time window large enough to accommodate the planned service upgrade activity. In a non high availability environment only one WSRR system will be available for access by the broker runtime. During service upgrade this system will be moved offline and so the broker established web service connection will be lost. In such a scenario, broker managed caches cannot honor WSRR look-ups if a timeout on an specific entry has elapsed. Such entries are invalidate, causing their stale data to be destroyed, despite the fact that an active web service connection is not available.

The next step is to follow the recommended upgrade process that comes with the fix pack or the latest version.

Promotion with cache notification

For synchronous promotions under WSRR with cache notification enabled, events tied to a given promotion will not be committed to the SuccessTopic until the transaction has completed successfully. Therefore, following a successful WSRR life-cycle transition, which triggers a promotion, production teams should look out for that life-cycle transition's success event being processed by broker.

Confirmation of the processing of this event will show up in broker service trace, if enabled, where this event represents the last event tied to a given promotion; therefore it is not anticipated after this point that the WSRR cache will be re-invalidated by further events associated to the same promotion committed under transactional control within WSRR.

At this point it is safe for the production team to switch off service trace and re-enable or start their WSRR flows, so that messages can continue to be processed against their fully updated WSRR system. For most production teams, switching on service trace will not be an option. Best practise advises the creation of a service flow for deployment within each effected execution group. This flow would consist of a JMSInput node connected to a TraceNode. Such a flow may be started prior to promotion; where its purpose would be to capture JMS success topic events published by WSRR and write them to file.

The Location JNDI bindings JMS connection property set on the JMSInput node should be set to the same value that is set for the ServiceRegistries configurable service's locationJNDIBinding property. So the assumption above is to disable message flows during the promotion process, ensuring that no messages are processed against a WSRR system or broker cache; where either maybe in a state of transition and thus their contents could be inconsistent.

However, in the case where integration with the target WSRR system is restricted to EndpointLookup nodes only, inconsistency is not an issue i.e. you either get an old or new endpoint. In this scenario there is no requirement to stop flows during promotion activity.

For registry look-up the same situation is not true. We don't want the cache holding inconsistent data i.e. all updates (barring deletes) causes the whole cache to be invalidated, but deletes only invalidate specific affected entries. Hence, in theory we could work with a mix of stale and up to date data following a promotion.

No events are published to the success topic until the whole transaction is committed following a promotion. The order of event messages and their types, which are pulled by broker and processed, is unknown; so the cache may not be immediately invalidated and thus forcing a fetch for all subsequent query requests.

Promotion with cache enabled

Here we assume that we have followed best practise guidance and thus isolated all WSRR specific flows into their own execution groups. Prior to WSRR production registry upgrade through promotion, the production team should stop the WSRR execution groups so that no more messages are processed against the production registry and the entire cache is invalidated.

They should then proceed and transition entities in their staged system so they are promoted, along with their target entities in the same governed collection, to the production system. They should monitor the synchronous promotions in WSRR and confirm the transitions have completed successfully. The final step is to restart the WSRR execution groups and continue to process new messages against the upgraded production registry. All query result sets will be initially acquired by a direct call to the production system. Fetched results are combined with their query to populate the cache with a new query entry. Subsequent calls are fetched directly from the cache, where the default timeout value for such entries should be extended to suit the team's operational requirements.

In the case where integration with the target WSRR system is restricted to EndpointLookup nodes only, inconsistency in the registry's contents during artefact promotion to it, is not an issue. In this scenario, as above, there is no requirement to stop flows during promotion activity.

Troubleshooting

Error connecting to WSRR from a broker

To diagnose this general fault, carry out the following checks:

Check the endpoint address value that points to your registry server. From the command console, run the following broker command, substituting <broker_name> for the name of your broker to display the WSRR properties associated with it:

mqsireportproperties <broker_name> -c ServiceRegistries 
–o DefaultWSRR -r

The above command will produce a response similar to the following output:

ReportableEntityName=''
ServiceRegistries
DefaultWSRR=''
connectionFactoryName='jms/SRConnectionFactory'
enableCacheNotification='false'
endpointAddress='http://<host>:<port><web service interface url>'
....

If you are not connecting to a secure WSRR server, you should be able to copy the reported endpointAddress URL value into a browser and get a valid response back i.e. not a 404 Not Found or 5xx server error returned as a status code. For example, consider the following invalid URL and the returned response:

http://localhost:9080/WSRR_9/services/WSRRCoreSDOPort
Error 404: 
com.ibm.ws.webcontainer.servlet.exception.NoTargetForURIException: 
No target servlet configured for uri: /WSRR_9/services/WSRRCoreSDOPort

Here are examples of valid URLs and their corresponding responses when sent to a WSRR V8.0 system:

http://localhost:9080/WSRRCoreSDO/services/WSRRCoreSDOPort            
{http://www.ibm.com/xmlns/prod/serviceregistry/6/0/ws/sdo}WSRRCoreSDOPort
Hi there, this is a Web service!

http://localhost:9080/WSRR6_1/services/WSRRCoreSDOPort
{http://www.ibm.com/xmlns/prod/serviceregistry/6/1/ws/sdo}WSRRCoreSDOPort
Hi there, this is a Web service!

http://localhost:9080/WSRR7_5/services/WSRRCoreSDOPort
{http://www.ibm.com/xmlns/prod/serviceregistry/7/5/ws/sdo}WSRRCoreSDOPort
Hi there, this is a Web service!

http://localhost:9080/WSRR8_0/services/WSRRCoreSDOPort
{http://www.ibm.com/xmlns/prod/serviceregistry/8/0/ws/sdo}WSRRCoreSDOPort
Hi there, this is a Web service!

If the previous action doesn't work, the URL isn't entered correctly. Usually there is an issue with the <host> or <port> value. Ensure that the hostname provided is fully qualified. For example, a fully qualified host name is: pic.dhe.ibm.com rather than pic.dhe.

For different versions of WSRR, there are different URLs supported for accessing their publicly exposed web service interface. Higher versions of WSRR support their own URL and for backward compatibility, previous versions of the same URL.

  • For example the Core web service interface URL supported by WSRR version 6.0.2 is: /WSRRCoreSDO/services/WSRRCoreSDOPort,
  • Where the same URL for V6.1 is: /WSRR6_1/services/WSRRCoreSDOPort.

Therefore, check the value reported for <web service interface url>, to ensure it is compatible with your target registry system. For example, V7.0.0.3 through V7.0.0.6 (plus APAR IC84693) provides access to WSRR V8 support, when using either V6.1, V6.2, or V6.3 compatibility URLs rather than V6.0.2. With the same broker level accessing WSRR V7.5, the V6.2 and V6.3 compatibility URLs aren't available.

Use the highest web service interface version available and supported by your combined broker and WSRR systems. For example, for V9, use the V6.3 core web service interface URL: WSRR6_3/services/WSRRCoreSDOPort.

See the Related topics section for a link to information on what web service interface URLs are available for use against a specific WSRR version. Another link shows system requirements for specific broker versions and platforms; more specifically, capabilities on offer by different versions of WSRR listed under the product collaboration section.

When using WebSphere Application Server V8.0, the default IIOP secure setting must be set to SSL-supported if you wish to enable and use JMS cache notification support. For later versions of WebSphere Application Server, the default JMS security setting was changed from supported to enforced. For more information, see Related topics.

Enable and examine broker user trace from within MQ Explorer. The specified endpoint address may be correct, but you may still see an invalid response. For example consider the response GIOP. It is generic, meaning that something went wrong when connecting with WSRR. This might indicate an exception occurred while binding to WebSphere Application Server where an error occurred when negotiating the General Inter-ORB network Protocol.

The next step in this instance, therefore, is to produce a broker service trace. Such a trace might capture an exception, if any, generated when the products attempted interaction. It could highlight whereabouts in the broker an error happened, whether it was sent back from WSRR or whether the broker hasn't handled something correctly. Consider the example service trace snippet below tied to such a response:

...'WSRRMbLogHandler:FINEST' , 'BUFFER RELEASED: Calling Element: 
com.ibm.ws.http.channel.impl.HttpServiceContextImpl.clear
(HttpServiceContextImpl.java:993) Main ID: 0'

...'WSRRMbLogHandler:FINEST' , 'BUFFER RELEASED: Calling Element: 
com.ibm.ws.genericbnf.impl.BNFHeadersImpl.clearBuffers
(BNFHeadersImpl.java:502) Main ID: 0'

...'WSRRMbLogHandler:FINER' , 'close(), 
com.ibm.ws.tcp.channel.impl.TCPConnLink@3d66c39 Entry'

...'WSRRMbLogHandler:FINEST' , 'Closing the connection: remote host 
name: wsrrhost.corp.xyz.com remote host address: x.x.x.x local host 
name: brokerhost local host address: x.x.x.x'

...'WSRRMbLogHandler:FINEST' , 'Connection closed with exception: 
Connection close: Read failed.  Possible end of stream encountered. '

...'WSRRMbLogHandler:FINER' , 'destroy(Exc) Connection close: Read 
failed. Possible end of stream encountered.  Entry'

...'WSRRMbLogHandler:FINEST' , 'Destroying the connection: remote host 
name: wsrrhost.corp.xyz.com remote host address: x.x.x.x local host 
name: brokerhost local host address: x.x.x.x'

...'WSRRMbLogHandler:FINER' , 'close Entry'

...'WSRRMbLogHandler:FINEST' , 'SocketChannel close starting, local: 
brokerhost/x.x.x.x:38775 remote: wsrrhost.corp.xyz.com/x.x.x.x:9080'

...'WSRRMbLogHandler:FINE' , 'SocketChannel close, local: 
brokerhost/x.x.x.x:38775 remote: wsrrhost.corp.xyz.com/x.x.x.x:9080'

From looking through the service trace, it looks like an IOException (Connection close: Read failed) is being raised. The thin client successfully makes the connection and reads some data, but then for some reason the connection gets closed.

Tip: To filter on just the WSRR thin client trace, all the broker log entries from WSRR will contain WSRRMbLogHandler. Filtering this way enables the skipping of all classloading entries.


Downloadable resources


Related topics


Comments

Sign in or register to add and subscribe to comments.

static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Middleware
ArticleID=970933
ArticleTitle=Integrating IBM Integration Bus with WebSphere Service Registry and Repository: Part 7: Configuring the WebSphere Service Registry and Repository cache in IBM Integration Bus
publish-date=04302014