WS-Notification troubleshooting tips

A JAX-WS application that is a raw consumer of brokered notifications must recognize a notification broker SOAP action

The PublisherRegistrationManager.wsdl file is not successfully parsed by wsimport unless you include a JAX-WS bindings file

[ERROR] the following naming conflicts occurred: 
com.ibm.websphere.wsn.publisher_registration_manager.ResourceNotDestroyedFault
line 2 of file:/path_to_wsdl/PublisherRegistrationManager.wsdl

This fault occurs because the WSDL for the publisher registration manager uses both the WS-Notification and the WS-ResourceLifetime specifications; both of which refer to a ResourceNotDestroyedFault element that shares the same message name. Here is the relevant part of the publisher registration manager WSDL file:

  <wsdl:operation name=DestroyRegistration>
    <wsdl:input name=DestroyRegistrationRequest message=wsn-brw:DestroyRegistrationRequest 
      wsam:Action=http://docs.oasis-open.org/wsn/brw-2/PublisherRegistrationManager/DestroyRegistrationRequest 
      wsaw:Action=http://docs.oasis-open.org/wsn/brw-2/PublisherRegistrationManager/DestroyRegistrationRequest/>
    <wsdl:output name=DestroyRegistrationResponse message=wsn-brw:DestroyRegistrationResponse 
      wsam:Action=http://docs.oasis-open.org/wsn/brw-2/PublisherRegistrationManager/DestroyRegistrationResponse 
      wsaw:Action=http://docs.oasis-open.org/wsn/brw-2/PublisherRegistrationManager/DestroyRegistrationResponse/>
    <wsdl:fault name=ResourceUnknownFault message=wsrf-rw:ResourceUnknownFault 
      wsam:Action=http://docs.oasis-open.org/wsrf/fault 
      wsaw:Action=http://docs.oasis-open.org/wsrf/fault/>
    <wsdl:fault name=ResourceNotDestroyedFault message=wsn-brw:ResourceNotDestroyedFault 
      wsam:Action=http://docs.oasis-open.org/wsn/fault wsaw:Action=http://docs.oasis-open.org/wsn/fault/>
  </wsdl:operation>

<!-- Some parts have been omitted -->

<!-- An extract from WS-ResourceLifetime -->
  <wsdl:operation name=Destroy>
    <wsdl:input name=DestroyRequest message=wsrf-rlw:DestroyRequest 
      wsam:Action=http://docs.oasis-open.org/wsrf/rlw-2/ImmediateResourceTermination/DestroyRequest 
      wsaw:Action=http://docs.oasis-open.org/wsrf/rlw-2/ImmediateResourceTermination/DestroyRequest/>
    <wsdl:output name=DestroyResponse message=wsrf-rlw:DestroyResponse 
      wsam:Action=http://docs.oasis-open.org/wsrf/rlw-2/ImmediateResourceTermination/DestroyResponse 
      wsaw:Action=http://docs.oasis-open.org/wsrf/rlw-2/ImmediateResourceTermination/DestroyResponse/>
    <wsdl:fault name=ResourceNotDestroyedFault message=wsrf-rlw:ResourceNotDestroyedFault 
      wsam:Action=http://docs.oasis-open.org/wsrf/fault 
      wsaw:Action=http://docs.oasis-open.org/wsrf/fault/>
    <wsdl:fault name=ResourceUnknownFault message=wsrf-rw:ResourceUnknownFault 
      wsam:Action=http://docs.oasis-open.org/wsrf/fault 
      wsaw:Action=http://docs.oasis-open.org/wsrf/fault/>
    <wsdl:fault name=ResourceUnavailableFault message=wsrf-rw:ResourceUnavailableFault 
      wsam:Action=http://docs.oasis-open.org/wsrf/fault 
      wsaw:Action=http://docs.oasis-open.org/wsrf/fault/>
  </wsdl:operation>

To resolve this naming conflict, you must include the JAX-WS bindings file ibm-wsn-jaxws.xml as an argument to the wsimport command. This bindings file ensures that the conflicting elements are mapped to different class names.

c:\was\bin\wsimport -b ibm-wsn-jaxws.xml -keep PublisherRegistrationManager.wsdl

WS-Notification service receives a triggerActionNotSupportedFault

You have a JAX-WS demand-based publisher registered with a Version 7.0 type of WS-Notification service. When the service invokes any of the operations on the PausableSubscriptionManager interface of the publisher, the publisher responds with a triggerActionNotSupportedFault exception message. The operations that trigger this message are Renew, Unsubscribe, PauseSubscription or ResumeSubscription.

You see messages similar to the following text in the SystemOut.log file for the server. In this example the fault message is triggered in response to the broker attempting to unsubscribe from the demand-based publisher.

triggerActionNotSupportedFault triggerActionNotSupportedFault: messageContext: 
[MessageContext: logID=urn:uuid:13616A3EB4F278A3DC1221827497002] problemAction: 
http://docs.oasis-open.org/wsn/bw-2/SubscriptionManager/UnsubscribeRequest

CWSJN1029W: An attempt was made to perform an operation on a remote NotificationProducer
 located with endpoint Address[[xxx/prod_service/NotificationProducerPort]], 
ReferenceParams[] but the operation could not be completed. This operation will be 
retried automatically in 2 seconds. 
The operation failed due to com.ibm.ws.sib.wsn.webservices.WSNWSException: 

CWSJN5035E: An internal error occurred: Unable to unsubscribe from remote publisher 
at endpoint reference Address[[xxx/prod_service/PausableSubscriptionManagerPort]], 
ReferenceParams[] due to javax.xml.ws.soap.SOAPFaultException: 
The [action] cannot be processed at the receiver.

The server continues to unsuccessfully attempt to unsubscribe from the publisher at ever-increasing time intervals.

To resolve this problem, in the WSDL file for your demand-based publisher you must explicitly specify the SOAP action to use for each of the operations on the PausableSubscriptionManager interface. The action URI to use for each operation is defined in the Web Services Base Notification 1.3 (WS-BaseNotification) specification available from https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=wsn For example, the WS-Addressing action for the Unsubscribe operation is defined in the specification as http://docs.oasis-open.org/wsn/bw-2/SubscriptionManager/UnsubscribeRequest, therefore you must use this action for the Unsubscribe operation in your publisher WSDL file. Here is an excerpt from the binding section of such a WSDL file:

<wsdl:binding name=PausableSubscriptionManagerBinding type=wsn-bw:PausableSubscriptionManager>
<soap:binding style=document transport=http://schemas.xmlsoap.org/soap/http />

<wsdl:operation name=Unsubscribe>
<soap:operation soapAction=http://docs.oasis-open.org/wsn/bw-2/SubscriptionManager/UnsubscribeRequest />

Similarly you must specify the corresponding actions for the Renew, PauseSubscription and ResumeSubscription operations in your publisher WSDL file.

There are Connection timed out errors in your broker server logs

If you see WebServicesFault( "Connection timed out" ) errors in your WS-Notification broker server logs, and you have a large number of consumers subscribed to your WS-Notification service, the broker might have exceeded the maximum number of connections in the HTTP outbound connector connection pool when sending outbound notification messages to subscribed consumers.

To increase the number of outbound HTTP connections in the pool, set the com.ibm.websphere.webservices.http.maxConnection custom property on the server or servers on which your broker is running. For more information about this property, see HTTP transport custom properties for web services applications.

There are Out of memory errors in your broker server logs

If you see Out of memory errors in your WS-Notification broker server logs, and you have a large number of consumers subscribed to your WS-Notification service, the broker might have exceeded the available JVM heap size allocated to the server on which it is running.

To increase the maximum heap size for the server or servers on which your broker is running, see Tuning the IBM virtual machine for Java.

WebSphere Application Server Version 6.1 client applications must handle the additional error conditions

The WSDL and schema files shipped with WebSphere Application Server Version 7.0 or later are updated to reflect the final Version 1.3 WS-Notification standards.

You need not change your existing WS-Notification services. Your existing client applications will also continue to work unchanged, but if they are now also working with new WS-Notification services, and you want them to explicitly handle the new fault conditions, then regenerate the client stubs by using the WSDL file from the new service.

Exception occurs because the SDO repository is not configured correctly

If you try to create a Version 6.1 WS-Notification service, and you get the following stack trace, then the SDO repository is not configured correctly. To resolve this problem, see Installing and configuring the SDO repository.

java.lang.Exception: com.ibm.ws.sib.webservices.admin.config.SIBConfigException: CWSWS5010E: 
Failed to store WSDL located at https://www.ibm.com/websphere/wsn/notification-broker 
due to the following exception: com.ibm.ws.sib.webservices.exception.SIBWSUnloggedException: 
CWSWS1007E: The following exception occurred: 
com.ibm.ws.sdo.config.repository.impl.RepositoryRuntimeException: 
javax.transaction.TransactionRolledbackException: CORBA TRANSACTION_ROLLEDBACK 0x0 No; 
nested exception is: org.omg.CORBA.TRANSACTION_ROLLEDBACK: 
javax.transaction.TransactionRolledbackException: ; nested exception is: 
javax.ejb.TransactionRolledbackLocalException: ; nested exception is: 
com.ibm.ws.ejbpersistence.utilpm.PersistenceManagerException: 
PMGR1014E: Exception occurred when getting connection factory: 
com.ibm.websphere.naming.CannotInstantiateObjectException: 
threw NameNotFoundException while the JNDI NamingManager was processing a 
javax.naming.Reference object. [Root exception is javax.naming.NameNotFoundException: 
Context: smeagolNode03Cell/nodes/smeagolNode03/servers/server1, name: 
jdbc/com.ibm.ws.sdo.config/SdoRepository: 
First component in name com.ibm.ws.sdo.config/SdoRepository not found. 
[Root exception is org.omg.CosNaming.NamingContextPackage.NotFound: 
IDL:omg.org/CosNaming/NamingContext/NotFound:1.0]] vmcid: 0x0 minor code: 0 completed: No.

Multiple messages are received by a notification consumer for each event notification

In some situations you might receive more notifications at a given notification consumer than the number of event notifications that have been inserted into the notification broker by a publisher. For example you might publish 4 messages, and receive 8, 12, 16 (or some other multiple of four) messages at the notification consumer.

This is usually caused by there being two or more active subscriptions that target the notification consumer - a situation that can occur if the subscriber application is run more than once. Each time the Subscribe operation is called, a new subscription must be created by the notification broker (see section 4.2 of the Web Services Base Notification specification), which causes duplicate messages to be delivered if a previous subscription exists.

To check whether this is what is happening, examine the SubscriptionReference property of the notifications received by the notification consumer. This endpoint reference contains the identifier of the subscription that caused the notification to be sent. If you find several different subscription identifiers, then there is more than one subscription active.

Subscriber applications should tidy up subscriptions when they are not required (or register them with a timeout), however you can tidy up subscriptions administratively using the runtime panels.

Problems can occur when deleting administered subscribers and messaging engines

Deleting administered subscribers

You should be wary of deleting and recreating messaging engines on bus members for which WS-Notification-administered subscribers have been configured, because in some cases this can leave the remote web service subscription active (and passing notification messages to the local server) even though there is no longer any record of it.

To avoid this situation you should delete the WS-Notification configuration, or just the administered subscribers, in a separate step to deleting the messaging engine. When the dynamic configuration update is then processed, or the server restarted, the remote web service subscription is tidied up cleanly.

Note: This problem does not occur if it is only the WS-Notification configuration that is modified; it only occurs if the messaging engine is also deleted.

Administered subscribers in a cluster

When you remove messaging engines from a cluster, remove them in numerical order from highest to lowest so as to avoid a situation where, for example, there are messaging engines numbered 001 and 002 and not 000. This is to avoid problems if you use WS-Notification, which attaches special significance to the first-created messaging engine in a cluster.

In a clustered topology there can be more than one messaging engine running on the bus member (cluster). Administered subscribers are defined against a service point (bus member) and so there are several alternatives when choosing the messaging engine that is responsible for creating the subscription to the remote web service. In this situation, the first messaging engine in the cluster is responsible for making the subscription. For example in a cluster containing three messaging engines the messaging engines will have names following the pattern xxx-000-yyy, xxx-001-yyy, xxx-002-yyy, and the administered subscriber subscriptions will be managed by the 000 messaging engine.

If you delete the 000 messaging engine from the cluster then restart the servers, the administered subscriptions are now managed by the 001 messaging engine - being the lowest number engine in the cluster. However, as previously mentioned, deleting and recreating messaging engines on bus members for which administered subscribers have been configured can leave the remote web service subscription active (and passing notification messages to the local server) even though there is no longer any record of it. Therefore if another messaging engine is later added to the cluster and there is no xxx-000-yyy messaging engine currently defined the new engine takes on the name of xxx-000-yyy. Therefore, in this instance it is possible for two messaging engines to concurrently believe that they manage the administered subscription, resulting in multiple subscriptions being made to the remote web service.

In the unlikely event that you have to re-create messaging engine xxx-000-yyy, you can avoid duplicate messages from an administered subscription by completing the following steps:

• Delete the administered subscriptions defined to this cluster.
• Allow the existing messaging engines to observe the change. This results in messaging engines in the cluster deleting the administered subscriptions they believe they manage.
• Create the new messaging engine in the cluster.
• Re-create the administered subscriptions.

There are differences when using bus destinations with Version 6.1 WS-Notification services

Usually, a bus destination is used as described in Bus destinations. However, this is not the case for Version 6.1 WS-Notification services. The destination that is associated with a Version 6.1 WS-Notification service does not relate to the topics for which the WS-Notification service can handle requests, and you should not alter or mediate the destination. In WS-Notification, the configuring of topics is handled through topic namespaces. For more information, see Creating a new WS-Notification permanent topic namespace.

An inbound (application to broker) notification is not successful

Applications that want to publish event notifications into the broker make use of the Notify operation. This is defined as a one-way (web services) operation which means that it is not possible to return a fault (exception) if it is not possible to complete the operation. Thus the application will assume that the notification was successful, but subscribing applications will not receive the notification message.

You should monitor this type of exception closely, because it might indicate a denial of service attack and certainly indicates that the application is not functioning correctly. The first time an inbound notification fails from a particular producing application, a warning message is sent to the SystemOut log of the server. If there are further notification failures for that producer, subsequent timed warning messages are logged at 30 minute intervals. Additional information is provided with each timed message to indicate how many failed notifications were received for that producer during the 30 minute interval.

Outbound (broker to application) notification is not successful

An outbound web service invocation (broker to application) does not succeed when a remote application is unavailable for invocation. This might be the result of an application failure, a network error, or a firewall configuration issue. When event notifications are not passed to subscribed applications, messages build up on the subscriptions held on the server. The messages held on a given subscription can be observed by using the runtime panels. Subscriptions for which the most recent event notification attempt has failed in this way are marked as being in ERROR state when viewed in the WS-Notification subscription runtime administration panel.

If the WS-Notification service point does not successfully notify a NotificationConsumer application, a warning message is sent to the application server SystemOut log and the subscription is told to wait for 2 minutes. Reasons for a failure of this type might be that the remote web service is not currently available, or that network conditions prevent contact between the local server and the service.

After 2 minutes, the notification is retried. If delivery is still not possible then the subscription is put back into a wait state for another 2 minutes. If the failure is caused by a transient I/O error, this pattern is repeated indefinitely, until the notification is either successfully delivered or you delete the subscription. If the error is caused by an application failure on the remote side then the notification will be retried up to the number of times defined in the Maximum failed deliveries setting of the service integration bus topic space destination from which the message is being received. After the first warning message is output to the SystemOut log, subsequent timed warning messages are logged at 30 minute intervals.

Tidying up stateful resources that are not automatically deleted

The act of subscribing to the broker or registering a publisher creates a stateful resource on the server that consumes system resources while it is active. Usually an application specifies a termination time as part of the act of creating these resources, and thus they are automatically deleted when the termination time is reached. However it is also possible for the application to request an infinite lifetime for the resource. If this is done then it is possible for resources to remain on the server indefinitely even though the application might never be coming back to use (or destroy) the resource.

You can view the stateful resources (subscriptions and publisher registrations) by using the runtime panels. These panels also provide the ability to administratively delete the items if required. Only do this if you are sure that the application is no longer using the resource because it will cause application failures if the resource is referenced after being deleted.

Durable subscription that was created by WS-Notification cannot be deleted when using the service integration bus panel

When you create a subscription by using a WS-Notification application, that is by using the Subscribe operation, one or more durable subscriptions are created in the relevant service integration bus topic space destination. You can view these durable subscriptions in the service integration bus runtime panels for the publication point.

To delete a subscription that was created by a WS-Notification application, use the runtime panels provided by the WS-Notification implementation. This approach closes the active consumer and automatically deletes the related service integration bus durable subscriptions.

Administrative stop of a messaging engine

WebSphere Application Server depends on being able to access a running service integration bus messaging engine to send and receive messages, and to create and retrieve state for the various web service resources that are created.

You can stop a messaging engine by using the MBean interface or runtime panels. This prevents WS-Notification from successfully servicing any requests from applications that might come in during the time that the messaging engine is stopped. In this situation, error messages are logged as described in An inbound (application to broker) notification is not successful and Outbound (broker to application) notification is not successful. When you stop a messaging engine, all WS-Notification processing stops and all messaging applications cease to function. When you restart the messaging engine, WS-Notification processing resumes.

Failures as a result of changes in topic space and topic namespace configurations

The WS-Notification configuration artifacts often depend on objects defined in other areas of the server configuration. For example (for Version 6.1 WS-Notification services) the endpoint listeners through which application requests are received, and the service integration bus topic spaces to and from which messages are sent.

The following items describe the action that is taken by the WS-Notification runtime code when it meets relevant changes in the objects upon which it depends.

Deleting a service integration bus topic space

The service integration bus topic space is the primary messaging object upon which WS-Notification depends at run time. Notification messages from an application are published to the topic space specified by the (permanent) topic namespace mapping specified by the administrator.

Deleting a service integration bus topic space has the following effects upon new and existing WS-Notification applications:

• RegisterPublisher requests that use a WS-Notification topic namespace that references the deleted topic space receive a TopicNotSupportedFault error message.
• Notify requests for a topic associated with the deleted topic space do not publish the message to the topic space (because it has been deleted). The application is not informed because no faults are thrown by the Notify operation (see An inbound (application to broker) notification is not successful).
• Subscribe requests that use a WS-Notification topic namespace that references the deleted topic space receive a SubscribeCreateFailedFault error message.
• No further messages are delivered to applications that have existing subscriptions to the deleted topic space. The existing subscription is deleted, and any attempt to invoke operations on the subscription (for example getCreationTime) results in a ResourceUnknownFault error message.
• Deleting and recreating a service integration bus topic space is considered as two separate steps. Existing subscriptions are deleted in response to the first step, and therefore do not exist when the topic space is recreated.

Deleting a permanent topic namespace mapping

Deleting the topic namespace mapping that was used to establish a (currently active) subscription has the same effect as deleting the underlying service integration bus topic space as defined previously, and subscriptions that were created using this namespace mapping are deleted.

Publisher registrations and pull points associated with the deleted topic namespace mapping are also deleted.

Changing a permanent topic namespace mapping

The fields of a permanent topic namespace mapping are read-only fields, so the only way to change the fields is to delete the namespace mapping and recreate it with new values. The effect of deleting a permanent topic namespace mapping is described in the previous item.

Unable to create a Version 6.1 WS-Notification service without a configured SDO repository

java.lang.Exception: com.ibm.ws.sib.webservices.admin.config.SIBConfigException: CWSWS5010E: Failed to 
store WSDL located at https://www.ibm.com/websphere/wsn/notification-broker due to the following 
exception:
    com.ibm.ws.sib.webservices.exception.SIBWSUnloggedException: CWSWS1007E: The following exception 
        occurred: com.ibm.ws.sdo.config.repository.impl.RepositoryRuntimeException: 
        javax.transaction.TransactionRolledbackException: CORBA TRANSACTION_ROLLEDBACK 0x0 No; nested 
        exception is: org.omg.CORBA.TRANSACTION_ROLLEDBACK: 
        javax.transaction.TransactionRolledbackException: ;
    nested exception is: javax.ejb.TransactionRolledbackLocalException: ;
    nested exception is: com.ibm.ws.ejbpersistence.utilpm.PersistenceManagerException: PMGR1014E: 
        Exception occurred when getting connection factory: 
        com.ibm.websphere.naming.CannotInstantiateObjectException: threw NameNotFoundException while 
        the JNDI NamingManager was processing a javax.naming.Reference object.
    [Root exception is javax.naming.NameNotFoundException: Context: 
        KADGINNode01Cell/nodes/KADGINNode01/servers/server1, name: 
        jdbc/com.ibm.ws.sdo.config/SdoRepository: First component in name 
        com.ibm.ws.sdo.config/SdoRepository not found.
    [Root exception is org.omg.CosNaming.NamingContextPackage.NotFound: 
        IDL:omg.org/CosNaming/NamingContext/NotFound:1.0]] vmcid: 0x0 minor code: 0 completed: No.

Exception occurs when a JAX-WS client that does not have internet access attempts to contact a WS-Notification service

WSDLException (at /definitions/import[1]): faultCode=OTHER_ERROR: 
Unable to resolve imported document at 'http://docs.oasis-open.org/wsn/brw-2.wsdl', 
relative to 'http://localhost:9082/WSNService1WSNServicePt1NB/Service/WEB-INF/wsdl
/NotificationBroker.wsdl': java.net.UnknownHostException: docs.oasis-open.org

Configure the client to use a local copy of the WSDL file instead.