Host blocking

A blocked host is no longer available as a resource for that application or service, so that the system does not continuously try to start services on a host that is lacking vital software or hardware requirements. The host blocking feature prevents IBM® Spectrum Symphony from repeatedly trying to run a service on a host that does not have adequate hardware or software resources.

You can configure host blocking to take effect on timeout or exit for each of your services, or when a service throws an exception or sends a specific return code.

About host blocking

When host blocking takes effect, IBM Spectrum Symphony creates a blocked host list for the application with which the service is associated. A host that appears on the blocked host list can no longer be used by the application until you intentionally unblock the host, or the application is re-registered or disabled and enabled again.

By default, host blocking is enabled for a version mismatch or communication timeout between the session manager and the service instance manager. You can also configure host blocking for a service instance error, a service instance exit, or a service instance method timeout. By default, host blocking is enabled for the following service instance methods:

Method	Event types
Register	Timeout Exit
CreateService	Timeout Exit Failure exception Fatal exception
SessionEnter	Timeout Exit
SessionUpdate	Timeout Exit

The following illustrations show the benefits of using the host blocking feature.

Without host blocking (feature disabled)

Default view of host blocking (feature is disabled)

With host blocking enabled

Host blocking triggers

Host blocking triggers automatically when the session manager version on the management host does not match the service instance manager version on the compute host.

You can configure additional host blocking based on the requirements of your application so that IBM Spectrum Symphony triggers host blocking for any of the following reasons:

A service method times out, exits or crashes, throws an exception, or returns certain control codes.
The service instance manager does not communicate with the session manager before the configured timeout period expires (controlled by the startUpTimeout value).
The service instance does not communicate with the service instance manager before the configured timeout period expires (controlled by the setting for the Register method actionOnSI attribute).

Slot blocking for IBM Spectrum Symphony Developer Edition

IBM Spectrum Symphony Developer Edition blocks slots (not hosts) under the same conditions that trigger host blocking for a production grid. Symptoms of blocked slots include fewer resources than expected or no resources serving your application, more tasks in the PENDING state, a slower rate of workload completion, and clients that hang. You can check for blocked slots by looking in the ssm.hostname.app_name.log file and searching for WARN or ERROR messages about blocked hosts. If you see a blocked host message, one or more slots might be blocked. You can unblock slots by disabling and then enabling the application or by restarting the Developer Edition cluster.

Slot blocking for IBM Spectrum Symphony Developer Edition is supported on all host types supported by the product. Note that for IBM Spectrum Symphony Developer Edition, only slots (not hosts) are blocked.

Configuration to enable host blocking

Host blocking is enabled in the application profile for each application. You can configure host blocking at the service instance manager level, the service instance level, or both.

Table 1. Application profile configuration to enable host blocking
Section	Attribute name and syntax	Behavior
SOAM > SIM	blockHostOnTimeout="true"	Enables host blocking for the application when the service instance manager times out while trying to communicate with the session manager. Used with the startUpTimeout attribute.
SOAM > SIM	startUpTimeout="`seconds`"	Number of seconds to wait for the service instance manager to communicate with the session manager. This attribute works in conjunction with blockHostOnTimeout. When the process times out, the session manager requests a new host from EGO and tries to start a new service instance manager on the new host.
Service > Control > Method > Timeout	actionOnSI=blockHost	When a timeout is reached on the method, terminates the running service instance on this host and does not use this host to start any other service instance for the application. Used with the duration attribute. You can specify the blockHost option for the following methods: Register Deployment CreateService SessionEnter SessionUpdate Invoke SessionLeave
Service > Control > Method > Exit	actionOnSI=blockHost	When the service instance exits or crashes during execution of the method, the system does not use this host to start any other service instance for the application You can specify the blockHost option for the following methods: Register Deployment CreateService SessionEnter SessionUpdate Invoke SessionLeave
Service > Control > Method > Return	actionOnSI=blockHost	When the method returns normally or with a specified control code, terminates the running service instance on this host and does not use this host to start any other service instance for the application. You can specify the blockHost option for the following methods: CreateService SessionEnter SessionUpdate Invoke SessionLeave
Service > Control > Method > Exception	actionOnSI=blockHost	When the specified exception (failure or fatal exception) occurs, terminates the running service instance on this host and does not use this host to start any other service instance for the application. You can specify the blockHost option for the following methods: CreateService SessionEnter SessionUpdate Invoke SessionLeave

Host blocking behavior

When host blocking is triggered, the system creates a blocked host list for the application. The following example illustrates the host blocking process triggered at the service instance level.

Example of the host blocking process

Configuration to modify host blocking behavior

Not applicable. There are no attributes that change the way that host blocking works other than those attributes configured in the application profile.

Host blocking actions

Actions to monitor

You can monitor host blocking through the cluster management console, the command line, and through the IBM Spectrum Symphony log files located in the logs directory of SOAM_HOME. You can also trap SNMP events to receive notifications when a service triggers the system to block a host.

Here are the actions that each user can monitor:

User	Action	Description
Cluster administrator	From the cluster management console:Symphony Workload > `application_name` > Hosts Blocked for Application	Displays a list of blocked hosts for the selected application.
Cluster administrator Consumer administrator	From the command line: egosh alloc view	Displays detailed information about all allocations, including the allocation ID, current users, consumer, resource groups, resource requirements, minimum and maximum slots requested, whether it has exclusive use of the host, names of the allocated hosts, and any blocked hosts.

You can find information about host blocking in the session manager log file ssm.host_name.app_name.log, which can be found at:

Linux®: $SOAM_HOME/logs
Windows: %SOAM_HOME%\logs.

Look for the SOA_SERVICE_BLOCKED event, an error level message that indicates that host blocking has occurred.

Actions to control

Typically, a cluster administrator removes a blocked host when the host has been modified (by means of a software or hardware upgrade, for example), to meet the requirements of the service. A host can be removed from the blocked host lists in one of two ways:

Directly from the cluster management console
Indirectly, by disabling and re-enabling the application associated with the blocked host

Here are the actions that each user can control:

Table 2. IBM Spectrum Symphony supported control actions for different types of users
User	Action	Behavior
Cluster administrator	From the cluster management console:Workload > Symphony > Applications, then select `application_name` > Hosts Blocked for Application, then select the host name, and click Unblock.	The system removes the host from the blocked host list. The application can start a service on the previously blocked host
Cluster administrator	From the cluster management console:Workload > Symphony > Applications, then select `application_name`, and click Disable.	Disables the application, which clears the blocked host list for the disabled application No clients can be served by the disabled application
Cluster administrator Consumer administrator Consumer user	From the command line: soamcontrol app disable `application_name`	Disables the application, which clears the blocked host list for the disabled application No clients can be served by the disabled application For information about how to use the soamcontrol command to disable and enable applications, see soamcontrol.
Cluster administrator Consumer administrator	From the cluster management console: Workload > Symphony > Application Profiles > `application_name` > Basic Configuration > Save Workload > Symphony > Application Profiles > `application_name` > Advanced Configuration > Save	The system first disables and then re-registers the application, which clears the blocked host list for the modified application.

For IBM Spectrum Symphony Developer Edition, you can unblock slots by disabling and then enabling the application, or by restarting the Developer Edition cluster.

Table 3. IBM Spectrum Symphony Developer Edition supported control actions for different types of users
User	Action	Behavior
Developer	From the command line: soamcontrol app disable `application_name`	Disables the application, which unblocks slots for the disabled application No clients can be served by the disabled application
Developer	From the command line: soamcontrol app enable `application_name`	Enables the application, which can start services on any previously blocked slot
Developer	Windows: Right-click on the IBM Spectrum Symphony Developer Edition tray icon and choose Stop Symphony DE on all hosts. Once the IBM Spectrum Symphony Developer Edition cluster shuts down, right-click on the IBM Spectrum Symphony Developer Edition tray icon and choose Start Symphony DE on all hosts. Linux: soamshutdown soamstartup	Shuts down and then restarts IBM Spectrum Symphony Developer Edition Unblocks slots for all applications running on the IBM Spectrum Symphony Developer Edition cluster

Actions to display configuration

User	Command	Behavior
Cluster administrator Consumer administrator	From the cluster management console: Workload > Symphony > Application Profiles > `application_name` > Basic Configuration Workload > Symphony > Application Profiles > `application_name` > Advanced Configuration	Displays application profile settings for the selected application.
Cluster administrator Consumer administrator Consumer user	From the command line: soamview app `app_name` -p	Displays application profile settings for the selected application.

You can also view an application profile using an XML editor.