Performance tuning and optimization

Using standby services to reduce service start times
Standby services minimize the need to restart services at the time resources are allocated to an application by allowing these services to run idle when there is no workload. You can configure a default standby service, and additionally, non-default (most currently running) standby services. Standby services is at the consumer (not cluster) level, and is supported on slot-based scheduling and multidimensional scheduling.
Using global (cluster) standby services to share service instances
The global (or cluster scope) standby services feature (on-demand shutdown of idle SIs) minimizes the need to restart services when resources are allocated to an application by allowing these services to run idle when there is no workload and reusing them when workload arrives, either within an application or across multiple applications. Global standby services is supported on slot-based scheduling and multidimensional scheduling using IBM® Spectrum Symphony.
Viewing standby service instances
You can view information related to standby instances using the egosh command for both slot-based scheduling and multidimensional scheduling standby services.
Terminating standby service instances
To manage workload more efficiently, you may at times want to terminate slot-based scheduling or multidimensional scheduling standby services. You can terminate all instances in the cluster or specify a list of standby instances to be terminated.
Limiting the number of service instances that can run on a host
Use this feature to limit the number of service instances that an application can run on a host; for example, you have an application that consumes a lot of memory and you want to limit the amount of memory that the application consumes on a host.
Imposing limits on the number of services per session
Limit the number of services for a session by configuring maximum and minimum limits for the number of services per session. This feature is supported on all operating systems supported by IBM Spectrum Symphony.
Resource-aware scheduling for sessions
Resource-aware scheduling for sessions provides a means for the application developer to state a resource preference for host attributes that provide the best match for the workload in a session. You can use it with IBM Spectrum Symphony. It is not supported on IBM Spectrum Symphony Developer Edition.
Optimizing slot allocation through incremental allocation
To improve the efficiency of slot allocation, you can configure the session manager (SSM) to slow down application slot allocations during periods of high demand. By using incremental slot allocation, you can configure the SSM to gradually increase the number of requested slots for an application, rather than request the maximum number of slots based on the resource plan and demand. This features is supported on all IBM Spectrum Symphony supported operating systems.
Delaying the release of slots after running workload
You can configure the amount of time that an unassigned service will remain idle with the application before releasing its corresponding slot to EGO (and terminating the service). An unassigned service is defined as a service not assigned to any session. If new workload arrives before the configured time elapses, these unassigned services can be used to handle the new workload. Once an unassigned service becomes assigned to a session again, its idle timer is reset.
Workload redirection across multiple applications
The workload redirection feature enhances scalability by enabling an IBM Spectrum Symphony client to submit workload to logical sessions of logical applications, which in turn are mapped to physical applications. Thus, workload is redirected from the logical application to several physical applications, enabling each logical application to use multiple SSMs. With workload redirection, you can also choose the type of redirection for each client: either at the session level or at the task level.
Service instance resource usage limits
You can set limits on resource usage (for memory or CPU usage) when IBM Spectrum Symphony handles service instances (SI) running on compute hosts.
Host-level memory usage limits
To prevent applications from encountering out-of-memory issues on compute hosts, set a memory usage limit for all processes created by PEM (for Windows) and the sub-PEM (for Linux®), and descendant processes on each compute host. Use the memory usage limit to ensure that the total memory (virtual memory for Windows, and physical memory for Linux) consumed by these processes during workload execution does not exceed the total memory of the host.
Dynamic service
The dynamic service is a feature that reduces capital and operating expenses by improving utilization of resources in mixed environments. It can also reduce input and output, and memory contention in multi-core environments. The dynamic service feature allows dynamic control of service-to-slot mapping to optimize core, memory, and input and output use. This enables efficient running of workload made up of multi-threaded, single-threaded, input and output-intensive, and compute-intensive tasks from a single application or a mix of applications in a multi-core and heterogeneous environment. Dynamic service is available on all supported operating systems.
Data-aware scheduling
The data-aware scheduling (data affinity) framework and plug-in allows IBM Spectrum Symphony to intelligently schedule application tasks and improve performance by taking into account data location when dispatching the tasks. By directing tasks to resources that already contain the required data, application runtimes can be significantly reduced. As well, this feature can help to meet the challenges of latency requirements for real time applications.
Changing the session scheduling interval
The frequency at which the scheduler reassesses resource assignments between sessions is called the scheduling interval.
How resources are scheduled by the session manager
IBM Spectrum Symphony supports several session scheduling policies.
Prioritizing sessions for time-critical workload
The session manager provides a priority scheduling policy where all resources are assigned to the highest priority session before lesser priority sessions. If the highest priority session cannot make use of all of the resources, any unused resources are assigned to the next highest priority session. This cycle repeats to the next highest priority session, and so on. If there are still some slots remaining after all the sessions are satisfied, these slots are not assigned to any session.
Filtering resource groups for sessions
When resource groups are specified at the application level, it means that any sessions serving the application will be allocated resources from the specified resource groups. Sometimes sessions for an application may want to use different resource groups. For example, with a parent/child workload pattern in one application, the parent sessions may only want to use the resource group for which it has 100% ownership, while child session can share resources with other applications. The resource group filter can serve this purpose. Each session can use a resource group filter to narrow down the choices of resource groups specified at the application level.
Specifying criteria for resource selection
When you put your application on the grid in IBM Spectrum Symphony, you specify a resource requirement string in the application profile to set criteria for selecting a resource or restricting which resources are available to the application.
Controlling when applications request or release resources through high- and low-water marks
Sharing scheduling policy using tag-based resource sharing for applications
IBM Spectrum Symphony provides a share policy that enables you to distribute an application's resources among tags in a hierarchy based on the tag's relative position and predefined share ratios in the hierarchy. Use this share policy when you want to limit the number of resources that each user can use. This topic is applicable to the R_TagHierarchy share policy.
Monitoring application health through SNMP alerts
Monitor the health of your cluster at the application level by setting up workload thresholds. When IBM Spectrum Symphony determines that an SSM is not working efficiently based on those settings, it generates an SNMP alert notifying the cluster administrator to check the status of the cluster.
Performance tuning for large clusters
There are a number of system and IBM Spectrum Symphony parameters available that can be changed from their default settings to provide better performance for large, high-utilization clusters. These parameters are defined in configuration files and the Windows registry. As to what constitutes a large cluster, this is somewhat arbitrary. For the purposes of this discussion, we will consider a large cluster to have over 20,000 cores and over 100 applications. Since qualification of cluster size is not an exact science, relatively smaller clusters may also benefit to some degree from adjusting the parameters described here.
Optimizing session director (SD) performance
Optimize performance for the IBM Spectrum Symphony session director by tuning settings within the application profile, session director environment variables, and applicable configuration files.
Optimizing session manager (SSM) performance
You can optimize the IBM Spectrum Symphony session manager (SSM) using various techniques.
Stacking sessions to improve cluster performance
The fragmentation of a session's tasks across hosts can impact cluster performance. For example, when common data is used with the common data optimization feature, the more hosts that are assigned to a session, the more common data that needs to be transferred on the network. You can configure resource preference to enable the SSM to minimize the number of hosts that are selected to run workload for a session. When this feature is enabled, the SSM tries to stack slots allocated to a session on one host. This features is supported on all supported operating systems. Note that MapReduce framework does not support this feature.
Optimizing memory consumption with tcmalloc or jemalloc
When most of your memory is allocated in a large number of noncontiguous blocks (for example, small chunks of common data), memory fragmentation can cause resource starvation. To avoid resource starvation and optimize memory consumption and transactional throughput, try applying a different memory allocation library in your cluster. IBM Spectrum Symphony is certified with tcmalloc (gperftools 2.2) and jemalloc (3.0.0).
Releasing inactive session managers
An IBM Spectrum Symphony Session Manager (SSM) can be configured to shut itself down if it is inactive for a specified amount of time.
Detecting unresponsive service instance managers
Configure a heartbeat interval for your applications, enabling the session manager (SSM) to check for any disconnected or unresponsive SIMs and release the SIM's resources after the response timeout expires. Use this configuration to prevent phantom allocations when unavailable SIMs consume cluster resources.
Preventing a SYN flood denial of service attack response for Windows
If an IBM Spectrum Symphony server daemon (such as SD, SSM, or RS) does not accept remote connection requests in an efficient enough manner, a large number of pending connections can accumulate, and cause Windows Server 2012 64-bit to trigger a SYN flood denial of service attack response. This blocks incoming connections to the host. To avoid this, increase the server daemon connections accepted together. You can also increase the number of communications threads that are performing input and output operations.
Tuning performance for deployment
Repository server scalability allows you to resolve performance issues in the service deployment.