Table of Contents (exploded view)
Abstract for MVS Setting Up a Sysplex
Summary of changes
Summary of changes for z/OS Version 2 Release 2 (V2R2), as updated December, 2015
Summary of changes in z/OS Version 2 Release 2 (V2R2)
Summary of changes for z/OS Version 2 Release 1 (V2R1) as updated March 2014
z/OS Version 2 Release 1 summary of changes
Introduction
Characteristics of a sysplex
Recommendations for redundancy
Coupling facility and the sysplex
Defining sysplex parameters and policies
Parmlib parameters and members
Sharing parmlib members
Additional information
Policies
Setting up couple data sets for policy data
Defining policies
Setting up couple data sets for z/OS UNIX
Using signaling for sysplex communications
Tuning the signaling services and coupling facility
Installing systems in a sysplex
Sharing a time source among LPARs
Planning parmlib members for a sysplex
Planning the IEASYSxx member in parmlib
Considerations for the SYSNAME system parameter
Planning the PLEXCFG system parameter
Planning for XCF-local mode
Planning the GRS system parameter
Summary of PLEXCFG and GRS system parameters
Planning the GRSCNF system parameter
Planning the GRSRNL system parameter
Planning the CLOCK system parameter
Setting the TOD clock
Synchronizing time stamps in a sysplex
Adjusting local time in a sysplex
Understanding PLEXCFG configurations with a Sysplex Timer
Consequences of resetting the TOD clock in a sysplex
Planning the COUPLE system parameter
Specifying COUPLE=**
Planning the IXGCNF system parameter
Planning the DRMODE System Parameter
Planning the COUPLExx member in parmlib
The COUPLE statement
The FUNCTIONS statement
CLASSDEF, PATHIN, PATHOUT, and LOCALMSG statements
The DATA statement
Modifying COUPLExx values
Planning the CONSOLxx member in parmlib
Naming your consoles
Understanding message routing
Related parmlib members to consider
Planning the CLOCKxx member in parmlib
Planning the XCFPOLxx member in parmlib
Planning the IEASYMxx member in parmlib
Specifying the SPINRCVY statement in the EXSPAT parmlib member
Preparing to use HyperPAV storage subsystems using the HYPERPAV statement in the IECIOSxx parmlib member
HyperPAV operations
Planning the couple data sets
Considerations for all couple data sets
Considerations for a sysplex couple data set
Formatting a sysplex couple data set
Group Names and members for MVS components and subsystems
APPC
Console services
DAE
DFSMS/MVS and PDSE sharing
ENF
Global resource serialization
IOS
JES2 multi access spool (MAS)
JES3 complex
RRS
VLF
WLM
XCF
XES
Defining a sysplex couple data set to MVS
Connectivity requirements for a sysplex couple data set
Sysplex couple data sets and joining a sysplex
Sample scenario
Implications of the MAXSYSTEM value for the sysplex couple data set
Considerations for function couple data sets
Formatting couple data sets
Defining function couple data sets to MVS
Defining and activating policies
Updating policies in a sysplex
Connectivity requirements for function couple data sets
Providing coupling facility information to a recovery manager
Managing coupling facility resources
Planning for a coupling facility
Defining a coupling facility
Configuring a processor and a coupling facility
Understanding the coupling facility level (CFLEVEL)
CFLEVEL and operating system level coexistence
Specifying coupling facility non-volatility
Planning for coupling facility failure-independence
Determining your coupling facility requirements
Managing a coupling facility
Two ways to start a CFRM policy
Authorizing coupling facility requests
Planning a coupling facility policy
Identifying the coupling facility
Identifying the coupling facility structures
Requesting structure size
How MVS initially allocates the structure
Structure allocation when rebuilding a structure
Warning about CF structure size growth: structure rebuild/duplexing, and insufficient CF white space storage capacity for recovery
Alter consideration
Understanding MINSIZE/INITSIZE/SIZE correlation
Requesting the use of storage-class memory
Understanding preference and exclusion lists
Developing preference and exclusion lists
How MVS uses the lists
Specifying a rebuild threshold
How MVS decides to initiate a rebuild
Monitoring structure utilization
Allowing a structure to be altered automatically
Updating the CFRM policy
Structure full avoidance
Cache structure reclaim avoidance
Relieving coupling facility storage constraints
Frequency of system-initiated alters
Conditions preventing a system-initiated alter
Determining the amount of coupling facility dump space
Starting a CFRM policy
Updating a CFRM policy
Formatting a new CFRM couple data set
Updating an existing CFRM couple data set
Summarizing CFRM policy setup requirements
Comparing message-based processing and policy-based processing
An installation guide to duplexing rebuild
Understanding system-managed duplexing rebuild requirements
Hardware requirements
Software requirements
Deciding whether to duplex a structure
Migration steps
Controlling the use of the duplexing rebuild process
Policy changes affecting duplexing rebuild processes
Overview of the system-managed duplexing rebuild process
Duplexing rebuild monitoring
Selecting a coupling facility for the duplexed pair of structures
Performance enhancements for lock structures with record data
Common problems and solutions
The role of CFRM in a sysplex
Cleaning up status data
Performing system level initialization
Reconciling CFRM policy data
Transitioning to a new CFRM policy
Resource allocation in the coupling facility
Coupling facility storage considerations
Coupling facility resource allocation “rules”
CFRM policy specification
Authorized application specification
Coupling facility storage increment
Understanding coupling facility use of processor resources
Operating in a coupling facility environment
Removing a coupling facility from a sysplex
Transferring a coupling facility to another sysplex
Using REALLOCATE or POPULATECF to relocate coupling facility structures
POPULATECF function
REALLOCATE process
Comparison of POPULATECF function and REALLOCATE process
Sample procedure for coupling facility maintenance
Summary of MVS operator commands
Coupling facility structures for IBM products
Planning signaling services in a sysplex
Recommended steps for setting up signaling
Overview of available signaling options
Advantages of implementing signaling through list structures
Setting up outbound and inbound signaling paths
Implementing Signaling through CTC Connections
Implementing signaling paths through coupling facility list structures
Planning a CFRM policy for signaling
Avoiding a single point of failure
Supporting dynamic alteration of signaling structures
Determining the Size of a List Structure for Signaling
COUPLExx specifications and commands
Handling signaling path failures
Through CTC devices
Through coupling facility list structures
Handling signaling connectivity failures
Deleting inoperative signaling paths
Communicating signaling path information to operators
Planning the RETRY limit for paths
Planning for optimum signaling performance
XCF groups
XCF group names
Message Characteristics for XCF Groups
Planning the transport classes
Defining Transport Classes
Assigning XCF groups to a transport class
Undesignated Groups
Using the default transport class
Specifying the lengths of messages for the transport class
Planning the signaling paths for the transport class
Planning the message buffer space
Specifying Message Buffer Space
Kinds of message buffers
Outbound and Local Message Buffers
Inbound message buffers
Calculating message buffer space
Summary
Adding and deleting signaling paths
Adding a signaling path
Through CTC devices
Through coupling facility list structures
Deleting a signaling path
Through CTC devices
Through coupling facility list structures
Considerations
Sample signaling scenarios
Example 1 — Using COUPLExx defaults
Example 1 — Diagram of PLEXOF2
Example 1 — SYSA COUPLExx parmlib member
The COUPLE statement
/* Devices that lead to SYSB. */
/* Devices that come from SYSB. */
Example 1 — SYSB COUPLExx parmlib member
Example 2 — Creating a transport class for large messages
Example 2 — Diagram of PLEXOF2B
Example 2 — SYSA COUPLExx parmlib member
The COUPLE statement
The CLASSDEF statement
/* Devices that lead to SYSB. */
/* Devices that come from SYSB. */
Example 3 — Creating a transport class for a group
Example 3 — Diagram of PLEXOF2A
Example 3 — SYSA COUPLExx parmlib member
The COUPLE statement
The CLASSDEF statement
/* Devices that lead to SYSB. */
/* Devices that come from SYSB. */
Example 4 — Defining signaling paths through the coupling facility
Example 4 — Diagram of PLEXOF2C
Example 4 — SYSA COUPLExx parmlib member
The COUPLE statement
The DATA Statement
The CLASSDEF statement
/* Paths out. */
/* Paths in. */
Example 5 — Signaling through a coupling facility and CTC connections
Example 5 — Diagram of PLEXOF2D
Example 5 — SYSA COUPLExx parmlib member
The COUPLE statement
The DATA Statement
The CLASSDEF statement
/* Paths out. */
/* Paths in. */
Handling signalling path problems
Problems with signalling path definitions
Incorrect signalling path definitions
Not enough signalling path definitions
Unable to Establish Connectivity
Loss of some signalling paths between systems
Loss of all signalling paths between some systems
Loss of all signalling paths between all systems
Signaling sympathy sickness
WSC flash — Parallel Sysplex performance: XCF performance considerations
Transport classes
Message Buffers
Message buffer size
Message buffer pools
Outbound messages
Inbound messages
Local messages
Signaling paths
Number of paths
Type of signaling path
Capacity planning
CTC configuration planning
Case study
Planning XCF Note Pad Services in a sysplex
Steps for setting up XCF Note Pad Services
Defining the XCF note pad catalog structure in the CFRM policy
Name of the XCF note pad catalog structure
Determining the size of the XCF note pad catalog structure
Other considerations for the XCF note pad catalog structure
Deleting the XCF note pad catalog structure
Defining XCF note pad structures in the CFRM policy
Naming conventions for the XCF note pad structures
Determining the sizes of the XCF note pad structures
Other considerations for the XCF note pad structures
Deleting XCF note pad structures
Authorizing XCF note pad requests
Restricting IXLCONN access to XCF catalog and note pad structures
Tuning a sysplex
RMF reports and the sysplex
XCF activity report
Tuning the signaling service
Tuning the message buffer size (class length)
Segregating message traffic
Choosing a class length
Best practices for choosing a class length
Summary
Tuning the maximum message buffer space
Adjusting message buffer space
Using RMF reports to analyze message buffer space
Analyzing a lack of outbound message buffer space
Analyzing a lack of inbound message buffer space
Tuning the signaling paths
Signaling path capacity
Signaling path performance
Analyzing message counts
Impact of signaling path restarts
Impact of having no paths in a class
Tuning the transport classes
Hints
Capacity Planning
Using the RMF reports
Remote Message Traffic
Local message traffic
Using the DISPLAY command
RMF XCF Activity Report — Sample
Tuning UNIX System Services performance in a sysplex
Tuning coupling facilities
Coupling facility activity report
How z/OS performs synchronous to asynchronous conversion of coupling facility requests
SYNC and ASYNC service
CHNGD requests
Coupling facility request response time (service time)
Potential sources of change and delay in service times
Insufficient CF capacity
Contention for CF paths
Evaluating Path Busy Conditions and taking Actions to Reduce Them
z/OS Subchannel Tuning
IOP contention
Shortage of CF subchannels
Usage of coupling facilities by structure
Planning sysplex availability and recovery
Controlling availability and recovery through COUPLExx
Planning the failure detection interval and operator notification interval
Planning the cleanup interval
Controlling system availability and recovery through the SFM Policy
Overview and requirements of SFM policy
Requirements for using the SFM policy
Relationship between SFM policy and XCF PR/SM policy
Planning for a status update missing condition
Prompting the operator
Isolating a failing system
Resetting or deactivating a failed system on PR/SM
Handling signaling connectivity failures
Handling coupling facility connectivity failures
Handling signaling sympathy sickness
XCF messages
Planning PR/SM reconfigurations
Setting Up an SFM policy
Defining SFM policies
Starting an SFM policy
Updating an SFM policy
Controlling availability and recovery through XCFPOLxx (PR/SM only)
Implementing XCFPOLxx
Examples
Example 1: Failure management in a PR/SM environment — one processor
Objective
SFM policy specifications
Scenario
Corresponding XCFPOLxx specifications
Example 2: Failure management in a PR/SM environment — two processors and a coupling facility
Objective
SFM specifications
Scenario
Corresponding XCFPOLxx specifications
Example 3: Failure management in a PR/SM environment — one processor
Objective
SFM specifications
Scenario
Corresponding XCFPOLxx specifications
Example 4: Failure management with SFM policy and a coupling facility
Objective
SFM Policy specifications
Scenario
Controlling job availability and recovery through automatic restart management
Requirements for participating in automatic restart management
Using automatic restart management
Customizing the automatic restart management policy
Considerations
Using exits to customize automatic restart management processing
Using the system status detection partitioning protocol and BCPii for availability and recovery
Handling concurrent system and couple data set failures
Planning disaster recovery actions
Adding MVS systems to a sysplex
XCF-local mode configuration
Single-system sysplex configuration
Multisystem sysplex configuration without an existing global resource serialization complex
Multisystem sysplex configuration with an global resource serialization complex
The first system IPL
The second system IPL
The third system IPL
The fourth system IPL
Multisystem sysplex configuration on one processor under z/VM
Multisystem sysplex configuration on PR/SM
Multisystem sysplex configuration with a coupling facility
Multisystem sysplex configuration of greater than eight systems
Removing a system from the sysplex
Lock structure considerations
Multisystem application considerations
Rejoining a sysplex
Planning for system logger applications
What is system logger?
Coupling facility log stream
DASD-only log stream
The system logger configuration
The system logger component
Indicating the IXGLOGR address space is not to start on a system
The LOGR couple data set (LOGR policy)
Log data on the coupling facility
Log data on DASD log data sets
Duplexing log data
Duplexing log data on a coupling facility log stream
Duplexing for a DASD-only log stream
Offloading log data from interim storage by freeing and/or moving it to DASD
Other events that trigger offloading
Using the high and low thresholds to control offloading
Offloading and log data deletion
Coupling facility size and offloading
Staging data set size and offloading
Coupling facility log stream
DASD-only log streams
Offload and service task monitoring
Monitoring log stream interim storage consumption
System clock considerations
Logger configuration resource value ranges/limits
Finding information for system logger applications
Finding information for CICS log manager
Finding information for OPERLOG log stream
Finding information for logrec log stream
Finding information for APPC/MVS
Finding information for IMS common queue server log manager
Finding information for resource recovery services
Finding information for system management facilities (SMF)
Preparing to use system logger applications
Understand the requirements for system logger
Coupling facility requirement for system logger
Sysplex requirement
GRS complex considerations
System managed storage (SMS) requirement for system logger
Authorization requirements for system logger
Logger resource configuration consistency requirements
Plan the system logger configuration
Determine the number of log streams required
Determine the right kind of log stream for each application
Determine which log streams map to which coupling facility structures
The LOGSNUM parameter
Separating system logger production and test log streams
Examples of coupling facility log stream configurations
Example - merging log data from across the sysplex
Example - maintaining multiple log streams in the same coupling facility structure
Plan your configuration for log data recovery
Examples - peer connectors recovering coupling facility log data for a failing system
Managing log data in interim storage to minimize movement to DASD
Determine the size of each coupling facility structure
Develop a naming convention for system logger resources
Naming conventions for the log stream and DASD data sets
Naming conventions for system logger coupling facility structures
Plan DASD space for system logger
Plan space for DASD log data sets
Increasing the space available for DASD log data sets
Plan space for staging data sets
Plan staging data sets for coupling facility log streams
Plan DRXRC-type staging data sets for coupling facility log streams
Sizing staging data sets for coupling facility log streams
Change the staging data set size for coupling facility log streams
Plan staging data sets for DASD-only log streams
Sizing the staging data sets for DASD-only log streams
Changing the staging data set size for a DASD-only log stream
Monitoring staging data set usage log streams
Set up the SMS environment for DASD data sets
Define share options for DASD data sets
Add the DASD data sets to the GRSRNL inclusion list
Managing logger log stream data sets
Log stream data set migration and recall considerations
Copying log stream offload data sets
Testing log data set parameter modifications
Managing log data: How much? For how long?
Choosing the right retention period and automatic deletion policy for a log stream
Example 1. Delete data after no more than 3 days, no matter what
Example 2. Keep data for at least 30 days, even if it was marked for deletion
Example 3. Keep data in the log stream until deletion
Example 4. Don't keep my log data at all
Using a resource manager and auto deletion together
Define authorization to system logger resources
Define authorization for the system logger address space
Define authorization for setting up policies
Authorization for system logger applications
Format the LOGR couple data set and make it available to the sysplex
Prevent a z/OS image from accessing LOGR couple data sets
Add information about log streams and coupling facility structures to the LOGR policy
Specifying whether the log stream is DASD-only
Specifying the average log block size
Specifying SMS data set characteristics for DASD data sets
Selecting a method of duplexing coupling facility log data
What is a single point of failure?
Using the STG_DUPLEX and DUPLEXMODE parameters
Example - Using staging data sets when the coupling facility shares a CPC with a system
Example - Using staging data sets when the coupling facility is volatile
Logger and coupling facility duplexing combinations
Migration steps to enable system-managed duplexing of logger structures
Base requirements
Activation steps when there are no current logger structure connections
Activation steps when there are existing logger structure connections
Things to consider when logger structure connections exists
Associating a log stream with a resource manager
Deleting log streams from the LOGR policy
Define the coupling facility structures attributes in the CFRM function couple data set
Activate the LOGR subsystem
Deleting log data and log data sets
Deleting log data
Deleting log data sets
When is my log data or log data set physically deleted?
Finding and deleting orphaned log data sets
Upgrading an existing log stream configuration
Changing the GROUP value
Example 1–Change the GROUP value of a log stream that is the only log stream defined to a structure
Example 2 - Change the GROUP value for all the log streams in a structure
Example 3 - Have log streams with both values of GROUP(PRODUCTION) and GROUP(TEST)
Upgrading a log stream from DASD-only to coupling facility
Plan for a DASD-only log stream upgrade
Match the MAXBUFSIZE values for a DASD-only log stream upgrade
Staging data sets and upgrading a DASD-only log stream
Updating a log stream to use a different coupling facility structure
Updating a log stream's attributes
Renaming a log stream dynamically
System logger recovery
Operations log and logrec application recovery
Recovery performed for DASD-only log streams
When an MVS system fails
Recovery for a failing system in a single system sysplex
When a sysplex fails
When the coupling facility survives the sysplex failure
When the coupling facility fails with the sysplex
DRXRC considerations and DRMODE=YES IPL option
When the system logger address space fails or hangs
When the system logger address space hangs
When the system logger address space fails
When the coupling facility structure fails
Damage to or failure of the coupling facility structure
Loss of connectivity to the coupling facility structure
When the coupling facility structure volatility state changes
A coupling facility becomes volatile
A coupling facility becomes non-volatile
When the coupling facility structure duplex/simplex mode changes
When the coupling facility space for a log stream becomes full
When a staging data set becomes full
When a log stream is damaged
When DASD log data set space fills
When unrecoverable DASD I/O errors occur
When unrecoverable DASD I/O errors occur during offload
When staging data set unrecoverable DASD I/O errors occur
Preparing for z/OS IBM zAware log stream client usage
Format utility for couple data sets
Using the couple data set format utility
Understanding couple data set coexistence
Coding the couple data set format utility
Sysplex parameters for format utility
Automatic restart management parameters for format utility
ARM couple data set versioning — new format level
CFRM parameters for format utility
Considerations for supporting CFRM functions
LOGR parameters for format utility
LOGR couple data set versioning — new format levels
LOGR couple data set use considerations
SFM parameters for format utility
z/OS UNIX parameters for format utility
BPXMCDS couple data set versioning — new format level
WLM parameters for format utility
Return codes
Examples of using the IXCL1DSU format utility
Formatting a sysplex couple data set
Formatting an automatic restart management couple data set
Formatting a CFRM couple data set
Formatting a LOGR couple data set
Formatting an SFM couple data set
Formatting a UNIX System Services couple data set
Formatting a WLM couple data set
Sample output
Administrative data utility
Using the administrative data utility
Authorizing use of the utility
Coding the administrative data utility
Automatic restart management parameters for administrative data utility
Using wildcard characters
Examples of matching
Using system symbol substitution
CFRM parameters for administrative data utility
LOGR keywords and parameters for the administrative data utility
DEFINE LOGSTREAM keywords and parameters
DEFINE STRUCTURE keywords and parameters
UPDATE LOGSTREAM keywords and parameters
DELETE LOGSTREAM keyword and parameters
DELETE STRUCTURE keyword and parameters
LIST LOGSTREAM keywords and parameters
LIST STRUCTURE keywords and paramaters
Using wildcards with the LIST request
CONTINUE Keyword
SFM parameters for the administrative data utility
Assigning defaults using an *
Return codes
Examples of using the IXCMIAPU utility
Administrative data utility for automatic restart manager Policy Data
Administrative data utility for CFRM policy data
Administrative data utility for the LOGR policy
Administrative data utility for SFM policy data
Deletion utility for XCF group members
Output messages from the IXCDELET utility
Deletion utility for XCF note pads
Output messages from the IXCDELNP utility
Calculating sizes for signaling services
Calculating the size of a list structure for signaling
Additional considerations
Required space
Recommended space
Possible consequences of too little space
Vector space required on the MVS systems
Message information for some MVS groups
Calculating message buffer space — An example
Calculating the size of list structures for system logger applications
Sysplex configurations
Under VM - Physical configuration of a sysplex with MVS Guests
Components of the z/VM Guest Coupling Simulation Support
The CF service machine
The coupled guest
The message facility simulation
Setting up a guest coupling environment under z/VM
Define the z/VM service machines to MVS
Establish data sharing among MVS virtual machines
Modify the MVS clock definitions
Change and synchronize the time-of-day Clocks
Understanding the node descriptor
On PR/SM - Physical configuration of a sysplex on LPARs
Configuration of a sysplex on LPAR and Non-LPAR systems
Coupling Facility Guidelines
A Review of Some Basic Concepts
A Review of the Use of Structure Data
A Review of the Concept of Persistence
Specifying Structure and Connection Persistence
Sample Persistence Scenarios
Persistent Scenarios from a Connector's View
Normal disconnect
Abnormal disconnect
Reconnect
Persistent Scenarios in Coupling Facility Failure Situations
Guidelines for coupling facility reconfiguration
Adding a Coupling Facility
Adding a structure
Planning for a Coupling Facility Shutdown
Shutting down a coupling facility for maintenance
Removing a coupling facility from the configuration
Upgrading or replacing a coupling facility
Upgrading or replacing all coupling facilities
Transitioning from a Parallel Sysplex to a base sysplex
Removing structures from a coupling facility for shutdown
Removing a structure from a coupling facility
Actions to remove a structure from a coupling facility
Rebuilding a structure into another coupling facility
Additional processing for rebuilding structures
Deleting a structure from the CFRM active policy
Deleting structures
Deleting the JES2 checkpoint structure
Deleting DB2 cache structures for group buffer pools
Deleting the DB2 list (SCA) structure
Deleting CICS temporary storage queue pool structures
Deleting CICS coupling facility data table structures
Deleting CICS named counter structures
Deleting the RACF database cache structures
Deleting IMS CQS log stream structures
Deleting IMS CQS message queue (EMHQ, EMHQ overflow, MSGQ, and MSGQ overflow) structures
Deleting the IMS fastpath MADSIOT structure
Deleting IMS OSAM and VSAM cache structures
Deleting IMS CQS resource structures
Deleting the IMS fastpath VSO cache structure
Deleting the IRLM lock structure
Deleting the XCF signaling structure
Deleting the system logger structures
Deleting the VSAM RLS lock and cache structures
Deleting the global resource serialization lock structure
Deleting the VTAM structure for generic resources
Deleting the VTAM structure for MNPS
Deleting the TCP/IP structure for Sysplexports
Deleting the TCP/IP structure for sysplex-wide security associations (SWSA)
Deleting the enhanced catalog sharing structure
Deleting the DFSMShsm common recall queue structure
Deleting the WLM structures (IRD and enclaves)
Deleting the WMQ shared queued structure
Deleting structures with SETXCF FORCE
Structure rebuild recovery
Steps for handling a structure rebuild hang
Example of a rebuild hang
Guidelines for unplanned outages
Guidelines for recovering from coupling facility failures
CASE 1 - An alternate coupling facility is available
All structures rebuild
Some structures do not rebuild
Structure or connection persistence occurs
Handling failed coupling facility structures
Structure rebuild fails
Coupling facility storage constraints exist
CASE 2 - An alternate coupling facilityis not available
Handling failing or disconnecting connection states
Determining the connection states
Role of an active connector
Steps for diagnosis and recovery
Identify the connector or systems that require recovery actions
Determine recovery actions
Rebuild the structure
Collect documentation
Recover the Hung connectors
Best practices for updating a coupling facility
"Push/pull" of a coupling facility or POR of a processor with a coupling facility
POR of a CPC with a Coupling Facility with no Physical or Logical Changes to the CF
Disruptive CF Upgrade
Using automatic tape switching (ATS STAR)
Setting up automatic tape switching
Recommendations for automatic tape switching