CCR directory structure and recommendations for configuring CCR

The working directory of CCR is at /var/mmfs/ccr on every node in the cluster. Directory structure on quorum node and non-quorum node is slightly different.

On quorum nodes, the following directory structure is available inside this directory:

# ls -al /var/mmfs/ccr
total 32
drwxr-xr-x. 4 root root 4096 Jul  8 16:00 .
drwxr-xr-x. 9 root root 4096 Jul  8 16:00 ..
drwxr-xr-x. 2 root root 4096 Jul  8 16:00 cached
-rw-r--r--. 1 root root    0 Jul  8 16:00 ccr.disks
-rw-r--r--. 1 root root    4 Jul  8 16:00 ccr.noauth
-rw-r--r--. 1 root root  114 Jul  8 16:00 ccr.nodes
-rw-------. 1 root root 4096 Jul  8 16:00 ccr.paxos.1
-rw-------. 1 root root 4096 Jul  8 16:00 ccr.paxos.2
drwxr-xr-x. 2 root root 4096 Jul  8 16:00 committed

On non-quorum node, the files like ccr.paxos.1, ccr.paxos.2, and ccr.disks do not exist.

The directory structure of all CCR files is also known as the CCR state.

The cached directory might contain a file that is named as ccr.paxos. This file is used by the CCR clients to speed up the process for locating quorum nodes. This file is a binary file.

The ccr.disks and ccr.nodes files are the configuration files of CCR. The ccr.disks file contains the CCR configuration of tiebreaker disks if the cluster is configured with tiebreaker disks. The ccr.nodes file contains the CCR configuration of the quorum nodes. Without having this file, the CCR client and server cannot establish a connection to exchange data with the CCR server on other quorum nodes. The master copies of both these files are stored in the committed directory on all quorum nodes as shown in the following example:

# ls -al /var/mmfs/ccr/committed/ccr.*
-rw-r--r--. 1 root root   0 Jul  8 16:00 /var/mmfs/ccr/committed/ccr.disks.2.2.ffffffff.01076f
-rw-r--r--. 1 root root 114 Jul  8 16:00 /var/mmfs/ccr/committed/ccr.nodes.1.7.f20ea9e3.01076e

The ccr.paxos.1 and ccr.paxos.2 files are used by the CCR to reach consensus among the quorum nodes, if an update happens to the CCR. At least one good copy of these files must be available. It can be used if the last write that modified these files failed for some reason. Both files are binary files.

The committed directory contains the files that are committed to the CCR. Changing those files manually might result in damaging CCR operation and state.

A complete copy of the CCR directory is available on every quorum node. The number of quorum nodes is limited to eight. Use an odd number of quorum nodes when no tiebreaker disks are configured.

CCR can be configured with up to three tiebreaker disks. It is strongly recommended to use an odd number of tiebreaker disks. The space that is reserved for the CCR is limited on these tiebreaker disks. Therefore, only the CCR Paxos state, which has nearly the same content of the ccr.paxos.1 and ccr.paxos.2 files are stored on the tiebreaker disks. That is, the files that are committed to the CCR are not stored on tiebreaker disks and it leads to a limitation for the combination of two quorum nodes and tiebreaker disks. For more information, see Limitations of CCR.

The CCR service remains available even if GPFS is shut down on a particular quorum node. This capability is achieved by including the CCR server functions in the following two GPFS daemon executable:

A CCR server runs inside the mmfsd daemon when GPFS is up.
A CCR server instance runs inside the mmsdrserv daemon when GPFS is down.

One of those daemons is running all the time on at least the quorum nodes. The only exceptions to this are the cases in which the mmsdrserv must be stopped by design. For example, in disaster-recovery scenarios.

Nearly all IBM Storage Scale administrator commands use the CCR to evaluate whether a new GPFS configuration file is available. This means that when the CCR is not working, most of the IBM Storage Scale administrator commands also do not work and the CCR must return into a working state first to run any administrator commands.

CCR uses the GSKit security library for authentication purposes.