UID Mapping for GPFS in a
Multi-cluster Environment

4.1. Credential mapping for permission checking and file ownership

When an application on a remote node accesses a GPFS file system in another cluster (e.g., opens or creates a file), GPFS will map the UID/GIDs of the thread issuing the I/O operation into a set of IDs in the file system home cluster ID space, and execute the I/O request using the remapped IDs. This ensures that permission checking and file ownership will be handled just as if the application were running on a node in the file system home cluster. GPFS will perform this mapping by executing the following operations:

(1)

pass the credentials of the I/O thread (UID and GIDs) to mmuid2name

(2)

receive one or more GUNs back

(3)

send output received in (2) to a node in the home cluster of the file system in question

(4)

on the home cluster node, pass GUN(s) to mmname2uid

(5)

receive a set of credentials (a single UID and one or more GID) back

(6)

send this set of credentials back to the node where the I/O request is being executed and use these credentials for the duration of the I/O request

In this process, an IRHF command will be invoked twice: once on the node where the I/O thread is running to map local IDs to global names, and once on a node in the file system home cluster to map global names to local IDs:

• In step (1) mmuid2name is executed with intent set to credentials. Input shall consist of a single UID and one or more GIDs as seen in the I/O thread context. Output should consist of one or more GUNs (see discussion below). Also, in this case all environment variables will be set prior to exec’ing mmuid2name exactly as they are set in the I/O thread context to facilitate the mapping process.

• In step (4) mmname2uid is executed with intent set to credentials. The input shall consist of a verbatim copy of mmuid2name output from step (1). The output should be a list of numeric IDs, starting with UID, followed by the primary GID, followed by zero of more supplemental GIDs.

If the user attempting to execute the I/O request is properly authorized to do so, as is normally the case, then there will be a GUN associated with the local ID, and the user will have an account in the file system home cluster, and thus both remapping requests will succeed, at least for mapping the UID. Otherwise, mmname2uid should return a set of credentials that correspond to an “invalid” or “nobody” user, as defined on the file system home cluster.

In the environments where GIDs cannot be mapped individually (e.g. because global names are defined for users, but not for groups), the proposed treatment is to replace the entire GID list with that associated with the given user on the home cluster. In other words, in this case, mmuid2name would ignore the GID list and produce a single GUN as output; mmname2uid would be called with a single GUN and return the UID and the GID list associated with the given user’s account in the file system home cluster. This will result in a consistent approach to group permission checking no matter where the application is running (although it will render changing the effective GID, e.g. with newgrp, inconsequential for cross-cluster GPFS I/O).

Note that the exact behavior for mapping groups and for handling unknown users is defined by the IRHF implementation, since GPFS will pass the output from mmuid2name to mmname2uid without interpreting its content; GPFS does not require or expect the number of the returned GUNs to match that given on input. For example, it is up to the user-supplied IRHF how user “nobody” is represented as a GUN or what UID/GIDs are assigned to such a user.

If the entire set of GIDs is to be replaced, as discussed above, the number of supplemental GIDs naturally should not exceed the maximum allowed by the host OS. It should be noted that since different OSes may impose different limits on the maximum number of supplemental GIDs, one should expect the possibility of the GID list being truncated on the remote cluster.

Since the case of credential remapping for permission checking and file ownership is particularly performance-sensitive, we propose to implement a caching mechanism to avoid costly lookup operations where possible, as described in Section 5.

ID remapping in GPFS is disabled by default, and can be enabled by setting enableUIDRemap GPFS config option to yes.

Example: On a node that remotely mounts a GPFS file system, mmuid2name is called to map credentials of an I/O thread into GUNs. It is given 1 UID and 3 GIDs as input, and the output is a single GUN corresponding to the given UID:

mmuid2name sdsc.edu credentials 1 3
stdin:
15001
2000
2001
3000
<EOF>

stdout:
/C=US/O=NPACI/OU=SDSC/CN=Jane Doe
<EOF>

On a node in the file system home cluster, mmname2uid is then called to convert the output above into a set of numeric IDs that would compose the I/O thread credentials. The output is the UID of the local account belonging to Jane Doe (30001), and the corresponding set of 4 GIDs (IDs of the groups that Jane Doe belongs to on the home cluster):

mmname2uid sdsc.edu credentials 1 0
stdin:
/C=US/O=NPACI/OU=SDSC/CN=Jane Doe
<EOF>

stdout:
30001
2001
2002
3001
3002
<EOF>

4.2. Stat system call

When a stat system call is issued, e.g. as a result of the “ls –l” command, GPFS will need to return a set of UID/GID pairs for a set of files. If GPFS simply returns the IDs as stored on disk,

i.e. IDs in the home cluster ID space, these IDs may get locally interpreted in a manner that some users may find confusing: “ls –l”, for example, would display those IDs that happen to be present in the local /etc/passwd and /etc/group as symbolic names of local users/groups, and display the rest as numeric IDs. If this is considered to be undesirable, one way to address the problem is to perform a “reverse ID remapping”, i.e. translate UIDs/GIDs from the ID space of the file system home cluster to the local ID space, using a scheme similar to that outlined in Section 4.1. However, we cannot reasonably expect all such remapping requests to succeed, since only a subset of users in any given cluster will normally have both (1) a GUN defined and

(2) an account in every cluster where remote GPFS mounts are desired. There is fundamentally no good solution for this problem. If one chooses to remap IDs, then only a small subset of IDs may get translated into meaningful values, with the rest showing up as ‘nobody’, which is probably not very useful in most cases. If one chooses to not remap any IDs, the output of something like “ls –l” command may potentially be confusing to some users. It should be noted that the latter is the standard semantics for other network file systems, such as NFS or AFS, and users familiar with those environments will not be confused by GPFS following the same semantics; it is also clearly much more efficient than performing any UID remapping. Reverse ID mapping for stat calls is disabled by default, and can be enabled (separately from credentials ID mapping) by setting enableStatUIDRemap GPFS config option to yes.

If enabled, ID mapping for stat will follow the same steps as for credentials mapping, except in the reverse direction: GPFS will first invoke mmuid2name to map UIDs/GIDs to a GUNs on a node in the file system home cluster, and then call mmname2uid to map to local UIDs/GIDs on the node where stat was called:

• mmuid2name is executed on a node in the file system home cluster with intent set to stat. Input shall consist of a list of numeric IDs that need to be remapped for the purposes of a stat system calls. Output should consist of the corresponding GUNs, exactly one line of output for every line of input

• mmname2uid is executed on the node where stat call originated with intent set to stat. In this particular case, two lines of input shall be given for each ID: a GUN on one line (as found in mmuid2name output in the previous step), followed by the original numeric ID associated with this GUN. The rationale for providing both GUN strings and numeric IDs is as follows. Either of the remapping steps may fail for some or all of the presented IDs, and we expect that different sites may have different policies for such a case. Therefore, we provide mmname2uid enough input to make a decision: remap some IDs based on GUNs and remap the rest to “nobody”, remap none at all, etc. Output should consist of a list of numeric ID, exactly one line of output for every line of input.

Example: On a node in the file system home cluster, mmuid2name is called to map a set of IDs to GUNs during processing of a set of stat calls on a remote node. It is given a list of 4 UIDs (of which 3 are unique) and 4 GIDs (of which 2 are unique) as input, and only two of the UIDs and none of the GIDs have GUNs associated with them:

mmname2uid sdsc.edu stat 4 4
stdin:
30001
30001
21843
18304
2001
2001
4002
4002
<EOF>

stdout:
/C=US/O=NPACI/OU=SDSC/CN=Jane Doe
/C=US/O=NPACI/OU=SDSC/CN=Jane Doe
/C=US/O=National Center for Supercomputing Apps/CN=John Doe

<EOF>

In this example, mmuid2name produces an empty line for IDs that cannot be mapped to a GUN; however, any other convention can be used to represent unmappable IDs here, since GPFS passes the output of mmuid2name to mmname2uid without interpreting the content of each output line. We only require that mmuid2name produces exactly one output line for each input line. On the remote node (the node where the stat calls originated from), mmname2uid is then called to convert mmuid2name output (accompanied by a set of original numeric IDs from the home cluster ID space) into a new set of numeric IDs corresponding to the local ID space. In this example, the local policy is to remap IDs during stat when a valid mapping exists, and otherwise map IDs to local “nobody” (UID 99) and “nogroup” (GID 65534) IDs. In this case, Jane Doe has a local account, but John Doe does not.

mmname2uid sdsc.edu stat 4 4
stdin:
/C=US/O=NPACI/OU=SDSC/CN=Jane Doe
30001
/C=US/O=NPACI/OU=SDSC/CN=Jane Doe
30001
/C=US/O=National Center for Supercomputing Applications/CN=John
Doe
21843

18304

2001

2001

4002

4002
<EOF>

stdout:
15001
15001
99
99
65534
65534
65534
65534
<EOF>

4.3. ACL operations

Since operations for displaying or changing access control lists (ACLs) in GPFS are performed by invoking specific GPFS commands (mmgetacl, mmputacl) rather than system calls that pass numerical UID values, there is no intrinsic need to convert between numerical ID values in two different clusters. Instead, we can directly convert between numerical ID values in the file system home cluster (UID/GID values stored in ACLs on disk) and user or group names used in the output or input of GPFS ACL commands. Unlike for stat operation, it makes little sense to map IDs not known on a node where an ACL command is issued to “nobody” or “unknown”. For example, running mmputacl with the output from mmgetacl as input should leave the ACL unchanged rather than setting some of the IDs in the ACL (those not known on the node where the commands were issued) to “nobody”. Hence, GPFS handles each ACL command as if the command had been issued on a node in the file system home cluster. When an ACL command is issued for a file that resides on a remote file system, the command will issue a warning stating that it is executing on a remote node, and therefore all symbolic user and group names are the names as known on this node.

4.4. Changing file ownership (chown/chgrp)

Processing a chown or chgrp system call issued on a remote node requires mapping a single UID or GID from the remote ID space to the ID space in the file system home cluster. In environments where GIDs cannot be mapped individually (e.g., where groups do not have global names), it will not be possible support chgrp() on a remote node. Furthermore, only root is allowed to perform chown, and we expect that root access to remote file systems would normally be disallowed. Therefore, we chose to disallow chown and chgrp operations on remotely mounted GPFS file systems. If needed, a chown/chgrp command can simply be executed in the file system home cluster.

5.1 The rationale for caching ID remapping results

It is of paramount importance to implement ID remapping in a way that does not impose a significant performance penalty. Clearly, performing a full remapping operation on every I/O request is prohibitively expensive. A remapping request necessarily involves invoking the IRHF twice, meaning running an external application, with the associated overhead and the actual mapping cost, and also sending a message to another node, which means additional latency. Altogether, we cannot reasonably expect that a remapping request should be very fast. This dictates the need for some form of caching, so that the majority of the remapping requests on performance-sensitive paths are satisfied out of the cache. We implement two distinct types of caching: client-side caching (numeric ID to numeric ID mapping on remote I/O nodes), and server-side caching (GUN to numeric ID mapping on the node in the file system home cluster where the corresponding part of the remapping was performed).

5.2 Ambiguous UID to GUN mappings

The basic concept of ID remapping for credentials checking is complicated by the fact that a given local user may be associated with more than one GUN, and this dictates that some assumptions are made. The key assumption that we make is that a given ID to GUN mapping is effective on a node-wide basis (and not, for example, on a session-wide or a process-group-wide basis, so that the mapping can be done with the sole input being a list of UID/GIDs), and the mapping remains valid for at least a moderately long period of time, which would allow us to effectively cache ID remapping results. Note that a full complement of environmental variables, as set in the user process, will be supplied to mmuid2name to facilitate the task of disambiguating UID to GUN mapping. However, on different I/O nodes the same user may have different GUNs associated with her at the same time, so ID to ID caching cannot be done outside of the I/O node.

Note that while it makes sense to cache individual ID to ID mappings when each ID is somewhat independent, e.g. when processing a stat request, it is not efficient to take the same approach to caching the results of mappings obtained for the purposes of credentials checking. We chose a more practical approach of caching UID to a set of credentials (UID/GIDs list) mappings in the latter case.

5.3 Cache invalidation

There is two mechanisms available for removing a cached ID remapping entry. First, both client-side and server-side cache entries are automatically removed from cache when they expire. The GPFS configuration parameter uidExpiration specifies the maximum lifetime of a remapping cache entry in seconds. The default value is 36000 seconds (10 hours). Second, any given client-side remapping cache entry can be invalidated using tsctl command with the uidinvalidate parameter (See Appendix A). Note that explicit server-side cache invalidation should never be necessary. While it is anticipated that a client-side mapping may change somewhat frequently, e.g. when a user decides to switch the identity under which the remote cluster is being accessed (meaning a different UID-to-GUN mapping should be used), the server-side mappings cached on the home cluster should normally be significantly more static. If a user has a given GUN in the home cluster associated with a particular UID (which is the mapping cached on the server side), the GUN-to-UID mapping normally never changes unless the user account is removed, and automatic cache expiration is sufficient to address such a case.