Benefits and implementation of Spectrum Scale sudo wrappers
Nils Haustein 060000NX38 Visits (2928)
Thanks to Felipe Knop and Sandeep Ramesh for their valuable feedback and contributions.
In this blog article I would like to share my experience configuring IBM Spectrum Scale™ with sudo wrappers. I will start with a short introduction of sudo and explain the objective of sudo wrappers in the context of a Spectrum Scale cluster. Afterwards I will disclose an approach to implement the four eye principle which relies on two distinct users controlling each other for secure administration. At the end I will demonstrate how to configure and use sudo wrappers in a Spectrum Scale cluster. I assume that you are familiar with the administration of a Spectrum Scale cluster.
What is sudo
Sudo is a program that allows computer users to run programs with the security privileges of another user, by default the superuser (commonly the root user). The name “sudo” originally stood for "superuser do", but sudo is no longer limited to running commands with the privileges of root. It can also run commands with privileges of other (restricted) users.
The commands a user can run with the security privileges of another user are configured in the sudo configuration file /etc/sudoers (also called sudoer file). It is recommended to use the command visudo to edit this file because it does a syntax check after saving and closing it. The sudoer file allows to configure privileges for running commands, including enabling commands only from the invoking terminal; requiring a password per user or group; requiring re-entry of a password every time or never requiring a password at all for a particular command line. It can also be configured to permit passing arguments or multiple commands.
A simple explanation of the syntax to configure privileges of a user is shown in the following clause:
%user host=(as-user) PASSWD: passwd-cmds NOPASSWD: nopasswd-cmds
Here is a brief explanation of this clause:
In the following example clause the user admin is allowed to run the all mm-commands with password and the command mmremote without password.
%admin ALL=(ALL) PASSWD: /usr
It is also possible to disallow commands by adding a “!” in front of the command. In the following example clause the user admin is not allowed to run the command mmfsadm:
%admin ALL=(ALL) !/us
Instead of spelling out every command that is allowed or forbidden in the clause, so called command aliases can be created. The following example shows a command alias for some networking commands:
Cmnd_Alias NET = /sbin/route, /sbin/ifconfig, /bin/ping, …
This alias can now be embedded into a clause defining user privileges. In the example below the user is allowed to run the above networking commands requiring his password:
%admin ALL=(ALL) PASSWD: NET
In general the sudoer file can include multiple clauses for a given user. Be aware, however, that the file is interpreted from top to bottom. So if there is a clause that allows a command and later on there is a clause that disallows a command for given user, the latter one becomes effective.
Another important feature in sudo are the logging capabilities. Sudo can be configured to log all commands that a given named user runs in the sudo context. In the following example all commands (input and output) the user admin runs with sudo are logged to the directory: =/va
%admin ALL=(root) PASSWD: LOG_INPUT: LOG_OUTPUT: cmds
Alternatively the clauses Defaults logdir=directory or Defaults syslog=auth can be used to enforce logging.
For more information about the features and syntax in sudo refer to the sudo man pages:
What are Spectrum Scale sudo wrappers
Spectrum Scale does not have its own user management for administrative users using the command line interface (CLI). CLI users are authenticated by the operating system and have privileges to perform commands according to user configuration in the operating system. By default the Spectrum Scale cluster requires root privileges for administration.
In order to eliminate the need for direct root access Spectrum Scale allows to configure and enable so called sudo wrappers. With sudo wrappers commands are executed under the named user with root privileges but no longer as root user itself. Sudo wrappers rely on a sudo configuration that allows the named user to run certain commands with root privileges.
The key advantages of Spectrum Scale sudo wrappers are:
In contrast, sudo wrappers do not assure that a named user cannot elevate its privileges. Therefore additional operational processes are required to assure that that the sudo definitions cannot be changed without permission and without being audited. In addition sudo in general does not allow managing access permissions to files, only for commands. File access permission needs to be regulated using file system permissions and Access Control Lists (ACL).
With the current implementation (Spectrum Scale version 5.0) sudo wrappers do not support the following Spectrum Scale functions:
Configuration and Operations guidance
Users administering a Spectrum Scale cluster should not be able to tamper any data stored in the file system. Therefore it might be appropriate to allow only commands (on a Spectrum Scale and operating system level) that are required for normal operations executed by administrative users. In special cases the privileges of these users can be temporarily increased by another user. This calls for the implementation of the four-eye principle where the administrative user requires the authorization of another user (security administrator) to perform a special task.
For the implementation of the four-eye principle create two user groups and assign named users to each of these groups in the operating system. In this example one group is named gpfsadmin and another group is named secadmin. The following principles should apply:
The four-eye principle allows users of the group gpfsadmin to execute a limited set of commands that are required for normal operations. If a user of the group gpfsadmin requires more privileges (e.g. due to a problem) he can contact a user of the group secadmin. The user of the group secadmin can now grant the user of the group gpfsadmin further privileges temporary by changing the sudo definitions.
Configure Spectrum Scale sudo wrappers according to the instructions in the knowledge center (Link for version 4.2.3). Allow users in group gpfsadmin to run commands that are required to administer the Spectrum Scale cluster. Use the default sudo configuration provided with the file sudoers.sample, adjust and test this according to the needs. Find an example of the sudo definitions in section Configure sudo. In general consider the following guidance
Users in group secadmin can easily elevate their privileges because they control sudo. Persons with this privilege should be trusted. In addition, operational processes can be established that audit the involvement of these users and their activities on the system. Consider implementing a process that only allows users of the secadmin group to access a cluster node when approved by a user of the gpfsadmin group.
The Spectrum Scale GUI has a separate user management. It is recommended to use the same user names in the GUI and the CLI for the same user. In addition the roles of the user in GUI should match the roles the user has in CLI.
Configuring sudo wrappers with four eye principle
Now I will show how to implement sudo wrappers in a Spectrum Scale cluster. Use this as a high level guidance and always consult the Spectrum Scale knowledge center for the exact and most current procedure (Link for version 4.2.3). I do not guarantee that this works for any environment.
In this example we have three cluster nodes named g1_node1, g1_node2 and g1_node3 running Red Hat Enterprise Linux version 7. All cluster nodes are administrative nodes which means that Spectrum Scale commands can be executed on any of these nodes.
Configure user and groups
We will create a group gpfsadmin and add user admin to this. This user - and all other users that might be added to this group - are Spectrum Scale administrators, able to run all mm-commands (with some exceptions). In addition we configure a group secadmin and add user secco to this. This user is allowed to change the sudo configuration and temporarily grant additional privileges to the admin user using the visudo command.
Create group gpfsadmin and user admin on all nodes (as root user). The group and user ID (1001) are arbitrarily chosen; you can use different IDs. Just make sure that these IDs are not used by other users and groups (file /etc/passwd):
# groupadd -g 1001 gpfsadmin
# useradd -c 'gpfs admin' -g gpfsadmin -m -u 1001 admin
Set password for user admin on all nodes (as root user):
# passwd admin
Create group secadmin and user secco on all nodes (as root user):
# groupadd -g 1002 secadmin
# useradd -c 'security admin' -g secadmin -m -u 1002 secco
Set password for user secco on all nodes for new user (as root user):
# passwd secco
When required, generate the ssh-key pair for the root user on all nodes (as root user). Do not enter a passphrase when asked, but instead press enter a couple of times:
Copy the public key of root user to the authorized_keys file of the admin user. This has to be done from all admin nodes to all other nodes, including the admin node itself. In our example all three nodes are admin nodes, so we have to do this on all nodes (as root user):
# ssh-copy-id admin@g1_node1
# ssh-copy-id admin@g1_node2
# ssh-copy-id admin@g1_node3
The root user must be able to perform password-less ssh-communication with the remote nodes as user admin (not root).Test password-less ssh-authentication for the new user admin from all admin nodes to all other nodes, including the admin node itself:
# ssh admin@g1_node1
# ssh admin@g1_node2
# ssh admin@g1_node3
The two steps above do not have to be done for the user secco because this user will not do cluster administration.
Add the Spectrum Scale command path to the profile for the user admin and secco on all nodes (login as admin or secco user respectively):
# su - admin
# vi .bash_profile
Optionally you can create a key pair for the admin user and exchange this from all nodes to all other nodes. This enables the admin user to access other nodes seamlessly without using passwords. Perform the following steps on all nodes:
1. Login as admin user and create the ssh-key pair, do not enter a passphrase, but instead press enter a couple of times:
2. Copy the public ssh key of the admin user to all other nodes including the node itself:
# ssh-copy-id <node-name>
3. Test the ssh-connection to be password less with all other nodes including the node itself:
You have now prepared the users and groups for the sudo configuration. Continue to the next section.
Save the current sudoer file and copy the sudoers.sample file to /etc/sudoers on all nodes (as root user):
# mv /etc/sudoers /etc/sudoers.backup
# cp /usr
If the current sudoer file (/et
Make the necessary adjustments to the sudoer file on all nodes (as root user). In the example clauses below users in the group gpfsadmin are allowed to run all mm-commands, with some exceptions. The users in group secadmin are allowed to run visudo. In addition we configure logging for all commands issued from users in the sudo context:
For example, the following adjustments can be considered when required:
Please note that the users of the gpfsadmin group are allowed to do all mm-commands. However, these users are not allowed to execute these commands:
The disallowed commands in this example are pretty extensive. These commands may allow for privilege escalation and changes of the file system structure and cluster configuration. You have to decide which commands you disallow and not.
Makes sure that these adjustments are done on all nodes, regardless if admin node or not. You may consider doing these adjustments to the sudoer file on one node using visudo and then copy the file to all other nodes. For example if the adjustments have been done on g1_node1, you can copy the sudoer file to node 2 and 3 as shown below:
# scp /etc/sudoers g1_n
# scp /etc/sudoers g1_n
On each admin node run the sudo wrappers test as admin user. These tests have to be run without errors. Do not continue if a single test fails:
# su - admin
# sudo /usr
# sudo /usr
# sudo /usr
# sudo /usr
# sudo /usr
# sudo /usr
Enable and test sudo wrappers
If the above test completes without errors you can configure sudo wrappers in the cluster:
# su - admin
# sudo mmchcluster --use-sudo-wrappers
Now perform some tests by logging on with the user admin and running mm-commands. Do not only run list kind of commands, also run command that perform changes on the cluster. You have to run commands using sudo:
# sudo mmchfs fsname
# sudo mmcrfileset fsname fsetname …
Also test the privileges of the user secadmin. For example allowing or disallowing certain commands for the user in the gpfsadmin group. Logon with the user secco and edit the sudoer file to your test scenario:
# sudo visudo
If these tests do not work and you are not able to fix it by adjusting the sudo configuration then you can revert from sudo wrappers to non-sudo-wrappers configuration using this command:
# sudo mmchcluster --no
When the Spectrum Scale GUI is used, make the following adjustment to the GUI configuration:
On all nodes running the GUI open the file /usr
Than restart the GUI:
# systemctl restart gpfsgui
Disable root login
Optionally you disable direct root login on all cluster nodes.
This requires root privileges, so logon as root user or allow another user to become root:
# vi /etc
Than restart the ssh daemon:
# systemctl restart ssh
Operations with sudo wrappers
During normal operations the user in the group gpfsadmin login with their named user credentials (e.g. admin) and perform commands using sudo:
# sudo command
There might be cases where the user of the group gpfsadmin requires further privileges. Let us assume a fileset needs to be deleted. In the sudo clauses above we have prohibited the admin user to run the mmchconfig commands. In order to temporarily allow the admin user to run the mmchconfig command, the secco user has to be contacted to enable this commands in the sudo configuration. The secco user logs to one admin node and enables the mmchconfig command by removing it from the commands that are disallowed:
Allow the mmchconfig command by removing it from the list of disallowed commands:
Subsequently the changed sudoer file must be propagated to all other nodes. Either the file has to be changed on all nodes by the secco user or this user copies the files to all other nodes using scp. This requires that the users in the secadmin group are allowed to use the command: /usr/bin/scp
# sudo scp /etc/sudoers g1_n
# sudo scp /etc/sudoers g1_n
Now the admin user can perform the required operations. All operations done in the sudo context can be logged. When the user is done he contacts the user secco in order to disallow the command.
Spectrum Scale sudo wrappers eliminate the need to allow password-less ssh-communication between the cluster nodes. Furthermore, they allow to enforce named user login and tracing of their command activities in the sudo context. Thus the root user login can be eliminated. However, sudo wrappers do not entirely prevent experienced users from increasing their privileges.
The setup of sudo wrappers requires a few steps and has to be done with care. Especially the testing of sudo wrappers must be done thoughtfully. For the distribution of the sudoer files federation tools make life easier because they allow to change one instance of the sudoer file and distribute this to all nodes.
Security by design is one of the key design principles in the Spectrum Scale development. Given the more than 20 year history of Spectrum Scale (alias GPFS) there is some work to do in order to improve security in the context of a Spectrum Scale cluster.
If you want to learn more about Spectrum Scale administration and functions then consider registering for one of our Spectrum Scale hands-on workshops delivered by experienced Spectrum Scale service experts:
Spectrum Scale Administration and Problem Determination:
Spectrum Scale Architectures, concepts and functions:
This document provides guidance for certain configuration and operational aspects. IBM does not guarantee that this guidance complies with laws or regulation. In order to obtain a compliance assessment an independent auditor has to be engaged by the client. IBM cannot be made liable for any findings or violations of laws and regulations.
The software nature of the solution may allow malicious hackers to exploit the system and elevate privileges of users. The use of Spectrum Scale sudo wrappers does not completely assure that there is no way to escape the sudo environment and elevate privileges of users. In addition it might be possible to use certain undocumented Spectrum Scale commands to exploit the system and elevate privileges of users. IBM cannot be made liable for any damage resulting of the use of exploits.
The guidance given herein does not imply warranty that the commands given or their intention satisfies the purpose. IBM cannot be made liable upon damage caused by any of the commands or guidance.