IBM GPFS is a high-performance enterprise file management platform widely used on commercial and scientific applications. As part of the GPFS configuration, it is required that remote connections for the root user are enabled so that the nodes that are part of the GPFS cluster can communicate among them to run the commands that require root privileges.
Allowing remote root connections is not recommended as a security best practice. Typically, root login connections must be done exclusively from the console, or through a method that maintains accountability to an individual of such connections (for example,
Actions must be taken in order to mitigate the risk that this situation represents to the whole environment.
This article describes the changes that can be applied to the Secure Shell (SSH) configuration file on each one of the GPFS nodes so that remote login operations with the root user are allowed to be established only from the GPFS nodes.
A basic GPFS network configuration
The following figure shows a basic GPFS network configuration. It consists of two nodes, and each one of them has configured two network cards: one is connected to the corporate network and the other has been assigned to be used for GPFS purposes.
Figure 1. A basic GPFS network configuration
Both nodes are running the Linux® operating system (the recommendations of this article can be applied to any UNIX®-like operating system) and the communication mechanism among nodes was determined to be SSH [GPFS supports other communication mechanisms such as remote shell (rsh)].
sshd_config file is the configuration file for SSH, and the parameter
that allows or denies root connections is called
PermitRootLogin. For typical
GPFS installations the value for this parameter is
yes, as shown in the following example.
# grep ^PermitRootLogin /etc/ssh/sshd_config PermitRootLogin yes #
This value applies globally to all the SSH behaviour, meaning that the remote root login connections are allowed on all the network interfaces on which SSH is listening (commonly on all the available).
Recording and maintaining access logs helps to associate the responsibility of actions taken on any computer system by individuals accessing them. By allowing root remote login connections as required by GPFS, a door is let to be open to potential attacks or non-authorized access. Accountability of the user who accessed the system using root credentials is lost and investigations are harder to perform.
The following scenarios show some of the situations that might occur on systems that are not configured to take proper actions to mitigate the risk allowing remote root login connections.
Scenario 1: Individuals in possession of the root password
An individual can be knowledgeable of the root's user password by approved means (for example, a system administrator with a valid business need) or by unauthorized ways (for example, an ex-employee who kept this information and the new system administrator did not change the password right away, or someone who found it on an unprotected text file on a shared folder).
With this information on the user’s possession, the user can start a remote connection from the corporate network to both GPFS nodes with the root identity.
Listing 1. Connection to Node A
$ ssh -l root 172.19.230.35 firstname.lastname@example.org's password: ********** Last login: Mon Jul 29 11:14:07 2013 from gpfsnodeb === Welcome to GPFS Node A #
Listing 2. Connection to Node B
$ ssh -l root 172.19.230.36 email@example.com's password: ********** Last login: Mon Jul 29 11:30:16 2013 from gpfsnodea === Welcome to GPFS Node B #
SSH connections with similar characteristics as explained earlier can be initiated from the nodes themselves. A user can start an SSH connection from the user’s personal computer or from other non-GPFS server with the user’s personal user ID (or a hijacked user ID in case this connection is part of an attack) to one of the nodes and soon after obtaining access, the user can jump to the other node using the root credentials.
Listing 3. Connecting to Node A and then jumping to Node B
$ ssh -l kath1406 172.19.230.35 firstname.lastname@example.org's password: ********** Last login: Thu Aug 15 11:30:16 2013 from pc0212 === Welcome to GPFS Node A [kath1406@gpfsnodea ~]$ [kath1406@gpfsnodea ~]$ ssh -l root 192.168.10.13 email@example.com's password: ********** Last login: Mon Jul 29 13:26:19 2013 from gpfsnodea === Welcome to GPFS Node B #
Listing 4. Connecting to Node B and then jumping to Node A
$ ssh -l kath1406 172.19.230.36 firstname.lastname@example.org's password: ********** Last login: Thu Aug 15 11:03:10 2013 from pc0212 === Welcome to GPFS Node B [kath1406@gpfsnodeb ~]$ [kath1406@gpfsnodeb ~]$ ssh -l root 192.168.10.12 email@example.com's password: ********** Last login: Mon Jul 29 14:02:02 2013 from gpfsnodeb === Welcome to GPFS Node A #
As it can be seen, remote root login connections are accepted no matter where they come from (the corporate network or the dedicated GPFS subnet), restrictions are non-existent and a potential risk of obtaining unauthorized root access is latent.
Figure 2. Unrestricted SSH connections can be started from any point in the network
Scenario 2: Using stolen root SSH encryption keys
In order to enable GPFS to perform automated administrative tasks among the nodes using SSH, encryption keys are created and exchanged among the nodes. But, in GPFS installations, these encryption keys are typically created without a passphrase, thus automation gets preeminence over security.
An individual who has had previous root access to the GPFS nodes can make a copy of the root's encryption keys and further use them to gain unauthorized root access.
Because encryption keys are not changed as frequently as passwords, the inherent risk is that the user in possession of the keys would be able to continue accessing the nodes even when the root password had been changed.
Encryption keys are usually located in the
.ssh subdirectory under the
home directory of the user to whom the keys belong. In the case of the root user, they can be located
/root/.ssh folder. By default, the
command used to create the keys uses the Rivest-Shamir-Adleman (RSA) algorithm. Then, a copy of the
id_rsa.pub files (which contain the
private and public keys respectively) can be made for further use.
After the keys have been copied, the root private key can be used to log in to the server from the user’s computer.
Listing 5. Connection to Node A using stolen root SSH encryption keys
$ eval `ssh-agent` Agent pid 22047 $ ssh-add gpfsnodea_id_rsa Identity added: gpfsnodea_id_rsa (gpfsnodea_id_rsa) $ ssh -l root 172.19.230.35 Last login: Thu Aug 15 13:27:35 2013 from gpfsnodeb === Welcome to GPFS Node A #
Listing 6. Connection to Node B using stolen root SSH encryption keys
$ eval `ssh-agent` Agent pid 24040 $ ssh-add gpfsnodeb_id_rsa Identity added: gpfsnodeb_id_rsa (gpfsnodeb_id_rsa) $ ssh -l root 172.19.230.36 Last login: Thu Aug 15 13:30:12 2013 from gpfsnodea === Welcome to GPFS Node B #
As it can be seen, by using the stolen root SSH encryption keys, a password to access the nodes is not even required. The interchange of the SSH encryption keys among the nodes defines a trust relationship. Therefore, even if the root password is changed, the connections continue to be established regardless of what the root password is.
Scenario 3: Injecting rogue SSH encryption keys
In the previous scenario it was shown how, by using previously created SSH encryption keys for the root user, unauthorized privileged access can be achieved using them. It was assumed that the required SSH configuration for the use of encryption keys for remote connections was already in place as part of the GPFS installation pre-requisites.
This scenario covers the situation when a malicious user injects the user’s own created encryption keys in order to gain root access to the GPFS nodes. It is assumed that this user has had previous access to the nodes with root authority in order to perform this task.
The first step is the creation of a pair of encryption keys. The keys can be created on the user's personal computer
on one of the GPFS nodes or from other non-GPFS server. The
ssh-keygen command has to be
run as shown in the following listing (this example uses an empty passphrase as required by a default GPFS installation):
Listing 7. Creation of rogue SSH encryption keys
$ ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/home/left173/.ssh/id_rsa): ./rogue_rsa Enter passphrase (empty for no passphrase): (Just type the 'Enter' key) Enter same passphrase again: (Type the 'Enter' key again) Your identification has been saved in ./rogue_rsa Your public key has been saved in ./rogue_rsa.pub The key fingerprint is: c2:ae:ac:af:dd:71:35:35:39:c1:07:b2:f8:cc:71:82 left173@pc2604 The key's randomart image is: +--[RSA 2048]----+ | ..o. | | o o.o. | | E + *. | | . + = o | | o S * | | . . . . | | o . | | o o o | | o+= . | +----------------+ $
Next step is to install the rogue public encryption key
each of the GPFS nodes. It has to be transmitted or copied to the
directory on each of the GPFS nodes. Then, it must be appended to the end of the
authorized_keys file, as shown in the following listing.
Listing 8. Injecting the rogue SSH public key on GPFS nodes
# pwd /home/root/.ssh # cat rogue_id_rsa.pub >> authorized_keys # rm rogue_id_rsa.pub #
authorized_keys file maintains a list of public encryption keys that are
allowed to be used to connect to the server. This is where the trust relationship is established among the GPFS nodes and in general, with anyone who tries to connect to the servers.
After accomplishing these steps, the malicious user would be prepared to log in as a root user, as shown in the following listing.
Listing 9. Connection to Node A using rogue encryption keys
$ eval `ssh-agent` Agent pid 25036 $ ssh-add rogue_id_rsa Identity added: rogue_id_rsa (rogue_id_rsa) $ ssh -l root 172.19.230.35 Last login: Thu Aug 15 14:15:00 2013 from gpfsnodeb === Welcome to GPFS Node A #
Listing 10. Connection to Node B using rogue encryption keys
$ eval `ssh-agent` Agent pid 25140 $ ssh-add rogue_id_rsa Identity added: rogue_id_rsa (rogue_id_rsa) $ ssh -l root 172.19.230.36 Last login: Thu Aug 15 14:15:00 2013 from gpfsnodea === Welcome to GPFS Node B #
By using this new pair of rogue keys, it does not matter whether the root password changes or its encryption keys change, now the malicious user has a way to enter the servers regardless of any change on the access credentials of the root user.
Mitigating the risk by strengthening the SSH configuration
The three scenarios depicted previously are just examples of the behaviors that malicious users can perform by misusing a configuration that is required by GPFS to work properly.
More sophisticated alternatives that can take advantage of this backdoor can exist or might be developed in the future. Our objective is to be prepared for unknown current and future malicious techniques and mitigate the exposure to a reasonable extent.
The rationale followed is that allowing remote root login is mandatory for GPFS installations (as per August 2013) to work properly, and therefore, this configuration must be kept. But also, it can be restricted through SSH configuration, specifying which IP addresses are allowed to connect directly with root and which are not allowed. Those IP addresses belonging to the dedicated GPFS subnet (see Figure 1) will be allowed to connect directly with root, and other IPs will continue to be able to connect to the SSH service but not directly with root.
Controlling conditionally remote root login connections
Match directive allows to include a conditional validation. If the condition established is met, then the parameters that would have been defined to have a certain value can be changed here.
This directive can be configured in multiple ways and can receive different arguments, but as the objective is to restrict connections coming from certain IPs, focus will be given to the
Using the Match directive
According to Figure 1, the IP addresses for the GPFS dedicated subnetwork are 192.168.10.12 (Node A) and 192.168.10.13 (Node B), and therefore, users starting connections from these IP addresses would be able to log in remotely with the root user ID. It is important to consider that any
Match directive must be included
at the very end of the
sshd_config configuration file.
The SSH configuration for Node A and Node B would be as shown in Listing 11 and Listing 12.
Listing 11. SSH configuration for Node A
# egrep -n "^PermitRootLogin|Match" sshd_config 38:PermitRootLogin no 114:Match Address 192.168.10.13 115:PermitRootLogin yes #
Listing 12. SSH configuration for Node B
# egrep -n "^PermitRootLogin|Match" sshd_config 38:PermitRootLogin no 114:Match Address 192.168.10.12 115:PermitRootLogin yes #
Notice that the whole contents of the
sshd_config file is not shown,
and instead, the
egrep command is used to look for the strings
Match inside the file.
-n flag is used to show the line numbers where the strings are
found and the numbers returned by
egrep are not part of the contents of
On line 38 of the configuration files for both nodes, notice that the
parameter has been set to a value of
no. Because this configuration
appears practically at the beginning of the file, this is considered as a global setting. Therefore,
the value applies to all connections, which means that by default any remote root
login connection is rejected. This is the recommended value to have.
Nevertheless in line 114 (at the end of both files), the
appears and receives as arguments the string
Address and the IP address on
the dedicated GPFS subnet for their each counterpart cluster node.
If the IP address from which a connection trying to log in directly with the root ID is listed in the
Match Address directive, then the
yes (as it is defined in line 115), which overrides the global value defined in line 38 and allows the connection coming from the authorized IP address to log in with the root ID.
If the GPFS cluster consists of more than two nodes, then multiple stanzas of the
Match Address directive can be declared, each one containing the IP address
of each one of the cluster's nodes.
Using exclusively encryption keys authentication
PermitRootLogin parameter is configured to the value
yes, both authentication methods can be used to connect to the SSH server by providing the root password or by using its SSH encryption keys.
Because GPFS requires that some automated processes run and connect to other nodes with the root identity, the encryption keys authentication is usually the alternative chosen and configured. GPFS will not require to use the root password to start connections, and therefore,
can be configured to the
At first sight, it might seem to cause confusion to have this value configured, but instead, it is actually a more stringent value because it disables the password authentication for the root ID, leaving the encryption key authentication method as the only option allowed for root. The SSH configuration file would contain the code in Listing 13.
Listing 13. Restricting root connections to use only encryption keys
# egrep -n "^PermitRootLogin|Match" sshd_config 38:PermitRootLogin no 114:Match Address 192.168.10.12 115:PermitRootLogin without-password #
Figure 3. Remote root login connections are restricted to be available just for the dedicated GPFS IP addresses
adminMode configuration parameter
In GPFS version 3.3 or later, the
adminMode configuration parameter has been introduced in order to control and limit the number of nodes that can run administrative commands and therefore need to directly access the root user remotely.
There are two possible values available for the
adminMode is set to
allToall, GPFS understands that all the nodes have access among them, and therefore, all of them can run administrative commands on their cluster counterparts.
adminMode is set to the value
central, GPFS acts as if the node on which the command is being run is the only one with root access to all of the other nodes. This allows additional control over which GPFS nodes can run administrative commands.
Notice that by setting
central, security is strengthened
even more, but this parameter itself does not interfere on how the SSH service behaves, and therefore, both
adminMode parameter from the GPFS side and the
directive from the SSH side) should be used in conjunction.
Conclusions and recommendations
By implementing conditional configuration of the
parameter through the
Match directive in SSH,
the risk of allowing remote root login connections from non-authorized or non GPFS cluster servers or networks is
mitigated, and access is only granted to those who strictly need it. Malicious users from non-authorized
locations trying any of the scenarios depicted previously will not be able to connect directly
Notice that controlling the login access to the root user is not the only conditional control that can be used with the SSH
Match directive. Also, the solution described in this article is not exclusive for GPFS. Any system or installation that requires this kind of granular control over SSH remote accesses can benefit from its implementation.
The ultimate objective is to balance productivity, automation, and security that must be associated with them. They cannot been seen as separate and confronted positions. This article shows that having a productive and a more secure GPFS environment than those installed with the default configurations is perfectly achievable.
- Dedicate a specific subnet solely for GPFS communication purposes if possible.
- This can help to control and limit the IP addresses that can use the remote root login configuration.
GPFS can use more than one network in order to allow greater network throughput by concurrently
communicating with its daemons running on the cluster nodes. If network performance is an issue, then
ensure that you configure only the authorized GPFS nodes' IP addresses with their correspondent
Matchdirective on the
sshd_configconfiguration file of each cluster node.
- Use the
adminModeparameter of GPFS in order to limit the nodes that can run administrative commands on all the other nodes on the cluster. This adds another security layer to the GPFS installation.
- Restrict conditionally remote root login connections with SSH to
those strictly required by implementing the
Update your OpenSSH server if its version is lower or equal to 4.3p2 in order
to be able to use the
Generate a different pair of root's SSH encryption keys for each GPFS node.
- Treat encryption keys as passwords: they must be different for each system.
- This reduces the risk of unauthorized access to multiple systems with the same credentials.
Make sure that the root's encryption keys are accessible only to the root user ID.
- Ensure that the permissions of the root's private key are 700 or more stringent.
Regularly create new SSH root's encryption keys and discard the old ones.
- As it is recommended with passwords, encryption keys for automated processes should be changed periodically in order to reduce the probability of the occurrence of a situation that is similar the one depicted in Scenario 2.
Periodically check the contents of the
- Unauthorized keys found on the file must be deleted and investigations about the configuration files contents' changes must be initiated.
- The probability of the occurrence of a situation similar to the one depicted in Scenario 3 is reduced.
Periodically review the contents of the
- Check if unauthorized changes have been made on the
- Check if new IP addresses have been added to allow remote root login connections or
if the global
PermitRootLoginparameter has been modified from
- Check if unauthorized changes have been made on the
- Additional network access restrictions to those configured on the SSH service through the
Matchdirective can be done by implementing (either or both)
tcpwrappersor local firewall (for example,
iptables) access controls on each one of the GPFS nodes.
Implement privilege access controls as
sudoin order to avoid the indiscriminate disclosure of the root password among several individuals. This helps to reduce the probability of the occurrence of a situation similar to the one depicted in Scenario 1.
- Logging of privileged activities performed can be kept and accountability to an individual can be tracked.
- Sudo events can be sent to a syslog server so that if security has been compromised locally on any of the GPFS nodes, the activity logs would reside on an external and independent server for further analysis and forensic investigation.
- See how the use of remote root login connections without the use of a passphrase when using encryption keys is part of the pre-requisites for a GPFS installation and configuration on AIX.
- To learn more about the working of GPFS network
communication and the
adminModeconfiguration parameter, refer to the GFPS Network communication overview document.
- The OpenSSH
version 4.4 release notes (later than 4.3p2) shows that the inclusion of the
Matchdirective was introduced as a new functionality.
- For an extended explanation of the SSH parameters, and
Matchdirective, refer to the
sshd_configconfiguration file man page.
- See how to integrate