Strengthening the SSH configuration used in GPFS installations to make them more secure

Mitigating the risk of unrestrictedly allowing direct remote login connections with root users

Currently, a prerequisite of any IBM General Parallel File System (IBM GPFS™) installation is that all the nodes on the cluster must be able to communicate with each other using the root user ID and that this is done without a password in order to facilitate automated tasks. But this configuration represents a security exposure, because security best practices forbid that the root user can log in remotely. The configuration presented in this article resolves this dilemma by allowing the GPFS cluster nodes to continue communicating with each other using the root ID and at the same time blocking this type of connection to other users and nodes.

Share:

Jose Eduardo Martinez Cordero (emartine@mx1.ibm.com), IT Security Architect, IBM

Image of Jose EduardoJose Eduardo Martinez Cordero is an IT Security Architect and an IBM Certified IT Security Specialist Level 2 with 15 years of experience on the field. He is knowledgeable about IBM AIX and Linux operating systems and has co-authored an IBM AIX Redbooks. He has been working with IBM Mexico helping with outsourcing customers on insurance, retail, banking, and telecommunications industries on defining, implementing or improving their IT security policies, assisting on IT audits and vulnerability analysis, and remediation.



19 September 2013

Overview

IBM GPFS is a high-performance enterprise file management platform widely used on commercial and scientific applications. As part of the GPFS configuration, it is required that remote connections for the root user are enabled so that the nodes that are part of the GPFS cluster can communicate among them to run the commands that require root privileges.

Allowing remote root connections is not recommended as a security best practice. Typically, root login connections must be done exclusively from the console, or through a method that maintains accountability to an individual of such connections (for example, sudo).

Actions must be taken in order to mitigate the risk that this situation represents to the whole environment.

This article describes the changes that can be applied to the Secure Shell (SSH) configuration file on each one of the GPFS nodes so that remote login operations with the root user are allowed to be established only from the GPFS nodes.

A basic GPFS network configuration

The following figure shows a basic GPFS network configuration. It consists of two nodes, and each one of them has configured two network cards: one is connected to the corporate network and the other has been assigned to be used for GPFS purposes.

Figure 1. A basic GPFS network configuration
A network diagram that shows two GPFS nodes and their IP addresses

Both nodes are running the Linux® operating system (the recommendations of this article can be applied to any UNIX®-like operating system) and the communication mechanism among nodes was determined to be SSH [GPFS supports other communication mechanisms such as remote shell (rsh)].

The sshd_config file is the configuration file for SSH, and the parameter that allows or denies root connections is called PermitRootLogin. For typical GPFS installations the value for this parameter is yes, as shown in the following example.

# grep ^PermitRootLogin /etc/ssh/sshd_config
PermitRootLogin yes
#

This value applies globally to all the SSH behaviour, meaning that the remote root login connections are allowed on all the network interfaces on which SSH is listening (commonly on all the available).


Risk scenarios

Recording and maintaining access logs helps to associate the responsibility of actions taken on any computer system by individuals accessing them. By allowing root remote login connections as required by GPFS, a door is let to be open to potential attacks or non-authorized access. Accountability of the user who accessed the system using root credentials is lost and investigations are harder to perform.

The following scenarios show some of the situations that might occur on systems that are not configured to take proper actions to mitigate the risk allowing remote root login connections.

Scenario 1: Individuals in possession of the root password

An individual can be knowledgeable of the root's user password by approved means (for example, a system administrator with a valid business need) or by unauthorized ways (for example, an ex-employee who kept this information and the new system administrator did not change the password right away, or someone who found it on an unprotected text file on a shared folder).

With this information on the user’s possession, the user can start a remote connection from the corporate network to both GPFS nodes with the root identity.

Listing 1. Connection to Node A
$ ssh -l root 172.19.230.35
root@172.19.230.35's password: **********
Last login: Mon Jul 29 11:14:07 2013 from gpfsnodeb
=== Welcome to GPFS Node A
#
Listing 2. Connection to Node B
$ ssh -l root 172.19.230.36
root@172.19.230.36's password: **********
Last login: Mon Jul 29 11:30:16 2013 from gpfsnodea
=== Welcome to GPFS Node B
#

SSH connections with similar characteristics as explained earlier can be initiated from the nodes themselves. A user can start an SSH connection from the user’s personal computer or from other non-GPFS server with the user’s personal user ID (or a hijacked user ID in case this connection is part of an attack) to one of the nodes and soon after obtaining access, the user can jump to the other node using the root credentials.

Listing 3. Connecting to Node A and then jumping to Node B
$ ssh -l kath1406 172.19.230.35
kath1406@172.19.230.35's password: **********
Last login: Thu Aug 15 11:30:16 2013 from pc0212
=== Welcome to GPFS Node A
[kath1406@gpfsnodea ~]$
[kath1406@gpfsnodea ~]$ ssh -l root 192.168.10.13
root@192.168.10.13's password: **********
Last login: Mon Jul 29 13:26:19 2013 from gpfsnodea
=== Welcome to GPFS Node B
#
Listing 4. Connecting to Node B and then jumping to Node A
$ ssh -l kath1406 172.19.230.36
kath1406@172.19.230.35's password: **********
Last login: Thu Aug 15 11:03:10 2013 from pc0212
=== Welcome to GPFS Node B
[kath1406@gpfsnodeb ~]$
[kath1406@gpfsnodeb ~]$ ssh -l root 192.168.10.12
root@192.168.10.12's password: **********
Last login: Mon Jul 29 14:02:02 2013 from gpfsnodeb
=== Welcome to GPFS Node A
#

As it can be seen, remote root login connections are accepted no matter where they come from (the corporate network or the dedicated GPFS subnet), restrictions are non-existent and a potential risk of obtaining unauthorized root access is latent.

Figure 2. Unrestricted SSH connections can be started from any point in the network
A network diagram that shows how unrestricted SSH connections can be started from any point in the network

Scenario 2: Using stolen root SSH encryption keys

In order to enable GPFS to perform automated administrative tasks among the nodes using SSH, encryption keys are created and exchanged among the nodes. But, in GPFS installations, these encryption keys are typically created without a passphrase, thus automation gets preeminence over security.

An individual who has had previous root access to the GPFS nodes can make a copy of the root's encryption keys and further use them to gain unauthorized root access.

Because encryption keys are not changed as frequently as passwords, the inherent risk is that the user in possession of the keys would be able to continue accessing the nodes even when the root password had been changed.

Encryption keys are usually located in the .ssh subdirectory under the home directory of the user to whom the keys belong. In the case of the root user, they can be located in the /root/.ssh folder. By default, the ssh-keygen command used to create the keys uses the Rivest-Shamir-Adleman (RSA) algorithm. Then, a copy of the id_rsa and id_rsa.pub files (which contain the private and public keys respectively) can be made for further use.

After the keys have been copied, the root private key can be used to log in to the server from the user’s computer.

Listing 5. Connection to Node A using stolen root SSH encryption keys
$ eval `ssh-agent`
Agent pid 22047
$ ssh-add gpfsnodea_id_rsa
Identity added: gpfsnodea_id_rsa (gpfsnodea_id_rsa)
$ ssh -l root 172.19.230.35
Last login: Thu Aug 15 13:27:35 2013 from gpfsnodeb
=== Welcome to GPFS Node A
#
Listing 6. Connection to Node B using stolen root SSH encryption keys
$ eval `ssh-agent`
Agent pid 24040
$ ssh-add gpfsnodeb_id_rsa
Identity added: gpfsnodeb_id_rsa (gpfsnodeb_id_rsa)
$ ssh -l root 172.19.230.36
Last login: Thu Aug 15 13:30:12 2013 from gpfsnodea
=== Welcome to GPFS Node B
#

As it can be seen, by using the stolen root SSH encryption keys, a password to access the nodes is not even required. The interchange of the SSH encryption keys among the nodes defines a trust relationship. Therefore, even if the root password is changed, the connections continue to be established regardless of what the root password is.

Scenario 3: Injecting rogue SSH encryption keys

In the previous scenario it was shown how, by using previously created SSH encryption keys for the root user, unauthorized privileged access can be achieved using them. It was assumed that the required SSH configuration for the use of encryption keys for remote connections was already in place as part of the GPFS installation pre-requisites.

This scenario covers the situation when a malicious user injects the user’s own created encryption keys in order to gain root access to the GPFS nodes. It is assumed that this user has had previous access to the nodes with root authority in order to perform this task.

The first step is the creation of a pair of encryption keys. The keys can be created on the user's personal computer on one of the GPFS nodes or from other non-GPFS server. The ssh-keygen command has to be run as shown in the following listing (this example uses an empty passphrase as required by a default GPFS installation):

Listing 7. Creation of rogue SSH encryption keys
$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/left173/.ssh/id_rsa): ./rogue_rsa
Enter passphrase (empty for no passphrase): (Just type the 'Enter' key)
Enter same passphrase again: (Type the 'Enter' key again)
Your identification has been saved in ./rogue_rsa
Your public key has been saved in ./rogue_rsa.pub
The key fingerprint is:
c2:ae:ac:af:dd:71:35:35:39:c1:07:b2:f8:cc:71:82 left173@pc2604
The key's randomart image is:
+--[RSA 2048]----+
|         ..o.   |
|        o o.o.  |
|       E + *.   |
|     .  + = o   |
|      o S *     |
|     . . . .    |
|      o .       |
|   o o o        |
|  o+= .         |
+----------------+
$

Next step is to install the rogue public encryption key rogue_rsa.pub on each of the GPFS nodes. It has to be transmitted or copied to the /root/.ssh directory on each of the GPFS nodes. Then, it must be appended to the end of the authorized_keys file, as shown in the following listing.

Listing 8. Injecting the rogue SSH public key on GPFS nodes
# pwd
/home/root/.ssh
# cat rogue_id_rsa.pub >> authorized_keys
# rm rogue_id_rsa.pub
#

The authorized_keys file maintains a list of public encryption keys that are allowed to be used to connect to the server. This is where the trust relationship is established among the GPFS nodes and in general, with anyone who tries to connect to the servers.

After accomplishing these steps, the malicious user would be prepared to log in as a root user, as shown in the following listing.

Listing 9. Connection to Node A using rogue encryption keys
$ eval `ssh-agent`
Agent pid 25036
$ ssh-add rogue_id_rsa
Identity added: rogue_id_rsa (rogue_id_rsa)
$ ssh -l root 172.19.230.35
Last login: Thu Aug 15 14:15:00 2013 from gpfsnodeb
=== Welcome to GPFS Node A
#
Listing 10. Connection to Node B using rogue encryption keys
$ eval `ssh-agent`
Agent pid 25140
$ ssh-add rogue_id_rsa
Identity added: rogue_id_rsa (rogue_id_rsa)
$ ssh -l root 172.19.230.36
Last login: Thu Aug 15 14:15:00 2013 from gpfsnodea
=== Welcome to GPFS Node B
#

By using this new pair of rogue keys, it does not matter whether the root password changes or its encryption keys change, now the malicious user has a way to enter the servers regardless of any change on the access credentials of the root user.


Mitigating the risk by strengthening the SSH configuration

The three scenarios depicted previously are just examples of the behaviors that malicious users can perform by misusing a configuration that is required by GPFS to work properly.

More sophisticated alternatives that can take advantage of this backdoor can exist or might be developed in the future. Our objective is to be prepared for unknown current and future malicious techniques and mitigate the exposure to a reasonable extent.

The rationale followed is that allowing remote root login is mandatory for GPFS installations (as per August 2013) to work properly, and therefore, this configuration must be kept. But also, it can be restricted through SSH configuration, specifying which IP addresses are allowed to connect directly with root and which are not allowed. Those IP addresses belonging to the dedicated GPFS subnet (see Figure 1) will be allowed to connect directly with root, and other IPs will continue to be able to connect to the SSH service but not directly with root.

OpenSSH and the Match directive

The Match directive only can be used on OpenSSH versions later than 4.3p2.

Controlling conditionally remote root login connections

The Match directive allows to include a conditional validation. If the condition established is met, then the parameters that would have been defined to have a certain value can be changed here.

This directive can be configured in multiple ways and can receive different arguments, but as the objective is to restrict connections coming from certain IPs, focus will be given to the Address argument.

Using the Match directive

According to Figure 1, the IP addresses for the GPFS dedicated subnetwork are 192.168.10.12 (Node A) and 192.168.10.13 (Node B), and therefore, users starting connections from these IP addresses would be able to log in remotely with the root user ID. It is important to consider that any Match directive must be included at the very end of the sshd_config configuration file.

The SSH configuration for Node A and Node B would be as shown in Listing 11 and Listing 12.

Listing 11. SSH configuration for Node A
# egrep -n "^PermitRootLogin|Match" sshd_config
38:PermitRootLogin no
114:Match Address 192.168.10.13
115:PermitRootLogin yes
#
Listing 12. SSH configuration for Node B
# egrep -n "^PermitRootLogin|Match" sshd_config
38:PermitRootLogin no
114:Match Address 192.168.10.12
115:PermitRootLogin yes
#

Notice that the whole contents of the sshd_config file is not shown, and instead, the egrep command is used to look for the strings PermitRootLogin and Match inside the file. (the -n flag is used to show the line numbers where the strings are found and the numbers returned by egrep are not part of the contents of the file).

On line 38 of the configuration files for both nodes, notice that the PermitRootLogin parameter has been set to a value of no. Because this configuration appears practically at the beginning of the file, this is considered as a global setting. Therefore, the value applies to all connections, which means that by default any remote root login connection is rejected. This is the recommended value to have.

Activating the SSH configuration

It is important to consider that the SSH server has to be restarted (for example in Linux by running the service sshd restart command) in order to activate the changes made on the configuration file.

Nevertheless in line 114 (at the end of both files), the Match directive appears and receives as arguments the string Address and the IP address on the dedicated GPFS subnet for their each counterpart cluster node.

If the IP address from which a connection trying to log in directly with the root ID is listed in the Match Address directive, then the PermitRootLogin value is yes (as it is defined in line 115), which overrides the global value defined in line 38 and allows the connection coming from the authorized IP address to log in with the root ID.

If the GPFS cluster consists of more than two nodes, then multiple stanzas of the Match Address directive can be declared, each one containing the IP address of each one of the cluster's nodes.

Using exclusively encryption keys authentication

When the PermitRootLogin parameter is configured to the value yes, both authentication methods can be used to connect to the SSH server by providing the root password or by using its SSH encryption keys.

Because GPFS requires that some automated processes run and connect to other nodes with the root identity, the encryption keys authentication is usually the alternative chosen and configured. GPFS will not require to use the root password to start connections, and therefore, PermitRootLogin can be configured to the without-password value.

At first sight, it might seem to cause confusion to have this value configured, but instead, it is actually a more stringent value because it disables the password authentication for the root ID, leaving the encryption key authentication method as the only option allowed for root. The SSH configuration file would contain the code in Listing 13.

Listing 13. Restricting root connections to use only encryption keys
# egrep -n "^PermitRootLogin|Match" sshd_config
38:PermitRootLogin no
114:Match Address 192.168.10.12
115:PermitRootLogin without-password
#
Figure 3. Remote root login connections are restricted to be available just for the dedicated GPFS IP addresses
A network diagram that shows how remote root login connections are restricted to be available just for the dedicated GPFS IP addresses.

The GPFS adminMode configuration parameter

In GPFS version 3.3 or later, the adminMode configuration parameter has been introduced in order to control and limit the number of nodes that can run administrative commands and therefore need to directly access the root user remotely.

There are two possible values available for the adminMode parameter: allToall and central.

When adminMode is set to allToall, GPFS understands that all the nodes have access among them, and therefore, all of them can run administrative commands on their cluster counterparts.

When adminMode is set to the value central, GPFS acts as if the node on which the command is being run is the only one with root access to all of the other nodes. This allows additional control over which GPFS nodes can run administrative commands.

Notice that by setting adminMode to central, security is strengthened even more, but this parameter itself does not interfere on how the SSH service behaves, and therefore, both configurations (adminMode parameter from the GPFS side and the Match directive from the SSH side) should be used in conjunction.


Conclusions and recommendations

By implementing conditional configuration of the PermitRootLogin parameter through the Match directive in SSH, the risk of allowing remote root login connections from non-authorized or non GPFS cluster servers or networks is mitigated, and access is only granted to those who strictly need it. Malicious users from non-authorized locations trying any of the scenarios depicted previously will not be able to connect directly with root.

Notice that controlling the login access to the root user is not the only conditional control that can be used with the SSH Match directive. Also, the solution described in this article is not exclusive for GPFS. Any system or installation that requires this kind of granular control over SSH remote accesses can benefit from its implementation.

The ultimate objective is to balance productivity, automation, and security that must be associated with them. They cannot been seen as separate and confronted positions. This article shows that having a productive and a more secure GPFS environment than those installed with the default configurations is perfectly achievable.

Recommendations

  • Dedicate a specific subnet solely for GPFS communication purposes if possible.
    • This can help to control and limit the IP addresses that can use the remote root login configuration.
    • GPFS can use more than one network in order to allow greater network throughput by concurrently communicating with its daemons running on the cluster nodes. If network performance is an issue, then ensure that you configure only the authorized GPFS nodes' IP addresses with their correspondent Match directive on the sshd_config configuration file of each cluster node.
  • Use the adminMode parameter of GPFS in order to limit the nodes that can run administrative commands on all the other nodes on the cluster. This adds another security layer to the GPFS installation.
  • Restrict conditionally remote root login connections with SSH to those strictly required by implementing the Match directive.
  • Update your OpenSSH server if its version is lower or equal to 4.3p2 in order to be able to use the Match directive.
  • Generate a different pair of root's SSH encryption keys for each GPFS node.
    • Treat encryption keys as passwords: they must be different for each system.
    • This reduces the risk of unauthorized access to multiple systems with the same credentials.
  • Make sure that the root's encryption keys are accessible only to the root user ID.
    • Ensure that the permissions of the root's private key are 700 or more stringent.
  • Regularly create new SSH root's encryption keys and discard the old ones.
    • As it is recommended with passwords, encryption keys for automated processes should be changed periodically in order to reduce the probability of the occurrence of a situation that is similar the one depicted in Scenario 2.
  • Periodically check the contents of the authorized_keys file.
    • Unauthorized keys found on the file must be deleted and investigations about the configuration files contents' changes must be initiated.
    • The probability of the occurrence of a situation similar to the one depicted in Scenario 3 is reduced.
  • Periodically review the contents of the sshd_config file.
    • Check if unauthorized changes have been made on the Match directives.
    • Check if new IP addresses have been added to allow remote root login connections or if the global PermitRootLogin parameter has been modified from no to yes.
  • Additional network access restrictions to those configured on the SSH service through the Match directive can be done by implementing (either or both) tcpwrappers or local firewall (for example, iptables) access controls on each one of the GPFS nodes.
  • Implement privilege access controls as sudo in order to avoid the indiscriminate disclosure of the root password among several individuals. This helps to reduce the probability of the occurrence of a situation similar to the one depicted in Scenario 1.
    • Logging of privileged activities performed can be kept and accountability to an individual can be tracked.
    • Sudo events can be sent to a syslog server so that if security has been compromised locally on any of the GPFS nodes, the activity logs would reside on an external and independent server for further analysis and forensic investigation.

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into AIX and Unix on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=AIX and UNIX
ArticleID=945221
ArticleTitle=Strengthening the SSH configuration used in GPFS installations to make them more secure
publish-date=09192013