Topic
  • 9 replies
  • Latest Post - ‏2014-01-14T23:45:19Z by ufa
ufa
ufa
148 Posts

Pinned topic security: remote shell as root

‏2013-12-05T11:34:02Z |

Hi,

there is a number of people who dislike the idea of direct root access through ssh. Confronting them with exactly that requirement of GPFS raises their eyebrows, usually.

However, the GPFS documentation is very clear here: At least one system in the cluster must be able to execute commands as root on all others via a remote shell program (such as rsh or ssh, the latter will be the widely chosen one).

The applicable restrictions I'd think of is restricting ssh access for root on the cluster nodes to originate from the admin node, forbidding any other connections.

However, compromising that admin machine is opening full root access to all other systems in the cluster. One can discuss whether there is anything to save if guys have hacked into one node anyway, but the question stands nevertheless: What is the "safest" setup of GPFS WRT remote shell access which is still operational ?

My bet is:

adminMode=central

admin machine is (the) one with valuable data (as it might be protected best anyway) - chosing any other would nevertheless compromise it due to the remote access for troot.

on all nodes, restrict root ssh access to originate from the admin machine only.

Use a long rsa key.

Anything else to do, or any better approaches?

 

ufa

 

Updated on 2013-12-05T11:34:36Z at 2013-12-05T11:34:36Z by ufa
  • yuri
    yuri
    303 Posts
    ACCEPTED ANSWER

    Re: security: remote shell as root

    ‏2013-12-05T21:23:29Z  

    Things are not quite so bleak.  The current GPFS code has this requirement: batch (i.e. passwordless/promptless) remote shell access to all nodes must be available from the tty where an mm command is running, for the duration of the command execution.  These two points are pretty important.  You don't have to authorize an entire node at all times.  With adminMode=central, GPFS won't do any remote shell command execution unless a sysadmin runs an mm command.  The intent here is: one could set up ssh public/private key authentication, and protect the private key with a passphrase.  ssh-agent could be used to input the passphrase and authorize a given tty to execute ssh commands without prompting.  Once you're done, the tty can be closed, and so is the window for remote command execution.  If someone hacks into this host and steals the private key, it won't be catastrophic, because the key is still protected by the passphrase.  

    This is a pretty slippery slope, of course.  If someone could steal your ssh private key, it's pretty hard to tell what other damage has been done and what else got compromised -- all bets are off in this case, pretty much.  While having a batch ssh channel open for spreading the infection obviously makes it easier for the attacker, not having such a channel will stop a script kiddie but not a skilled hacker, who would perhaps install a quality rootkit and wait patiently for you to type enough passwords in sudo.

    yuri

     

  • HajoEhlers
    HajoEhlers
    253 Posts
    ACCEPTED ANSWER

    Re: security: remote shell as root

    ‏2013-12-06T14:24:52Z  
    • ufa
    • ‏2013-12-06T13:34:24Z

    I might not completely understand the use of containers here. There is at least one system, which has to (temporarily / potentially) be able to root-access all other GPFS nodes, i.e. entities of OSs running an mmfsd in order to mount one or more mmfs-type file systems. I am not sure how containers might help here - still there is one system which needs to root-access all others at times. I am not too familiar with the containers, it might be they do not run an mmfsd of their own but use the host system for vfs access. But that is not the issue - also in a "normal" setup without containers those machines would not be priviledged.  The container approach cannot eliminate the existence of one priviledged machine though - and this is the concern. Of course I am with you that setting up that node in a way that no normal user has access to it lowers the risk of exploits being used, but if a malicious guy has rooted the priviledged machine, the container hosts are vulnerable (assuming they run the mmfsd, mount the GPFS filesystems and pass them on to the containers somehow), and thus the containers as well: I do not know whether root on the container host might manipulate the containers to get access to their data space, but the attacker could at least plainly shutdown/crash/destroy the containers. The unspecific paranoia is not of mine but of the customer. We all know that to make a system secure against hacks and attacks one has to switch it off completely and prevent it from being switched on (a milder form is to cut off all communication lines to the system). It is also clear that large organizations tend to establish quite static rules without thinking of each individual case.

    As I said - I might have a misconception of how containers can help here.

    ufa

     

    See a Container as a chroot environment.

    Example: 

    Node1 GPFS Network  - VLAN 100 , IP 10.1.0.1/29 ( Admin Node )

    Node2 GPFS Network - VLAN 100 , IP 10.1.0.2/29 ( Client Node )

    Serving LXC1 - VLAN 200 , IP 192.1.1.1/..... ( User Node )

    Node3 GPFS Network - VLAN 100 , IP 10.1.0.3/29 ( Client Node )

    Serving LXC2 - VLAN 200 , IP 192.1.1.2/.....  (User Node )

     

    Access:

    Admin -> Node1 -> NodeX -> LXCx ) 

    Client -> LXCxy 

    Attack vector for the client:

    1. Connecting to the NodeXY is not possible at all ( Env. is not even known )

    2. Connecting to LXCx

     - in case the container is save only DOS are possible even in case the attacker gets root rights within the container.

     - in case the container is un save, the attacker could break out of the container and reach the underlying system. Root access could be archived but attacker could not jump to the Admin Node ( Node 1 ).  SSH access from Node[2,3] to the Admin Node could be even disabled via a firewall role.

    With an 2 Cluster setup it would be even more secure and more flexible since between the 2 cluster no root access is needed at all.

    IO cluster <-> Remote Cluster ( root squash ) /  LXC <- Client

    but the attacker could at least plainly shutdown/crash/destroy the containers

    Thats a different issue and in case your customer does not want any risk he should disable access AT ALL to this system. Best would be to provide NO service at all. No service no risk ^_^

    A talk about secure linux containers - http://lwn.net/Articles/515034/ or search for "secure linux container"

  • YannickBergeron
    YannickBergeron
    35 Posts
    ACCEPTED ANSWER

    Re: security: remote shell as root

    ‏2013-12-17T19:23:16Z  

    This: PermitRootLogin without-password

    would be better than: PermitRootLogin yes

     

    But this: PermitRootLogin forced-commands-only

    would be even better, but you would need a list of all commands that can be executed in the cluster and add them in the root authorized_keys

    It would be great if GPFS support could provide this list of command per GPFS level

     

    Best regards,

    Yannick Bergeron

    Also take a look at this article
    http://www.ibm.com/developerworks/aix/library/au-aix-modifying-ssh-configuration/index.html?ca=drs

  • HajoEhlers
    HajoEhlers
    253 Posts

    Re: security: remote shell as root

    ‏2013-12-05T13:58:42Z  

    > dislike the idea of direct root access through ssh

    You should point out to these people that on a normal computer cluster processes with root right even communicating directly ( RDMA ) WITHOUT the usage of ssh at ALL...... scnr

    Depending of the paranoia level you could run an IO cluster ( Define it as a single machine )  and access is it  via a remote  cluster ( with no local disks )

    This way you can have 2 security domains with different security level

    On the remote cluster ( Or even on the IO Cluster )  you run  kerberos RSH .... Does somebody tried to use kerberos  rsh in combination with GPFS ?

    BTW:  In case you use for the remote cluster admin mode central you take after the initial  configuration the master node offline. No administration at all is possible.....

    More ideas ?

     

     

  • yuri
    yuri
    303 Posts

    Re: security: remote shell as root

    ‏2013-12-05T21:23:29Z  

    Things are not quite so bleak.  The current GPFS code has this requirement: batch (i.e. passwordless/promptless) remote shell access to all nodes must be available from the tty where an mm command is running, for the duration of the command execution.  These two points are pretty important.  You don't have to authorize an entire node at all times.  With adminMode=central, GPFS won't do any remote shell command execution unless a sysadmin runs an mm command.  The intent here is: one could set up ssh public/private key authentication, and protect the private key with a passphrase.  ssh-agent could be used to input the passphrase and authorize a given tty to execute ssh commands without prompting.  Once you're done, the tty can be closed, and so is the window for remote command execution.  If someone hacks into this host and steals the private key, it won't be catastrophic, because the key is still protected by the passphrase.  

    This is a pretty slippery slope, of course.  If someone could steal your ssh private key, it's pretty hard to tell what other damage has been done and what else got compromised -- all bets are off in this case, pretty much.  While having a batch ssh channel open for spreading the infection obviously makes it easier for the attacker, not having such a channel will stop a script kiddie but not a skilled hacker, who would perhaps install a quality rootkit and wait patiently for you to type enough passwords in sudo.

    yuri

     

  • ufa
    ufa
    148 Posts

    Re: security: remote shell as root

    ‏2013-12-06T09:50:28Z  
    • yuri
    • ‏2013-12-05T21:23:29Z

    Things are not quite so bleak.  The current GPFS code has this requirement: batch (i.e. passwordless/promptless) remote shell access to all nodes must be available from the tty where an mm command is running, for the duration of the command execution.  These two points are pretty important.  You don't have to authorize an entire node at all times.  With adminMode=central, GPFS won't do any remote shell command execution unless a sysadmin runs an mm command.  The intent here is: one could set up ssh public/private key authentication, and protect the private key with a passphrase.  ssh-agent could be used to input the passphrase and authorize a given tty to execute ssh commands without prompting.  Once you're done, the tty can be closed, and so is the window for remote command execution.  If someone hacks into this host and steals the private key, it won't be catastrophic, because the key is still protected by the passphrase.  

    This is a pretty slippery slope, of course.  If someone could steal your ssh private key, it's pretty hard to tell what other damage has been done and what else got compromised -- all bets are off in this case, pretty much.  While having a batch ssh channel open for spreading the infection obviously makes it easier for the attacker, not having such a channel will stop a script kiddie but not a skilled hacker, who would perhaps install a quality rootkit and wait patiently for you to type enough passwords in sudo.

    yuri

     

    Thanks (to both of you),

    if the ssh path is not needed permanently (depending on automated tasks like scheduled waiter state checking) then Kerberos might indeed be another step to tighten the setup. Are there any experiences with using Kerberos here?

    Are there any changes planned for GPFS WRT root-authorized remote commands?  I could imagine that some special GPFS user could execute GPFS admin commands via a special restricted shell (which would only allow for that mm* or ts* subset of commands), in order to relieve the direct root login requirement.

    ufa

     

  • HajoEhlers
    HajoEhlers
    253 Posts

    Re: security: remote shell as root

    ‏2013-12-06T12:25:33Z  
    • ufa
    • ‏2013-12-06T09:50:28Z

    Thanks (to both of you),

    if the ssh path is not needed permanently (depending on automated tasks like scheduled waiter state checking) then Kerberos might indeed be another step to tighten the setup. Are there any experiences with using Kerberos here?

    Are there any changes planned for GPFS WRT root-authorized remote commands?  I could imagine that some special GPFS user could execute GPFS admin commands via a special restricted shell (which would only allow for that mm* or ts* subset of commands), in order to relieve the direct root login requirement.

    ufa

     

    Please forget about the root requirement and step back and be aware what your GPFS cluster is. A service for data access.

    But it is not a must for the user to access this service directly. 

    On AIX system you can provide WPARs(*) and on Linux LXC(*)   which decouples(**) the GPFS service from the User and 

    So a user would not access the GPFS node directly but the container the node is providing. ( Using namefs (AIX) and bind (LX) mount )

    In such a setup with admin node central and the admin node does NOT even serve any user access i would feel quite well regarding the security.

    To force a requirement of:  "no remote root access" without questioning what problem - since there might no problem at all - it shall solve is IMHO pretty stupid.

    happy weekend

    Hajo

     *  Both are container type environment.

    ** With KVM you might be able in the near future to access a GPFS "native" from a VM. This would solve IMHO almost all security constrains. Except your security requirement is equal to lock and protect every door in your flat.

     

  • ufa
    ufa
    148 Posts

    Re: security: remote shell as root

    ‏2013-12-06T13:34:24Z  

    Please forget about the root requirement and step back and be aware what your GPFS cluster is. A service for data access.

    But it is not a must for the user to access this service directly. 

    On AIX system you can provide WPARs(*) and on Linux LXC(*)   which decouples(**) the GPFS service from the User and 

    So a user would not access the GPFS node directly but the container the node is providing. ( Using namefs (AIX) and bind (LX) mount )

    In such a setup with admin node central and the admin node does NOT even serve any user access i would feel quite well regarding the security.

    To force a requirement of:  "no remote root access" without questioning what problem - since there might no problem at all - it shall solve is IMHO pretty stupid.

    happy weekend

    Hajo

     *  Both are container type environment.

    ** With KVM you might be able in the near future to access a GPFS "native" from a VM. This would solve IMHO almost all security constrains. Except your security requirement is equal to lock and protect every door in your flat.

     

    I might not completely understand the use of containers here. There is at least one system, which has to (temporarily / potentially) be able to root-access all other GPFS nodes, i.e. entities of OSs running an mmfsd in order to mount one or more mmfs-type file systems. I am not sure how containers might help here - still there is one system which needs to root-access all others at times. I am not too familiar with the containers, it might be they do not run an mmfsd of their own but use the host system for vfs access. But that is not the issue - also in a "normal" setup without containers those machines would not be priviledged.  The container approach cannot eliminate the existence of one priviledged machine though - and this is the concern. Of course I am with you that setting up that node in a way that no normal user has access to it lowers the risk of exploits being used, but if a malicious guy has rooted the priviledged machine, the container hosts are vulnerable (assuming they run the mmfsd, mount the GPFS filesystems and pass them on to the containers somehow), and thus the containers as well: I do not know whether root on the container host might manipulate the containers to get access to their data space, but the attacker could at least plainly shutdown/crash/destroy the containers. The unspecific paranoia is not of mine but of the customer. We all know that to make a system secure against hacks and attacks one has to switch it off completely and prevent it from being switched on (a milder form is to cut off all communication lines to the system). It is also clear that large organizations tend to establish quite static rules without thinking of each individual case.

    As I said - I might have a misconception of how containers can help here.

    ufa

     

  • HajoEhlers
    HajoEhlers
    253 Posts

    Re: security: remote shell as root

    ‏2013-12-06T14:24:52Z  
    • ufa
    • ‏2013-12-06T13:34:24Z

    I might not completely understand the use of containers here. There is at least one system, which has to (temporarily / potentially) be able to root-access all other GPFS nodes, i.e. entities of OSs running an mmfsd in order to mount one or more mmfs-type file systems. I am not sure how containers might help here - still there is one system which needs to root-access all others at times. I am not too familiar with the containers, it might be they do not run an mmfsd of their own but use the host system for vfs access. But that is not the issue - also in a "normal" setup without containers those machines would not be priviledged.  The container approach cannot eliminate the existence of one priviledged machine though - and this is the concern. Of course I am with you that setting up that node in a way that no normal user has access to it lowers the risk of exploits being used, but if a malicious guy has rooted the priviledged machine, the container hosts are vulnerable (assuming they run the mmfsd, mount the GPFS filesystems and pass them on to the containers somehow), and thus the containers as well: I do not know whether root on the container host might manipulate the containers to get access to their data space, but the attacker could at least plainly shutdown/crash/destroy the containers. The unspecific paranoia is not of mine but of the customer. We all know that to make a system secure against hacks and attacks one has to switch it off completely and prevent it from being switched on (a milder form is to cut off all communication lines to the system). It is also clear that large organizations tend to establish quite static rules without thinking of each individual case.

    As I said - I might have a misconception of how containers can help here.

    ufa

     

    See a Container as a chroot environment.

    Example: 

    Node1 GPFS Network  - VLAN 100 , IP 10.1.0.1/29 ( Admin Node )

    Node2 GPFS Network - VLAN 100 , IP 10.1.0.2/29 ( Client Node )

    Serving LXC1 - VLAN 200 , IP 192.1.1.1/..... ( User Node )

    Node3 GPFS Network - VLAN 100 , IP 10.1.0.3/29 ( Client Node )

    Serving LXC2 - VLAN 200 , IP 192.1.1.2/.....  (User Node )

     

    Access:

    Admin -> Node1 -> NodeX -> LXCx ) 

    Client -> LXCxy 

    Attack vector for the client:

    1. Connecting to the NodeXY is not possible at all ( Env. is not even known )

    2. Connecting to LXCx

     - in case the container is save only DOS are possible even in case the attacker gets root rights within the container.

     - in case the container is un save, the attacker could break out of the container and reach the underlying system. Root access could be archived but attacker could not jump to the Admin Node ( Node 1 ).  SSH access from Node[2,3] to the Admin Node could be even disabled via a firewall role.

    With an 2 Cluster setup it would be even more secure and more flexible since between the 2 cluster no root access is needed at all.

    IO cluster <-> Remote Cluster ( root squash ) /  LXC <- Client

    but the attacker could at least plainly shutdown/crash/destroy the containers

    Thats a different issue and in case your customer does not want any risk he should disable access AT ALL to this system. Best would be to provide NO service at all. No service no risk ^_^

    A talk about secure linux containers - http://lwn.net/Articles/515034/ or search for "secure linux container"

  • YannickBergeron
    YannickBergeron
    35 Posts

    Re: security: remote shell as root

    ‏2013-12-16T20:35:49Z  

    This: PermitRootLogin without-password

    would be better than: PermitRootLogin yes

     

    But this: PermitRootLogin forced-commands-only

    would be even better, but you would need a list of all commands that can be executed in the cluster and add them in the root authorized_keys

    It would be great if GPFS support could provide this list of command per GPFS level

     

    Best regards,

    Yannick Bergeron

  • YannickBergeron
    YannickBergeron
    35 Posts

    Re: security: remote shell as root

    ‏2013-12-17T19:23:16Z  

    This: PermitRootLogin without-password

    would be better than: PermitRootLogin yes

     

    But this: PermitRootLogin forced-commands-only

    would be even better, but you would need a list of all commands that can be executed in the cluster and add them in the root authorized_keys

    It would be great if GPFS support could provide this list of command per GPFS level

     

    Best regards,

    Yannick Bergeron

    Also take a look at this article
    http://www.ibm.com/developerworks/aix/library/au-aix-modifying-ssh-configuration/index.html?ca=drs

  • ufa
    ufa
    148 Posts

    Re: security: remote shell as root

    ‏2014-01-14T23:45:19Z  

    Thanks to all for your answers,

    I think I see a bit clearer now what can be done to get that remote root access requirement fulfilled with lower risk.

    ufa