Topic
  • 4 replies
  • Latest Post - ‏2018-12-12T17:10:56Z by yiannis.gkoufas
RenataZakirova
RenataZakirova
2 Posts

Pinned topic Authorized_ccr_keys Error

‏2015-11-24T14:20:17Z | authorization ccr error gpf spectrumscale

Dear colleagues.

We've installed IBM Spectrum Scale on RHEL 7.1 LE. But during the cluster creation process we have the following error:

root@bi_mngmnt_node ssl]# mmcrcluster -N /tmp/gpfs/nodes.lst -p bi_mngmnt_node -C BI_Cluster -r /usr/bin/ssh -R /usr/bin/scp -A --ccr-enable                                                                            I_cluster -r /usr/bin/ssh -R /usr/bin/scp -A
mmcrcluster: Performing preliminary node verification ...
mmcrcluster: Processing quorum and other critical nodes ...
bi_mngmnt_node: authentication refresh for node 2 (bi_data_node1) failed (err 79)
bi_mngmnt_node: authentication refresh for node 3 (bi_data_node2) failed (err 79)
bi_mngmnt_node: authentication refresh for node 4 (bi_data_node3) failed (err 79)
bi_mngmnt_node: authentication refresh for node 5 (bi_data_node4) failed (err 79)
bi_mngmnt_node: authentication refresh for node 6 (bi_data_node5) failed (err 79)
bi_mngmnt_node: authentication refresh for node 7 (bi_data_node6) failed (err 79)
bi_mngmnt_node: authentication initialization failed (err 79)
bi_mngmnt_node: ccr server nodes to be initialized:
bi_mngmnt_node: 1 172.17.204.164:1191 bi_mngmnt_node
bi_mngmnt_node: 2 172.17.204.165:1191 bi_data_node1
bi_mngmnt_node: 3 172.17.204.166:1191 bi_data_node2
bi_mngmnt_node: 4 172.17.204.167:1191 bi_data_node3
bi_mngmnt_node: 5 172.17.204.168:1191 bi_data_node4
bi_mngmnt_node: 6 172.17.204.169:1191 bi_data_node5
bi_mngmnt_node: 7 172.17.204.170:1191 bi_data_node6
bi_data_node1: authentication refresh for node 1 (bi_mngmnt_node) failed (err 79)
bi_data_node1: authentication refresh for node 3 (bi_data_node2) failed (err 79)
bi_data_node1: authentication refresh for node 4 (bi_data_node3) failed (err 79)
bi_data_node1: authentication refresh for node 5 (bi_data_node4) failed (err 79)
bi_data_node1: authentication refresh for node 6 (bi_data_node5) failed (err 79)
bi_data_node1: authentication refresh for node 7 (bi_data_node6) failed (err 79)
bi_data_node1: authentication initialization failed (err 79)
bi_data_node1: ccr server nodes to be initialized:
bi_data_node1: 1 172.17.204.164:1191 bi_mngmnt_node
bi_data_node1: 2 172.17.204.165:1191 bi_data_node1
bi_data_node1: 3 172.17.204.166:1191 bi_data_node2
bi_data_node1: 4 172.17.204.167:1191 bi_data_node3
bi_data_node1: 5 172.17.204.168:1191 bi_data_node4
bi_data_node1: 6 172.17.204.169:1191 bi_data_node5
bi_data_node1: 7 172.17.204.170:1191 bi_data_node6
bi_data_node2: authentication refresh for node 1 (bi_mngmnt_node) failed (err 79)
bi_data_node2: authentication refresh for node 2 (bi_data_node1) failed (err 79)
bi_data_node2: authentication refresh for node 4 (bi_data_node3) failed (err 79)
bi_data_node2: authentication refresh for node 5 (bi_data_node4) failed (err 79)
bi_data_node2: authentication refresh for node 6 (bi_data_node5) failed (err 79)
bi_data_node2: authentication refresh for node 7 (bi_data_node6) failed (err 79)
bi_data_node2: authentication initialization failed (err 79)
bi_data_node2: ccr server nodes to be initialized:
bi_data_node2: 1 172.17.204.164:1191 bi_mngmnt_node
bi_data_node2: 2 172.17.204.165:1191 bi_data_node1
bi_data_node2: 3 172.17.204.166:1191 bi_data_node2
bi_data_node2: 4 172.17.204.167:1191 bi_data_node3
bi_data_node2: 5 172.17.204.168:1191 bi_data_node4
bi_data_node2: 6 172.17.204.169:1191 bi_data_node5
bi_data_node2: 7 172.17.204.170:1191 bi_data_node6
bi_data_node3: authentication refresh for node 1 (bi_mngmnt_node) failed (err 79)
bi_data_node3: authentication refresh for node 2 (bi_data_node1) failed (err 79)
bi_data_node3: authentication refresh for node 3 (bi_data_node2) failed (err 79)
bi_data_node3: authentication refresh for node 5 (bi_data_node4) failed (err 79)
bi_data_node3: authentication refresh for node 6 (bi_data_node5) failed (err 79)
bi_data_node3: authentication refresh for node 7 (bi_data_node6) failed (err 79)
bi_data_node3: authentication initialization failed (err 79)
bi_data_node3: ccr server nodes to be initialized:
bi_data_node3: 1 172.17.204.164:1191 bi_mngmnt_node
bi_data_node3: 2 172.17.204.165:1191 bi_data_node1
bi_data_node3: 3 172.17.204.166:1191 bi_data_node2
bi_data_node3: 4 172.17.204.167:1191 bi_data_node3
bi_data_node3: 5 172.17.204.168:1191 bi_data_node4
bi_data_node3: 6 172.17.204.169:1191 bi_data_node5
bi_data_node3: 7 172.17.204.170:1191 bi_data_node6
bi_data_node4: authentication refresh for node 1 (bi_mngmnt_node) failed (err 79)
bi_data_node4: authentication refresh for node 2 (bi_data_node1) failed (err 79)
bi_data_node4: authentication refresh for node 3 (bi_data_node2) failed (err 79)
bi_data_node4: authentication refresh for node 4 (bi_data_node3) failed (err 79)
bi_data_node4: authentication refresh for node 6 (bi_data_node5) failed (err 79)
bi_data_node4: authentication refresh for node 7 (bi_data_node6) failed (err 79)
bi_data_node4: authentication initialization failed (err 79)
bi_data_node4: ccr server nodes to be initialized:
bi_data_node4: 1 172.17.204.164:1191 bi_mngmnt_node
bi_data_node4: 2 172.17.204.165:1191 bi_data_node1
bi_data_node4: 3 172.17.204.166:1191 bi_data_node2
bi_data_node4: 4 172.17.204.167:1191 bi_data_node3
bi_data_node4: 5 172.17.204.168:1191 bi_data_node4
bi_data_node4: 6 172.17.204.169:1191 bi_data_node5
bi_data_node4: 7 172.17.204.170:1191 bi_data_node6
bi_data_node5: authentication refresh for node 1 (bi_mngmnt_node) failed (err 79)
bi_data_node5: authentication refresh for node 2 (bi_data_node1) failed (err 79)
bi_data_node5: authentication refresh for node 3 (bi_data_node2) failed (err 79)
bi_data_node5: authentication refresh for node 4 (bi_data_node3) failed (err 79)
bi_data_node5: authentication refresh for node 5 (bi_data_node4) failed (err 79)
bi_data_node5: authentication refresh for node 7 (bi_data_node6) failed (err 79)
bi_data_node5: authentication initialization failed (err 79)
bi_data_node5: ccr server nodes to be initialized:
bi_data_node5: 1 172.17.204.164:1191 bi_mngmnt_node
bi_data_node5: 2 172.17.204.165:1191 bi_data_node1
bi_data_node5: 3 172.17.204.166:1191 bi_data_node2
bi_data_node5: 4 172.17.204.167:1191 bi_data_node3
bi_data_node5: 5 172.17.204.168:1191 bi_data_node4
bi_data_node5: 6 172.17.204.169:1191 bi_data_node5
bi_data_node5: 7 172.17.204.170:1191 bi_data_node6
bi_data_node6: authentication refresh for node 1 (bi_mngmnt_node) failed (err 79)
bi_data_node6: authentication refresh for node 2 (bi_data_node1) failed (err 79)
bi_data_node6: authentication refresh for node 3 (bi_data_node2) failed (err 79)
bi_data_node6: authentication refresh for node 4 (bi_data_node3) failed (err 79)
bi_data_node6: authentication refresh for node 5 (bi_data_node4) failed (err 79)
bi_data_node6: authentication refresh for node 6 (bi_data_node5) failed (err 79)
bi_data_node6: authentication initialization failed (err 79)
bi_data_node6: ccr server nodes to be initialized:
bi_data_node6: 1 172.17.204.164:1191 bi_mngmnt_node
bi_data_node6: 2 172.17.204.165:1191 bi_data_node1
bi_data_node6: 3 172.17.204.166:1191 bi_data_node2
bi_data_node6: 4 172.17.204.167:1191 bi_data_node3
bi_data_node6: 5 172.17.204.168:1191 bi_data_node4
bi_data_node6: 6 172.17.204.169:1191 bi_data_node5
bi_data_node6: 7 172.17.204.170:1191 bi_data_node6
mmcrcluster: Processing quorum and other critical nodes ...
mmcrcluster: Removing GPFS cluster files from the nodes in the cluster . . .
mmcrcluster: Command failed. Examine previous error messages to determine cause.

The Node.lst file:

bi_mngmnt_node:quorum-manager
bi_data_node1:quorum
bi_data_node2:quorum
bi_data_node3:quorum
bi_data_node4:quorum
bi_data_node5:quorum
bi_data_node6:quorum

Passwordless SSH verification:

[root@bi_mngmnt_node ~]# sh script_sshTest.sh
Testing bi_mngmnt_node to bi_mngmnt_node: Tue Nov 24 17:19:15 MSK 2015
Testing bi_mngmnt_node to bi_data_node1: Tue Nov 24 17:19:16 MSK 2015
Testing bi_mngmnt_node to bi_data_node2: Tue Nov 24 17:19:16 MSK 2015
Testing bi_mngmnt_node to bi_data_node3: Tue Nov 24 17:19:15 MSK 2015
Testing bi_mngmnt_node to bi_data_node4: Tue Nov 24 17:19:16 MSK 2015
Testing bi_mngmnt_node to bi_data_node5: Tue Nov 24 17:19:16 MSK 2015
Testing bi_mngmnt_node to bi_data_node6: Tue Nov 24 17:19:17 MSK 2015
Testing bi_data_node1 to bi_mngmnt_node: Tue Nov 24 17:19:17 MSK 2015
Testing bi_data_node1 to bi_data_node1: Tue Nov 24 17:19:18 MSK 2015
Testing bi_data_node1 to bi_data_node2: Tue Nov 24 17:19:18 MSK 2015
Testing bi_data_node1 to bi_data_node3: Tue Nov 24 17:19:17 MSK 2015
Testing bi_data_node1 to bi_data_node4: Tue Nov 24 17:19:18 MSK 2015
Testing bi_data_node1 to bi_data_node5: Tue Nov 24 17:19:18 MSK 2015
Testing bi_data_node1 to bi_data_node6: Tue Nov 24 17:19:19 MSK 2015
Testing bi_data_node2 to bi_mngmnt_node: Tue Nov 24 17:19:19 MSK 2015
Testing bi_data_node2 to bi_data_node1: Tue Nov 24 17:19:20 MSK 2015
Testing bi_data_node2 to bi_data_node2: Tue Nov 24 17:19:20 MSK 2015
Testing bi_data_node2 to bi_data_node3: Tue Nov 24 17:19:19 MSK 2015
Testing bi_data_node2 to bi_data_node4: Tue Nov 24 17:19:20 MSK 2015
Testing bi_data_node2 to bi_data_node5: Tue Nov 24 17:19:20 MSK 2015
Testing bi_data_node2 to bi_data_node6: Tue Nov 24 17:19:21 MSK 2015
Testing bi_data_node3 to bi_mngmnt_node: Tue Nov 24 17:19:21 MSK 2015
Testing bi_data_node3 to bi_data_node1: Tue Nov 24 17:19:22 MSK 2015
Testing bi_data_node3 to bi_data_node2: Tue Nov 24 17:19:22 MSK 2015
Testing bi_data_node3 to bi_data_node3: Tue Nov 24 17:19:22 MSK 2015
Testing bi_data_node3 to bi_data_node4: Tue Nov 24 17:19:22 MSK 2015
Testing bi_data_node3 to bi_data_node5: Tue Nov 24 17:19:22 MSK 2015
Testing bi_data_node3 to bi_data_node6: Tue Nov 24 17:19:23 MSK 2015
Testing bi_data_node4 to bi_mngmnt_node: Tue Nov 24 17:19:23 MSK 2015
Testing bi_data_node4 to bi_data_node1: Tue Nov 24 17:19:24 MSK 2015
Testing bi_data_node4 to bi_data_node2: Tue Nov 24 17:19:24 MSK 2015
Testing bi_data_node4 to bi_data_node3: Tue Nov 24 17:19:24 MSK 2015
Testing bi_data_node4 to bi_data_node4: Tue Nov 24 17:19:24 MSK 2015
Testing bi_data_node4 to bi_data_node5: Tue Nov 24 17:19:24 MSK 2015
Testing bi_data_node4 to bi_data_node6: Tue Nov 24 17:19:25 MSK 2015
Testing bi_data_node5 to bi_mngmnt_node: Tue Nov 24 17:19:25 MSK 2015
Testing bi_data_node5 to bi_data_node1: Tue Nov 24 17:19:26 MSK 2015
Testing bi_data_node5 to bi_data_node2: Tue Nov 24 17:19:26 MSK 2015
Testing bi_data_node5 to bi_data_node3: Tue Nov 24 17:19:26 MSK 2015
Testing bi_data_node5 to bi_data_node4: Tue Nov 24 17:19:26 MSK 2015
Testing bi_data_node5 to bi_data_node5: Tue Nov 24 17:19:26 MSK 2015
Testing bi_data_node5 to bi_data_node6: Tue Nov 24 17:19:27 MSK 2015
Testing bi_data_node6 to bi_mngmnt_node: Tue Nov 24 17:19:27 MSK 2015
Testing bi_data_node6 to bi_data_node1: Tue Nov 24 17:19:28 MSK 2015
Testing bi_data_node6 to bi_data_node2: Tue Nov 24 17:19:28 MSK 2015
Testing bi_data_node6 to bi_data_node3: Tue Nov 24 17:19:28 MSK 2015
Testing bi_data_node6 to bi_data_node4: Tue Nov 24 17:19:28 MSK 2015
Testing bi_data_node6 to bi_data_node5: Tue Nov 24 17:19:28 MSK 2015
Testing bi_data_node6 to bi_data_node6: Tue Nov 24 17:19:29 MSK 2015

Could you please assist what could it be? What should we do to resolve this issue?

Thank you in advance!

 

  • ScaleoutSean
    ScaleoutSean
    107 Posts

    Re: Authorized_ccr_keys Error

    ‏2015-11-25T09:42:23Z  

    1) Check firewall settings (see GPFS Install Guide)

    2) Make sure your /etc/hosts map to correct IP addresses (or check SSH manually, to same host names that the mmcrcluster tries to use)

  • RenataZakirova
    RenataZakirova
    2 Posts

    Re: Authorized_ccr_keys Error

    ‏2015-11-25T10:32:46Z  

    1) Check firewall settings (see GPFS Install Guide)

    2) Make sure your /etc/hosts map to correct IP addresses (or check SSH manually, to same host names that the mmcrcluster tries to use)

    Thank you for the reply!

    Your recommendation (Check firewall settings) helps us. We've done the following commands (on all nodes)

     firewall-cmd --permanent --add-port=1191/tcp
     firewall-cmd --permanent --add-port=22/tcp
     systemctl restart firewalld.service
    

    Helpful links:

    https://www-01.ibm.com/support/knowledgecenter/STXKQY_4.1.1/com.ibm.spectrum.scale.v4r11.ins.doc/bl1ins_protocolsprerequisites.htm

    http://www-01.ibm.com/support/knowledgecenter/STXKQY_4.1.1/com.ibm.spectrum.scale.v4r11.adv.doc/bl1adv_firewallforinternalcommn.htm

     

     

  • Vexata
    Vexata
    3 Posts

    Re: Authorized_ccr_keys Error

    ‏2018-02-22T02:00:22Z  

    mmcrcluster command tries to start the CCR server on the quorum nodes

    If the CCR server fails to start , you can get this error which is cryptic to understand.

    Ended up spending  time verifying the password less connection between nodes .

    After looking at  mmsdrserv.log, found that  CCR server fails to start with error

     

    mmsdrserv: /project/spreltac500/build/rtac5001749d/src/avs/fs/mmfs/ts/classes/basic/ticktime.C:97: void logAssertFailed(UInt32, const char*, UInt32, Int32, Int32, UInt32, const char*, const char*): Assertion `!"cl
    ock_gettimeP is NULL"' failed.

     

    This was because of missing  gcc libraries. Do yum install gcc* on all the nodes.
     

    Updated on 2018-02-22T02:01:27Z at 2018-02-22T02:01:27Z by Vexata
  • yiannis.gkoufas
    yiannis.gkoufas
    1 Post

    Re: Authorized_ccr_keys Error

    ‏2018-12-12T17:10:56Z  

    Hi,

    I am observing the same behavior when we run version 5.0.2.0 inside an ubuntu container.

    xxxxxxxxx: authentication refresh for node 1 (xxxxxxxxx) failed (err 79)
    xxxxxxxxx: authentication initialization failed (err 79)
    xxxxxxxxx: ccr server nodes to be initialized:
    xxxxxxxxx: 1 xxxxxxxxx:1191 xxxxxxxxx
    xxxxxxxxx: ccrctl:error:230:error from ccr init (rc=149):

    With version 5.0.0.0 we had it working without a problem.