Topic
12 replies Latest Post - ‏2013-08-15T03:18:00Z by Eric Covener
XVQG_Anwarul_Azim
XVQG_Anwarul_Azim
15 Posts
ACCEPTED ANSWER

Pinned topic LIb security error issue during plugin loadin for liberty cluster

‏2013-07-25T17:17:00Z |

Hi,

I have a liberty cluster and I generated the plugin (attached) using the admin tool from liberty cluster. I have now installed IBM http server and trying to forward request to liberty cluster. When I load the plugin, it works for http but not for https. I get error as like below in plugin.log file: I have included ssl feature in each cluster member too.

Please let me know how can I solve this issue.

[25/Jul/2013:13:05:28.03875] 000033fe bf6acb80 - ERROR: lib_security: logSSLError: str_security (gsk error 408):  GSK_ERROR_BAD_KEYFILE_PASSWORD
[25/Jul/2013:13:05:28.03881] 000033fe bf6acb80 - ERROR: lib_security: initializeSecurity: Failed to initialize GSK environment. Secure transports are not possible.
[25/Jul/2013:13:05:28.03882] 000033fe bf6acb80 - ERROR: ws_transport: transportInitializeSecurity: Failed to initialize security. Secure transports are not possible.
[25/Jul/2013:13:05:28.03887] 000033fe bf6acb80 - ERROR: ws_server: serverAddTransport: Failed to initialize security. Secure transports are not possible.
[25/Jul/2013:13:05:28.03889] 000033fe bf6acb80 - ERROR: ws_server: serverAddTransport: HTTPS Transport is skipped. IMPORTANT: If a HTTP transport is defined, it will be used for communication to the application server.
 

Attachments

  • Eric Covener
    Eric Covener
    16 Posts
    ACCEPTED ANSWER

    Re: LIb security error issue during plugin loadin for liberty cluster

    ‏2013-07-26T12:21:49Z  in response to XVQG_Anwarul_Azim

    Your generated plugin-cfg.xml references a keyring, plugin-key.kdb. You're responsible for creating it and making it trust whoever signed the certificates for your liberty servers (if you use collectives, that will usually be a single CA). 

    The password error means the file is either corrupted or the corresponding plugin-key.sth does not have the correct password stashed to it. 

    ikeyman in any IBM JRE can create and manipulate the keystore.

    • XVQG_Anwarul_Azim
      XVQG_Anwarul_Azim
      15 Posts
      ACCEPTED ANSWER

      Re: LIb security error issue during plugin loadin for liberty cluster

      ‏2013-08-07T05:26:13Z  in response to Eric Covener

      Hi,

      I have created WASplugin.kdb using IKEYMAN following link here:http://pic.dhe.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=%2Fcom.ibm.websphere.express.doc%2Finfo%2Fexp%2Fae%2Ftsec_httpserv.html

      and placed this key and password file under /root/host1/wlp/usr/servers/keys/newkeys/. Attached defaultCluster-plugin-cfg.xml has this location for key kdb file and sth file.

      Now, I have added the following code in my IBM HTTP server conf file for SSL connection and restarted the httpd server. The log file shows the plugin is loaded. KeyFile and SSLStashFile have the values for their location.

      LoadModule ibm_ssl_module modules/mod_ibm_ssl.so
      Listen 443
      <VirtualHost dl160:443>
       ServerName dl160
       SSLEnable
       SSLClientAuth none
       </VirtualHost>
       SSLDisable
       KeyFile /root/host1/wlp/usr/servers/keys/newkeys/WASplugin.kdb
       SSLStashFile /root/host1/wlp/usr/servers/keys/newkeys/WASplugin.sth

      But now if I want to use the sample application: https://dl160:443/snoop or https://dl160/snoop. I get page error after trusting the self certificate.

      I am not sure what step I am missing or what is the wrong for configuration. Is there any tutorial for configuring liberty servers with IBM httpd server to get SSL connection.

       

      Thanks-Anwarul

       

       

      • Eric Covener
        Eric Covener
        16 Posts
        ACCEPTED ANSWER

        Re: LIb security error issue during plugin loadin for liberty cluster

        ‏2013-08-07T12:42:08Z  in response to XVQG_Anwarul_Azim

        The likely causes is that the WAS plugin did not trust the backend liberts servers certificate.

         

        1) update the plugin-cfg.xml to point to a fully qualified http_plugin.log

        2) review the KeyFile in plugin-cfg.xml and confirm you've create a keystore and stash file in the path actually encoed in plugin-cfg.xml (it doesn't match the file of the same name you also referenced in httpd.conf)

        3) Try again

         

        4) review the http_plugin.log to see if there's an error loading the security library (should be rare for Plugin running inside of IHS) or if it is a more specific handshake error. Either way, post here in plain text.

        • XVQG_Anwarul_Azim
          XVQG_Anwarul_Azim
          15 Posts
          ACCEPTED ANSWER

          Re: LIb security error issue during plugin loadin for liberty cluster

          ‏2013-08-07T22:00:43Z  in response to Eric Covener

          Thanks Eric for your reply.  I am still having the issues following all the steps you mentioned.

          I have only linux CLI environment and that's why I am using CLI for generating keys/certificate. The steps I followed as follows:

          1) I have WAS plugin installed in /etc/wasplugin/ directory. I went to the directory /etc/wasplugin/java/jre/bin to generate keys and self certificate as like following:

          [root@dl160 bin]# ./ikeycmd -keydb -create -db plugin-key.kdb -pw "pac" -type cms -expire 365 -stash
          [root@dl160 bin]# ./ikeycmd -cert -create -db plugin-key.kdb  -pw "pac" -size 1024 -dn "CN=dl160, O=IBM" -label liberty -default_cert yes

          Then I copied plugin-key.kdb and plugin-key.sth file to /etc/wasplugin/etc/ directory.

          2) My liberty plugin for webserver has all the configuration for loading plugin-key.kdb, plugin-key.sth file. defaultCluster-plugin-cfg.xml is attached here. It has also fully qualified http_plugin.log name as you mentioned.

          3) I have also added this code snippet to 2 of my cluster members including the controller:

          <pluginConfiguration webserverPort="80"
                                webserverSecurePort="443"
                                sslKeyringLocation="/etc/wasplugin/etc/plugin-key.kdb"
                                sslStashfileLocation="/etc/wasplugin/etc/plugin-key.sth"
                                />
           

          4) Restarted the liberty cluster and then restarted the ibm http server but I still get same issues (screenshot1.jpg and screenshot2.jpg). Screen1.jpg is logical since I signed the certificate myself.

          After trying to browse the site: https://dl160/snoop, I get the following error in http_plugin.log file at the end

           

          [07/Aug/2013:17:54:33.20890] 00004964 85188b80 - PLUGIN: Plugins loaded.
          [07/Aug/2013:17:54:33.20898] 00004964 85188b80 - PLUGIN: --------------------System Information-----------------------
          [07/Aug/2013:17:54:33.20901] 00004964 85188b80 - PLUGIN: Bld version: 8.5.0
          [07/Aug/2013:17:54:33.20903] 00004964 85188b80 - PLUGIN: Bld date: May  7 2013, 16:05:23
          [07/Aug/2013:17:54:33.20905] 00004964 85188b80 - PLUGIN: Webserver: IBM_HTTP_Server
          [07/Aug/2013:17:54:33.20907] 00004964 85188b80 - PLUGIN: OS : Linux x86_64
          [07/Aug/2013:17:54:33.20909] 00004964 85188b80 - PLUGIN: Hostname = dl160
          [07/Aug/2013:17:54:33.20911] 00004964 85188b80 - PLUGIN: NOFILES = hard: 1024, soft: 1024
          [07/Aug/2013:17:54:33.20913] 00004964 85188b80 - PLUGIN: MAX COREFILE SZ = hard: INFINITE, soft: 0
          [07/Aug/2013:17:54:33.20915] 00004964 85188b80 - PLUGIN: DATA = hard: INFINITE, soft: INFINITE
          [07/Aug/2013:17:54:33.20917] 00004964 85188b80 - PLUGIN: --------------------------------------------------------------
          [07/Aug/2013:17:54:47.82862] 00004968 43810940 - ERROR: lib_stream: openStream: Failed in r_gsk_secure_soc_init: GSK_ERROR_BAD_CERT(gsk rc = 414) PARTNER CERTIFICATE DN=CN=dl160.eng.platformlab.ibm.com,OU=member2,O=ibm,C=us, Serial=54:28:c7:3f:7b:16
          [07/Aug/2013:17:54:47.82875] 00004968 43810940 - ERROR: ws_common: websphereGetStream: Could not open stream
          [07/Aug/2013:17:54:47.82881] 00004968 43810940 - ERROR: ws_common: websphereExecute: Failed to create the stream
          [07/Aug/2013:17:54:47.82884] 00004968 43810940 - ERROR: ws_server: serverSetFailoverStatus: Marking default_node_defaultServer0_0 down
          [07/Aug/2013:17:54:47.82886] 00004968 43810940 - ERROR: ws_common: websphereHandleRequest: Failed to execute the transaction to 'default_node_defaultServer0_0'on host 'dl160.eng.platformlab.ibm.com'; will try another one
          [07/Aug/2013:17:54:47.84077] 00004968 43810940 - ERROR: lib_stream: openStream: Failed in r_gsk_secure_soc_init: GSK_ERROR_BAD_CERT(gsk rc = 414) PARTNER CERTIFICATE DN=CN=dl160.eng.platformlab.ibm.com,OU=member1,O=ibm,C=us, Serial=53:4c:f1:15:99:1d
          [07/Aug/2013:17:54:47.84086] 00004968 43810940 - ERROR: ws_common: websphereGetStream: Could not open stream
          [07/Aug/2013:17:54:47.84091] 00004968 43810940 - ERROR: ws_common: websphereExecute: Failed to create the stream
          [07/Aug/2013:17:54:47.84094] 00004968 43810940 - ERROR: ws_server: serverSetFailoverStatus: Marking default_node_defaultServer0_1 down
          [07/Aug/2013:17:54:47.84096] 00004968 43810940 - ERROR: ws_common: websphereHandleRequest: Failed to execute the transaction to 'default_node_defaultServer0_1'on host 'dl160.eng.platformlab.ibm.com'; will try another one
          [07/Aug/2013:17:54:47.84098] 00004968 43810940 - ERROR: ws_common: websphereWriteRequestReadResponse: Failed to find an app server to handle this request
          [07/Aug/2013:17:54:47.84100] 00004968 43810940 - ERROR: ESI: getResponse: failed to get response: rc = 2
          [07/Aug/2013:17:54:47.84103] 00004968 43810940 - ERROR: ws_common: websphereHandleRequest: Failed to handle reques

           

          I get the following in my  error_log file after accessing at the end.

          [Wed Aug 07 17:29:09 2013] [notice] WebSphere Plugins loaded.
          [Wed Aug 07 17:29:09 2013] [notice] Bld version: 8.5.0
          [Wed Aug 07 17:29:09 2013] [notice] Bld date: May  7 2013, 16:05:38
          [Wed Aug 07 17:29:09 2013] [notice] Webserver: IBM_HTTP_Server
          [Wed Aug 07 17:29:09 2013] [notice] Using GSKit version 8.0.14.27
          [Wed Aug 07 17:29:09 2013] [notice] Using config file /opt/IBM/ihs85/conf/httpd.conf
          [Wed Aug 07 17:29:09 2013] [notice] IBM_HTTP_Server/8.5.5.0 (Unix) configured -- resuming normal operations
          [Wed Aug 07 17:29:09 2013] [notice] Core file limit is 0; core dumps will be not be written for server crashes
          [Wed Aug 07 17:29:41 2013] [error] [client 9.29.143.251] File does not exist: /opt/IBM/ihs85/htdocs/favicon.ico

          What steps I am missing or what should I do to fix this issue?

          Thanks-Anwarul

          • Eric Covener
            Eric Covener
            16 Posts
            ACCEPTED ANSWER

            Re: LIb security error issue during plugin loadin for liberty cluster

            ‏2013-08-08T02:30:27Z  in response to XVQG_Anwarul_Azim

            GSK_ERROR_BAD_CERT in the plugin log means the WAS plugin (really its configured KDB) does not trust the issuer of the liberty servers certificate.

            If this liberty is part of a collective, you can trust the root self-signed CA presented any time you connect to the controller over SSL. The members' certs are all signed by the same root CA, so only 1 entry is needed.

            I usually just use openssl s_client -connect example.com:9443 -showcerts and copy/paste the self-signed cert returned into a file. Then ikeycmd -cert -add can add it to the plugins KDB as trusted.

            If this is a standalone liberty server, just do the same to any HTTPS port that is active.

            • XVQG_Anwarul_Azim
              XVQG_Anwarul_Azim
              15 Posts
              ACCEPTED ANSWER

              Re: LIb security error issue during plugin loadin for liberty cluster

              ‏2013-08-08T05:09:55Z  in response to Eric Covener

              Thanks for your reply.

              Did you mean the following steps:

              1) To copy the output from the command "openssl s_client -connect example.com:9443 -showcerts" in a file like e.g (plugin-key.kdb)

              2) ikeycmd -cert -add dB plugin-key.kdb -pw pac -label liberty -format binary -trust enable  -file <file>

              Is my upper command correct? What should I use as file name here.

              3) As I know, I need to refer same kth and ssh file in httpd.conf and defaultCluster-plugin-cfg.xml file. So will I get those two files from step 1 and step 2?

              Thanks-Anwarul

               

               

               

              • XVQG_Anwarul_Azim
                XVQG_Anwarul_Azim
                15 Posts
                ACCEPTED ANSWER

                Re: LIb security error issue during plugin loadin for liberty cluster

                ‏2013-08-08T19:16:43Z  in response to XVQG_Anwarul_Azim

                I copied the output from the command #openssl s_client -connect dl160:9443 -showcerts >>liberty.kdb

                into a file named liberty.kdb It looks the it is properly copied because the following command shows the issuer.

                [root@dl160 bin]# openssl x509 -noout -in liberty.kdb -issuer
                issuer= /DC=com.ibm.ws.collective/O=98ca040c-b69d-469c-8f42-6f75e0ef3e4b/OU=controllerRoot

                Now, I tried with  ikeycmd with different options but no luck....

                [root@dl160 bin]# ./ikeycmd -cert -add  liberty.kdb

                Unknown parameter 'liberty.kdb'.

                -Correct command usage-
                -db or -crypto                      Required
                -relativeSlotNumber or -tokenlabel  Required if -crypto present
                -file                               Required
                -label                              Optional
                -pw                                 Optional
                -type                               Optional if -db present <cms | jceks | jks | kdb | p12 | pkcs12>
                -format                             Optional  <ascii | binary>
                -trust                              Optional  <disable | enable>
                -secondaryDB                        Optional if -crypto present
                -secondaryDBpw                      Optional if -secondaryDB present
                -secondaryDbType                    Optional if -secondaryDB present <cms | jceks | jks | kdb | p12 | pkcs12>

                 

                [root@dl160 bin]# ./ikeycmd -cert -add -db liberty.kdb

                A required value for the command was not specified:
                -file                               Required

                -Correct command usage-
                -db or -crypto                      Required
                -relativeSlotNumber or -tokenlabel  Required if -crypto present
                -file                               Required
                -label                              Optional
                -pw                                 Optional
                -type                               Optional if -db present <cms | jceks | jks | kdb | p12 | pkcs12>
                -format                             Optional  <ascii | binary>
                -trust                              Optional  <disable | enable>
                -secondaryDB                        Optional if -crypto present
                -secondaryDBpw                      Optional if -secondaryDB present
                -secondaryDbType                    Optional if -secondaryDB present <cms | jceks | jks | kdb | p12 | pkcs12>

                [root@dl160 bin]# ./ikeycmd -cert -add -db liberty.kdb -file f1.file //f1.file is random file name

                Cannot load keystore:
                Invalid KeyStore Format.

                Ensure that the keystore is valid and of the correct type.

                 




                 

                • Eric Covener
                  Eric Covener
                  16 Posts
                  ACCEPTED ANSWER

                  Re: LIb security error issue during plugin loadin for liberty cluster

                  ‏2013-08-08T23:46:53Z  in response to XVQG_Anwarul_Azim

                  You would copy and paste just the begin/end of the root CA to a file, not overwriting the .kdb file. Then pass that filename to the -cert -add command as the -file argument.

                  • XVQG_Anwarul_Azim
                    XVQG_Anwarul_Azim
                    15 Posts
                    ACCEPTED ANSWER

                    Re: LIb security error issue during plugin loadin for liberty cluster

                    ‏2013-08-09T22:32:47Z  in response to Eric Covener

                    Hi, I did the following steps to try https connection for liberty cluster with one controller and two members, but still no luck:

                     

                    Step1:

                    From my /etc/wasplugin/java/jre/bin/ directory I ran the ikeycmd command

                    [root@dl160 bin]# ../ikeycmd -keydb -create -db new-key1.kdb -pw "pac" -type cms -expire 365 -stash

                    Step2:


                    [root@dl160 bin]# ./ikeycmd -cert -create -db new-key1.kdb  -pw "pac" -size 1024 -dn "CN=dl160, O=IBM" -label liberty -default_cert yes

                    Step3:

                    Then copied the output begin/end of the root CA into keyfile.txt from the command #openssl s_client -connect dl160:9443 -showcerts

                    Step4:

                    [root@dl160 bin]# ./ikeycmd -cert -add -db new-key1.kdb -file keyfile.txt -pw pac -trust enable

                    Step5: I have one controller (controller1) and two member servers (member1 and member2). I copied two files new-key1.kdb and new-key1.sth file to all these collective members /root/host1/wlp/usr/servers/member1/resources, /root/host1/wlp/usr/servers/member2/resources, /root/host1/wlp/usr/servers/member1/resources, /root/host1/wlp/usr/servers/controller1/resources.

                    Step6:  Added the following code to all server.xml files for (member1, member2, controller1)

                    <pluginConfiguration webserverPort="80"
                                          webserverSecurePort="443"
                                          sslKeyringLocation="${server.config.dir}/resources/security/new-key1.kdb"
                                          sslStashfileLocation="${server.config.dir}/resources/security/new-key1.sth"
                                          />
                     

                    Step7: Restarted the cluster and generated the plugin by provided jythin script:

                    jython /root/platform/lab-materials/sample_scripts/genClusterPlugin/genClusterPlugin.py defaultCluster --host=dl160.eng.platformlab.ibm.com --port=9443 --user=admin --password=adminpwd --truststore=/root/host1/wlp/usr/servers/controller1/resources/security/trust.jks --truststorePassword=password

                    Step8: Copied the generated plugin file defaultCluster-plugin-cfg.xml into my  ibm httpd server conf directory (/opt/IBM/ihs85/conf) . The plugin file is attached with this post.

                     

                    Step9:  in httpd.conf file I added following code snippet

                    LoadModule was_ap22_module /etc/wasplugin/bin/64bits/mod_was_ap22_http.so
                    WebSpherePluginConfig  "/opt/IBM/ihs85/conf/defaultCluster-plugin-cfg.xml"

                     

                    LoadModule ibm_ssl_module modules/mod_ibm_ssl.so
                    Listen 443
                    <VirtualHost dl160:443>
                    ServerName  dl160.eng.platformlab.ibm.com
                    SSLClientAuth None
                    SSLEnable
                    #SSLServerCert default
                    </VirtualHost>
                    SSLDisable
                    Keyfile "/etc/wasplugin/java/jre/bin/new-key1.kdb"
                    SSLStashFile "/etc/wasplugin/java/jre/bin/new-key1.sth"

                    Step10:

                    I modified the following line in  defaultCluster-plugin-cfg.xml

                    <Log LogLevel="Trace" Name="/opt/IBM/ihs85/logs/http_plugin.log"/>

                    Step11:

                    Restarted httpd ./apachectl stop and ./apachectl start

                    But unfortunately I still get same issue as error when I try to browse https://dl160/snoop .

                    Please let me know what I am doing wrong. I believe I followed all the necessary steps to get https connection for liberty cluster. I appreciate your help.

                     

                    Thanks-Anwarul

                     

                    [09/Aug/2013:17:55:30.82812] 00001bca 42b53940 - DEBUG: ws_common: websphereGetStream: Setting socket to non-block for ServerIOTimeout over HTTP
                    [09/Aug/2013:17:55:30.82815] 00001bca 42b53940 - DEBUG: ws_common: websphereGetStream: socket 13 connected to dl160.eng.platformlab.ibm.com:9445 timeout=900
                    [09/Aug/2013:17:55:30.82818] 00001bca 42b53940 - DEBUG: lib_stream: openStream: Opening the SSL stream soc=13
                    [09/Aug/2013:17:55:30.82906] 00001bca 42b53940 - TRACE: lib_stream: openStream: setting GSK_USER_DATA (timeout=900)
                    [09/Aug/2013:17:55:30.83034] 00001bca 42b53940 - TRACE: lib_rio: Blocking for read, waiting 900
                    [09/Aug/2013:17:55:30.84329] 00001bca 42b53940 - ERROR: lib_stream: openStream: Failed in r_gsk_secure_soc_init: GSK_ERROR_BAD_CERT(gsk rc = 414) PARTNER CERTIFICATE DN=CN=dl160.eng.platformlab.ibm.com,OU=member2,O=ibm,C=us, Serial=54:28:c7:3f:7b:16
                    [09/Aug/2013:17:55:30.84333] 00001bca 42b53940 - DEBUG: lib_stream: destroyStream: Destroying the stream
                    [09/Aug/2013:17:55:30.84343] 00001bca 42b53940 - ERROR: ws_common: websphereGetStream: Could not open stream
                    [09/Aug/2013:17:55:30.84350] 00001bca 42b53940 - DEBUG: ws_common: websphereGetStream: socket 13 closed - failed to open stream
                    [09/Aug/2013:17:55:30.84352] 00001bca 42b53940 - ERROR: ws_common: websphereExecute: Failed to create the stream
                    [09/Aug/2013:17:55:30.84354] 00001bca 42b53940 - ERROR: ws_server: serverSetFailoverStatus: Marking default_node_defaultServer0_0 down
                    [09/Aug/2013:17:55:30.84356] 00001bca 42b53940 - STATS: ws_server: serverSetFailoverStatus: Server default_node_defaultServer0_0 : pendingRequests 0 failedRequests 1 affinityRequests 0 totalRequests 0.
                     

                     

                     

                     

                     

                    • Eric Covener
                      Eric Covener
                      16 Posts
                      ACCEPTED ANSWER

                      Re: LIb security error issue during plugin loadin for liberty cluster

                      ‏2013-08-09T22:53:55Z  in response to XVQG_Anwarul_Azim

                      Your keyring references /opt/IBM/ihs8.5/keys/WASplugin.kdb but you customized one by a different name.  I am not very familiar with the plugin-cfg.xml generation, or cluster merge, in Liberty -- but if you have the ability to override the path to the KDB, it would need to be relative to the webservers local filesystem. In other words, I do not think the KDB's get collected and merged during that operation

                       

                      So perhaps your override A) was not interpreted correctly, but I am not sure the value you specified would have been useful at runtime in your webserver.  Ultimately you can see in the plugin-cfg.xml you must have a local path to your customized *.kdb in each <Server> stanza.

                      • XVQG_Anwarul_Azim
                        XVQG_Anwarul_Azim
                        15 Posts
                        ACCEPTED ANSWER

                        Re: LIb security error issue during plugin loadin for liberty cluster

                        ‏2013-08-14T21:54:18Z  in response to Eric Covener

                        I am having exactly the same issue for liberty as it is mentioned for WAS.

                        https://www.ibm.com/developerworks/community/forums/html/topic?id=77777777-0000-0000-0000-000013957703

                        The link mentioned here to setup the keys is not working. Is there other link/guideline so that I can follow.

                         

                        • Eric Covener
                          Eric Covener
                          16 Posts
                          ACCEPTED ANSWER

                          Re: LIb security error issue during plugin loadin for liberty cluster

                          ‏2013-08-15T03:18:00Z  in response to XVQG_Anwarul_Azim

                          If you're still getting a 414 error, you just need to establish trust in the keyfile actually being used by the WAS plugin.  There is no tooling for it in liberty, beyond the certificate management toos provided with IHS and the WAS plugin.