IBM Support

Securing the Infosphere Information Server Zookeeper and Kafka services

Question & Answer


Question

How can I secure Apache zookeeper, kafka and solr services installed by the Infosphere Information Server?

Cause

After installing Governance Rollup Patch 1 on Infosphere Information Server 11.5 or after installing Infosphere Information Server, 11.5.0.1 or later versions, the newly installed Apache zookeeper, kafka and solr cloud services are running without security scheme. Each user who knows the hostname and the port number of the services is allowed to connect to them, add, update or delete data and metadata.

Answer

Prerequisite: this document applies to Infosphere Information Server version 11.5 with Governance Rollup Patch 1 and to later versions.

Conventions:

  • symbols in italic and in angle brackets are placeholders. Change them with the appropriate values of your configuration..
  • If not otherwise indicated, all paths given below are relative paths from the directory where the Shared Open Source patch is installed. When needed, this path is indicated below as <shared-open-source>. The default installation path is:
    AIX or Linux:
    cd /opt/IBM/InformationServer/shared-open-source
    Windows:
    cd C:\IBM\InformationServer\shared-open-source

Firewall

To prevent unallowed access to the Apache zookeeper, kafka and solr cloud services, we recommend to protect the access to the service ports through your firewall configuration. The configuration depends on the firewall you use and is not explained here.

You can find the port numbers in the configuration files of the services listed below.

  • If you have Infosphere Information Server version 11.5 with Governance Rollup Patch 1 or Infosphere Information Server version 11.5.0.1:
    - property port=59092 in shared-open-source/kafka/conf/server.properties
    - option -p 58983 in shared-open-source/solr/start-solr.*
    - property clientPort=52181 in shared-open-source/zookeeper/conf/zookeeper-config.cfg
  • If you have Infosphere Information Server version 11.5.0.1 with Governance Rollup Patch 2 or later, all ports are in the single configuration file conf/env.sh on Unix or conf\env.cmd on Windows. Refer to the values of variables KAFKA_PORT, SOLR_PORT and ZOOKEEPER_CLIENT_PORT.

Zookeeper

Additionally, zookeeper provides the ability to protect each zookeeper node with ACL (see documentation below), restricting the ability to read, write, create, delete or administrate rights to one or several authenticated users, hostnames or IP addresses. The two last possibilities would be an alternative to the firewall protection above. Be aware that the protection is not recursive and that all nodes of a subtree have to be protected in zookeeper and not only the root node of this subtree.



How to:
  1. define a user and a password for zookeeper, in this example zookeeper / ZooRocks. Get the digest of the password by executing following command from the Information Server installation directory:
    AIX or Linux:
    java -cp shared-open-source/zookeeper/install/zookeeper-3.4.6.jar:shared-open-source/zookeeper/install/lib/log4j-1.2.16.jar:shared-open-source/zookeeper/install/lib/slf4j-api-1.6.1.jar:shared-open-source/zookeeper/install/lib/slf4j-log4j12-1.6.1.jar org.apache.zookeeper.server.auth.DigestAuthenticationProvider zookeeper:ZooRocks
    zookeeper:ZooRocks->zookeeper:GbWYBq2bCcogf6pbZDKhIISoYpY=
    Windows:
    java -cp shared-open-source/zookeeper/install/zookeeper-3.4.6.jar;shared-open-source/zookeeper/install/lib/log4j-1.2.16.jar;shared-open-source/zookeeper/install/lib/slf4j-api-1.6.1.jar;shared-open-source/zookeeper/install/lib/slf4j-log4j12-1.6.1.jar org.apache.zookeeper.server.auth.DigestAuthenticationProvider zookeeper:ZooRocks
    zookeeper:ZooRocks->zookeeper:GbWYBq2bCcogf6pbZDKhIISoYpY=
  2. Connect to the zookeeper server by calling the zkCli utility (make it executable first on Unix).
    AIX or Linux:
    shared-open-source/zookeeper/install/bin/zkCli.sh -server localhost:52181
    Windows:
    shared-open-source\zookeeper\install\bin\zkCli.cmd -server localhost:52181
  3. authenticate yourself as the zookeeper user. (for this session)
    addauth digest zookeeper:ZooRocks
  4. Set ACL to the zookeeper nodes of your choice.
    setAcl / digest:zookeeper:GbWYBq2bCcogf6pbZDKhIISoYpY=:cdrwa,world:anyone:cr

In this example, we restrict the ability to create, modify or delete root-level nodes to the user zookeeper, all other users are allowed to list the root-level nodes.

Caution:
Please be aware that restricting zookeeper node access may break one or several services of the Shared Open Source patch, as well as client side applications of these services like the Information Server Data Quality Exception Console. Those services and applications do not authenticate themselves and use the zookeeper nodes with the authorization scheme "world".

Following nodes can be considered for restricting access:
/The root node can be protected against changes after that the Shared Open Source packages have been installed and started. All root-level nodes should have been created then.
/security.jsonThis node contains the authentication and authorization definition of the solr cloud service. It may be protected in order not to be overwritten or not to have its rights changed. The solr cloud REST API for authorization and authentication cannot be used afterwards to modify this file.
/collectionsThis node contains 2 solr collections that should not be deleted. Restricting the delete capabilities on this node may prevent unallowed deletion.
/brokers/topicsThis node contains kafka topics that should not be deleted. Restricting the delete capabilities on this node may prevent unallowed deletion.

Kafka


Starting with RUP7, Information Server uses Kafka 0.10.0.1 that provides security features such as SSL over the wire.

The following procedure describes how to enable SSL secured client to broker communication as well as how to enable SSL for Information Server Kafka events. In this example self signed certificates are used.

Note: This example is limited to a single node Kafka configuration. The Kafka broker configuration (server.properties) is overwritten by any subsequent RUP installation.

Procedure for enabling the Kafka broker SSL feature:

Generate SSL key and SSL certificate for the broker.

  1. Make sure you have at least openssl 1.0.2 installed and in the path. In order to check the version, use the following command:
    openssl version
  2. Generate the key in a temporary keystore in the desired location by using java. In this example the keystore is created in the current directory. The key is valid for 10 years.
    keytool -keystore kafka.server.keystore.jks -alias localhost -validity 3650 -genkey -keypass keypass -storepass storepass -dname "CN=mdmdemowin, OU=Data Quality, O=IBM, L=Boeblingen, S=BW, C=DE"
  3. Generate a certificate authority (CA) that is later used to sign certificates.
    openssl req -new -x509 -keyout ca-key -out ca-cert -days 3650 -passout pass:ca-password -subj "/C=DE/ST=BW/L=Boeblingen/O=IBM/OU=Data Quality/CN=mdmdemowin"
  4. Add the generated CA to the truststore.
    keytool -keystore kafka.server.truststore.jks -alias CARoot -import -file ca-cert -storepass storepass
  5. Sign the certificates in the keystore by using the CA generated before.
    • Export the certificate from the keystore.
      keytool -keystore kafka.server.keystore.jks -alias localhost -certreq -file cert-file -storepass storepass -keypass keypass
    • Sign the certificate.
      openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 3650 -CAcreateserial -passin pass:ca-password
  6. Import both the certificate of the CA and the signed certificate into the keystore.
    • Import the CA certificate.
      keytool -keystore kafka.server.keystore.jks -alias CARoot -import -file ca-cert -storepass storepass
    • Import the signed certificate.
      keytool -keystore kafka.server.keystore.jks -alias localhost -import -file cert-signed -storepass storepass -keypass keypass
  7. Move the keystore and truststore files to a location of your choice.
    move *store.jks C:\temp
  8. Configure the Kafka broker for secure client to broker communication.

    Update the server.properties file. The following example shows the default name and location.
    \InformationServer\shared-open-source\kafka\conf\server1.properties
    • Comment out or remove the port information:
      Change
      port=59092
      to
      #port=59092
    • Add the keystore and truststore information as well as the port information. In this example the SSL port is 59093. Adjust the passwords and file location as needed.

      listeners=PLAINTEXT://localhost:59092,SSL://localhost:59093
      ssl.keystore.location=C:/temp/kafka.server.keystore.jks
      ssl.keystore.password=storepass
      ssl.key.password=keypass
      ssl.truststore.location=C:/temp/kafka.server.truststore.jks
      ssl.truststore.password=storepass
    • Restart the Kafka service. To check whether the configuration was successful, use the following command:
      openssl s_client -connect localhost:59093 -tls1

      The output shows a successful connection and certificate details as provided during certificate creation.

      Example output:
      Certificate chain
      0 s:/C=DE/ST=BW/L=Boeblingen/O=IBM/OU=Data Quality/CN=mdmdemowin
      i:/C=DE/ST=BW/L=Boeblingen/O=IBM/OU=Data Quality/CN=mdmdemowin
All clients can now connect to port 59093 using SSL as protocol. A truststore with the self signed certificate needs to be present on the client. The following examples show how to connect using SSL.


The following examples show how Kafka clients can connect using secured client to broker communication.

Command line producer example:

The producer.config file needs to be enhanced as follows:
security.protocol=SSL
ssl.truststore.location=C:/temp/kafka.server.truststore.jks
ssl.truststore.password=storepass


kafka-console-producer --broker-list localhost:59093 --topic test --producer.config ..\..\config\producer.properties

JAVA client connection example:

Properties props = new Properties();



props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, “brokerHost:brokerPort");
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,StringSerializer.class.getName());
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,StringSerializer.class.getName());
//SSL
props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, SecurityProtocol.SSL.name());
props.put("ssl.truststore.location","C:/temp/kafka.server.truststore.jks");
props.put("ssl.truststore.password","storepass");

KafkaProducer<String,String> kafkaProducer = new KafkaProducer<String,String>(props);

Procedure for enabling SSL for Information Server Kafka events:

Note: This feature is available in Rollup Patch 8, and later.

  1. Upload the kafka.server.truststore.jks that was created before to Zookeeper. The default Zookeeper node name is '/kafka.server.truststore.jks'. You can specify antoher location by setting the Information Server key 'com.ibm.iis.events.kafka.truststoreZookeeperURI'.
    shared-open-source\solr\install\server\scripts\cloud-scripts\zkcli -zkhost localhost:52181 -cmd putfile /kafka.server.truststore.jks C:\temp\kafka.server.truststore.jks
  2. Setup the Kafka security protocol (SSL) Information Server key:
    AIX or Linux:
    ASBServer/bin/iisAdmin.sh -s -k com.ibm.iis.events.kafka.securityProtocol -value SSL
    Windows:
    ASBServer\bin\iisAdmin.bat -s -k com.ibm.iis.events.kafka.securityProtocol -value SSL
  3. Encrypt the truststore password for Information Server:
    AIX or Linux:
    ASBServer/bin/encrypt.sh storepass
    {iisenc}pXOSozvaBVpxwcWJmG6Dnw==
    Windows:
    ASBServer\bin\encrypt.bat storepass
    {iisenc}pXOSozvaBVpxwcWJmG6Dnw==
  4. Setup the truststore password Information Server key to the just encrypted password value:
    AIX or Linux:
    ASBServer/bin/iisAdmin.sh -s -k com.ibm.iis.events.kafka.truststorePassword -value {iisenc}pXOSozvaBVpxwcWJmG6Dnw==
    Windows:
    ASBServer\bin\iisAdmin.bat -s -k com.ibm.iis.events.kafka.truststorePassword -value {iisenc}pXOSozvaBVpxwcWJmG6Dnw==
  5. Restart Information Server.


Solr cloud (collections authentication)

If you have Infosphere Information Server version 11.5.0.1 with Governance Rollup Patch 3 or later, the Information Server components that use Solr for indexing their data support authentication. We recommend to protect the solr collections as follows.


How to:
  1. Enable authentication and authorization for solr, define an admin user and a non-admin user, and restrict security access to the admin user and collections access to both users.
    1. Create a file named security.json with the content below. This file specifies an admin user "solr" with password "SolrRocks". You can change this password later by using solr REST API "set-user", refer to the sample command in item c.
    2. { "authentication":
      { "class": "solr.BasicAuthPlugin",
      "credentials": { "solr": "IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0= Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c="}
      },
      "authorization":
      { "class": "solr.RuleBasedAuthorizationPlugin",
      "user-role": { "solr": ["admin", "user"] },
      "permissions": [ { "name": "security-edit", "role": "admin" } ]
      }
      }
    3. Load the file security.json above into zookeeper to zNode /security.json. All solr nodes will instantly pick up the file and enable security. We assume that the file created in item 1 is located in /tmp on AIX or Linux, and in c:\temp on Windows. If not, modify the command to refer to the actual path.
      AIX or Linux:
      shared-open-source/solr/install/server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:52181 -cmd putfile /security.json /tmp/security.json
      Windows:
      shared-open-source\solr\install\server\scripts\cloud-scripts\zkcli.bat -zkhost localhost:52181 -cmd putfile /security.json c:\temp\security.json
    4. Add a user "solruser" with password "infosphere" and give it role "user". The sample commands below use executable curl that is available on Linux by default. It can be installed on AIX and Windows too. You can of course use any other REST client of your choice to post the indicated messages to the solr REST API. Do not forget to specify credentials "solr:SolrRocks" in the POST request, since the solr security-edit authorization is already enabled.
      curl -X POST -H "Content-Type: application/json" -u solr:SolrRocks -d '{ "set-user": { "solruser": "infosphere" } }' localhost:58983/solr/admin/authentication
      curl -X POST -H "Content-Type: application/json" -u solr:SolrRocks -d '{ "set-user-role": { "solruser": "user" } }' localhost:58983/solr/admin/authorization
    5. Restrict access to collections dqecExceptionSets and da-datasets to users having role "user". The sample commands below use executable curl that is available on Linux by default. It can be installed on AIX and Windows too. You can of course use any other REST client of your choice to post the indicated messages to the solr REST API. Do not forget to specify credentials "solr:SolrRocks" in the POST request, since the solr security-edit authorization is already enabled.
      curl -X POST -H "Content-Type: application/json" -u solr:SolrRocks -d '{ "set-permission": { "name": "collections", "collection": ["dqecExceptionSets", "da-datasets"], "path": "/*", "role": "user" } }' localhost:58983/solr/admin/authorization
  2. Declare into Information Server the non-admin user that is allowed to access the solr collections.
    1. Encrypt the Solr authentication password for Information Server:
      AIX or Linux:
      ASBServer/bin/encrypt.sh infosphere
      {iisenc}3Jz+rq7CxMI57umgoCHPEw==
      Windows:
      ASBServer\bin\encrypt.bat infosphere
      {iisenc}3Jz+rq7CxMI57umgoCHPEw==
    2. Setup the Solr authentication user and password (encrypted) in Information Server keys:
      AIX or Linux:
      ASBServer/bin/iisAdmin.sh -s -k com.ibm.iis.solr.search.user -value solruser
      ASBServer/bin/iisAdmin.sh -s -k com.ibm.iis.solr.search.password -value "{iisenc}3Jz+rq7CxMI57umgoCHPEw=="
      Windows:
      ASBServer\bin\iisAdmin.bat -s -k com.ibm.iis.solr.search.user -value solruser
      ASBServer\bin\iisAdmin.bat -s -k com.ibm.iis.solr.search.password -value "{iisenc}3Jz+rq7CxMI57umgoCHPEw=="



Solr cloud (admin UI authentication)

If you have Infosphere Information Server version 11.5 with Governance Rollup Patch 1 installed, or if you have Infosphere Information Server version 11.5.0.1 but not yet Governance Rollup Patch 3 installed, the Information Server components that use Solr for indexing their data do not support authentication. Hence, we recommend to protect the access to the solr cloud admin UI so that creating, modifying or deleting collections is password-protected.

Even if you have Infosphere Information Server version 11.5.0.1 with Governance Rollup Patch 3 installed, or a later version, you may consider protecting the access to the solr cloud admin UI in addition to protect the solr collections as explained above. The parts of the admin UI that display or modify collections are password-protected as soon as the above solr collections authentication is in place. But other parts of the admin UI may display informations that you may want to protect too.


How to:
  1. Append the following content to the solr jetty context file shared-open-source/solr/install/server/contexts/solr-jetty-context.xml, at the end but inside of the <Configure> element:
            <!-- security handling -->
            <Get name="securityHandler">
              <Set name="loginService">
                <New class="org.eclipse.jetty.security.HashLoginService">
                   <Set name="name">sos-realm</Set>
                   <Set name="config"><SystemProperty name="jetty.base" default="."/>/resources/sos-realm.properties</Set>
                </New>
              </Set>
            </Get>
  2. Create a file shared-open-source/solr/install/server/resources/sos-realm.properties containing one line per user with the user name, the obfuscated password and the list of roles of this user.
    For example:
    solr: OBF:1s9r1w8z1u2w1xmq1lkv1xmk1u2e1w8r1sbj,admin
    The solr user has the role admin and the password SolrRocks. To get the obfuscated password, call the following:
    java -cp shared-open-source/solr/install/server/lib/jetty-util-9.2.11.v20150529.jar org.eclipse.jetty.util.security.Password solr SolrRocks
    SolrRocks
    OBF:1s9r1w8z1u2w1xmq1lkv1xmk1u2e1w8r1sbj
    MD5:511f970aa30f04bde97c831f2dd316eb
    CRYPT:so3oR2FANmQ4Y
  3. Append the following to the file shared-open-source/solr/install/server/solr-webapp/webapp/WEB-INF/web.xml, at the end but inside of the <web-app> element:
            <security-constraint>
              <web-resource-collection>
                 <url-pattern>/admin/*</url-pattern>
               </web-resource-collection>
               <auth-constraint>
                <role-name>admin</role-name>
              </auth-constraint>
            </security-constraint>
            <login-config>
              <auth-method>BASIC</auth-method>
              <realm-name>sos-realm</realm-name>
            </login-config>
  4. restart the InfoSrvSolrCloud service as follows:
    AIX:
    stopsrc -s InfoSrvSolrCloud
    startsrc -s InfoSrvSolrCloud

    Linux:
    service InfoSrvSolrCloud restart
    Windows:
    net stop InfoSrvSolrCloud
    net start InfoSrvSolrCloud


CAUTION:

Please be aware that the modifications to Zookeeper and Solr Cloud that are described in this document apply to the version of the Shared Open Source patch currently installed. Those modifications will disappear the next time you install (the next version of) the Shared Open Source patch. Before the next installation, you may want to save the modified files and merge them back manually after the installation.

[{"Product":{"code":"SSZJPZ","label":"IBM InfoSphere Information Server"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"IBM Stewardship Center","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF016","label":"Linux"},{"code":"PF033","label":"Windows"}],"Version":"11.5","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}}]

Document Information

Modified date:
16 June 2018

UID

swg21976111