Installing Kafka

Learn how to install Kafka. Apache Kafka is a high-throughput distributed messaging system that you can use to facilitate scalable data collection.

Before you begin

As a restadmin user, run the which java command and ensure that Java 11 is being used. If required, set the JAVA_HOME parameter. If it is not already installed, install Java 11. For more information, see Installing Java 11.

Procedure

  1. Navigate to the intended installation location (for example cd /opt/IBM) and unpack the Kafka tarball by using the following command: 
    tar zxf ~scadmin/Downloads/predictiveInsights1.3.6_<iFix_version>_el7_x86_64/REST_Mediation_utility/kafka_2.13-3.6.1.tgz
    Where <iFix_version> is the latest Interim Fix, for example iFix8.

    After you decompress the file, the archive files are in the <install_path>/kafka_2.13-3.6.1 directory. Where <install_path> is the path to the directory where you decompressed the file.

  2. It is strongly recommended to copy the JAR files from the ~scadmin/Downloads/predictiveInsights1.3.6_<iFix_version>_el7_x86_64/REST_Mediation_utility/kafka_extra_jars/*jar directory to the $KAFKA_HOME/libs directory.
  3. To access Kafka, add the location of the Kafka installation directory to the ~/.bashrc files.
    Add the location of the Kafka installation directory:
    export KAFKA_HOME=/opt/IBM/kafka_2.13-3.6.1
  4. Update the config files by replacing /tmp.
    For Example: cd $KAFKA_HOME
    mkdir .tmp
    cd .tmp
    pwd
    In the following property files, replace /tmp with the output of the pwd command:
    • $KAFKA_HOME/config/connect-standalone.properties change offset.storage.file.filename=/tmp/connect.offsets
    • $KAFKA_HOME/config/server.properties where log.dirs=/tmp/kafka-logs
    • $KAFKA_HOME/config/zookeeper.properties where dataDir=/tmp/zookeeper

What to do next

Start Kafka with the following commands: 
nohup $KAFKA_HOME/bin/zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.properties &
nohup $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties & 
ps -ef | grep kafka.Kafka