Creating an Apache Kafka data source

With a Kafka data source, you can process real-time data feeds.

Before you begin

Before you create a new Apache Kafka data source, install Apache Kafka and specify the basic authentication.
Install Apache Kafka and ksqlDB
  1. Install Docker desktop by completing the steps that are described in the Docker documentation.
  2. Create a directory named <docker>/<ksqldb> and create a file named docker-compose.yml in that directory.
  3. Copy the Start a Kafka broker for me script from the ksqlDB Quickstart page into the docker-compose.yml file.
  4. At the command prompt, change the directory to <docker>/<ksqldb> and run the docker compose up -d command.
  5. Wait for all the containers to start.
Specify the basic authentication
  1. Open the docker-compose.yml file.
  2. Add the following code to ksqldb-server: after environment:
    KSQL_SASL_MECHANISM: PLAIN
KSQL_OPTS: "-Dauthentication.method=BASIC -Dauthentication.realm=KsqlServer-Props -Dauthentication.roles=admin -Djava.security.auth.login.config=/etc/ksqldb/jaas/jaas_config.file"
volumes: - "./etc/ksqldb/jaas:/etc/ksqldb/jaas/"
  3. Create the folders etc/ksqldb/jaas in the directory where docker-compose.yml is stored.
  4. Create the password and jass_config files in the jaas folder.
  5. Specify the user details in the password file.
  6. Enter the following details in the jass_config file:
    KsqlServer-Props {org.eclipse.jetty.jaas.spi.PropertyFileLoginModule required file="/etc/ksqldb/jaas/password-file" debug="false"; };
  7. At the command prompt, change the directory to <docker>/<ksqldb> and run the following command: docker-compose up -d
  8. Verify the login by running the following command: docker exec -it ksqldb-cli ksql -u username -p letmein http://ksqldb-server:8088
Note: For advanced steps, refer to the information mentioned in Configure Security for ksqlDB.

About this task

You can connect to Kafka data source by either specifying the host name and the host port or by setting the connection parameters.

To create a Kafka data source, complete the following steps:

Procedure

  1. Select File > New > Other > Repository > Kafka Data Source.
    The Create Kafka Data Source wizard opens.
  2. On the Select data source availability type dialog, select Shared or Personal.
    Note: For more information about types of data source availability, see Data sources.
  3. Type the data source name in the Data Source Name field.
  4. Specify the necessary parameters in the Connection Parameters area.
    • In the Host name field, specify the location where Kafka is installed.
    • In the Host Port field, specify the port on which the ksqldb server is running. The default port number for Kafka data source is 8088.
  5. To specify the necessary user parameters, click Set User Information.
  6. To verify the connection to the Kafka data source, click Test Connection.
  7. Click Finish.
    A new Kafka data source gets created and the Create Kafka Data Source wizard closes.

Results

The Kafka data source is added in the Repository Explorer under the Kafka Data Source node with three sub-nodes: Streams, Topics, and Tables.