Managing a Processing Application

This section presents procedures to manage Processing Application deployments.

Installing and using the management CLI

A management service command-line interface (CLI) is provided for you to manage Processing Applications. The CLI can be used also to send events, start a Kafka consumer or producer for development purposed. In production, the CLI must only be used to management Processing Applications configuration and lifecycle.

Installing the command line interface

  1. Make sure that you have Java™ 17, bash 3.2, curl 7, and jq 1.5, or later versions, installed.
  2. Retrieve the URL, username, and password of the management service as instructed in Accessing Business Automation Insights services.
  3. Download the archive:
    curl -k https://${MANAGEMENT_URL}/cli --header 'Authorization: Basic X' -o ${DESTINATION_DIR}/bai-command.tar.gz
  4. Unpack the archive.
    tar xvf <file>

  5. If you want to be able to run the management-cli commands from any directory, remember to update your PATH environment variable.


Using the command line interface

  1. Login to the management service in order to use the CLI.
    management-cli env login --management-url=${MANAGEMENT_URL}
  2. Use the help function to display available commands.
    management-cli --help

Creating and restarting from savepoints

The management CLI offer the capability to create flink savepoints for your Processing Application and restart them.

  1. Login into the management service using the management cli.
    management-cli env login --management-url=${MANAGEMENT_URL} --management-username=${MANAGEMENT_USERNAME} --management-password=${MANAGEMENT_PASSWORD}
  2. List the job and note the id of jobs to which you want to create a savepoint.
    management-cli processing-jobs list
  3. Cancel a job and create a savepoint using the following command. Take note of the savepoint-path under result.operation.location displayed after running the command.
    management-cli processing-jobs cancel --job-id=<jobid> --create-savepoint
  4. When deploying a processing-application you can specify the savepoint-path noted in the previous step.
    management-cli processing-app deploy --savepoint-path=<savepoint-path>
Note: You can also cancel and create savepoints for all the running jobs using the create-savepoints command: 
management-cli processing-jobs create-savepoints --cancel

Cleaning up the Processing Application

When you deploy a Processing Application, the management service takes care of creating or updating various resources declared in the processing configuration such as OpenSearch indices and Kafka topics. However, during the development phase, you must regularly clean up your environment to avoid a proliferation of resources and Processing Applications.

  1. Stop the processing of the application. See How to stop the processing.
  2. Delete the topic. See How to delete a topic.
  3. Delete the OpenSearch configuration of the index. See How to delete an index configuration.
  4. Delete the Processing Application along with its configuration: 
    management-cli processing-app delete --name=<app_name>

How to stop the processing

When you deploy an application, a processing job (Flink job) starts. Follow this procedure to stop it.

  1. List the jobs to identify the one that matches your application name and whose state is running.
    management-cli processing-jobs list
  2. If you find a match, use the job ID to cancel the job.
    management-cli processing-jobs cancel --job-id=<job-id>

    The cancel operation is synchronous.

  3. Run the command in step 1 again to verify that the stats of the job is now CANCELED.

How to delete a topic

  1. Stop the processing of the application.
  2. List the topics to confirm the name of the topic that you want to delete.
    management-cli kafka list-topics
  3. Delete the topic. See Adding and removing topics External link opens a new window or tab] from Kafka documentation.

How to delete an index configuration

Retrieve the OpenSearch URL and credentials.
export NAMESPACE=<namespace>
OPENSEARCH_URL="https://$(oc get routes opensearch-route -o jsonpath="{.spec.host}" -n "$NAMESPACE")"
OPENSEARCH_USERNAME=$(oc get secret/opensearch-admin-user -o json -n "$NAMESPACE" | jq -r '.data|keys[0]')
OPENSEARCH_PASSWORD=$(oc extract secret/opensearch-admin-user --keys="$OPENSEARCH_USERNAME" --to=- -n "$NAMESPACE" 2>/dev/null

Deleting an index configuration consists of deleting the indices, aliases, and the index template that is related to the index name declared in the processing configuration.

CAUTION:
The following procedure deletes all the data of the index configuration.
  1. Stop the processing of the application.
  2. Set the variable paIndexName with the name of the index declared in the processing configuration.
  3. Identify one or more indexes instance names to delete.
    curl -sku ${OPENSEARCH_USERNAME}:${OPENSEARCH_PASSWORD}  ${OPENSEARCH_URL}/_cat/indices/${paIndexName}-idx-*?h=index
  4. Delete each index.
    indexInstanceName=<index_instance_name>
   curl -sku ${OPENSEARCH_USERNAME}:${OPENSEARCH_PASSWORD} -XDELETE ${OPENSEARCH_URL}/${indexInstanceName}
  5. Identify the index template name.
    curl -sku ${OPENSEARCH_USERNAME}:${OPENSEARCH_PASSWORD} ${OPENSEARCH_URL}/_cat/templates | grep ${paIndexName} | awk '{print $1;}'
  6. Delete the template.
    templateName=<template_name>
   curl -sku ${OPENSEARCH_USERNAME}:${OPENSEARCH_PASSWORD} -XDELETE ${OPENSEARCH_URL}/_template/${templateName}