Customizing Kafka resource

You can modify Kafka resource settings, including CPU, memory, and replica count, to improve system performance or make more efficient use of system resources. You might need to make these changes during setup, scaling, or non-production maintenance activities.

Before you begin

Before you start customizing Kafka resources, review the following prerequisites to ensure a smooth and safe update process.:

Ensure you have cluster administrator privileges.
Verify that the Kafka custom resource (CR) is deployed and accessible.
Install the OpenShift® CLI (oc) and configure access to your cluster.
Back up any critical Kafka data if operating outside a test or development environment.

Procedure

Edit the Kafka custom resource (CR) to configure CPU, memory, and replica settings.

kafka:
  replicas: 3
  resources:
    limits:
      cpu: 1
      memory: 2Gi
    requests:
      cpu: 1
      memory:

Save the CR and exit the editor.
Wait for Kafka to reconcile the changes. Kafka pods restart automatically.
Verify that the updated configuration is applied.
- Use the OpenShift console
- Or run the following command:
```
oc get pods | grep kafka
```
To reduce Kafka replicas in a non-production environment, scale down the events operator to stop event flow.
```
oc get deploy | grep ibm-events-operator
oc scale --replicas=0 deployment <ibm-events-operator>
```
Warning: This procedure deletes Kafka deployments and persistent volumes, which causes data loss. Do not perform this procedure in production environments or where data integrity is required.
Delete the Kafka deployment.
```
oc delete kafka iaf-system -n bai
```
Check the availability of the Kafka CR.
```
oc get kafka -o yaml
```
When the CR is available, edit it to reduce replicas and resource allocations.
```
kafka:
  replicas: 3
  resources:
    limits:
      cpu: 1
      memory: 2Gi
    requests:
      cpu: 1
      memory: 1Gi
```
Example adjustment for large profile:
- Replicas: from 6 to 3
- CPU: from 4 to 2
- Memory: from 8Gi to 4Gi
Update Kafka storage size by editing the Kafka CR:
```
oc edit kafka iaf-system
```
Locate the storage section under spec.kafka and update the size value:
```
spec:
  kafka:
    storage:
      class: fakeblockstorage-nfs
      size: 10Gi   #--> update this value
      type: persistent-claim
```
Note: To increase Kafka storage, update the spec.kafka.storage.size field in the Kafka CR (iaf-system). Do not edit the the Kafka persistent volume claim (PVC) directly.
Delete the Kafka PVC.
```
oc delete pvc data-iaf-system-kafka-0
```
Note: If the PVC remains in a terminating state, run the following command:
```
oc edit pvc data-iaf-system-kafka-0
```
Then remove the finalizers entry to override PVC protection.

Scale the events operator back to 1 replica.

oc scale --replicas=1 deployment <ibm-events-operator>

Verify that Kafka pods are recreated with the updated replica count, CPU, and memory settings.