Flink parameters

Provide the appropriate values to configure Apache Flink processing, the Apache Flink persistent volume claim (PVC), and the Docker image that is used by the initialization containers in pods for deploying Flink event processors.

Notation

Configuration parameters for custom resources start with the spec.bai_configuration. prefix.

For example:
spec:
  bai_configuration:
    flink:
      task_manager_memory: '2728mb'

Measurement units

The task_manager_memory and job_manager_memory parameters are used for configuring Flink parameters. For the list of supported units, see the Flink documentation External link opens a new window or tab for Flink configuration.

The task_manager_memory_request, task_manager_memory_limit, job_manager_memory_request, and job_manager_memory_limit parameters are used for configuring requests and limits for Kubernetes memory and CPU resources. Find the valid units on the following pages.
  • For memory requests and limits, see the Meaning of Memory External link opens a new window or tab page of the Kubernetes documentation.
  • For CPU requests and limits, see the Meaning of CPU External link opens a new window or tab page of the Kubernetes documentation.

Flink processing

The following table lists the parameters for configuring Apache Flink processing.
Table 1. Configuration of Flink processing
Parameter Description Default value
flink.rocks_db_properties_config_map The name of a ConfigMap object that is already deployed to Kubernetes and contains RocksDB properties. For more information, see Specifying configuration maps for RocksDB properties. None
flink.job_checkpointing_interval The interval between checkpoints of Apache Flink jobs (in milliseconds) 5000
flink.log_level The logging level for the Flink event processor. Use this parameter to increase log verbosity when Flink jobs process events from custom sources through the event forwarder. Valid values are INFO, WARN, ERROR, FATAL, DEBUG, and TRACE.
Note: This parameter only works for the first time install flink, you have to update the rootLogger.level from FlinkDeployment CR directly.
 INFO
flink.job_manager_memory The total size of the Apache Flink job manager process. Flink parameter: jobmanager.memory.process.size 1728mb
flink.job_manager_cpu_request The CPU request for pods of Flink job managers (in cpu units). Kubernetes parameter: resources.requests.cpu 1
flink.job_manager_cpu_limit The CPU limit for pods of Flink job managers (in cpu units). Kubernetes parameter: resources.limits.cpu 1
flink.job_manager_memory_request The memory request for pods of Apache Flink job managers. Kubernetes parameter: resources.requests.memory 1728Mi
flink.job_manager_memory_limit The memory limit for pods of Flink job managers. Kubernetes parameter: resources.limits.memory 1728Mi
flink.task_manager_memory The total size of the Flink task manager process. Flink parameter: taskmanager.memory.process.size 1728mb
flink.task_manager_memory_request The memory request for pods of Apache Flink task managers. Kubernetes parameter: resources.requests.memory 1728Mi
flink.job_manager_replicas Use the replicas parameter to set the number of flink job manager pod replicas. The default value is 1.

If set to a value greater than 1, Flink HA (high-availability) is enabled.

See Enabling Flink Job Manager high availability

 1
flink.task_manager_memory_limit The memory limit for pods of Flink task managers. Kubernetes parameter: resources.limits.memory 1728Mi
flink.task_manager_cpu The number of CPUs that are used by Flink task managers (in CPU units). Flink parameter: kubernetes.taskmanager.cpu 1
flink.task_manager_cpu_request The CPU request for pods of Flink task managers (in CPU units). Kubernetes parameter: resources.requests.cpu 1
flink.task_manager_cpu_limit You can use this parameter to set the CPU limit for pods of Flink task managers (in CPU units). Kubernetes parameter: resources.limits.cpu 1
flink.additional_task_managers Use this parameter to set the number of additional Flink task managers that must be created to host custom processing applications. This parameter is mandatory only if you use custom processing applications. 0
flink.elasticsearch_max_actions You can use this parameter to set the number of create, delete and update actions to be performed on document indices in a single request. 1
flink.elasticsearch_flush_interval_ms The time interval, in milliseconds, at which to flush the buffered actions, regardless of the number of actions that are specified in the elasticsearch_max_actions parameter. The value is ignored if the elasticsearch_max_actions parameter is set to 1. 1000 if the elasticsearch_max_actions parameter is greater than 1.
flink.verbose_logs Set this parameter to true to increase log verbosity when Flink jobs process fixed-format events. false

Flink persistent volume claim (PVC)

Table 2. Persistent Volume Claim for Flink
Parameter Description Default value
flink_pv.capacity The capacity of the persistent volume (PV) None
flink_pv.storage_class_name Inherits the default value of the shared_configuration.sc_dynamic_storage_classname parameter. None
flink_pv.existing_claim_name The name of an existing claim if one is available. By default, a new persistent volume claim is created. New persistent volume

Initialization image

Table 3. Initialization image
Parameter Description Default value
init_image.image.repository The name of the Docker image for initialization containers bai-init
init_image.image.tag The tag of the Docker image Inherits the default value of the shared_configuration.sc_image_tag parameter. If that value is not set, the image digest that is predefined by the ICP4BA operator image is used instead.