Custom resources for advanced configuration

IBM® Automation foundation uses Kubernetes custom resources (CRs) for configuration. All the CRs are in the API groups that end with automation.ibm.com. These API groups have a version of v1beta1.

The following custom resources are used internally by IBM Automation foundation but awareness of them is useful for advanced configuration tasks.

The custom resource definitions (CRDs) for these kinds are declared in IBM Automation foundation's ClusterServiceVersion.

Similar to a Kubernetes resource, these custom resources contain the following elements:

• a metadata field that describes the resource
• a specification in a spec field
• a status field

Note the following points about a CRD:

• In the representations of the following CRDs, the field that is suffixed with [] indicates an array and the field that is suffixed with {} indicates default configuration.
• Where a child element is marked as mandatory for a parent that is optional, that child element is mandatory only if the parent element is included.

Cartridge

The Cartridge custom resource represents a cartridge that extends IBM Automation foundation. Complete information on Cartridge registration is found at Registration.

Cartridge CustomResourceDefinition

Cartridge YAML structure

The Cartridge definition is organized in the following structure.

apiVersion: core.automation.ibm.com/v1beta1
kind: Cartridge
metadata: ~
spec: 
  description: ~
  version: ~
status: 
  components: 
    ui: 
      endpoints: 
        caSecret: 
          secretName: ~
          key: ~
        casecret: 
          secretname: ~
          key: ~
        name: ~
        scope: ~
        type: ~
        uri: ~

Cartridge details

• Cartridge:
  • metadata (required): Refer to the Kubernetes API documentation for the fields of the metadata field. Make sure that the name field follows the naming conventions that are described in Technical naming of cartridges.
  • spec (required): _CartridgeSpec_ defines the wanted specifications of the Cartridge.
    • description (optional): Provides a description of the Cartridge.
    • version (mandatory): Desired version of the operand.
  • status: IBM Automation foundation instance status.
    • components (optional): The status of the components. Returned only when at least one of the components is enabled.
      • ui: Information that is needed to access the IBM Automation foundation user interface. Endpoints of types route, service, or both are returned here. The endpoint of type route is configured with the passthrough TLS method. You can change the Route configuration.
        • endpoints: Access details to access the user interface.
          • name: A name that represents this endpoint information.
          • type: The type of endpoint, UI.
          • scope: External.
          • uri: The route URL to access the UI.
          • caSecret (optional): Represents that tls is enabled. A reference to the public certificate that is used to protect the endpoint and verify the provided issuer.
            • secretName: Name of the secret.
            • key: Key of the public key of CA within the secret.
          • casecret: Equivalent to the previous field but deprecated as it is not consistent with the API used elsewhere. This is filled out along with the previous field for 1.1.x Cartridge Operands.
            • secretname:
            • key:
      • managedResources: Inventory resources managed by this custom resource.
      • conditions: Defines status of subresources that need to be ready before Cartridge is ready.

Cartridge sample YAML

The following example shows a Cartridge definition:

apiVersion: core.automation.ibm.com/v1beta1
kind: Cartridge
metadata:
  name: com.acme.smartdecisions-dashboard
  namespace: acme-iaf
spec:
  description: "ACME Smart Decisions Dashboard"
  version: "v1.0"
  license:
    accept: true

Cartridge status

The returned status section takes the following form:

status:
  components:
    ui:
      endpoints:
        type: UI
        name: external-route-https
        scope: External
        uri: https://iaf-system-ui.acme-iaf.acme.com
        caSecret:
          secretName: iaf-system-ui-tls-secret
          key: ca.crt
        casecret:
           secretname: iaf-system-ui-tls-secret
           key: ca.crt
  conditions:
  - lastTransitionTime: '2021-03-03T14:12:00Z'
    status: 'True'
    type: BedrockReady
  - lastTransitionTime: '2021-03-03T14:26:01Z'
    message: Cartridge successfully registered
    reason: Registered
    status: 'True'
    type: Ready

CartridgeRequirements

The CartridgeRequirements CR represents a cartridge that extends IBM Automation foundation. This CR is to be issued after the Cartridge CR is issued and need not wait until the Cartridge CR is Ready.

CartridgeRequirements CustomResourceDefinition

CartridgeRequirements YAML structure

The CartridgeRequirements definition is organized in the following structure:

apiVersion: base.automation.ibm.com/v1beta1
kind: CartridgeRequirements
metadata: 
  annotations: ~
spec: 
  requirements: ~
  externalCartridges: ~  
status: 
  components: 
    apicurio: 
      endpoints: 
        name: ~
        scope: ~
        type: ~
        uri: ~
    elasticsearch: 
      endpoints: 
        authentication: 
          secret: 
            secretName: ~
          type: ~
        caSecret: 
          key: ~
          secretName: ~
        name: ~
        scope: ~
        type: ~
        uri: ~
    kafka: 
      endpoints: 
        authentication: 
          secret: 
            secretName: ~
          type: ~
        bootstrapServers: ~
        caSecret: 
          key: ~
          secretName: ~
        name: ~
        scope: ~
        type: ~
  conditions: ~
  managedResources: ~

CartridgeRequirements details

• CartridgeRequirements:
  • metadata (required): Refer to the Kubernetes API documentation for the fields of the metadata field.
    • annotations (required): com.ibm.automation.cartridge represents the Cartridge name.
  • spec (required): CartridgeRequirementsSpec.
    • requirements [] (optional): List of the IBM Automation foundation capabilities that the Cartridge is to use. Allowed values: Events, EventProcessors (Deprecated), OperationalDataStore.
    • externalCartridges [] (optional): List of technical Cartridge names that the Cartridge is to expect shared data from.
  • status: CartridgeRequirements status.
    • components: The status of the components that satisfy the requirements.
      • apicurio (optional): Information that is needed to access the Apicurio registry.
        • endpoints: List of endpoints that are available to access the Apicurio registry.
          • type: The type of endpoint, API.
          • name: A name that represents this endpoint information.
          • scope: Internal, External.
          • uri: An endpoint-specific address that is used for connections.
      • elasticsearch: Information that is needed to access Elasticsearch.
        • endpoints: List of endpoints that are available to access Elasticsearch.
          • type: The type of endpoint, API.
          • name: A name that represents this endpoint information.
          • scope: Internal, External.
          • uri: Complete endpoint access URL.
          • authentication: Authentication details to access the Elasticsearch.
            • type: The type of the authentication mechanism. Default is BasicSecret.
            • secret: For type: BasicSecret, this references a basic auth type secret, which contains a username and a password field.
              • secretName: Name of the secret.
          • caSecret (optional): Represents that tls is enabled. A reference to the public certificate that the endpoint is protected with.
            • secretName: Name of the secret.
            • key (optional): Key of the public key of CA within the secret.
      • kafka (optional): Information that is needed to access the Kafka cluster. Available only when Events is a requirement in the CartridgeRequirementsSpec.
        • endpoints: List of endpoints that are available to access Kafka.
          • type: The type of endpoint, Kafka.
          • name: The name that represents this endpoint information.
          • scope: Internal, External.
          • bootstrapServers: Complete endpoint access URL.
          • authentication: Authentication details to access the Kafka.
            • type: The type of the authentication mechanism. Default is ScramSha512Secret.
            • secret: For type: ScramSha512Secret, this references a secret where its name is the username and it contains a password field.
              • secretName: Name of the secret.
          • caSecret (optional): Represents that tls is enabled. A reference to the public certificate that the endpoint is protected with.
            • secretName: Name of the secret.
            • key (optional): Key of the public key of CA within the secret.
    • managedResources: Inventory resources that are managed by this custom resource.
    • conditions: Defines status of subresources that need to be ready before CartridgeRequirements is ready.
  • Events: Specifying Events gets Kafka and Apicurio in status. Events is mandatory for EventProcessors requirements.

Note:

• If Kafka is not present in AutomationBase CR, then the prerequisite for Events is not met and the CartridgeRequirements CR does not go to the Ready state. It waits for a successful KafkaReady condition in the AutomationBase CR. Similarly, for the EventProcessors requirement, an instance of ElasticSearch is required to be created successfully and an ElasticReady status must be present in the AutomationBase CR for the CartridgeRequirements CR to reconcile successfully.
• Flink does not get installed here, if Flink is required, EventProcessor CR must be issued separately.
• When the operational data store (Elasticsearch) is not provided in the AutomationBase CR and a CartridgeRequirements CR is successfully reconciled, an update to the AutomationBase CR to include the operational data store does not update this addition in the existing CartridgeRequirements CR. The workaround is to delete the existing CartridgeRequirements CR and to create it after the operational data store is successfully created.

Note: While deleting AutomationBase and CartridgeRequirements, you will have to manually delete the corresponding KafkaComposite too. See Troubleshooting for more details.

An alternative that is a temporary workaround is to include a dummy update in the CartridgeRequriements CR after AutomationBase CR shows ElasticReady status. For example, you can add the following field under the section status > components in the CartridgeRequirments CR, which triggers the reconcile again for CartridgeRequirements and all the operational data store details are updated in the CartridgeRequirements CR with a new Elasticsearch user created.

  "elasticsearch: {}"

CartridgeRequirements sample YAML

Following is an example:

apiVersion: base.automation.ibm.com/v1beta1
kind: CartridgeRequirements
metadata:
  name: acme-smartdecisions-dashboard
  namespace: acme-iaf
  annotations:
    com.ibm.automation.cartridge: com.acme.smartdecisions-dashboard
spec:
  license:
    accept: true
  version: 1.1.0
  requirements:
    - Events
    - EventProcessors (Deprecated)
    - OperationalDataStore

CartridgeRequirements status

The returned status section takes the following form:

status:
  components:
    elasticsearch:
      endpoints:
      - type: API
        name: external-route-https
        scope: External
        uri: https://iaf-system-es.acme-iaf.acme.com
        authentication:
          type: BasicSecret
          secret:
            secretName: <cartridge-technical-name>-es-<type>-auth
        caSecret:
          secretName: iaf-system-es-tls-secret
          key: ca.crt
      - type: API
        name: internal-service-https
        scope: Internal
        uri: https://iaf-system-es.acme-iaf
        authentication:
          type: BasicSecret
          secret:
            secretName: <cartridge-technical-name>-es-<type>-auth
        caSecret:
          secretName: iaf-system-es-tls-secret
          key: ca.crt
    kafka:
      endpoints:
      - type: Kafka
        name: internal-service-plain
        scope: Internal
        bootstrapServers: iaf-system-kafka.acme-iaf.svc:9092
      - type: Kafka
        name: internal-service-tls
        scope: Internal
        bootstrapServers: iaf-system-kafka.acme-iaf.svc:9093
        authentication:
          type: ScramSha512Secret
          secret:
            secretName: <cartridge-technical-name>-kafka-<type>-auth
        caSecret:
          secretName: iaf-system-kafka-tls-secret
          key: ca.crt
      - type: Kafka
        name: external-route-tls
        scope: External
        bootstrapServers: iaf-system-kafka.acme-iaf.acme.com:443
        authentication:
         type: ScramSha512Secret
          secret:
            secretName: <cartridge-technical-name>-kafka-<type>-auth
        caSecret:
          secretName: iaf-system-kafka-tls-secret
          key: ca.crt
  conditions:
  - lastTransitionTime: '2021-03-03T14:32:06Z'
    status: 'True'
    type: CartridgeReady
  - lastTransitionTime: '2021-03-03T14:31:35Z'
    status: 'True'
    type: ElasticUserReady
  - lastTransitionTime: '2021-03-03T14:32:05Z'
    status: 'True'
    type: KafkaUserReady
  - lastTransitionTime: '2021-03-04T05:01:03Z'
    message: CartridgeRequirements successfully registered
    reason: Registered
    status: 'True'
    type: Ready

Note:

• If Kafka is not present in AutomationBase CR, then the prerequisite for Events is not met and the CartridgeRequirements CR does not go to the Ready state. It waits for a successful KafkaReady condition in the AutomationBase CR.
• Flink does not get installed here, if Flink is required, EventProcessor CR must be issued separately.
• EventProcessors in the spec.requirements is deprecated but its use is supported until it is removed in a future release. EventProcessors is replaced by OperationalDataStore from v1.2 of IBM Automation foundation. For the OperationalDataStore requirement, an instance of ElasticSearch is required to be created successfully and an ElasticReady status must be present in the AutomationBase CR for the CartridgeRequirements CR to reconcile successfully.

Topic Sharing

The following currently applies to IBM Events only. For EventStreams, the KafkaTopic read/write permissions would be set manually using IAMs.

The field spec.externalCartridges can be provided in the CartridgeRequirements operand. The externalCartridges is a list of Cartridge names (Cartridges that already exist, or the Cartridges that will exist in the future) that are expected to provide shared data onto a predefined KafkaTopic, which would be relevant to the CartridgeRequirements corresponding to the Cartridge. Below is an example :

apiVersion: base.automation.ibm.com/v1beta1
kind: CartridgeRequirements
metadata:
  name: cartridge-requirements-a
  annotations:
    com.ibm.automation.cartridge: cartridge1-name
spec:
  license:
    accept: true
  version: vnext
  requirements:
    - Events

apiVersion: base.automation.ibm.com/v1beta1
kind: CartridgeRequirements
metadata:
  name: cartridge-requirements-b
  annotations:
    com.ibm.automation.cartridge: cartridge2-name
spec:
  license:
    accept: true
  version: vnext
  requirements:
    - Events
  externalCartridges:
    - cartridge1-name

apiVersion: base.automation.ibm.com/v1beta1
kind: CartridgeRequirements
metadata:
  name: cartridge-requirements-c
  annotations:
    com.ibm.automation.cartridge: cartridge3-name
spec:
  license:
    accept: true
  version: vnext
  requirements:
    - Events
  externalCartridges:
    - cartridge1-name
    - cartridge2-name

The above resources would have the following permissions:

• cartridge1-name would have read/write access for KafkaTopics prefixed with cartridge1-name.
• cartridge2-name would have read only access for KafkaTopics prefixed with cartridge1.shared, as well as read/write access for KafkaTopics prefixed with cartridge2-name.
• cartridge3-name would have read only access for KafkaTopics prefixed with cartridge1.shared or cartridge2.shared, as well as read/write access for KafkaTopics prefixed with cartridge3-name.

Providing an externalCartridge will only allow read access to the KafkaTopics prefixed with externalCartridge.shared. Data would have to be intentionally written to the .shared KafkaTopics by the relevant Cartridge in order for it to be shared.

It should be noted that the .shared KafkaTopic is made up of the technical name of the Cartridge, rather than the full name of the Cartridge. Notice that it is cartridge1.shared rather than cartridge1-name.shared. It is important to also consider this when adding shared data to the .shared KafkaTopic.

EventProcessor

Deprecation note: Event Processing Operand versions 1.0.0 and 2.0.0 are now deprecated in favour of Event Processing Operand version 3.0.0. However, you can continue to use these Operand versions until they are removed in 6 months from the release of IBM Automation foundation v1.2.

EventProcessor CustomResourceDefinition

EventProcessor YAML structure

The EventProcessor definition is organized in the following structure:

apiVersion: eventprocessing.automation.ibm.com/v1beta1
kind: EventProcessor
metadata: 
  annotations: ~
spec: 
  flink: 
    audit: ~
    authentication: ~
    env: ~
    image: ~
    jobManager: 
      nodeSelector: ~
      template: 
        pod: 
          metadata: 
            annotations: ~
          spec: 
            affinity: ~
            containers: 
              image: ~
              imagePullPolicy: ~
              name: ~
              resources: ~
            tolerations: ~
    logConfig: 
      log4j-console.properties: ~
      logback-console.xml: ~
    monitoring: ~
    properties: ~
    serviceAccountName: ~
    storage: 
      class: ~
      fsGroup: ~
      selector: ~
      size: ~
      supplementalGroups: ~
      volumeClaimTemplate: ~
    taskManager: 
      nodeSelector: ~
      replicas: ~
      template: 
        pod: 
          metadata: 
            annotations: ~
          spec: 
            affinity: ~
            containers: 
              image: ~
              imagePullPolicy: ~
              name: ~
              resources: ~
            tolerations: ~
    tls: 
      caSecret: 
        key: ~
        secretName: ~
      issuerRef: 
        group: ~
        kind: ~
        name: ~
    volumeMounts: ~
    volumes: ~
  license: ~
  version: ~
status: 
  conditions: ~
  endpoints: 
    authentication: 
      secretName: ~
      type: ~
    caSecret: 
      key: ~
      secretName: ~
    name: ~
    scope: ~
    type: ~
    uri: ~

EventProcessor details

• EventProcessor:
  • metadata (required): Refer to the Kubernetes API documentation for the fields of the metadata field.
    • annotations (required): com.ibm.automation.cartridge representing the Cartridge name.
  • spec (required): Specification.
    • version (required): The version of the operand to use.
    • license (required): Indicates whether you accepted the license or not. You must accept the license, and the value needs to be true to use the EventProcessor resource.
    • flink (required): Specify the use of Flink for this EventProcessor with optional configuration.
      • image (optional): Custom flink image for JobManager and TaskManager containers.
      • taskManager (optional): Configuration for Flink TaskManagers.
        • replicas (optional): Number of TaskManager replicas. Defaults to 1.
        • nodeSelector (optional): An optional key and value map of labels that are used to select the nodes on which the pods for this node group are to be scheduled.
        • template (optional): Template is the object that describes the pod overrides that are used when pods are created.
          • pod (optional): FlinkPod contains the pod information.
            • metadata (optional): Use FlinkPodMetadata to customize the task manager pods metadata.
              • annotations (optional): Set or override annotations on the task manager pods.
            • spec (optional): FlinkPodSpec contains the pod container and scheduling information.
              • containers (optional): FlinkPodContainer contains the specification for pod container overrides.
                • name (required): For example, taskmanager.
                • resources (optional): ResourceRequirements describes the compute resource requirements.
                • image (optional): Container image override.
                • imagePullPolicy (optional): ImagePullPolicy override.
              • affinity (optional): Affinity information for pod scheduling. core/v1/Affinity.
              • tolerations (optional): An optional core/v1/Tolerations array to specify toleration for scheduling taints. For more information, see Taints and Tolerations.
      • jobManager (optional): Configuration for Flink JobManagers.
        • nodeSelector (optional): An optional key and value map of labels that are used to select the nodes on which the pods for this node group are to be scheduled.
        • template (optional): Template is the object that describes the pod overrides that are used when pods are created.
          • pod (optional): FlinkPod contains the pod information.
            • metadata (optional): Use FlinkPodMetadata to customize the job manager pods metadata.
              • annotations (optional): Set or override annotations on the job manager pods.
            • spec (optional): FlinkPodSpec contains the pod container and scheduling information.
              • containers (optional): FlinkPodContainer contains the specification for pod container overrides.
                • name (required): For example, jobmanager or tls-proxy.
                • resources (optional): ResourceRequirements describes the compute resource requirements.
                • image (optional): Container image override.
                • imagePullPolicy (optional): ImagePullPolicy override.
              • affinity (optional): Affinity information for pod scheduling. core/v1/Affinity.
              • tolerations (optional): An optional core/v1/Tolerations array to specify toleration for scheduling taints. For more information, see Taints and Tolerations.
      • properties (optional): Properties for configuring the Flink cluster. Any standard Flink configuration property can be included.
      • logConfig (optional): The logging configuration, a string-to-string map that becomes the configmap that is mounted at /opt/flink/conf in started Flink pods. See the following examples for some possible keys to include:
        • log4j-console.properties: The contents of the log4j properties file to use. If not provided, a default that logs only to stdout is provided.
        • logback-console.xml: The contents of the logback XML file to use. If not provided, a default that logs to only stdout is provided.
        • Other arbitrary keys are also allowed, and will become part of the ConfigMap.
      • env [] (optional): Environment variables that are shared by all JobManager and TaskManager containers.
      • volumes [] (optional): Volumes that are shared by all JobManager and TaskManager pods.
      • volumeMounts [] (optional): Volume mounts that are shared by all JobManager and TaskManager pods.
      • serviceAccountName (optional): Service account to be used by all JobManager and TaskManager pods.
      • tls (required): Configure TLS for the Flink REST API. Specify {} here to create self-signed certificates.
        • issuerRef (optional): IssuerRef is a reference to the issuer for this certificate. The name field in this stanza is always required.
          • name (mandatory if using issuerRef): Name of the issuer resource.
          • kind (optional): If not set, or set to Issuer, an issuer resource with the given name in the namespace. If set to ClusterIssuer, a ClusterIssuer with the provided name is used.
          • group (optional): Specifies the group of the issuer resource.
        • caSecret (optional): A reference to the certificate authority (CA) public certificate that is used to verify the provided issuer.
          • secretName (mandatory if using caSecret): Name of the secret.
          • key (optional): Key of the public key of CA within the secret.
      • authentication (required): Enables Basic authentication for the Flink REST API. Specify {} here to enable this authentication.
      • audit (optional): Enables audit logging in the Job Manager tls-proxy container. Specify {} here to enable this option.
      • storage (optional): Represents the storage spec for the EventProcessor. This value creates a shared RWX PVC for the JM and TM pods, enabling persisted checkpoints and savepoints.
        • size (optional): Maximum amount of storage that is required. Defaults to 2Gi.
        • class (optional): Name of the StorageClass that is required by the claim. Leave blank for the default. For more information, see Class.
        • selector (optional): A label query over volumes to consider for binding. For more information, see Label selectors.
        • volumeClaimTemplate (optional): Allows more detailed specification of the PersistentVolumeClaim.
        • fsGroup (optional): Sets the fsGroup on the JobManager and TaskManager containers security contexts.
        • supplementalGroups (optional): Sets the supplementalGroups on the JobManager and TaskManager containers security contexts.
      • monitoring (optional): Setting this value enables prometheus monitoring for the Flink cluster and exports the metrics into OpenShift Monitoring.
  • status: Status.
    • endpoints []: A list of available endpoints for the event processor. Currently, only the Flink REST API is shown. Only one item exists here.
      • name (required): The name of this endpoint.
      • type (required): The type of the endpoint, UI or API.
      • scope (required): The scope of the endpoint, Internal or External.
      • uri (optional): The URI for the endpoint. Required for type UI and API.
      • caSecret (optional): A reference to a CA public key for trusting the endpoint. Only present if tls is enabled.
        • secretName (required): Name of the secret.
        • key (optional): Key of the public key of CA within the secret.
      • authentication (optional): Any relevant information to authenticate with the endpoint. Only set if authentication is enabled.
        • type (required): The type of authentication. Currently, only BasicSecret.
        • secretName (optional): The name of the Basic Auth type secret. Required for type BasicSecret. Basic auth secrets have a username and password field.
    • conditions: Defines status of subresources, which need to be ready before EventProcessor is ready.

EventProcessor sample YAML

The following example shows an EventProcessor definition:

apiVersion: eventprocessing.automation.ibm.com/v1beta1
kind: EventProcessor
metadata:
  name: acme-smartdecisions-eventprocessor
  namespace: acme-cp4a
  annotations:
    com.ibm.automation.cartridge: com.acme.smartdecisions-dashboard
spec:
  version: "v2.0"
  license:
    accept: true
  flink:
    tls: {}
    authentication: {}
    taskManager:
      replicas: 2
  ...
status:
  endpoints:
    - name: FlinkJobManager
      type: API
      scope: Internal
      uri: https://some-service.namespace.svc:8081
      caSecret:
        secretName: my-ca-secret
        key: ca.crt
      authentication:
        type: BasicSecret
        secretName: my-auth-secret
  conditions:
    - lastTransitionTime: '2021-03-04T11:04:13Z'
      message: Event Processor is ready
      reason: Created
      status: 'True'
      type: Ready
    - lastTransitionTime: '2021-03-04T11:04:13Z'
      message: Default credentials should be updated for security reasons
      reason: RandomlyGeneratedJobManagerCredentials
      status: 'True'
      type: Warning

For more information about event processors, see Event Processors section.