Builds of managed entities fail

Builds of managed entities might fail with a Failed to save error.

Symptom

The builds of managed entities might fail with the following error:

Unable to create orchestration process: Failed to save process

Cause

This problem occurs if the content of the request is large.

Solution

To resolve this problem, complete the following steps:

  1. Log in to the Vault UI.
    1. To get the Vault route, run the following command:
      NAMESPACE NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
      lifecycle-manager cp4na-o-vault cp4na-o-vault-lifecycle-manager.apps.devmgmtcluster.example.com cp4na-o-vault 8200 passthrough/Redirect None

      The Vault route is displayed in the HOST/PORT column of the output.

    2. Append /ui/vault/auth to the Vault route, then open the URL in a browser. For example, open a URL similar to the following URL:
      https://cp4na-o-vault-lifecycle-manager.apps.devmgmtcluster.example.com/ui/vault/auth
    3. To get the Vault key, run the following command:
      oc get secret cp4na-o-vault-keys -o jsonpath --template '{.data.alm_token}' | base64 -d
    4. Log in to Vault UI with the Vault key.
  2. Add a maximum request size value for all applications, then the Galileo pod.
    1. In the Vault UI, click Secrets in the menu bar, and then click lm/1.0.0.
    2. Click application,custom.
    3. Switch the view to JSON in edit mode, then add the following value to the application secret:
      "spring.kafka.producer.properties.max.request.size": 4194304
    4. Click galileo,custom.
    5. Switch the view to JSON in edit mode, then add the following value to the Galileo secret:
      "alm.galileo.ldu.streams.max.request.size": 4194304
    6. Restart the pods for the Talladega, Galileo, and Daytona microservices:
      1. Update the StatefulSets and Deployments and complete a rolling restart of the affected pods:
        oc patch orchestration <instance name> --patch '{"spec":{"advanced": {"synchronizeVaultChanges": true}}}' --type=merge
      2. When the pods are restarted, reset the synchronizeVaultChanges flag:
        oc patch orchestration <instance name> --patch '{"spec":{"advanced": {"synchronizeVaultChanges": false}}}' --type=merge

      When all the pods have a status of Running, the new configuration is in effect.

  3. Increase the maximum bytes value for the Kafka configuration map.
    1. Edit the Kafka instance:
      oc edit kafka cp4na-o-events
    2. Increase the value of the message.max.bytes parameter to 4194304, as shown in the following sample excerpt from a configuration map:
      kafka:
            authorization:
              superUsers:
              - ANONYMOUS
              type: simple
            config:
              message.max.bytes: 4194304
              offsets.topic.replication.factor: 1
    3. Restart the Kafka pod by running the following command:
      oc delete pod cp4na-o-events-kafka-0
  4. Increase the maximum bytes value for the Ansible® lifecycle driver.
    1. To open the Ansible lifecycle driver configuration map, run the following command:
      oc edit configmap ansible-lifecycle-driver
    2. Locate the messaging section in the ald_config.yml: | section in the configmap.
    3. Add the max_request_size parameter with a value of 4194304 in the config list, as shown in the following sample excerpt from a configuration map:
      ald_config.yml: |
       ansible:
         max_unreachable_retries: 60
         output_prop_prefix: output__
         unreachable_sleep_seconds: 5
       messaging:
         api_version_auto_timeout_ms: 5000
         connection_address: cp4na-o-events-kafka-bootstrap:9092
         config:
           max_request_size: 4194304
    4. Restart the Ansible lifecycle driver pod by running the following command:
      oc delete pod ansible-lifecycle-driver
  5. If you are using other drivers, you must increase the maximum bytes value for those drivers in the same way that you did for the Ansible lifecycle driver. For example, if you are using the OpenStack driver, add the max_request_size parameter in the config list in the messaging section in the ovd_config.yml: | section in the configmap. Similarly, you can add max_request_size in the kubedriver_config.yml: | section in the Kubernetes driver configmap.