Troubleshooting Service Composer issues

Some known issues that are related to Service Composer or Portal-related issues in Managed services.

If the logged in user is not able to import services from GitLab and the following error occurs, then it indicates that the user does not have access to Helm repositories:
```
"Activity_id: ibmchartc7e3fb29, The template ibm-nodejs-sample does not exist."
```
As a resolution, grant access to Helm repository for the logged in user and then import service from GitHub/Gitlab/Bitbucket Server.
When you use special characters in JSON values that are used as an input parameter for a template, the provisioning fails with an error from IaaS.

For example,
```
"amis": { "today": "\"Thursday, January 24, 2019 11:01 AM\"" }
```
As a resolution, remove the special characters and retry provisioning. The following example is cleared off special characters:
```
"amis": { "today": "Thursday, January 24, 2019 11:01 AM" }
```
For some reason, if a service instance is stuck in IN_PROGRESS state for a long time without any backend activity, then delete the service instance entry from the database. This situation might arise because of an unplanned restart of any of the Managed services pods or any other related issues. Use Cleanup Service Instance in inprogress state API to delete the service instance. The API section provides you information about any ResourceLockError that might occur during the cleanup process.
If you do not see any content on the palette of the composer tab, then it is because of the connectivity issue between service composer and Business Process Manager or IBM Cloud Orchestrator or Helm plug-ins.

IaaS sample error message:
```
Bad return code from provider: { statusCode: 500, body: '{"error":{"statusCode":500,"message":"Internal Server Error"}}' }
```
Business Process Manager Plugin/Provider Error
```
[2019-05-08T06:29:47.422] [ERROR] http_utils - { Error: connect ETIMEDOUT <IP_ADDRESS>:9443 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1191:14) errno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'connect', address: '<IP_ADDRESS>', port: 9443 } Error: Error stack Logged at: at Logger.(anonymous function) [as error]
```
As a workaround, disable Business Process Manager plug-in that is causing the failure. For the actual procedure to disable a plug-in, see Enabling and disabling Business Process Manager and IBM Cloud Orchestrator.
Even after the service is deployed and is in a running state, the IBM Cloud Pak for AIOps or IBM Kubernetes Services cluster does not show up in the controller's cluster list.

It is possible that the service deployment had taken long enough that the controller token that is passed as a parameter on service deployment has been refreshed since the service deploy action. In this case, the IBM Cloud Pak for AIOps cluster is unable to register to the controller. To verify that this is the issue, check that the token on the controller instance has the same value as the one you have used when you deployed the Managed services. As a workaround for this issue, try any of the following methods:
- Method 1:
  1. Redeploy the service.
  2. Ensure that the controller token stays unchanged for the time it takes for the service to deploy the new cluster.
  3. Attach the token to the controller.
  4. Estimate the duration of service deployment.
  5. Set the refresh of the controller token beyond that time interval.
- Method 2
  1. Deploy the new cluster by using the Managed services templates.
  2. Use Loads_Images_and_Deploys_Chart_on_ICP service to install the chart.
  3. Register the cluster with the controller. Use Services/MCM/MCM_on_ICP/MCM_Klusterlet-Loads_Images_and_Deploys_Chart_on_ICP/MCM_Klusterlet-Loads_Images_and_Deploys_Chart_on_ICP.json from Git repository .
  Using this two-step cluster registration has the benefit that the long running step, which is the deployment of the cluster, gets separated from the chart install and connect operation. As the controller token is passed through during the service deploy, and this service deploy takes only minutes to deploy and run, it is unlikely for the token to be invalidated during the service deployment. If the issue still persists, redeploy just the chart service using the new token, rather than redeploying the entire cluster again.
When you keep multiple browser instances open for the same deployed instance (instance details or associated instance) that is in "wait" state, manually refresh the browser to see the latest updates.
Service may fail with unexpected EOF or may fail with both connection timeout and unexpected EOF. As a workaround, do the following steps:

Set the TERRAFORM_PARALLELISM_COUNT environment variable on the running cam-orchestration deployment of Managed services:
- Using the user interface:
  1. Login to IBM Cloud Pak for AIOps user interface and go to Workloads > Deployments.
  2. Search for cam-orchestration deployment.
  3. Select edit from the overflow menu.
  4. Edit the deployment by adding the following element to the env: array and then click Submit.
```
{
 "name": "TERRAFORM_PARALLELISM_COUNT",
 "value": "1"
}
```
  5. To verify whether your deployment is successful, double-click cam-orchestration deployment link.
  6. In the Overview tab > Pods section, check whether the cam-orchestration pod has restarted successfully.
- Using the command line:
  1. SSH to the Managed services deployment machine.
  2. Run the following command to edit the cam-orchestration deployment:
```
kubectl edit deployment cam-orchestration -n cp4aiops
```
  3. Under the - env: section, add the following lines.
    
    Note: Make sure you maintain the indentation same as other name-value pairs in this section.
```
name: TERRAFORM_PARALLELISM_COUNT
value: "1"
```
  4. Run the following command to ensure that the cam-orchestration pod has come up successfully after the change:
```
kubectl get pods -n cp4aiops | grep cam-orch
```
  5. Save these changes.
When you deploy a service that has non-permissible special characters in the value of any of its parameters, then the service instance creation fails with an error. For example, a service deployment fails because of a period (.) in the key value of a MongoDB parameter.
Mongo does not allow you to create a map with a key value having dot in the name. To resolve this Mongo limitation, use a list with key names that have hardcoded text without a dot.
If you use the same bind secret across multiple provisioned instances, the first bind instance is created correctly. From the second instance onwards, though a bind instance gets created, the following error occurs in IBM Cloud Pak for AIOps user interface:
```
ServiceBroker returned failure; bind operation will not be retried: Status: 400;
ErrorMessage: <nil>;
Description: <nil>;
ResponseError: <nil>;
```
When you create a service instance with bind action in IBM Cloud Pak for AIOps user interface, you cannot terminate the instance from IBM Cloud Pak for AIOps user interface without removing associated bind instances. Though you can terminate it from Managed services user interface, the bind action remains in active state.
For IBM Cloud Kubernetes Service template in Service Composition, do not select or enter values for datacenter and other networking attributes whenever the value of the machine_type is "free". This selection is not supported because the datacenter parameter along with other networking attributes are ignored by the Terraform provider for free clusters.
If you are logged into Managed services for a very long time, then its Access-Token on the browser expires. If you try to deploy a Service Instance after the Access-Token expires, the deployment fails with a 401 error. As a resolution, do the following steps:
1. Logout of Managed services.
2. Relogin to Managed services.
3. Retry service instance deployment.
When you drag-drop Helm charts in the composer, parameter details are not fetched properly. To resolve this error, synchronize the IBM Cloud Pak for AIOps Helm repositories in IBM Cloud Pak for AIOps user interface. For more information, see Managing Helm repositories.
Whenever an email notification is sent, the SMTP user name value is also used as the sender email address value. However, sometimes when you use services like sendgrid, the SMTP user name is not a valid email address. Though send a test email results in a message that says Test email sent successfully., no email is actually sent or received.
Terminate service instance may go to error state wherever the service instance has more than one Terraform template /Helm chart or is a combination of Terraform and Helm chart. A deployment failure in a template or helm chart can cause the entire terminate service instance to fail. However, the success of the destroy action of individual template or helm charts is based on their sequence in the composition.

For example, consider the following behavior whenever a Helm chart and Terraform template are placed in sequence and a failure occurs in Helm chart deployment:
- If Terraform template is positioned after the Helm chart in the sequence, it gets destroyed but the termination fails at the Helm chart. The rest of the sequence is not tried and the service instance termination fails.
- If the Helm chart is positioned after the Terraform template in the sequence, it is never attempted as the Helm failure occurs before the destroy action of Terraform template.
To resolve this issue, clean up Helm charts from backend. For the Terraform template, you can clean from backend or from the template tab.
Helm chart limitations:
- If you observe the following error message in the activity logs, then ensure the helm tiller has been configured on your IBM Kubernetes Service cluster:
```
Error: could not find tiller
```
- Helm deployments assume that you have precreated the required persistent volumes, otherwise it uses dynamic provisioning (for example GlusterFS).
- When you click the link for helm release in the View a deployed service instance page, you are forwarded to the IBM Cloud Pak for AIOps management console, and not to the Deployed templates page in Managed services.
- Helm release deployments are not visible from the deployed templates list view in Managed services. They are only visible from the Service instance details page and IBM Cloud Pak for AIOps Helm releases page.

When you rectify an invalid primary email server and redeploy a service that is already deployed, email notification may fail with an error message.

An example error message is as follows:

serviceblueprint_email.emailnot379ed899: Creating...
connection_used: "" => ""
fail_quietly: "" => "true"
message: "" => "Test"
sender_name: "" => "CAMadmin"
sequence_no: "" => ""
status: "" => ""
status_error_message: "" => ""
status_message: "" => ""
subject: "" => "Test"
to.#: "" => "1"
to.0: "" => "abc@in.ibm.com"
Error applying plan:

1 error(s) occurred:

serviceblueprint_email.emailnot379ed899: 1 error(s) occurred:
serviceblueprint_email.emailnot379ed899: unexpected EOF
Terraform does not automatically rollback in the face of errors
....
....

Do the following steps to resolve the error:

Go to IBM Cloud Pak for AIOps console and click Automate infrastructure>Managed services.
Delete primary and secondary SMTP connection.
Open new tab in same browser without closing the browser.

Open any REST Client extension in the browser and call the following API:

URL : https://<TSA_HOSTNAME>/cam/connections/call_proxy?service=composer&httpmethod=POST&relative_url=smtpconnections
Header: Content-Type : application/json
Method: POST
Payload:

Payload For Primary connection:

{
  "name": "PRIMARY",
  "description": "Primary SMTP Connection",
  "connection_parameters": [
    {
      "name": "username",
      "value": "<primary@somesmtpserver.com>"
    },
    {
      "name": "password",
      "value": "<password>"
    },
    {
      "name": "hostname",
      "value": "<smtp.somesmtpserver.com>"
    },
    {
      "name": "port",
      "value": "<587>"
    }
  ]
}

Payload For Secondary connection:

{
  "name": "SECONDARY",
  "description": "Secondary SMTP Connection",
  "connection_parameters": [
    {
      "name": "username",
      "value": "<secondary@somesmtpserver.com>"
    },
    {
      "name": "password",
      "value": "<password>"
    },
    {
      "name": "hostname",
      "value": "<smtp.somesmtpserver.com>"
    },
    {
      "name": "port",
      "value": "<587>"
    }
  ]
}

If you remove the mapping of service parameters in the Composition tab > Parameters tab of the template activity, it does not delete it from the main Parameters tab > Service Parameters section of the service and is also available while you create a service instance. If you do not want to see it while you create a service instance, you must delete it exclusively from the main Parameters tab > Service Parameters section.
There might be failures in the termination of a template instance in spite of a "Terminated" message in the user interface. After you terminate a service instance, check the template instance logs to reconfirm.
Whenever an error occurs during category creation, check for API-related errors in the log file.
When you order a service, do not map the same parameter to two different decisions in the Additional Parameters page. This causes both decisions to follow the same path.
If you do not provide the instance name in the composition tab for a template, then an error message is displayed in the Source Code tab during publish. To avoid such errors, whenever you add a template in the composition tab, provide a value for the instance name.
Examine the following logs for Service Composer or Portal-related issues:
- cam-orchestration
- cam-portal-api
- cam-iaas
If Managed services orchestration has multiple templates and a template fails in the workflow, then all subsequent templates are also not considered for deployment.