June 30, 2021 By Janardhana Anipireddy
Geetha Sathyamurthy
3 min read

IBM Cloud Schematics is used to provision IBM Cloud resources by describing their configurations in a Terraform template file.

If you have tried to provision IBM Cloud resources by using the user interface or command line, you would have noticed that some of them take a very long time (e.g., IBM Event Streams). In such scenarios, HashiCorp recommends the use of timeout blocks. In any case, you must be aware of the typical time taken by IBM Cloud to provision or configure its cloud resources in order to avoid failures in your automation.

Now, the question is, what if IBM Cloud takes more time to provision or configure than the predefined timeout values in the timeout block? How do Terraform and IBM Cloud Schematics respond to such a situation?

In our experience with the terraform apply command, we have seen IBM Cloud Schematics exit the automation with a timeout error message. It sounds harmless, but the real problem is that the IBM Cloud service-broker has not stopped the provisioning operation. In other words, even though the IBM Cloud Schematics has reported the timeout error, the IBM Cloud resource (such as IBM Event Streams) will continue to be provisioned in your account, and you will be charged for that IBM Event Streams instance.

As an cloud engineer, once you see the timeout error message in the logs, your natural instinct is to retry the provisioning of the same Terraform template and the resources therein. For the second time, IBM Cloud Schematics will behave in an unexpected manner. It will use the previously saved Terraform state file in the workspace, to re-run the provisioning operation. Unfortunately, Terraform would have marked the IBM Event Streams instance as tainted in the terraform state file due to the timeout error. 

The default behaviour of the Terraform engine is to destroy the tainted resource and reprovision it, by default. In other words, the IBM Event Streams instance that is eventually provisioned by the IBM Cloud service-broker — even after IBM Cloud  Schematics reported the timeout error — is destroyed again, and IBM Cloud Schematics tries to provision another instance of the IBM Event Streams instance. This results in double your costs.  

How can you stop IBM Cloud Schematics from reprovisioning the IBM Cloud resources, due to the “tainted” flag in the Terraform state file?

Handling timeout errors

You should use the following procedure to handle such timeout errors in the IBM Cloud Schematics logs.

  1. Once you see the timeout block in the IBM Cloud Schematics workspace logs, view the Resources user interface page for the IBM Cloud Schematics workspace and verify whether the corresponding Terraform resource is marked as tainted.
  2. You should wait few hours and verify whether the IBM Cloud resource is actually getting provisioned in the account. Then, view the respective IBM Cloud resources user interface pages to confirm the provision.
  3. If the IBM Cloud resource is actually provisioned in the Cloud account after a delay, then you should check the IBM Cloud Schematics untaint CLI command to remove the resource’s taint flag in the state file.
  4. Finally, run the IBM Cloud Schematics plan and apply command from the user interface or command line to complete the provisioning of the template.

Note: In the last step, IBM Cloud Schematics idempotently provisions the remaining resources in that template.

Based on this experience, you must fine tune the timeout values in the Terraform template so that the same error does not repeat again in another environment. You must normally keep a watch on the following IBM Cloud resources for timeout errors in IBM Cloud Schematics:

  • IBM Event Streams
  • IBM Cloud Databases
  • IBM Cloud Kubernetes Service
  • IBM Cloud Satellite

We wish you good luck with your Terraform and Ansible automation with IBM Cloud Schematics.

Was this article helpful?
YesNo

More from Cloud

IBM Tech Now: April 8, 2024

< 1 min read - ​Welcome IBM Tech Now, our video web series featuring the latest and greatest news and announcements in the world of technology. Make sure you subscribe to our YouTube channel to be notified every time a new IBM Tech Now video is published. IBM Tech Now: Episode 96 On this episode, we're covering the following topics: IBM Cloud Logs A collaboration with IBM watsonx.ai and Anaconda IBM offerings in the G2 Spring Reports Stay plugged in You can check out the…

The advantages and disadvantages of private cloud 

6 min read - The popularity of private cloud is growing, primarily driven by the need for greater data security. Across industries like education, retail and government, organizations are choosing private cloud settings to conduct business use cases involving workloads with sensitive information and to comply with data privacy and compliance needs. In a report from Technavio (link resides outside ibm.com), the private cloud services market size is estimated to grow at a CAGR of 26.71% between 2023 and 2028, and it is forecast to increase by…

Optimize observability with IBM Cloud Logs to help improve infrastructure and app performance

5 min read - There is a dilemma facing infrastructure and app performance—as workloads generate an expanding amount of observability data, it puts increased pressure on collection tool abilities to process it all. The resulting data stress becomes expensive to manage and makes it harder to obtain actionable insights from the data itself, making it harder to have fast, effective, and cost-efficient performance management. A recent IDC study found that 57% of large enterprises are either collecting too much or too little observability data.…

IBM Newsletters

Get our newsletters and topic updates that deliver the latest thought leadership and insights on emerging trends.
Subscribe now More newsletters