Runs Stuck Checking Sentinel Policies and 503 Errors on /api/v1/health/readiness in Terraform Enterprise

Troubleshooting

Problem

In Terraform Enterprise version 2.0.0 and 2.0.1, you may observe that some nodes in your installation begin to fail health checks. Load balancers or orchestration platforms may report these nodes as unhealthy, leading to service degradation. When you inspect the readiness endpoint, it returns a 503 Service Unavailable HTTP status code, and the response body indicates that the atlas check is in an ERROR state.

Symptom

Symptoms of this issue include failing health checks to /api/v1/health/readiness and runs becoming stuck at "Sentinel policies running".

Cause

This issue is caused by a known bug in Terraform Enterprise related to Policy Checks and Policy Evaluations. When these features are in use, large Policy Check/Evaluations results can cause a log pipe deadlock, causing the Puma web workers to block indefinitely while holding a database transaction. This exhausts the available web workers in the atlas container, preventing it from serving new requests, including health checks.

Environment

Terraform Enterprise 2.0.0 and 2.0.1
Policy Checks or Policy Evaluation enforced on workspaces

Diagnosing The Problem

Executing a curl command from within a failing container against the readiness endpoint shows the atlas check in status ERRORED:

$ curl http://127.0.0.1:8080/api/v1/health/readiness
{"node":"terraform-enterprise-b677f7fbf-t8k8x","status":"ERROR","checks":[{"check":"archivist","status":"OK"},{"check":"atlas","status":"ERROR"},{"check":"database","status":"OK"},{"check":"redis","status":"OK"},{"check":"task-worker","status":"OK"},{"check":"vault","status":"OK"}]}

Further investigation by directly querying the internal health check for the atlas service (the web platform) shows that the request hangs indefinitely.

$ curl -v http://127.0.0.1:9292/_health_check
*   Trying 127.0.0.1:9292...
* Connected to 127.0.0.1 (127.0.0.1) port 9292 (#0)
> GET /_health_check HTTP/1.1
> Host: 127.0.0.1:9292
> User-Agent: curl/7.81.0
> Accept: */*
>
## The request hangs here without receiving a response.

Resolving The Problem

Solution 1: Enable Supervisord (Recommended Workaround)

This solution restores stability without requiring you to disable Sentinel policies. By switching to the Supervisord process manager, Terraform Enterprise will no longer be impacted by the log pipe deadlock.

Enable Supervisord: Enable Supervisord in Terraform Enterprise by adding the following environment variable to your configuration (e.g., in your docker-compose.yml, Helm values, or via your orchestration platform):
```
TFE_USE_SUPERVISORD=true
```
Restart Terraform Enterprise: Restart the application to apply the new process supervisor configuration.
Verify Stability: Monitor the health check endpoint from a shell in the Terraform Enterprise nodes to ensure it remains OK during Sentinel evaluations:
```
curl http://127.0.0.1:8080/api/v1/health/readiness
```

Note: This is an internal break-glass configuration which will be removed in a future release. The setting should only be enabled as a temporary workaround until Solution 3 (Upgrade) can be implemented, at which point it should be removed.

Solution 2: Disable Policy Checks (Legacy Workaround)

If Solution 1 cannot be implemented immediately, you can prevent the deadlock by stopping the generation of large audit logs by disabling Policy Sets.

Disable Policy Sets: Navigate to Settings > Policy Sets within your Organizations.
Scope Modification: Remove workspaces from the scope of each Policy Set or unset the global enforcement setting, if it is enabled.
Clear Existing Locks: Restart the Terraform Enterprise nodes to terminate any currently hung Puma workers and release database transactions.

Solution 3: Upgrade Terraform Enterprise (Permanent Fix)

The long-term solution is to upgrade your Terraform Enterprise instance to version 2.0.2 where this bug has been resolved.

Disable Supervisord: If you used Solution 1 as a temporary workaround, remove the the TFE_USE_SUPERVISORD=true setting from the Terraform Enterprise configuration.
Perform Upgrade: Schedule and perform an upgrade of your Terraform Enterprise installation to version 2.0.2 by following the official upgrade documentation.

After applying either the workaround or the permanent fix, the atlas service will no longer experience web worker exhaustion. The /api/v1/health/readiness endpoint will consistently return an HTTP 200 status code, and all health checks within the response body will report an OK status. This will restore the stability of your Terraform Enterprise installation and prevent nodes from being incorrectly marked as unhealthy.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB77","label":"Automation Platform"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSGH5YK","label":"IBM Terraform Self-Managed"},"ARM Category":[{"code":"a8mgJ0000000DwdQAE","label":"Terraform-\u003ETFE Backend"}],"ARM Case Number":"TS022149046","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"2.0.0"}]

Tips