Terraform Enterprise Fails to Load Workspace due to Vault "JSON string value exceeds allowed length" Error

Resolving The Problem

There are four primary methods to resolve this issue:

1. Upgrade Terraform Enterprise: The recommended long-term solution.
2. Modify Terraform Configuration: Remove the large output from your code and re-apply.
3. Manually Edit the State File: Pull the state, remove the output, and push it back.
4. Build a Custom TFE Image: A temporary workaround to increase the Vault limit if an immediate upgrade is not possible.

Solution 1: Upgrade Terraform Enterprise (Recommended)

The most effective and sustainable solution is to upgrade your Terraform Enterprise instance to version 1.2.0 or later (Release Documentation).

Follow the official Terraform Enterprise upgrade documentation to plan and execute the upgrade for your specific installation.

Solution 2: Modify Terraform Configuration

If you can modify the Terraform code for the affected workspace, you can resolve the issue by removing the large state output.

1. In your Terraform root module, identify and remove the output block that generates the value larger than 1MB.
2. Create a new run in the workspace to apply the configuration changes. This will update the state file and remove the oversized output.
3. Verification: After the apply successfully completes, navigate to the workspace in the TFE UI. The page should now load without any errors.

Solution 3: Manually Edit the State File

You can also directly manipulate the state file to remove the problematic output. Use caution with this method, as incorrect modifications can corrupt your state file.

1. Configure your local environment with credentials for the affected TFE workspace.

2. Use the Terraform CLI to pull the current state file.

   $ terraform state pull > current.tfstate

3. Open the current.tfstate file in a text editor. Locate the "outputs" section and delete the entry corresponding to the large output value.
4. Increase the serial value +1.

5. Push the modified state file back to the Terraform Enterprise workspace.

   $ terraform state push current.tfstate

6. Verification: Refresh the workspace page in the TFE UI. The error should be resolved, and the page should load correctly.

Solution 4: Build a Custom TFE Image (Workaround)

If you cannot upgrade immediately or modify the state, you can build a custom TFE image with configuration fixes. This should be considered a temporary workaround.

1. Create a file named Dockerfile with the following content. Adjust the FROM instruction to match your current TFE version.

   FROM images.releases.hashicorp.com/hashicorp/terraform-enterprise:1.1.3 # adjust to version in use
   # Increase Vault JSON string limit to handle large state outputs
   RUN sed -i '/listener "tcp" {/a \ max_json_string_value_length = 33554432' /etc/vault/config.hcl.tmpl

This RUN command increases the JSON limit to 32MB (33554432 bytes).

For Active-Active (multi-node) TFE, due to a related Vault issue, in order for the above configuration change to properly apply, the Vault version also needs to be upgraded in the image. To do this, the following needs to be added to the Dockerfile:

USER root
RUN curl -L -o /tmp/vault.zip "https://releases.hashicorp.com/vault/1.21.4/vault_1.21.4_linux_amd64.zip" && \
        unzip -o /tmp/vault.zip vault -d /usr/local/bin && \
        rm /tmp/vault.zip && \
        cp /usr/local/bin/vault /usr/local/bin/vault-ipc-lock && \
        setcap cap_ipc_lock=+ep /usr/local/bin/vault-ipc-lock
USER terraform-enterprise

1. Build the custom Docker image.

   $ docker build -t tfe-custom:1.1.3 .

2. Update your Terraform Enterprise installation to use this new custom image (tfe-custom:1.1.3). The steps to do this will vary based on your deployment method (e.g. Docker, Kubernetes).

Outcome

After applying one of the solutions, you will be able to access the affected workspace in the Terraform Enterprise UI without encountering the Vault error. The workspace will be fully functional, allowing you to view state, manage variables, and queue runs.

Additional Information

• For more details on the underlying Vault security change, refer to the official HashiCorp security bulletin: HCSEC-2025-24 - Vault Denial of Service though Complex JSON Payloads.
• Terraform Enterprise 1.2.x Release Notes