Issue with OIDC - Vault key rotation mechanism to incorrectly identify the newest signing key: Invalid token signature

Problem

Workspaces using Dynamic Provider Credentials to authenticate providers may begin to fail with token signature verification errors. This occurs due to an issue in the OIDC signing key rotation mechanism, which can incorrectly identify the newest signing key after multiple rotations. This results in errors such as InvalidIdentityToken: Token signature invalid or failed to verify id token signature.

Example Error Messages

You may see one of the following errors, depending on the cloud provider in use.

AWS

Error: failed to refresh cached credentials, failed to retrieve credentials,exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: abcde123-a234-567d-abcde9874621, InvalidIdentityToken: Token signature invalid

Error: error configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found. Please see https://registry.terraform.io/providers/hashicorp/aws for more information about providing credentials. Error: WebIdentityErr: failed to retrieve credentials caused by: InvalidIdentityToken: Token signature invalid status code: 400, request id: bdc94a4b-d21e-4be0-9973-d6d3f03fc1b8

Vault

Operation failed: failed authenticating to Vault: Error making API request. URL: PUT https://your_vault_address.example.com:8200/v1/auth/jwt/tf/login Code: 400. Errors: * error validating token: error verifying token signature: failed to verify id token signature

Azure

Error: building account: could not acquire access token to parse claims: clientCredentialsToken: received HTTP status 401 with response: {"error":"invalid_client","error_description":"AADSTS7000274: Key 43ef235f1a1674828b08c529de1f1f1ffb1f4ff73f5f123456ba565dfc8509c4 was found, but use of the key to verify the signature failed. Trace ID: 12345678-a4a3-5552-001f-03132bd43c00 Correlation ID: 87123456-a0a1-23aa-92fd-bc24b9718c41 Timestamp: 2024-07-08 01:10:01Z","error_codes":[7000274],"timestamp":"2024-07-08 01:10:01Z","trace_id":"a6f8c29c-a60a-4446-991e-08132bd33a21","correlation_id":"cee699aa-0c6a-34cb-42ea-bc42d6268048","error_uri":"https://login.microsoftonline.com/error?code=7000274"}

GCP

oauth2/google: unable to generate access token: Post "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/{ACCOUNT_EMAIL_OR_UNIQUEID}:generateAccessToken": oauth2/google: status code 400: {"error":"invalid_grant","error_description":"Unable to verify the ID Token signature."}

Kubernetes

Error: Unauthorized

Prerequisites

• Terraform Enterprise versions v202207-1 through v202406-1.
• The instance has been running for approximately 810 days or more with the default 90-day OIDC signing key rotation period.

Cause

OIDC signing keys are rotated periodically and use a monotonically increasing integer for key versions. A code path sorted these versions lexicographically (like text) instead of numerically. When the key version reached 10, it was sorted incorrectly (e.g., 10 comes before 9 in a lexicographical sort), causing an older key to be selected for signing and leading to signature validation failures.

Solutions

This issue is resolved in HCP Terraform and requires an upgrade for Terraform Enterprise.

• HCP Terraform: This issue has been fixed and is not expected to cause further impact.
• Terraform Enterprise: The issue manifests in all versions prior to v202407-1 that support Dynamic Provider Credentials, typically after the OIDC signing key has rotated 9 times. The first version to support this feature was v202207-1.

Solution 1: Upgrade Terraform Enterprise (Recommended)

This issue was fixed in version v202407-1 of Terraform Enterprise. Upgrading to this version or a later release is the recommended permanent solution.

Solution 2: Forcefully Rotate and Trim Signing Keys (Workaround)

If an immediate upgrade is not possible, you can temporarily resolve the issue by using the Rails console to rotate the OIDC signing key and trim historic versions.

Run the following commands in the Rails console.

# Rotate the key 9 times to cycle past the problematic key versions.
9.times { OIDC::KeyManager.instance.rotate_key }

# Trim all keys except the latest one.
OIDC::KeyManager.instance.trim_key

This is not a permanent solution. Depending on your configured key rotation period (default is 90 days), it allows for safe rotations until the 100th rotation, at which point the issue will recur.

Solution 3: Extend the Key Rotation Period (Temporary Mitigation)

If you have not yet observed the issue and cannot upgrade or manually trigger a rotation, you can extend the key rotation period to delay its occurrence. Configure this via the WORKLOAD_IDENTITY_AUTO_ROTATE_PERIOD environment variable. The value is the number of days to wait between automatic rotations. Extending this period will delay the 10th key rotation.

Additional Information

• Terraform Enterprise v202407-1 - Bug Fixes
• Known Issue in Release Notes for v202208-1