TFC Agent Auto-Upgrade Failure While Updating to 1.28.9

Troubleshooting

Problem

Terraform Cloud Self-hosted agents fail during the automatic upgrade process when upgrading from 1.28.6+ to 1.28.9.

The issue occurs after the core binary is automatically updated and restarted.

Symptom

Agents configured with auto-update enabled may fail immediately during restart while agent core upgrading from:

1.28.6 → 1.28.9
1.28.7 → 1.28.9
1.28.8 → 1.28.9

Observed logs:

2026-05-13T03:22:15.823Z [INFO]  agent: Core update is available: version=1.28.9
2026-05-13T03:22:17.098Z [INFO]  agent: Core successfully updated: version=1.28.9
2026-05-13T03:22:17.101Z [INFO]  agent: Core plugin is shutting down
2026-05-13T03:22:17.452Z [INFO]  core: Shutdown complete
2026-05-13T03:22:17.462Z [INFO]  plugin: plugin process exited: plugin=/home/tfc-agent/bin/tfc-agent-core id=20
2026-05-13T03:22:17.578Z [INFO]  plugin: plugin process exited: plugin=/home/tfc-agent/bin/tfc-agent-core id=41
2026-05-13T03:22:17.578Z [ERROR] agent: Failed starting core plugin: error="failed configuring core: failed evaluating config: invalid job type \"native_task\""
2026-05-13T03:22:17.578Z [INFO]  agent: Shutting down
Graceful shutdown complete

The agent shuts down after failing to start the upgraded core plugin.

Cause

The issue appears to be related to the automatic core upgrade flow in certain 1.28.x versions.

This occurs specifically after the core replacement step during upgrades from 1.28.6+ versions to 1.28.9.

Direct installation of 1.28.9 is not affected.

Environment

HCP Terraform / Terraform Self-hosted Agents
Linux ARM64 environment
Agent versions:
- Affected:
  - 1.28.6
  - 1.28.7
  - 1.28.8
- Target version:
  - 1.28.9

Diagnosing The Problem

Review the agent startup logs and verify whether the failure occurs immediately after the automatic core update.

Affected agents show:

[INFO]  agent: Core successfully updated: version=1.28.9

followed by:

[ERROR] agent: Failed starting core plugin: error="failed configuring core: failed evaluating config: invalid job type \"native_task\""

Findings:

Direct 1.28.9 deployment works
Auto-upgrade regression isolated to later 1.28.x versions
Earlier upgrade paths (1.27.0, 1.28.0, 1.28.5) succeed

Resolving The Problem

Use one of the following workarounds.

Option 1 :

Deploy agent version 1.28.9 directly instead of relying on auto-upgrade from affected versions.

Direct installation of 1.28.9 works successfully.

Option 2 :

Disable automatic core upgrades by setting the following environment variable:

TFC_AGENT_AUTO_UPDATE=disabled

Example:

docker run \
  -e TFC_AGENT_AUTO_UPDATE=disabled \
  hashicorp/tfc-agent:1.28.8

After disabling auto-update, manually manage the agent version upgrades, it will not upgrade the core to 1.28.9 automatically.

Note :- For the VM hosted agents, only option 1 is applicable.

Related Information

Terraform Cloud Agents Documentation

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB77","label":"Automation Platform"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSTYDYO","label":"IBM Terraform"},"ARM Category":[{"code":"a8mgJ0000000DwYQAU","label":"Terraform-\u003ETFC-Agent"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":""}]

Tips