Azure DevOps Server: Unable to clone repository after upgrading to Terraform Enterprise v202309-1

Problem

Introduction

This article provides solutions for an issue where Terraform Enterprise version v202309-1 fails to clone a repository from Azure DevOps Server due to an SSH host key mismatch.

Terraform Enterprise fails to clone a repository from Azure DevOps Server and displays the following error message.

Failed to ingress slug: Failed to clone: Failed to reach repo: Failed on ls-remote: exit status 128
Output: Unable to negotiate with xx.xx.xx.xx port 22: no matching host key type found. Their offer: ssh-rsa

Prerequisites

• Terraform Enterprise v202309-1
• Azure DevOps Server 2020 or 2019

Cause

The Terraform Enterprise v202309-1 FDO/Consolidated Services docker image is based on Ubuntu 22.04.3 LTS and includes openssh-client 1:8.9p1-3ubuntu0.3. This version of OpenSSH client deprecates the ssh-rsa host key signature algorithm, as noted in the OpenSSH 8.2 release notes, in favor of the more secure RSA SHA-2 signature algorithms (rsa-sha2-256/512).

Azure DevOps Server 2020 and 2019, prior to specific patches, lack support for these newer algorithms, causing the negotiation to fail.

Solutions

Solution 1: Patch Azure DevOps Server (Recommended)

The preferred solution is to update your Azure DevOps Server instance to a version that supports the newer RSA SHA-2 signature algorithms. This resolves the incompatibility at the source.

• Patch Azure DevOps Server to Update 1.2 Patch 7 for version 2020.
• Patch Azure DevOps Server to Update 1.2 Patch 4 for version 2019.

Solution 2: Disable Consolidated Services (Replicated Workaround)

For Terraform Enterprise installations managed by Replicated, you can disable Consolidated Services as a workaround. This reverts to the previous architecture that does not have this SSH client restriction.

SSH into the Terraform Enterprise host and run the following commands to disable the service and apply the configuration change.

## SSH into the TFE host and disable Consolidated Services
# replicatedctl app-config set consolidated_services_enabled --value 0

## Restart the TFE app to apply the changes
# replicatedctl app apply-config

Solution 3: Allow ssh-rsa via SSH Configuration (FDO Docker Workaround)

For Terraform Enterprise FDO Docker installations, you can create a custom SSH configuration file on the host to re-enable the ssh-rsa algorithm for the Terraform Enterprise container.

1. Create a configuration file on the Terraform Enterprise host(s) and set its permissions.

   # echo 'PubKeyAcceptedAlgorithms=+ssh-rsa' | sudo tee /etc/ssh/tfe_allow_rsa
   # chmod 644 /etc/ssh/tfe_allow_rsa

2. Edit your compose.yaml file to add a volume bind mount. This makes the host's SSH configuration file available inside the container.

   volumes:
     # ...
     - type: bind
       source: /etc/ssh/tfe_allow_rsa
       target: /etc/ssh/ssh_config.d/tfe_allow_rsa

3. Save the file and redeploy your container with docker compose.

Outcome

After applying one of the solutions, Terraform Enterprise can successfully clone repositories from your Azure DevOps Server instance.

Additional Information

• Flexible Deployment Options
• Terraform Enterprise Consolidated Services Architecture