IBM Support

Vault Secrets Sync Fails With INTERNAL_VAULT_ERROR When Syncing to Cloud Provider Private Endpoints

Troubleshooting


Problem

After upgrading to a Vault version where Secrets Sync SSRF protection is enforced, some environments may begin to see failures when syncing secrets between Vault and a cloud provider destination.


This commonly occurs when the destination uses a private endpoint, private IP address, load balancer, proxy, or other cloud-provider networking that resolves to RFC1918 (private) address space.

Symptom

The sync association may show a status similar to:

 
INTERNAL_VAULT_ERROR
 

The Vault documentation defines INTERNAL_VAULT_ERROR as a sync status indicating that the operation failed due to an internal issue in Vault, reviewing the sync association status can help when troubleshooting failed synchronization operations.

Example Error:

 
failed to create store: unable to initialize store of type "azure-kv":
unauthorized request: failed to validate the client credentials with a no-op unsync operation:
dial tcp 10.x.x.x:443: prohibited IP address:
10.x.x.x is not a permitted destination
 

Cause

Vault Secrets Sync includes endpoint security controls designed to protect against server-side request forgery (SSRF). By default, Vault restricts the IP addresses and ports that sync clients are allowed to connect to. The documentation specifies that special-purpose IPv4 and IPv6 addresses are blocked by default, and only ports 80 and 443 are allowed unless explicitly customized.


When Vault attempts to initialize or validate the Secrets Sync destination, the SSRF-safe sync client may reject the resolved private IP address. This can prevent the destination from initializing successfully and cause existing associations to remain in a failed sync state.
 

Although Vault provides destination-level configuration options such as allowed_ipv4_addresses, allowed_ipv6_addresses, allowed_ports, and disable_strict_networking, environments may still encounter this issue when using private endpoints if the allowlist does not permit the resolved address. This behavior is not expected. A bug report is currently open, with no defined timeline for resolution. These configuration fields are documented in the Secrets Sync destination settings for AWS, Azure, and GCP.

Environment

Vault Secrets Sync enables KV v2 secrets managed by Vault to be synchronized to external destinations such as AWS Secrets Manager, Azure Key Vault, and GCP Secret Manager. Vault remains the system of record, while the external destination serves as the cloud-native location where applications consume the secret.

Prerequisites

Before proceeding, ensure the following:

  • Vault Enterprise or HCP Vault Dedicated with Secrets Sync enabled
  • A configured Secrets Sync destination (for example: aws-sm, azure-kv, or gcp-sm)
  • Access to read destination and association status under sys/sync
  • Access to update the Secrets Sync destination configuration
  • Confirmation that the cloud provider endpoint resolves to a private, internal, or otherwise restricted IP address
  • (Recommended) A non-production environment to validate the workaround before applying it in production

Diagnosing The Problem

Confirming the Issue

Read the associations for the affected destination:

vault read sys/sync/destinations/<destination_type>/<destination_name>/associations
 

Example:

vault read sys/sync/destinations/aws-sm/prod-secrets/associations
 

Look for associations reporting:

sync_status    INTERNAL_VAULT_ERROR
 

You may also see Vault operational logs indicating that the sync client was denied from connecting to a private IP address, for example:

prohibited IP address
not a permitted destination
denied by: 10.0.0.0/8

Resolving The Problem

Overview of Solution

At this time, the available workaround is to disable strict networking for the affected Secrets Sync destination by setting:

disable_strict_networking=true
 

Vault documents disable_strict_networking as disabling both IP address and port restrictions used by the SSRF-safe sync clients. It also notes that strict networking may need to be disabled in environments where IP addresses or ports are not static.

Example payload pattern:

{
  "disable_strict_networking": true
}
 

Because update requirements vary by destination type, ensure that all required fields for that destination are included in the update request. Take care to avoid exposing cloud credentials in shell history or logs.

Destination Update Endpoints

  • Azure Key Vault:

    sys/sync/destinations/azure-kv/<destination_name>

  • AWS Secrets Manager:

    sys/sync/destinations/aws-sm/<destination_name>

  • GCP Secret Manager:

    sys/sync/destinations/gcp-sm/<destination_name>
 

Retry the Association

After updating the destination, retry or refresh the affected association:

vault write sys/sync/destinations/<destination_type>/<destination_name>/associations/set \
  mount="<kv_mount>" \
  secret_name="<secret_name>"
 

The association set endpoint links a KV v2 secret to the destination and can also be used to retry a failed sync operation after the underlying issue is resolved.

Outcome

After applying disable_strict_networking=true, Vault should be able to connect to cloud provider endpoints that resolve to private IP address space.
 

Verify the result by attempting to sync your secrets again.
 

If the issue persists with INTERNAL_VAULT_ERROR, review Vault operational logs for the latest sync attempt and confirm whether the failure is due to another issue, such as:

  • Invalid cloud credentials
  • Insufficient provider permissions
  • Unavailable external service

Current Workarounds

These are currently the only two practical paths forward:

The following are currently the only practical options:

  • Use disable_strict_networking=true for Secrets Sync destinations that rely on private endpoints or private network routing
  • Wait for a future Vault release that resolves the known issue with allowed_ipv4_addresses not permitting expected private endpoint addresses

Additional Information

  • Vault Secrets Sync overview: Explains supported destinations, one-way sync behavior, associations, and sync statuses
  • Secrets Sync endpoint security: Describes SSRF protection, default IP and port restrictions, and when strict networking may need to be customized or disabled
  • Secrets Sync API: documents destination-level settings such as disable_strict_networking, allowed_ipv4_addresses, allowed_ipv6_addresses, and allowed_ports.

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB77","label":"Automation Platform"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSLC9Y2","label":"IBM Vault Self-Managed"},"ARM Category":[{"code":"a8mgJ0000000E4fQAE","label":"Vault-\u003ESecret Engine"}],"ARM Case Number":"TS021770491","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"and future releases;1.18.0"}]

Product Synonym

HashiCorp Vault; Vault Self-Managed; Vault

Document Information

Modified date:
03 June 2026

UID

ibm17271996

Page Feedback