Problem

The information contained in this article has been verified as up-to-date on the date of the original publication of the article.  HashiCorp endeavors to keep this information up-to-date and correct, but it makes no representations or warranties of any kind, express or implied, about the ongoing completeness, accuracy, reliability, or suitability of the information provided.

All information contained in this article is for general information purposes only.  Any reliance you place on such information as it applies to your use of your HashiCorp product is therefore strictly at your own risk.

Introduction

When facing connectivity challenges within your Consul service mesh, the consul-k8s CLI offers powerful troubleshooting tools. This guide consolidates best practices for diagnosing issues across key components, including Dataplane sidecar proxies, the Consul API, and Terminating, Mesh, and Ingress gateways.

Prerequisites

Installing consul-k8s CLI

To utilize the consul-k8s CLI, install it on your Kubernetes/OpenShift management machine, such as your laptop or a jumpbox.

Follow the Install the CLI instructions for your particular management platform distribution.

Kubernetes Contextual Flags 

When troubleshooting Consul dataplane and gateway proxies it's important to understand your Kuberenetes/OpenShift environment namespace and context to get the most out of using the consul-k8s CLI tool's proxy subcommand. Ensure you understand for multi-cluster and namespace isolated environments you'll need to be explicit in using the following flags:

For each of the scenarios described below it's assumed you understand know when/how to apply which of the above context flags t7o troubleshoot the correct pod for a given Kubernetes or OpenShift cluster.

Table of Contents

• Troubleshooting pod-to-pod service-mesh connectivity
• Troubleshooting pod-to-external service connectivity
• Troubleshooting cross-datacenter pod service connectivity
• Troubleshooting ingress service-mesh connectivity

Use Cases & Procedures

Troubleshooting pod-to-pod service-mesh connectivity

When you suspect a connectivity issue between two pods both running Consul service mesh sidecars, you can use a combination of the consul-k8s CLI commands to pinpoint and diagnose problems:

1. Verify that the proxy exists on each pod
   Use the consul-k8s proxy list command. For instance, if your pod names include “frontend” and “backend,” ensure that sidecars appear in the output:

   $ consul-k8s proxy list
   Namespace: default
   
   Name                        Type
   backend-658b679b45-d5xlb    Sidecar
   client-767ccfc8f9-6f6gx     Sidecar
   frontend-676564547c-v2mfq   Sidecar
   ...
   

   • If a sidecar proxy is missing, check that Consul’s automatic sidecar injection is enabled for that namespace or that your pod is correctly annotated for sidecar injection.

2. Inspect the pod’s Envoy configuration
   Use consul-k8s proxy read <PODNAME> to confirm that the sidecar is configured with:

   • Expected clusters (i.e., the services this sidecar is supposed to communicate with).
   • Properly configured listeners and routes.

   For instance:

   $ consul-k8s proxy read backend-658b679b45-d5xlb
   ...
   ==> Clusters (5)
   Name                    FQDN                                              Endpoints          ...
   client                  client.default.dc1.internal.<...>.consul         192.168.18.110:20000 ...
   ...
   

   • If your upstream services (for example, “frontend” or “backend”) are missing from the Clusters section, the Consul configuration may be incomplete or the service registration is incorrect.
   • Similarly, check the Listeners or Routes sections to verify the proxy is configured to accept or route traffic as expected.

3. Check Envoy logs
   If configuration looks correct but traffic still fails, you may need to inspect (and possibly adjust) Envoy’s log levels:

   # Show current log levels
   $ consul-k8s proxy log backend-658b679b45-d5xlb
   
   # Increase log level for detailed troubleshooting
   $ consul-k8s proxy log backend-658b679b45-d5xlb -update-level debug
   

   After adjusting log levels, re-check your application or environment logs to see if any relevant error messages appear, such as Envoy rejecting requests or failing to resolve DNS.

4. Inspect Envoy cluster stats
   Use consul-k8s proxy stats <PODNAME> to view Envoy cluster statistics for that proxy. Look for:

   • Circuit breaker state
   • Connection failures
   • Health check statuses

   For example:

   $ consul-k8s proxy stats backend-658b679b45-d5xlb
   cluster.<service>.assignment_stale: 0
   cluster.<service>.bind_errors: 0
   ...
   

   If you see non-zero values for errors or circuit breakers, investigate further (e.g., check resource usage, config for limitations, network connectivity, or intentions/ACLs).

5. Use the troubleshoot subcommands

   • consul-k8s troubleshoot upstreams -pod <pod>: Lists all upstreams (both explicit and transparent) that Envoy is configured to connect to from the specified pod.
     If a service you expect to see is missing, you may have missing or incorrect intentions or service definitions.
   • consul-k8s troubleshoot proxy -pod <pod> -upstream-ip <ip>: Checks the connectivity from one pod’s Envoy proxy to an upstream’s IP address. The CLI will tell you whether the configuration is valid, the certificates are correct, listeners are present, or if healthy endpoints exist.

   Example:

   $ consul-k8s troubleshoot proxy -pod frontend-767ccfc8f9-6f6gx -upstream-ip 10.4.6.160
   ==> Validation
   ✓ certificates are valid
   ✓ Envoy has 0 rejected configurations
   ...
   ! no healthy endpoints for cluster "backend2.default.dc1.internal..." found
   

   This output suggests a missing or unhealthy “backend2” upstream.

Troubleshooting pod-to-external service connectivity

Pods that communicate with external services (for example, external APIs, databases, or SaaS endpoints) rely on Terminating Gateways or direct egress rules in Consul’s configuration:

1. Check gateway proxies
   If using a Terminating Gateway, ensure it appears as a proxy when you run:

   $ consul-k8s proxy list -A
   

   You should see entries of type Terminating Gateway. If missing, the external connectivity may fail due to the gateway not existing or being in a misconfigured state.

2. Verify gateway config
   Use consul-k8s proxy read <pod-running-terminating-gateway> to confirm it has clusters and routes pointing to the external service.
3. Review logs and stats
   • consul-k8s proxy log to adjust or check log levels.
   • consul-k8s proxy stats to confirm successful connections or check for circuit breaker openings.
4. Use troubleshoot commands
   If your external service is configured as an upstream in your pods, consul-k8s troubleshoot upstreams -pod <pod> can show if the external IP or domain is recognized. If it’s not recognized, confirm your gateway or external service definition is set up properly in Consul.

Troubleshooting cross-datacenter pod service connectivity

For multi-datacenter or multi-cluster setups, Mesh Gateways are used to route traffic between different Consul clusters. Common steps include:

1. Check mesh gateway proxies
   Run:

   $ consul-k8s proxy list -A
   

   Look for entries of type Mesh Gateway. If these are missing in the relevant namespaces, cross-datacenter connectivity will fail.

2. Inspect the configuration
   Use:

   $ consul-k8s proxy read <mesh-gateway-pod-name>
   

   Confirm it has the clusters or endpoints referencing the remote datacenter’s services.

3. Check intentions/ACLs
   Ensure that the service identity used across datacenters has appropriate ACL tokens/permissions. If there are failing connections or “permission denied” errors in logs, you may have an ACL or intention blocking cross-datacenter traffic.
4. Troubleshoot from each datacenter
   Log in to each cluster’s environment:
   • Use consul-k8s troubleshoot proxy or consul-k8s proxy read from both sides to confirm that each gateway knows about the other’s services.

Troubleshooting ingress service-mesh connectivity

Ingress Gateways allow traffic from outside the cluster to enter the Consul service mesh:

1. Verify ingress gateway existence
   Make sure you see an Ingress Gateway in the output of:

   $ consul-k8s proxy list -A
   

   If you do not see your ingress gateway, confirm your Helm values or Kubernetes manifests to ensure it is deployed.

2. Inspect ingress gateway config

   $ consul-k8s proxy read <ingress-gateway-pod>
   

   Ensure that inbound listeners and routes match the intended external ports and that the correct cluster(s) (destination services) are present in its configuration.

3. Check logs and stats
   As with other proxies:
   • Use consul-k8s proxy log <ingress-gateway-pod> to adjust log levels.
   • Use consul-k8s proxy stats <ingress-gateway-pod> to view cluster stats, focusing on upstream errors, circuit breaker issues, or any health check failures.
4. Intentions and ACL checks
   Ingress traffic can also fail if intentions are not set to allow traffic from the gateway to the inside service. Verify that you have correct intentions for the ingress gateway’s service identity.

Additional Information

Commands for General Visibility

• consul-k8s status
  Provides a status summary of the Consul on Kubernetes installation, showing the deployed chart versions, replicas, and health status of servers and clients. Useful for a high-level health check before diving deeper into proxy-level issues.
• consul-k8s troubleshoot upstreams and consul-k8s troubleshoot proxy
  Both subcommands under the troubleshoot umbrella are particularly helpful for pinpointing misconfigurations or network issues from within a pod’s Envoy sidecar.

Conclusion

Using the consul-k8s command line tool consolidates multiple troubleshooting steps into a single interface. By examining proxy configurations, logs, and network stats, you can methodically isolate whether an issue stems from misconfigurations, intentions and ACLs, cluster connectivity, or gateway health. Combine these CLI steps with standard Kubernetes diagnostics (e.g., checking Pod and Service states, verifying your kubeconfig context/namespace) for a comprehensive troubleshooting workflow.

If you continue to experience issues, confirm you have followed Consul’s official installation guidelines for your environment, and that your pods are annotated/enabled for sidecar injection as required.