FinOps Agent Troubleshooting Steps

Steps

Steps for the Customer to Diagnose

Part 1 : Attach Debug container and find our domains

Step 1) Please have the agent up and running. To check run kubectl get pods -n ibm-finops-agent.

kubectl get pods -n ibm-finops-agent
NAME                                          READY   STATUS    RESTARTS   AGE
ibm-finops-agent-xxxxxxxxx-xxxxx   1/1     Running   0          3d21h

Step 2) Once the agent is up and running, attach a debug container to the pod. (kubectl exec will not work on the ibm-finops-agent as we run a lighter container image to prevent unnecessary vulnerabilities.

kubectl debug -it unified-agent-finops-agent-5744f85777-fdn2x --image=nicolaka/netshoot --target=finops-agent

*** What does kubectl debug -it do? This command attaches a container to the pod that has networking debugging tools like curl/nslookup.

Step 3) From within the debug container, perform a sanity check to ensure the finops-agent can hit both frontdoor and containers urls by running dig

dig frontdoor.apptio.com
dig api.cloudability.com
## Reminder these urls change depending on what region the customer is in, see "What are the networking requirements for the UA?" Q from this Q&A Doc

a. Success Response

dig frontdoor.apptio.com
; <<>> DiG 9.20.10 <<>> frontdoor.apptio.com
...
...
;; ANSWER SECTION:
frontdoor.apptio.com.	27	IN	CNAME	frontdoor-production.apps.apptio.com.
frontdoor-production.apps.apptio.com. 27 IN CNAME ig-4a5abe5e401b.apps.apptio.com.
ig-4a5abe5e401b.apps.apptio.com. 27 IN	A	54.71.82.164
ig-4a5abe5e401b.apps.apptio.com. 27 IN	A	35.163.213.160
ig-4a5abe5e401b.apps.apptio.com. 27 IN	A	35.166.107.199
...
dig api.cloudability.com
...
;; ANSWER SECTION:
api.cloudability.com.	30	IN	CNAME	cldy-api-us.apps.apptio.com.
cldy-api-us.apps.apptio.com. 30	IN	CNAME	ig-4a5abe5e401b.apps.apptio.com.
ig-4a5abe5e401b.apps.apptio.com. 30 IN	A	54.71.82.164
ig-4a5abe5e401b.apps.apptio.com. 30 IN	A	35.163.213.160
ig-4a5abe5e401b.apps.apptio.com. 30 IN	A	35.166.107.199

b. Failure Response

## Several errors can occur attempting dig commands on either of the two urls, listed are some example failures but may no capture all failure cases
;; connection timed out; no servers could be reached
;; communications error to 193.108.88.1#53: connection reset
## These errors indicate something on the networking side of your cluster is configured incorrectly, please reach out to your cluster networking specialist to discover why the agent can not find our domains

**** These errors indicate something on the networking side of your cluster is configured incorrectly, please reach out to your cluster networking specialist to discover why the agent can not find our domains

Part 2 Attempt to login to frontdoor from Container

Step 4)  Attempt to log in to frontdoor from within the debug container:

a. First grab necessary values for the request from outside the pod: 

<cluster_uid> = kubectl get ns default -o yanl | grep uid

b. Then make the curl request to gather apptio-opentoken (log into frontdoor)

curl -i -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "keyAccess": "<Your_keyAccess>",
    "keySecret": "<Your_KeySecret>"
  }' \
https://frontdoor.apptio.com/service/apikeylogin

= kubectl get ns default -o yaml | grep uid= kubectl get ns default -o yaml | grep uid= kubectl get ns default -o yaml | grep uid

c. Success Response

A successful response contains an apptio-opentoken and a message that the login was successful.

...
apptio-opentoken: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
...
{"result":"login successful"}

d. Failure response

i. Api keyAccess or keySecret is in an invalid format will result in frontdoor throwing a 400. Ensure your api key credentials are correctly formatted.

{"code":400,"message":"Unable to process JSON"}

ii. Api keyAccess or keySecret are not valid frontdoor will inform the user with the following response.

{"error":"Unable to log in user. Validate that the ApiKey access/secret are valid","translationKey":"FD_AUTH_APIKEY_UNABLE_TO_VALIDATE","substitutions":[]}

iii. A Forbidden or hanging response could indicate your cluster’s proxy is blocking the request. Follow step 5 below to attempt the curl request with the proxy configured

Part 3 ) Attempt to login to frontdoor with proxy (only needed if previous step failed)

Step 5) Try to log in to frontdoor with proxy enabled, only needed if step 4 is failing. 

curl -i -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "keyAccess": "<Your_keyAccess>",
    "keySecret": "<Your_KeySecret>"
  }' \
--proxy <your-proxy-outbound-url> \
https://frontdoor.apptio.com/service/apikeylogin

a. Success Response

A successful response contains an apptio-opentoken and a message that the login was successful

...
apptio-opentoken: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
...
{"result":"login successful"}

If this request succeeded but the curl request in step 4 failed, this means a proxy is blocking the finops-agent from logging into frontdoor and needs to be configured in the agent. Add the following environment variable to your ibm-finops-agent deployment: CLOUDABILITY_OUTBOUND_PROXY=

b. Failure Reponse

If both curls to obtain to log in are failing, this could indicate the customer has not whitelisted our endpoints on port 443 or some other VPC configuration is not allowing their cluster to correctly interact with out endpoints. We suggest the customer cut a support ticket with their cluster provider to help them troubleshoot why their requests to api.cloudability.com and frontdoor.apptio.com are being blocked.

Step 6) Attempt to acquire the presigned s3 url

curl -X POST \
  -H "Content-Type: application/json" \
  -H "Apptio-Environmentid: <Your_EnvID>" \
  -H "Apptio-Opentoken: <Your_OpenToken>" \
  -d '{
    "clusterUID": "<Your_cluster_uid>",
    "fileName": "<Your_cluster_uid>_2025-01-01-01-01-01.tgz",
    "agentVersion": "1.0.0",
    "uploadHash": "testingHash"
  }' \
https://api.cloudability.com/v3/internal/containers/clusters/upload

Success Response

A successful response contains a location to the apptio-cake-services-cldyctr s3 url.

{"location": "https://apptio-cake-services-cldyctr.s3.amazonaws.com/production/data/metrics-agent/XXXX/XXXX/XX/XX/<cluster_uid>/XXXX-<cluster_uid>-XXXXXXXX-XX-XX.tgz?AWSAccessKeyId=XXXXX&Signature=XXXXXXX&content-type=multipart%2Fform-data&content-md5=testinghash&x-amz-security-token=XXXXXXXXXXXXXXXXXXXXXXExpires=XXXX"}

Failure response

A Forbidden or hanging response could indicate your cluster’s proxy is blocking the request. Follow step 7 below to attempt the curl request with the proxy configured

{"message":"Forbidden"}/

Step 7 ) Attempt to acquire the presigned s3 url with proxy enabled. Only needed if step 6 is failing

curl -X POST \
  -H "Content-Type: application/json" \
  -H "Apptio-Environmentid: <Your_EnvID>" \
  -H "Apptio-Opentoken: <Your_OpenToken>" \
  -d '{
    "clusterUID": "<Your_cluster_uid>",
    "fileName": "<Your_cluster_uid>_2025-01-01-01-01-01.tgz",
    "agentVersion": "1.0.0",
    "uploadHash": "testingHash"
  }' \
--proxy <your-proxy-outbound-url> \
https://api.cloudability.com/v3/internal/containers/clusters/upload

a. Success Response

A successful response contains a location to the apptio-cake-services-cldyctr s3 url. If this request succeeded but the curl request in step 6 failed, this means a proxy is blocking the finops-agent from uploading and needs to be configured in the agent. Add the following environment variable to your ibm-finops-agent deployment: CLOUDABILITY_OUTBOUND_PROXY=<your-proxy-outbound-url>

### SUCCESS RESPONSE ###
{"location": "https://apptio-cake-services-cldyctr.s3.amazonaws.com/production/data/metrics-agent/XXXX/XXXX/XX/XX/<cluster_uid>/XXXX-<cluster_uid>-XXXXXXXX-XX-XX.tgz?AWSAccessKeyId=XXXXX&Signature=XXXXXXX&content-type=multipart%2Fform-data&content-md5=testinghash&x-amz-security-token=XXXXXXXXXXXXXXXXXXXXXXExpires=XXXX"}

b. Failure Response

If both curls to obtain the S3 URL are failing, this could indicate the customer has not whitelisted our endpoints on port 443 or some other VPC configuration is not allowing their cluster to correctly interact with out endpoints. We suggest the customer cut a support ticket with their cluster provider to help them troubleshoot why their requests to api.cloudability.com and frontdoor.apptio.com are being blocked.

### FAILURE RESPONSE ###
{"message":"Forbidden"}

Additional Queries

1 ) If the user is receiving an error like SSL certificate problem: unable to get local issuer certificate, it could be that the proxy is intercepting requests and re-encrypting with an untrusted CA.  How to deal with a proxy that uses a custom CA?

A: A simple option is to disable certificate verification with the agent.cloudability.outboundProxyInsecure=true setting. If this is not acceptably secure for the client, they can configure a custom CA as follows:

1. Create a secret named ca-certs-secret in the ibm-finops-agent’s namespace. It should contain the custom CA certificate. For example, something similar to this: kubectl create secret generic ca-certs-secret --from-file=./ca-cert.pem

2. Update the ibm-finops-agent helm release to set a value of true for global.updateCaTrust.enabled (--set global.updateCaTrust.enabled=true).

When the ibm-finops-agent pod starts up, it should add the certificate from the secret to the list of trusted CAs.

2 ) What are the networking requirements for the Unified Agent ?

The container that hosts the metrics agent should allow HTTPS requests to following endpoints:

• https://frontdoor.apptio.com port 443

  • https://frontdoor-eu.apptio.com if Cloudability is in EU

  • https://frontdoor-au.apptio.com if Cloudability is in AU

  • https://frontdoor-me.apptio.com if Cloudability is in ME

  • https://frontdoor-jp.apptio.com if Cloudability is in JP

  • https://frontdoor-sg.apptio.com if Cloudability is in SG

  • https://frontdoor-in.apptio.com if Cloudability is in IN

  • https://frontdoor-ca.apptio.com if Cloudability is in CA

• https://api.cloudability.com port 443

  • https://api-eu.cloudability.com if Cloudability is in EU

  • https://api-au.cloudability.com if Cloudability is in AU

  • https://api-me.cloudability.com if Cloudability is in ME

  • https://api-jp.cloudability.com if Cloudability is in JP

  • https://api-sg.cloudability.com if Cloudability is in SG

  • https://api-in.cloudability.com if Cloudability is in IN

  • https://api-ca.cloudability.com if Cloudability is in CA

The container that hosts the metrics agent should have write access to following Apptio S3 buckets:

• apptio* (s3 bucket prefixed with apptio)

  • If you require more detailed information on whitelisting requirements, please reach out to our support team