Performance issues

You might face watsonx Orchestrate installation issues due to slow response or high latency and resource utilization. Go though the following sections for resolutions of the problems.

Slow response time and timeout

Symptoms
  • Slow response times
  • Timeouts
Diagnosing the problem
Check resource usage:
oc adm top nodes
oc adm top pods -n $PROJECT_CPD_INST_OPERANDS
Check pod limits 
oc get pods -n watsonx-orchestrate -o json | \
 jq '.items[] | {name: .metadata.name, resources: .spec.containers[].resources}'
Check for throttling 
oc describe pod <pod-name> -n watsonx-orchestrate | grep -i throttl
Solution

To resolve such problems, complete the following steps:

  1. Review application logs for errors
  2. Check database performance: 
    SELECT * FROM pg_stat_activity WHERE state = 'active';
  3. Verify network latency between components
  4. Review and optimize database queries
  5. Implement caching strategies

High latency and resource utilization

Symptoms
  • High latency
  • High resource utilization
Solution
To resolve such problems, complete the following steps:
Increase resource limits
Edit CR to increase resources 
oc edit WatsonxOrchestrate watsonx-orchestrate-instance -n watsonx-orchestrate
Update resources section:
resources:
   requests:
     cpu: "8"
     memory: "32Gi"
   limits:
     cpu: "16"
     memory: "64Gi"
Enable Horizontal Pod Auto-scaling
Create HPA 
oc autoscale deployment watsonx-orchestrate-api \
 --min=3 --max=10 --cpu-percent=70 \
 -n watsonx-orchestrate