IBM Support

Collecting "Service Level" Trace for an ACE Integration Server

How To


Summary

IBM App Connect Enterprise (ACE) provides a trace facility for capturing runtime behavior.
Use this process to capture a "service level" trace of a particular Integration Server for providing to IBM Support for analysis.

Steps

The REST API is the primary method of communicating with an Integration Server.
This method of enabling trace requires the Rest Administration port to be active for the Integration Node or Server.

Int.Node owned Integration Server

  1. Clear trace (recommended): 
    curl -X POST http://{hostname}:{port}/apiv2/servers/{server}/reset-service-trace
  2. Start trace: 
    curl -X POST http://{hostname}:{port}/apiv2/servers/{server}/start-service-trace
  3. Re-create the runtime behavior.
     
  4. Stop trace: 
    curl -X POST http://{hostname}:{port}/apiv2/servers/{server}/stop-service-trace

Stand-alone Integration Server

  1. Clear trace (recommended): 
    curl -X POST http://{hostname}:{port}/apiv2/reset-service-trace
  2. Start trace: 
    curl -X POST http://{hostname}:{port}/apiv2/start-service-trace
  3. Re-create the runtime behavior.
     
  4. Stop trace: 
    curl -X POST http://{hostname}:{port}/apiv2/stop-service-trace

Stand-alone Integration Server running as a pod in CP4I (with container access)

In CP4I, ACE Integration Servers are created with "basicAuth" administrative security enabled.
By default, the connection is secured with TLS (although this is configurable when creating the Integration Server).
Additionally, the REST admin port for individual Integration Servers is not exposed inside the cluster. Therefore, it is necessary to trace from within the container or establish access from outside the container. These steps assume you can log into the pod/container, or that you have access to the OCP User Interface "Terminal" tab for the pod.
  1. Clear trace (recommended): 
    curl --unix-socket /home/aceuser/ace-server/config/IntegrationServer.uds -X POST http://localhost/apiv2/reset-service-trace
  2. Start trace: 
    curl --unix-socket /home/aceuser/ace-server/config/IntegrationServer.uds -X POST http://localhost/apiv2/start-service-trace
  3. Re-create the runtime behavior.
     
  4. Stop trace: 
    curl --unix-socket /home/aceuser/ace-server/config/IntegrationServer.uds -X POST http://localhost/apiv2/stop-service-trace
  5. Copy the trace outside of the container.

Stand-alone Integration Server running as a pod in CP4I (no container access)

In CP4I, ACE Integration Servers are created with "basicAuth" administrative security enabled.
By default, the connection is secured with TLS (although this is configurable when creating the Integration Server).
Additionally, the REST admin port for individual Integration Servers is not exposed inside the cluster. Therefore, it is necessary to trace from within the container or establish access from outside the container. These steps assume you can not log into the pod/container and must trace from the cluster level.
  1. Get the name of your Integration Server. 
    oc get IntegrationServer
  2. Store the name of the Integration Server in an environment variable, for ease of reuse. 
    export ISNAME=my-int-server-name
  3. Get the security credentials for the Integration Server. 
    oc get secrets $ISNAME-is -o template --template '{{.data.adminusers}}' | base64 -d

    The output format is: {userid} {password}

  4. Store the username and password in environment variables, for ease of reuse.

    export ISUSER=my-userid
export ISPASS=my-password


    Example: 

    % oc get integrationservers.appconnect.ibm.com
NAME            RESOLVEDVERSION   REPLICAS   AVAILABLEREPLICAS   CUSTOMIMAGES   STATUS   AGE
is-01-toolkit   11.0.0.12-r1      1          1                   false          Ready    6d8h

% export ISNAME=is-01-toolkit

% oc get secrets $ISNAME-is -o template --template '{{.data.adminusers}}' | base64 -d
ibm-ace-dashboard-admin fa105a4a-7f68-46e5-9dbe-3dc6bb932eeb

% export ISUSER=ibm-ace-dashboard-admin
% export ISPASS=fa105a4a-7f68-46e5-9dbe-3dc6bb932eeb

  5. Extract the certificates from the secrets and save them to files.

    echo $(oc get secrets $ISNAME-is-admincert -o template --template '{{index .data "tls.crt"}}') | base64 -d > tls.crt
echo $(oc get secrets $ISNAME-is-admincert -o template --template '{{index .data "tls.key"}}') | base64 -d > tls.key
    Note: If "AdminTLS" is not enabled, this step is not required.
     

  6. Get the pod name that corresponds to the Integration Server.

    oc get pods

  7. In a different console session, enable port forwarding for the pod:

    oc port-forward pod/my-pod-name 7600:7600
    While port is being forwarded, that console session cannot be used.
    The curl commands for the trace need the environment variables in the current console session.
     
  8. Clear any existing trace (recommended). 
    curl --cert tls.crt --key tls.key -k -X POST https://localhost:7600/apiv2/reset-service-trace -u $ISUSER:$ISPASS
  9. Start the trace. 
    curl --cert tls.crt --key tls.key -k -X POST https://localhost:7600/apiv2/start-service-trace -u $ISUSER:$ISPASS
  10. Re-create the runtime behavior.
     
  11. Stop the trace. 
    curl --cert tls.crt --key tls.key -k -X POST https://localhost:7600/apiv2/stop-service-trace -u $ISUSER:$ISPASS
  12. Copy the trace directory out of the pod. 
    oc cp my-pod-name:/home/aceuser/ace-server/config/common/log/ /path/to/local/folder/log/
This method of enabling trace requires an Integration Node.
If you are using a stand-alone Integration Server, you cannot use the mqsichangetrace command.
  1. Start trace: 
    mqsichangetrace <Int.Node> -e <Int. Server> -t -l debug -r -c 200000 
  2. Re-create the runtime behavior.
     
  3. Stop trace: 
    mqsichangetrace <Int.Node> -e <Int. Server> -t -l none 
  1. Open the server.conf.yaml file
    • Stand-alone Integration Server: workdir
    • Int.Node owned Int. Server: {work path}/components/{Int.Node}/servers/{Int. Server}/overrides
  2. Locate the lines for trace settings: 
    #trace: 'none' # choose 1 of : none|debug|diagnostic
#traceSize: '1G' # Set the service trace size
  3. Enable the trace by uncommenting and updating the lines. Create the lines if they do not exist: 
    trace: 'debug' # choose 1 of : none|debug|diagnostic
traceSize: '1G' # Set the service trace size
  4. Restart the Int. Server.
     
  5. Re-create the runtime behavior.
     
  6. Disable the trace by updating the 'trace' line to show 'none': 
    trace: 'none' # choose 1 of : none|debug|diagnostic
traceSize: '1G' # Set the service trace size
  7. Restart the Int. Server.

Additional Information

Output location
Traces are written to the Integration Server's work directory or the multi-user work path:
 - <workdir>/config/common/log
 - /var/mqsi/common/log (AIX and Linux)
 - C:\ProgramData\IBM\mqsi\common\log (Windows)
Performance Impact
A "service level" trace logs the internal function calls within Int. Server processing and therefore can have a significant impact to performance.
Take care enabling a trace in production and other high throughput environments.
 
Maximum file size
ACE component traces have a maximum file size.
When the maximum file size is reached, the tracing mechanism overwrites the oldest trace data.
The '-c' flag on the mqsichangetrace command, or the traceSize in the server.conf.yaml file, overrides the maximum size for the trace, in KB.
Capturing startup events
The mqsichangetrace command can be run with the Int. Server stopped.
Updating the server.conf.yaml file requires the Int.Node, Int. Server, or both to be restarted.
Enabling trace in either of these ways traces startup activities.
 
Capturing abends and errors
ACE uses the '-m temp' flag to auto stop the trace on events.  
This flag is set on the first mqsichangetrace command (when you enable the trace) and removes the need to run the command to stop the trace.
 - Use '-m temp' to stop tracing when an Int. Server stops (for example to capture an abend).
 - Use '-m temp=####' top stop tracing when a particular BIP message is written (for example to capture a specific error message).
The Int. Server must be running for this flag to be allowed!
 

 

Document Location

Worldwide

[{"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSDR5J","label":"IBM App Connect Enterprise"},"Component":"Trace;Documentation","Platform":[{"code":"PF016","label":"Linux"},{"code":"PF033","label":"Windows"}],"Version":"All Versions","Edition":"","Line of Business":{"code":"LOB67","label":"IT Automation \u0026 App Modernization"}}]

Product Synonym

ACE

Document Information

Modified date:
18 April 2024

UID

ibm10886367

Page Feedback