Kubernetes logging
The best practice for Kubernetes logging is to pipeline the log content to a centralized place that is easy to access, search, and manage retention. Most commercial Kubernetes platforms have a logging service such as ELK or EFK that collect all log data directed to console by default.
Keeping log files external to the Docker containers on volume does not work or is hard to track because different replicas of the same application can be up and down on different hosts or the same host. Often, this either ends up with a conflict (if you map to a local drive) or ends up with lots of log files for each container instance. It is hard to track those log files across host machines or on a network drive for searching and retention management.
For these reasons, it is a good idea to direct InfoSphere® MDM log files of interest to the Docker console. This enables the third party logging service to collect the logs for centralized storage, search, analysis, and retention management.
Viewing the Docker Console log
- Using the
kubectl
command -
- If the pod hosts a single container, use the following command
syntax:
kubectl logs <pod-name> -n <namespace>
- If the pod hosts multiple containers, use the following command
syntax:
kubectl logs <pod-name> <container-name> -n <namespace>
- If the pod hosts a single container, use the following command
syntax:
- Using Docker commands
-
- Run the
docker inspect
command on the worker nodes to get the Docker file location.docker inspect --format='{<.LogPath>}’ <container_id>
- Browse to the file location and use a text editor to view the log file.
- Run the
Piping logs to the Docker Console
To pipe logs to the Docker Console, append the following lines at the end of your startup scripts before creating your pods.
- For the MDM operational server image
- Edit the startup.sh file in the
<MDM_HOME> directory and append the following
lines:
PROFILE_NAME=AppSrv01 SERVER_NAME=server1 PID=$(ps -C java -o pid= | tr -d " ") tail -F /opt/IBM/WebSphere/AppServer/profiles/$PROFILE_NAME/logs/$SERVER_NAME/SystemOut.log --pid $PID -n +0 & tail -F /opt/IBM/WebSphere/AppServer/profiles/$PROFILE_NAME/logs/$SERVER_NAME/SystemErr.log --pid $PID -n +0 >&2 & while [ -e "/proc/$PID" ]; do sleep 1 done
- For the IBM® Stewardship Center for MDM image
- Edit the Configure_ISC.sh file in the /tmp directory
and append the following
lines:
WAS_PROFILE_HOME=/opt/IBM/BPM/v8.6/profiles/NodeProfilePS SERVER_NAME=SingleClusterMember1 PID=$(ps -C java -o pid= | tr -d " “) tail -F $/WAS_PROFILE_HOME/logs/$SERVER_NAME/SystemOut.log --pid $PID -n +0 & tail -F $/WAS_PROFILE_HOME/logs/$SERVER_NAME/SystemErr.log --pid $PID -n +0 >&2 & while [ -e "/proc/$PID" ]; do sleep 1 done
Example: Accessing the logs
Go to the node where the container is running and run the following commands to check the logs on console output.
root@node1:~# docker inspect --format='{{.LogPath}}’ 423020b44c7c
/var/lib/docker/containers/423020b44c7c788db0dc7bdd97799059bfbab5b65f8b4605cace517735623645/423020b44c7c788db0dc7bdd97799059bfbab5b65f8b4605cace517735623645-json.log
cat /var/lib/docker/containers/423020b44c7c788db0dc7bdd97799059bfbab5b65f8b4605cace517735623645/423020b44c7c788db0dc7bdd97799059bfbab5b65f8b4605cace517735623645-json.log
......
18-12-19T08:20:40.309521442Z"}
{"log":"ADMU0128I: Starting tool with the AppSrv01 profile\n","stream":"stdout","time":"2018-12-19T08:20:40.726765389Z"}
{"log":"ADMU3100I: Reading configuration for server: server1\n","stream":"stdout","time":"2018-12-19T08:20:40.727703342Z"}
{"log":"ADMU3201I: Server stop request issued. Waiting for stop status.\n","stream":"stdout","time":"2018-12- 19T08:20:43.523475999Z"}
{"log":"ADMU4000I: Server server1 stop completed.\n","stream":"stdout","time":"2018-12-19T08:20:54.138659675Z"}
{"log":"\n","stream":"stdout","time":"2018-12-19T08:20:54.140771822Z"}
{"log":"ADMU0116I: Tool information is being logged in file\n","stream":"stdout","time":"2018-12-19T08:20:56.589289061Z"}
{"log":"/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/server1/startServer.log\n","stream":"stdout","time":"2018-12-19T08:20:56.589323787Z"}
{"log":"ADMU0128I: Starting tool with the AppSrv01 profile\n","stream":"stdout","time":"2018-12-19T08:20:57.8524332Z"}
{"log":"ADMU3100I: Reading configuration for server: server1\n","stream":"stdout","time":"2018-12-19T08:20:57.855585484Z"}
{"log":"ADMU3200I: Server launched. Waiting for initialization status.\n","stream":"stdout","time":"2018-12-19T08:21:01.155541209Z"}
{"log":"ADMU3000I: Server server1 open for e-business; process id is 739\n","stream":"stdout","time":"2018-12-19T08:21:25.505406075Z"}