Troubleshooting
Problem
The journalctl command can be used to display messages from services, useful for troubleshooting errors and failures.
Symptom
An application framework or an application container service is not starting. The list of services responsible for the applications and framework functionality is published under technote Services responsible for the applications and framework functionality.
Diagnosing The Problem
Logs collected by systemd can be viewed by using
journalctl. The journal is implemented with the journald daemon and it retrieves messages from the kernel, systemd services, and other sources.These logs are gathered in a central location, which makes it easy to review. The log records in the journal are structured and indexed, and as a result,
journalctl is able to present your log information in various useful formats.Resolving The Problem
The journalctl utility can be used to access and manipulate the data held within the journal.
You can view logs by service within a date range:
journalctl -xu <service> --since "2020-01-01 12:10:10" --until "2020-02-01 12:10:10"
To view a journalctl service log, use the command:
journalctl -xu <service name>
Examples of journalctl commands.
For a specific service (docker in this case) you would run:
[root@apphost system]# journalctl -xu docker
-- Logs begin at Mon 2020-05-11 01:52:04 GMT, end at Mon 2020-05-11 17:45:48 GMT. --
May 11 01:52:22 74apphost.isslab.usga.ibm.com systemd[1]: Starting Docker Application Container Engine...
May 11 01:52:22 74apphost.isslab.usga.ibm.com configure-docker-network.sh[7752]: Configuring docker networking...
May 11 01:52:22 74apphost.isslab.usga.ibm.com configure-docker-network.sh[7752]: Requested docker interfaces: docker0 dockerApps
May 11 01:52:22 74apphost.isslab.usga.ibm.com configure-docker-network.sh[7752]: Configuring docker interface docker0...
May 11 01:52:22 74apphost.isslab.usga.ibm.com configure-docker-network.sh[7752]: Docker interface docker0 configured.
May 11 01:52:22 74apphost.isslab.usga.ibm.com configure-docker-network.sh[7752]: Configuring docker interface dockerApps...
May 11 01:52:22 74apphost.isslab.usga.ibm.com configure-docker-network.sh[7752]: Docker interface dockerApps configured.
May 11 01:52:22 74apphost.isslab.usga.ibm.com configure-docker-network.sh[7752]: Docker networking configured.
For a specific container service you would run:
[root@apphost system]# journalctl -xu container@13719764573200128895.service
-- Logs begin at Mon 2020-05-11 01:52:04 GMT, end at Mon 2020-05-11 17:44:34 GMT. --
May 11 01:52:29 74apphost.isslab.usga.ibm.com systemd[1]: Starting Container created and managed by the conman service...
May 11 01:52:29 74apphost.isslab.usga.ibm.com systemd[1]: Started Container created and managed by the conman service.
May 11 01:52:29 74apphost.isslab.usga.ibm.com conwrap[13618]: time="2020-05-11T01:52:29Z" level=info msg="conwrap starting" container=qapp-1055-9yXWC1G4 tid=135563
May 11 01:52:29 74apphost.isslab.usga.ibm.com conwrap[13618]: time="2020-05-11T01:52:29Z" level=error msg="Unable to retrieve data for URL" container=qapp-1055-9yX
May 11 01:52:29 74apphost.isslab.usga.ibm.com conwrap[13618]: time="2020-05-11T01:52:29Z" level=error msg="Failed to generate arguments for docker run." container=
May 11 01:52:29 74apphost.isslab.usga.ibm.com conwrap[13618]: time="2020-05-11T01:52:29Z" level=error msg="Failed to generate docker run arguments from the environ
May 11 01:52:29 74apphost.isslab.usga.ibm.com systemd[1]: container@13719764573200128895.service: main process exited, code=exited, status=1/FAILURE
May 11 01:52:29 74apphost.isslab.usga.ibm.com systemd[1]: Unit container@13719764573200128895.service entered failed state.
May 11 01:52:29 74apphost.isslab.usga.ibm.com systemd[1]: container@13719764573200128895.service failed.
May 11 01:52:32 74apphost.isslab.usga.ibm.com systemd[1]: container@13719764573200128895.service holdoff time over, scheduling restart.
May 11 01:52:32 74apphost.isslab.usga.ibm.com systemd[1]: Stopped Container created and managed by the conman service.
May 11 01:52:32 74apphost.isslab.usga.ibm.com systemd[1]: Starting Container created and managed by the conman service...
May 11 01:52:32 74apphost.isslab.usga.ibm.com systemd[1]: Started Container created and managed by the conman service.
Procedure to generate the journalctl output as files:
- Login as root with SSH.
- Make a directory to your preferred location:
# mkdir /tmp/appjournallogs -
Change to that directory:
# cd /tmp/appjournallogs -
Copy and paste the following line, and hit enter:
for service in {si-registry,docker,vault-qrd,qradarca-monitor,conman,traefik}; do journalctl -u $service --no-pager > "$service""_journal.txt"; done -
List the log files output generated by the journalctl:
[root@con-primary]# ls -lhtr /tmp/appjournallogs -rw-r--r-- 1 root root 17 May 11 18:38 conman_journal.txt -rw-r--r-- 1 root root 17 May 11 18:38 docker_journal.txt -rw-r--r-- 1 root root 3851 May 11 18:38 qradarca-monitor_journal.txt -rw-r--r-- 1 root root 17 May 11 18:38 si-registry_journal.txt -rw-r--r-- 1 root root 23642 May 11 18:38 traefik_journal.txt -rw-r--r-- 1 root root 17 May 11 18:38 vault-qrd_journal.txt
Document Location
Worldwide
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSBQAC","label":"IBM Security QRadar SIEM"},"ARM Category":[{"code":"a8m0z000000GndcAAC","label":"QRadar-\u003EApps-\u003EApp Framework"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.3.2;7.3.3;7.4.0","Line of Business":{"code":"LOB24","label":"Security Software"}}]
Was this topic helpful?
Document Information
Modified date:
04 January 2023
UID
ibm16208418