Logging and containers

Edit online

Information about supported log levels, container logs and some related errors.

Using json-file as the logging driver

By default, Faspex 5 uses journald as its logging driver. However, you can revert to using json-file, as in previous versions.

Steps to change the logging driver to json-file.

  1. Edit the core.yml and service.yml files located in /opt/aspera/faspex/conf/docker_compose_templates.
  2. Locate the logging section in each file and update it as follows:
    • Set the driver parameter to json-file in the docker-compose YAML file of any Faspex container.
    • Add the following options under the driver parameter: 
      options:
  max-size: "10m"
  max-file: "10"
  3. Example configuration after editing:
    Important: This template is meant to be only an example, do not copy and paste it.

    core.yml

    core:
    image: icr.io/ibmaspera/faspex-core:5.0.5
    container_name: faspex-core
    env_file: [ docker/core.env, docker/db.env ]
    environment: [ SECRET_KEY_BASE=$FASPEX_CORE_SECRET, OLD_SECRET_KEY_BASE=$FASPEX_CORE_OLD_SECRET ]
    ports: ["$FASPEX_CORE_PORT:$FASPEX_CORE_PORT"]
    logging:  { driver: "json-file", options: { max-size: "10m", max-file: "10" } }
 
    networks: [ docker_network ]
    depends_on: [ db ]

    service.yml

    service:
    image: icr.io/ibmaspera/faspex-service:5.0.5
    container_name: faspex-service
    env_file: [ docker/service.env, docker/db.env, docker/core.env ]
    ports: ["$FASPEX_SERVICE_PORT:$FASPEX_SERVICE_PORT"]
    logging:  { driver: "json-file", options: { max-size: "10m", max-file: "10" } }
 
    networks: [ docker_network ]
    depends_on: [ core ]
  4. Run the Faspex setup to apply these changes: 
    faspexctl setup

Redirecting container logs to syslog

While you can use the docker logs service command to access the logs for a container, you can also configure Docker to redirect output to system logs.

  1. Edit or create the /etc/docker/daemon.json file. 
    {
  "log-driver": "syslog",
  "log-opts": {
    "syslog-address": "unixgram:///dev/log",
    "tag": "docker/{{.Name}}" 
  }
}
  2. Restart the docker daemon: 
    systemctl restart docker
  3. Edit the core.yml and service.yml files located at /opt/aspera/faspex/conf/docker_compose_templates:
    1. Change the driver value to syslog in the docker-compose YAML file of any Faspex container.
    2. Remove the max-size: "10m"and max-file: "10"options under the driver parameter.
    For example, after editing core.yml and service.yml :
    Note: This template is only an example, you should not copy and paste it.
    # cat core.yml
  core:
    image: icr.io/ibmaspera/faspex-core:5.0.5
    container_name: faspex-core
    env_file: [ docker/core.env, docker/db.env ]
    environment: [ SECRET_KEY_BASE=$FASPEX_CORE_SECRET, OLD_SECRET_KEY_BASE=$FASPEX_CORE_OLD_SECRET ]
    ports: ["$FASPEX_CORE_PORT:$FASPEX_CORE_PORT"]
    logging:
      driver: "syslog"
 
    networks: [ docker_network ]
    depends_on: [ db ]
 
#####################################################################
 
# cat service.yml
  service:
    image: icr.io/ibmaspera/faspex-service:5.0.5
    container_name: faspex-service
    env_file: [ docker/service.env, docker/db.env, docker/core.env ]
    ports: ["$FASPEX_SERVICE_PORT:$FASPEX_SERVICE_PORT"]
    logging:
      driver: "syslog"
 
    networks: [ docker_network ]
    depends_on: [ core ]
  4. Run faspexctl setup to implement the changes.
  5. Read the /var/log/messages log file to see container logs.

    Faspex containers begin log entries with docker/faspex_container_name. For example, the Faspex core container begins log entries with docker/faspex-core.

    You can use grep and the container name to filter for or filter out specific log entries.
    Filter for the core container: 
    tail -f /var/log/messages | grep "docker/faspex-core"
    Filter out the core container: 
    tail -f /var/log/messages | grep -i "docker/faspex-core"

Log forwarding from Journald to Syslog in Podman

Note: This process applies only for Faspex 5.0.10 running on Podman, and Faspex 5.0.12 running Podman or Docker.

Follow these steps to modify the Journald configuration and enable log forwarding to Syslog in Podman.

  1. Open the Journald configuration file: 
    sudo nano /etc/systemd/journald.conf
  2. Uncomment the following values: 
    [Journal]
ForwardToSyslog=yes
MaxLevelSyslog=debug
  3. Save your changes and restart the Journald service: 
    systemctl restart systemd-journald
  4. To verify that the logs are now being forwarded to Syslog, check the /var/log/messages file.

Faspex log rotation

You can configure Faspex logs to rotate automatically using logrotate. To manage the rotation with logrotate you must be using Syslog.
Note: Journald already handles log rotation based on the configuration of the /etc/systemd/journald.conf file. The json-file rotation is defined once the logging driver is set.
Verify logrotate installation
  1. Check if logrotate is installed using this command: 
    logrotate --version
  2. If it is not installed, you can install it on Red Hat based environments with the following command: 
    sudo yum install logrotate -y
Configure log rotation

Once installed, logrotate configurations are stored in the /etc/logrotate.d/ directory. You can define custom rotation rules for different services by creating or modifying configuration files within this folder.

To apply log rotation for Faspex logs, you need to create/edit the rsyslog file configuration to manage the /var/log/messages file, which Syslog uses to capture Faspex logs.

Define log rotation rules

You can define the log rotation period to happen weekly, monthly, or daily for the /var/log/messages file. This is an example for a weekly configuration of the /etc/logrotate.d/rsyslog file:

/var/log/messages
{
    missingok
    weekly
    rotate 4
    compress
    dateext
    create 640 root root
}
Applying this configuration will result in the following:
  • Weekly log rotation
  • A compressed log file each week
  • Rotation of the last compressed files every 4 weeks (deleting the older .gz file every month).
  • The /var/log/messages file is cleared after rotation

If journald is active, modify the /etc/systemd/journald.conf file to configure the log rotation:

[Journal]
Storage=persistent   
SystemMaxUse=500M      
SystemKeepFree=1G    
SystemMaxFileSize=100M  
SystemMaxFiles=4        
Compress=yes        
MaxRetentionSec=4week

Viewing logs

Follow these steps to view all logs and container logs.

Viewing container logs in Docker

Run the following command to access the container logs in Docker:

docker logs <container_name>

Viewing logs with Journald in Podman

Starting with Faspex 5.0.10, the log driver used for containers running in Podman is going to be Journald.

To view all logs: 
journalctl -f
To view and follow new messages for a specific container: 
journalctl -f CONTAINER_NAME=<name-of-the-container>
Replace <name-of-the-container> with the actual name of the container. For example, for Faspex core the command would be: 
journalctl -f CONTAINER_NAME=faspex-core
To view all logs for a specific container: 
journalctl CONTAINER_NAME=<name-of-the-container>
Replace <name-of-the-container> with the actual name of the container. For example, for Faspex service the command would be: 
journalctl CONTAINER_NAME=faspex-service

Supported log levels

You can set the desired log level to filter out messages. They are listed in increasing order of severity.

FASPEX_SERVICE supported log levels:

  • debug
  • info
  • warning
  • error
  • fatal
  • panic

FASPEX_CORE supported log levels:

  • debug_sql
  • debug
  • info
  • warning
  • error

Setting log levels

You can configure log levels for Faspex by adding the corresponding variable to each .env file(s) located in /opt/aspera/faspex/conf/docker/ For example: FASPEX_SERVICE_LOG_LEVEL=debug

Variable .env file
FASPEX_UTILITY_LOG_LEVEL=debug utility.env
FASPEX_CORE_LOG_LEVEL=debug core.env
FASPEX_SERVICE_LOG_LEVEL=debug service.env
FASPEX_ROUTER_LOG_LEVEL=debug router.env
FASPEX_UI_LOG_LEVEL=debug ui.env

502 Bad Gateway error in the Web UI

For this error it is important to ensure consistency across the port numbers that the Faspex 5 containers use. If you want to make sure all environmental variables have the same port number, you can update the values in the /opt/aspera/faspex/conf/docker/router.env file. Here's an example of what your file should look like:

FASPEX_ROUTER_CORE_URL=core:3000

FASPEX_ROUTER_UI_URL=ui:3000

FASPEX_ROUTER_SERVICE_URL=service:3000

FASPEX_ROUTER_UTILITY_URL=utility:3000

Make sure that all four parameters (FASPEX_ROUTER_CORE_URL, FASPEX_ROUTER_UI_URL, FASPEX_ROUTER_SERVICE_URL, and FASPEX_ROUTER_UTILITY_URL) have the same port number to avoid any inconsistencies and potential 502 Bad Gateway errors. After making these changes, you'll need to restart faspex to apply the new configuration.

faspexctl restart

Faspex package debug helper

Follow these steps to increase logging levels to debug in core and service containers and send a new package.

Note: We're using pkg_id 123 as an example.

Steps for core container:

  1. To see the stats of all packages run: 
    docker logs -f faspex-core 2>&1 | grep 'Processing transfer_stats'
  2. To see the stats of a specific package run: 
    docker logs -f faspex-core 2>&1 | grep 'Processing transfer_stats: pkg_id: 123'

Steps for service container:

  1. To see the stats of all packages run: 
    docker logs -f faspex-service 2>&1 | grep 'PKG_ID'
  2. To see the stats of a specific package run: 
    docker logs -f faspex-service 2>&1 | grep 'PKG_ID: 123'

Steps in the UI:

Go to Admin app > Job queues and search for "delivery_id":123. If the search returns multiple pkg_poller records, click every record and look for direction: upload. If you don’t find it, that means Faspex finished polling on the package.