Docker Host sensor

The Docker Host Sensor (DHS) discovers Docker Hosts, host attributes, containers, networking, image and storage related information.

Sensor name that is used in the GUI and logs

DockerHostSensor

Elements discovered by the sensor

The sensor discovers the following elements:
  • Docker Hosts
  • Docker Containers
  • Docker Volumes
  • Docker Networks
  • Dockers Images

A Docker Host is represented by a blue-colored Docker whale design icon, and Docker container using square shaped four-stacked shipping containers.

The Docker Host Sensor uses REST APIs to retrieve the discovery related information from the Docker host machine running the 'dockerd' daemon process/application. The retrieved data primarily comprises of attribute data that is required to match naming rules and create valid model objects.

Prerequisites

  • The Docker daemon/application is running on a target Linux machine
  • For successful discovery of Docker Host, REST support must be enabled on the target machine
  • Ports for web service communication must be defined. By default, port value resulting from GenericServerSensor processing is used. If your Docker Host uses port-mapping, or, non-standard port, modify the value of the portList property in the discovery profile. For details, see 'Configuring the discovery profile'
  • Single set of TLS certificate is applicable for Agile Service Manager communication to all the Docker Host
  • Enable or Disable of TLS for discovery will have a uniform behavior across ALL Docker hosts defined within the scope
    • Either applicable to ALL, or, NONE Docker Hosts

A Docker Host is represented by a blue-colored Docker whale design icon, and, Docker container using square shaped four-stacked shipping containers.

The Docker Host Sensor uses REST APIs to retrieve the discovery related information from the Docker host machine running the 'dockerd' daemon process/application. The retrieved data primarily comprises of attribute data that is required to match naming rules and create valid model objects.

Security issues

No specific access-list entry is required. For TLS based security details, see Connection to Docker Host below:

Connection to Docker Host

The Docker Host Sensor can discover data from Docker Host through 2 modes: non-TLS mode, and, TLS mode.

Non-TLS mode

The non-TLS mode is the default mode. It retrieves data via web services and doesn’t require authentication. This mode is recommended in private network, or, private cloud deployments in customer premises.

TLS mode

The TLS mode is secure mode of communicating with the Docker Host. It verifies the TLS certificates installed in Agile Service Manager and target Docker Host. To use this mode, you must set the enableTLS property to true, along with configuring the certificate paths defined in discovery profile. For details, see 'Configuring the discovery profile'. For manually generating the TLS certificates for TADDM and Docker host, see 'Manual TLS certificate generation for non UCP'.

Model objects with associated attributes

The Docker Host Sensor creates model objects with associated attributes. The attributes indicate the type of information that the sensor collects about Docker Host resources in your IT environment.

The sensor creates the following model objects. The attributes that are associated with each model object are shown below the model object name.

app.docker.dockerhost.DockerHost

  • Name
  • VersionString
  • DockerContainers
  • DockerImages
  • DockerNetworks
  • DockerVolumes
  • Host
  • XA

    o Architecture

    o KernelVersion

    o OperatingSystem

    o OSType

    o RunningContainers

    o StoppedContainers

    o TotalContainers

app.docker.dockerhost.DockerContainer

  • Name
  • Parent
  • RuntimeProcesses
  • DockerContainerStatus
  • DockerImages
  • DockerNetworks
  • DockerVolumes

app.docker.dockerhost.DockerImage

  • DockerHost
  • ImageName
  • DockerContainer

app.docker.dockerhost.DockerNetwork

  • Name
  • SubnetAddress
  • DockerHost
  • DockerContainer

app.docker.dockerhost.DockerVolume

  • Name
  • DockerHost
  • DockerContainer

sys.RuntimeProcess (applicable to processes within a container)

  • PID
  • Command
  • PPID
  • User
  • CmdLine (refers to the full command)

Configuring the sensor

Before using the Docker Host Sensor, you must configure it.

Configuring the discovery profile:

By default, the Docker Host Sensor is enabled for a Level 3 discovery. Once enabled, it runs in a non-TLS mode by default. The sensor discovers all Docker Host containers, including the ones which are not running. To discover only containers which are running, or, for switching to TLS mode, create a discovery profile for the Docker Host sensor, and customize the sensor settings.

Properties

You can modify the following properties and attributes:

portList

It refers to the port(s) to be used for web service communication on Docker Host. By default, port value returned by GenericServerSensor processing is used. If your Docker Host uses port-mapping, or, non-standard port (or, comma-separated list of ports), specify the value accordingly.

enableTLS

It refers to the connection mode between Agile Service Manager and Docker Host.

The default value is false.

pathStore

Local path on Agile Service Manager discovery server where all the TLS/security certificates are placed.

caFileName

Certificate Authority file name.

cerFileName

Client certificate file name

keyFileName

Client key file name

Enable REST support on Docker Host

On a Docker Host following configuration changes need to be done:

  1. Enable the REST APIs on Docker host.
    • Login to the Docker host machine using 'root' credentials.
    • Create/Update the following file on Docker Host:
      vim /etc/systemd/system/docker.service.d/remote-api.conf

      with following content:

      [Service]
      ExecStart=
      ExecStart=/usr/bin/dockerd -H tcp://<DockerHost-IP>:2376 -H unix:///var/run/docker.sock
  2. Restart the 'dockerd' daemon and validate status via command line:
    service docker restart
    ps -aef | grep -i dockerd

Manual TLS certificate generation for non UCP

In case TLS mode is enabled and certificates are not available, you can manually generate these certificates in a Linux machine as below.

A. Docker Host:

On Docker host machine, generate CA private and public keys per the stepwise procedure mentioned below. Remember the key sample below is for example, and you must provision per your security standards.

  1. Login to the Docker Host machine using 'root', or, another user having 'super-user' privileges.
  2. Make a local directory via commands.
    mkdir docker_certificates
    cd docker_certificates
  3. Execute command:
    a. openssl genrsa -aes256 -out ca-key.pem 4096
    • 1. Enter any pass phrase for generating ca-key.pem and store it safely
    b. openssl req -new -x509 -days 365 -key ca-key.pem -sha256 -out ca.pem
    • 1. Enter the password entered in step (3.a.1)
    • 2. Enter the values asked
    • 3. Enter Docker Host 'domain.com' in Fqdn
  4. Using the CA, create a server key and certificate signing request (CSR) via commands
    a. openssl genrsa -out server-key.pem 4096
    b. openssl req -subj '/CN=$HOST' -sha256 -new -key server-key.pem -out server.csr
    • 1. Where, $HOST is hostname of the Docker Host.
  5. TLS connections can be made via IP address or DNS name, the IP addresses need to be specified when creating the certificate via command:
    a. echo subjectAltName = DNS:$HOST,IP:<DockerHost-IP> > extfile.cnf
    • 1. where, $HOST is hostname of the Docker Host.
    a.echo extendedKeyUsage = serverAuth >> extfile.cnf
  6. Now, generate the key via command:
    a.openssl x509 -req -days 365 -sha256 -in server.csr -CA ca.pem -CAkey ca-key.pem \
    -CAcreateserial -out server-cert.pem -extfile extfile.cnf
    • 1. Enter the password provided in step (3.a.1).
  7. Remove unnecessary files and set the permissions correctly:
    rm -v server.csr
    chmod -v 0400 ca-key.pem server-key.pem
    chmod -v 0444 ca.pem server-cert.pem
  8. Start the Docker daemon using TLS verfication:
    a.dockerd --tlsverify --tlscacert=ca.pem --tlscert=server-cert.pem --tlskey=server-key.pem -H=0.0.0.0:2376
    Note: In case of TLS support for multiple Docker Hosts, execute steps 1-3 ONLY once, and steps 4-8 need to be executed separately for each host to generate the necessary TLS certificates for the hosts.

B. Agile Service Manager Machine

TLS client certificates for Agile Service Manager machine (corresponding to the ones generated for Docker Host) can be generated manually. On the TADDM host machine, generate CA private and public keys by following the stepwise procedure mentioned below:

  1. Login to the Agile Service Manager machine using 'root' user credentials
  2. Make a local directory via commands
    a. mkdir taddm_certificates
    b. cd taddm_certificates
  3. Using the CA, create a server key and certificate signing request (CSR) via commands:
    a. openssl genrsa -out key.pem 4096
    b. openssl req -subj '/CN=client' -new -key key.pem -out client.csr
    c. echo extendedKeyUsage = clientAuth >> extfile.cnf
  4. Sign the private key via command:
    a. openssl x509 -req -days 365 -sha256 -in client.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out cert.pem -extfile extfile.cnf

    • 1. provide ca.pem and ca-key.pem generated in section A: step (3.a, 3.b)

    • 2. Enter the password provided in section A: step (3.a.1)

  5. Remove unnecessary files and set the permissions correctly:
    a. rm -v client.csr
    b. chmod -v 0400 ca-key.pem key.pem
    c. chmod -v 0444 ca.pem cert.pem
    d. cd ../
    e. chown -R taddmusr:taddmusr taddm_certificates
    f. chown -R taddmusr:taddmusr taddm_certificates
  6. Validate the TLS connection with DockerHost machine using the following commands:
    curl https://<Dockerhost-IP>:<Docker-Port>/_ping --cert ./cert.pem --key key.pem --cacert ca.pem
    e.g. curl https://<Dockerhost-IP>:<2376>/_ping --cert ./cert.pem --key key.pem --cacert ca.pem

Manual TLS certificate generation for UCP

  1. Login to the Agile Service Manager machine using 'root' user credentials.
  2. Make a local directory via commands:
    a. mkdir taddm_certificates
    b. cd taddm_certificates
  3. Validate the UCP url. It should up and running. For example: https://x.x.x.x:443
  4. Login to Docker UCP user interface.
  5. Download the certificates from UCP user interface:
    • Navigate to admin > My Profile > Client Bundles > New Client Bundle > Generate Client Bundle.
  6. Download the certificates zip file.
  7. Copy the certificates zip file to Agile Service Manager server in taddm_certificates directory created in step 2.
  8. Unzip the certificates zip file to Agile Service Manager server in taddm_certificates directory.
  9. Run the below command to change an ownership of Agile Service Manager certificates:
    chown -R taddmusr:taddmusr taddm_certificates

Troubleshooting the sensor

This topic describes common problems that occur with the Docker Host sensor and presents solutions for those problems.

The sensor fails with description ` CTJTD1585E Error – Docker host is not reachable on any of these ports:`

Problem : Remote web API may not be enabled on Docker host.

SolutionValidate using 'ps - eaf | grep dockerd' to see which port is used by 'dockerd' daemon process. The output should look something like below:

/usr/bin/dockerd -H tcp://9.158.143.51:2376 -H unix:///var/run/docker.sock

To enable remote API support, see 'Enable REST support' on Docker Host.

The sensor fails with description `CTJTD1587E/ CTJTD1590E Error – TLS configuration mismatch between Docker Host sensor and remote node:’

Problem : The problem occurs because of mismatch in configuration for Docker Host sensor and the Docker Host remote node. TLS may be enabled on one, and, disabled for the other.

Solution : Validate and configure the enableTLS property for Docker Host sensor correctly.

The sensor fails with description ` CTJTD1589E Error – Issue with TLS path-store directory`

Problem : Either, the pathstore directory configured in the Docker Host sensor configuration is invalid, or, is lacking the correct permissions.

Solution : Check for existence of the configured pathstore directory on the Agile Service Manager discovery server. If the directory exists, validate that it has been granted correct permissions.

drwxr-xr-x.  2 taddmusr taddmusr 4096 Nov 24 08:28 taddm_certificates

The sensor fails with description ` Failed: HTTP error code : 503`

Problem : In case Agile Service Manager is unable to connect via REST to Docker daemon/application on target node, the sensor can fail with an error message.

Solution : If dockerd process/application is running good, it becomes important to validate the specific port on which the daemon/process is listening using the command ps - Aef | grep dockerd. The port obtained from output must match with the one Agile Service Manager is trying to connect.

The sensor fails with a `CTJTD3520E Error – A storage error has occurred. Server id:`

Problem : In case there are any missing dependencies of java jars on character set conversion the sensor can fail with a storage error message shown above.

Solution : Validate for any missing Java jars and place the corresponding one in the below directory:

/opt/IBM/taddm/dist/lib/jdbc

Run discovery again.