Troubleshooting

If SELinux is installed on your host, then you need to create a policy to allow the agent to use eBPF. SELinux may prevent unconfined services similar to the host agent from issuing the bpf_* syscall that the eBPF sensor uses to instrument the Linux kernel. To verify, you must look in the log entries of the Audit system, which is stored by default in the /var/log/audit/audit.log.

The following example shows the steps to create policy for a Red Hat Linux machine:

1. Run the following command:

   $ cat /var/log/audit/audit.log | grep ebpf
   type=AVC msg=audit(1598891569.452:193): avc:  denied  { map_create } for  pid=1612 comm="ebpf-preflight-"
   scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:unconfined_service_t:s0
   tclass=bpf permissive=0
   type=SYSCALL msg=audit(1598891569.452:193): arch=c000003e syscall=321 success=no exit=-13
   a0=0 a1=7ffc0e1f5020 a2=78 a3=fefefefefefefeff items=0 ppid=1502 pid=1612 auid=4294967295
   uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="ebpf-preflight-"
   exe="/opt/instana/agent/data/repo/com/instana/ebpf-preflight/0.1.6/ebpf-preflight-0.1.6.bin"
   subj=system_u:system_r:unconfined_service_t:s0 key=(null)
   type=PROCTITLE msg=audit(1598891569.452:193):
   proctitle="/opt/instana/agent/data/repo/com/instana/ebpf-preflight/0.1.6/ebpf-preflight-0.1.6.bin"
   

   Note: Audit log files are usually rotated. Therefore, you must run this command not long after starting the host agent.

   In the log file, you might see the map_create syscall is denied. To allow the eBPF sensor to make the syscall, you must create the SELinux policy and the program audit2allow.

2. On Red Hat systems, install the policy as follows:

   yum install policycoreutils-python
   

3. With audit2allow, create raw policy files based on the log entries as shown in the following example:

grep ebpf /var/log/audit/audit.log | audit2allow -M instana_ebpf

The processing command creates the following files:

ls -Al | grep instana_ebpf

-rw-r--r--. 1 root                    root                      886 31. Aug 18:31 instana_ebpf.pp
-rw-r--r--. 1 root                    root                      239 31. Aug 18:31 instana_ebpf.te

The raw policy file instana_ebpf.te contains an instruction to allow the denied syscall as shown in the following example:

$ cat instana_ebpf.temodule instana_ebpf 1.0;require {
        type unconfined_service_t;
        class bpf map_create;
}#============= unconfined_service_t ==============#!!!! This avc is allowed in the current policy
allow unconfined_service_t self:bpf map_create;

This policy allows any application of type unconfined (very generic) to make the map_create syscall.

1. In addition, the eBPF sensor needs a few more syscalls. You must edit the instana_ebpf.te file as shown in the following example:

   $ cat instana_ebpf.te module instana_ebpf 1.0;require {
           type unconfined_service_t;
           class bpf { map_create map_read map_write prog_load prog_run };
   }#============= unconfined_service_t ==============#!!!! This avc is allowed in the current policy
   allow unconfined_service_t self:bpf { map_create map_read map_write prog_load prog_run };
   

2. Re-write the file to a binary format as the instana_ebpf.mod file:

   $ checkmodule -M -m -o instana_ebpf.mod instana_ebpf.te
   checkmodule:  loading policy configuration from instana_ebpf.te
   checkmodule:  policy configuration loaded
   checkmodule:  writing binary representation (version 19) to instana_ebpf.mod
   

3. Repackage the instana_ebpf.mod file as a loadable module:

   semodule_package -o instana_ebpf.pp -m instana_ebpf.mod
   

4. Apply the policy package:

semodule -i instana_ebpf.pp

Any unconfined process, such as the host agent can now make syscalls.