IBM Support

Howto tracing with LTTng

Technical Blog Post


Abstract

Howto tracing with LTTng

Body


image 
By:  Wainer dos Santos Moschetta. 

Overview
The Linux Trace Toolkit Next Generation (LTTng) is a toolkit for trace and visualization of events produced by both the Linux kernel and applications (user-space).
Version 2.x offers several improvements in relation to previous 1.x series, including:
 
 
  • Introduction of a new trace file format called CTF(Common Trace Format)
  • Beyond default kernel events, it allows trace of user-space applications
  • New implementation of ring buffer algorithm
  • Able to attach context information to events

Building & installing
Note: In this section I will show how to build LTTng from source although it is already delivered in some Linux distributions, for instance, in RPM packages for pcc64 in OpenSuse Linux and Fedora 17. So you might want to use a stable version from your chosen distro or just build latest yourself.

Its source code is version'ed in different git trees, one for each component (see table 1).

Table 1. LTTng source components

Component

Description

GIT URL

liburcu

Library that implement RCU (Read-Copy-Update)mechanism in user-space

git://git.lttng.org/userspace-rcu.git

LTTng-Tools

Provides main client that control execution of LTTng

git://git.lttng.org/lttng-tools.git

LTTng-modules

Kernel modules for tracing kernel events.

git://git.lttng.org/lttng-modules.git

LTTng-UST

Enable tracing in user-space

git://git.lttng.org/lttng-ust.git
 
 

# Cloning LTTng source code
#
$ git clone git://git.lttng.org/lttng-tools.git
$ git clone git://git.lttng.org/lttng-modules.git
$ git clone git://git.lttng.org/lttng-ust.git
$ git clone git://git.lttng.org/userspace-rcu.git

Requirements to build the components:
  • Common to all are GNU Autotools and Libtool
  • At the time this post is written, requires kernel >= 2.6.38 to build lttng-modules
  • Some components rely on third-party libraries so take a look at README file in each component


# Buiding and installing liburcu library
#
$ cd userspace-rcu
$ ./bootstrap
$ ./configure
$ make
$ make install
$ sudo ldconfig



# Bulding and installing lttng-ust
#
$ cd lttng-ust
$ ./bootstrap
$ ./configure
$ make
$ make install
$ ldconfig



# Building and installing lttng-tools
#
$ cd lttng-tools
$ ./bootstrap
$ ./configure
$ make
$ make install
$ ldconfig



# Building and installing lttng-modules
#
$ cd lttng-modules
$ make
$ sudo make modules_install
$ sudo depmod -a

There is a post-installation procedure that must be done in order to allow non-root users to (transparently) start LTTng daemon for monitoring kernel events. These users must be added in the tracing group as shown below (in Fedora 17):


# Create group if it doesn't exist
$ sudo groupadd -r tracing

# Add <username> to the group
$ sudo usermod -aG tracing <username>

Managing a trace session
 LTTng tracing relies on concept of session. The table 2 shows commands to manage a session lifecycle.

Table 2. Commands to manage tracing session

Command

Description

create NAME

Create a session with NAME. By default, tracing files are held in ~/lttng-traces but it may be redefined with option -o

set-session NAME

Used to switch between sessions, setting current to NAME.

start

Start tracing

stop

Stop tracing

destroy NAME

Destroy the session with NAME. The option -a or –all may be used to destroy all the sessions.

list NAME

Show information regarding session with NAME or list all sessions if NAME is omitted.

Below is an example of tracing session with LTTng where it is monitored system behavior after booting up an instance of Firefox browser.


$ sudo lttng-sessiond &
$ lttng create demo_session 
Session demo_session created. 
Traces will be written in /home/wainersm/lttng-traces/demo_session-20121030-233238
$ lttng start 
Tracing started for session demo_session
$ firefox & 
$ lttng stop 
Waiting for data availability 
Tracing stopped for session demo_session 
$ lttng destroy demo_session 
Session demo_session destroyed

Notice from example above lttng-sessiond (daemon) is initialized with sudo (i.e. root). The trace session may be started/stopped several times, allowing you to some parameters (i.e. add/remove events and context information).

Managing events in a trace session
The tool is able to trace events emitted by kernel and application which are made available through several infrastructures just like Kprobe, Ftrace, tracepoints and also processor PMU (Performance Monitoring Unit). Therefore, lttng has a set of commands to manage events to be monitored as shown in table 3.

Table 3. Commands to manage events tracing

Comand

Description

list [-k] [-u]

List available events of kernel (-k) and user-space (-u)

enable-event [options]

Add events to the session. The  [options] filter which events should be traced. For example, “-a -k --syscall” is used to add syscall events.

disable-event [opções]

Remove events to the session. The  [options] filter which events should be removed.

add-context -t [type]

Add information context to an event. As of this post is written, [type] may be pid, procname, prio, nice, vpid, tid, pthread_id, vtid, ppid, vppid as well as available PMU events.


Below listing show how to display all kernel events available for tracing.

$ lttng list -k 
Kernel events: 
------------- 
      timer_init (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
      timer_start (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
      timer_expire_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
      timer_expire_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
      timer_cancel (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
      hrtimer_init (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
      hrtimer_start (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
      hrtimer_expire_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
      hrtimer_expire_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
      hrtimer_cancel (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
      itimer_state (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
      itimer_expire (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
      lttng_statedump_start (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
      lttng_statedump_end (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
      lttng_statedump_process_state (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
      lttng_statedump_file_descriptor (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
      lttng_statedump_vm_map (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
      lttng_statedump_network_interface (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
      lttng_statedump_interrupt (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
      signal_generate (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
      signal_deliver (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
      signal_overflow_fail (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
      signal_lose_info (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
      sched_kthread_stop (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
      sched_kthread_stop_ret (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
      sched_wakeup (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
      sched_wakeup_new (loglevel: TRACE_EMERG (0)) (type: tracepoint) 
 (…)

The example below shows how to add all kernel events for tracing in a session: 

$ lttng enable-event -a -k 
All kernel events are enabled in channel channel0

Analyzing gathered data
Usually a tracing session generates a huge amount of data, which makes it unreadable by humans. So LTTng project provides some visualizing tools to easy analysis of traced data, as follows: 
  • Babeltrace
    • It is a library and command-line tool able to read and convert trace data stored in different formats (including CTF).
  • LTTV Viewer
    • It is a graphical visualizer (GTK+) able to make analysis of trace data produced by LTT, but it currently doesn't support CTF.
  • Eclipse LTTng
    • The Eclipse LTTng is a set of plug-ins able to visualize LTTng gathered data. Currently it is maintained by a pretty active community.

The LTTng plug-ins TMF by default in latest version (4.2.1) Eclipse IDE for C/C++ Developers, which may be found at http://www.eclipse.org/downloads/
Below is an screenshot of Eclipse view with a loaded LTTng tracing file.

image 
 
References

[{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"HW1W1","label":"Power ->PowerLinux"},"Component":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"","Edition":"","Line of Business":{"code":"","label":""}}]

UID

ibm16171537

Page Feedback