About cookies on this site Our websites require some cookies to function properly (required). In addition, other cookies may be used with your consent to analyze site usage, improve the user experience and for advertising. For more information, please review your options. By visiting our website, you agree to our processing of information as described in IBM’sprivacy statement. To provide a smooth navigation, your cookie preferences will be shared across the IBM web domains listed here.
Technical Blog Post
Abstract
Howto tracing with LTTng
Body
By: Wainer dos Santos Moschetta.
Overview
The Linux Trace Toolkit Next Generation (LTTng) is a toolkit for trace and visualization of events produced by both the Linux kernel and applications (user-space).
Version 2.x offers several improvements in relation to previous 1.x series, including:
Building & installing
Version 2.x offers several improvements in relation to previous 1.x series, including:
- Introduction of a new trace file format called CTF(Common Trace Format)
- Beyond default kernel events, it allows trace of user-space applications
- New implementation of ring buffer algorithm
- Able to attach context information to events
Building & installing
Note: In this section I will show how to build LTTng from source although it is already delivered in some Linux distributions, for instance, in RPM packages for pcc64 in OpenSuse Linux and Fedora 17. So you might want to use a stable version from your chosen distro or just build latest yourself.
Its source code is version'ed in different git trees, one for each component (see table 1).
Table 1. LTTng source components
|
Component |
Description |
GIT URL |
|---|---|---|
|
liburcu |
Library that implement RCU (Read-Copy-Update)mechanism in user-space |
git://git.lttng.org/userspace-rcu.git |
|
LTTng-Tools |
Provides main client that control execution of LTTng |
git://git.lttng.org/lttng-tools.git |
|
LTTng-modules |
Kernel modules for tracing kernel events. |
git://git.lttng.org/lttng-modules.git |
|
LTTng-UST |
Enable tracing in user-space |
git://git.lttng.org/lttng-ust.git |
# Cloning LTTng source code
#
$ git clone git://git.lttng.org/lttng-tools.git
$ git clone git://git.lttng.org/lttng-modules.git
$ git clone git://git.lttng.org/lttng-ust.git
$ git clone git://git.lttng.org/userspace-rcu.git
Requirements to build the components:
- Common to all are GNU Autotools and Libtool
- At the time this post is written, requires kernel >= 2.6.38 to build lttng-modules
- Some components rely on third-party libraries so take a look at README file in each component
# Buiding and installing liburcu library
#
$ cd userspace-rcu
$ ./bootstrap
$ ./configure
$ make
$ make install
$ sudo ldconfig
# Bulding and installing lttng-ust
#
$ cd lttng-ust
$ ./bootstrap
$ ./configure
$ make
$ make install
$ ldconfig
# Building and installing lttng-tools
#
$ cd lttng-tools
$ ./bootstrap
$ ./configure
$ make
$ make install
$ ldconfig
# Building and installing lttng-modules
#
$ cd lttng-modules
$ make
$ sudo make modules_install
$ sudo depmod -a
There is a post-installation procedure that must be done in order to allow non-root users to (transparently) start LTTng daemon for monitoring kernel events. These users must be added in the tracing group as shown below (in Fedora 17):
# Create group if it doesn't exist
$ sudo groupadd -r tracing
# Add <username> to the group
$ sudo usermod -aG tracing <username>
Managing a trace session
LTTng tracing relies on concept of session. The table 2 shows commands to manage a session lifecycle.
Table 2. Commands to manage tracing session
|
Command |
Description |
|---|---|
|
create NAME |
Create a session with NAME. By default, tracing files are held in ~/lttng-traces but it may be redefined with option -o |
|
set-session NAME |
Used to switch between sessions, setting current to NAME. |
|
start |
Start tracing |
|
stop |
Stop tracing |
|
destroy NAME |
Destroy the session with NAME. The option -a or –all may be used to destroy all the sessions. |
|
list NAME |
Show information regarding session with NAME or list all sessions if NAME is omitted. |
Below is an example of tracing session with LTTng where it is monitored system behavior after booting up an instance of Firefox browser.
$ sudo lttng-sessiond &
$ lttng create demo_session
Session demo_session created.
Traces will be written in /home/wainersm/lttng-traces/demo_session-20121030-233238
$ lttng start
Tracing started for session demo_session
$ firefox &
$ lttng stop
Waiting for data availability
Tracing stopped for session demo_session
$ lttng destroy demo_session
Session demo_session destroyed
Notice from example above lttng-sessiond (daemon) is initialized with sudo (i.e. root). The trace session may be started/stopped several times, allowing you to some parameters (i.e. add/remove events and context information).
Managing events in a trace session
The tool is able to trace events emitted by kernel and application which are made available through several infrastructures just like Kprobe, Ftrace, tracepoints and also processor PMU (Performance Monitoring Unit). Therefore, lttng has a set of commands to manage events to be monitored as shown in table 3.
Table 3. Commands to manage events tracing
|
Comand |
Description |
|---|---|
|
list [-k] [-u] |
List available events of kernel (-k) and user-space (-u) |
|
enable-event [options] |
Add events to the session. The [options] filter which events should be traced. For example, “-a -k --syscall” is used to add syscall events. |
|
disable-event [opções] |
Remove events to the session. The [options]
filter which events should be removed. |
|
add-context -t [type] |
Add information context to an event. As of this post is written, [type] may be pid, procname, prio, nice, vpid, tid, pthread_id, vtid, ppid, vppid as well as available PMU events. |
Below listing show how to display all kernel events available for tracing.
$ lttng list -k
Kernel events:
-------------
timer_init (loglevel: TRACE_EMERG (0)) (type: tracepoint)
timer_start (loglevel: TRACE_EMERG (0)) (type: tracepoint)
timer_expire_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
timer_expire_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
timer_cancel (loglevel: TRACE_EMERG (0)) (type: tracepoint)
hrtimer_init (loglevel: TRACE_EMERG (0)) (type: tracepoint)
hrtimer_start (loglevel: TRACE_EMERG (0)) (type: tracepoint)
hrtimer_expire_entry (loglevel: TRACE_EMERG (0)) (type: tracepoint)
hrtimer_expire_exit (loglevel: TRACE_EMERG (0)) (type: tracepoint)
hrtimer_cancel (loglevel: TRACE_EMERG (0)) (type: tracepoint)
itimer_state (loglevel: TRACE_EMERG (0)) (type: tracepoint)
itimer_expire (loglevel: TRACE_EMERG (0)) (type: tracepoint)
lttng_statedump_start (loglevel: TRACE_EMERG (0)) (type: tracepoint)
lttng_statedump_end (loglevel: TRACE_EMERG (0)) (type: tracepoint)
lttng_statedump_process_state (loglevel: TRACE_EMERG (0)) (type: tracepoint)
lttng_statedump_file_descriptor (loglevel: TRACE_EMERG (0)) (type: tracepoint)
lttng_statedump_vm_map (loglevel: TRACE_EMERG (0)) (type: tracepoint)
lttng_statedump_network_interface (loglevel: TRACE_EMERG (0)) (type: tracepoint)
lttng_statedump_interrupt (loglevel: TRACE_EMERG (0)) (type: tracepoint)
signal_generate (loglevel: TRACE_EMERG (0)) (type: tracepoint)
signal_deliver (loglevel: TRACE_EMERG (0)) (type: tracepoint)
signal_overflow_fail (loglevel: TRACE_EMERG (0)) (type: tracepoint)
signal_lose_info (loglevel: TRACE_EMERG (0)) (type: tracepoint)
sched_kthread_stop (loglevel: TRACE_EMERG (0)) (type: tracepoint)
sched_kthread_stop_ret (loglevel: TRACE_EMERG (0)) (type: tracepoint)
sched_wakeup (loglevel: TRACE_EMERG (0)) (type: tracepoint)
sched_wakeup_new (loglevel: TRACE_EMERG (0)) (type: tracepoint)
(…)
The example below shows how to add all kernel events for tracing in a session:
$ lttng enable-event -a -k
All kernel events are enabled in channel channel0
Analyzing gathered data
Usually a tracing session generates a huge amount of data, which makes it unreadable by humans. So LTTng project provides some visualizing tools to easy analysis of traced data, as follows:
- Babeltrace
- It is a library and command-line tool able to read and convert trace data stored in different formats (including CTF).
- LTTV Viewer
- It is a graphical visualizer (GTK+) able to make analysis of trace data produced by LTT, but it currently doesn't support CTF.
- Eclipse LTTng
- The Eclipse LTTng is a set of plug-ins able to visualize LTTng gathered data. Currently it is maintained by a pretty active community.
The LTTng plug-ins TMF by default in latest version (4.2.1) Eclipse IDE for C/C++ Developers, which may be found at
http://www.eclipse.org/downloads/
Below is an screenshot of Eclipse view with a loaded LTTng tracing file.
References
- LTTng project - https://lttng.org
- Eclipse LTTng plug-in - http://wiki.eclipse.org/Linux_Tools_Project/LTTng2/User_Guide
[{"Business Unit":{"code":"BU054","label":"Systems w\/TPS"},"Product":{"code":"HW1W1","label":"Power ->PowerLinux"},"Component":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"","Edition":"","Line of Business":{"code":"","label":""}}]
UID
ibm16171537