STLinux

Introduction

KPTrace is an innovative tracing solution for STLinux on ST40 and ARM. It allows dynamic tracepoints to be added anywhere in the system, and provides a simple interface to collect and view the trace. Over three hundred useful tracepoints are pre-defined, including interrupts, context switches, system calls, and much more.

New tracepoints can be added on any symbol without rebuilding or rebooting the system. All tracepoints can be enabled and disabled at runtime, and because only enabled tracepoints are actually present in the system, the system intrusion is minimised.

All the predefined tracepoints are inside the kernel, but tracepoints can be added anywhere: the kernel, kernel modules, or inside user space applications.

KPTrace can also be run from STWorkbench, which provides a sophisticated trace viewer

Check the STWorkbench online help for more details...

Build

To use KPTrace, the kernel must then be built with the following option enabled:

Instrumentation support -> KPTrace

This option will automatically select the other required options.

Usage

To use KPTrace, follow these steps:

1) Boot a kernel that has been built with KPTrace support.

2) Mount a debugfs filesystem on the target, e.g.

   target# mount -t debugfs nodev /sys/kernel/debug

You may find it more convenient to mount it by default from the fstab, by adding this line:

   debug           /sys/kernel/debug          debugfs defaults                        0 0

It is also necessary to have a sysfs filesystem mounted under /sys. This is usually the case by default.

3) Run the executable "kptrace". This will gather trace until stopped with Ctrl-C. Two output files are placed in the current working directory:

• The trace file (called kptrace.kpt by default).
• The system map file at the time of the trace (kptrace.kpm).

To set tracepoints in user space processes, use the option kptrace --attach:

   target% kptrace --attach <pid> <symbols...>

KPTrace provides various other command-line options, including the ability to change the tracebuffer size, to set the maximum callstack depth to be collected, and to use a "flight recorder" mode. Check the manpage on the target for more details.

4) Post-process the output on the host, using the kptrace.pl perl script:

   host% /opt/STM/STLinux-2.3/host/bin/kptrace.pl kptrace.kpt kptrace.kpm

Processed output goes to stdout. The trace is rendered in a human-readable format, and detailed system statistics from the traced period are generated.

Configuration

The trace is very configurable. Configuration is done on the target by editing the file /etc/kptrace.conf.

kptrace.conf defines which tracepoints are on, and whether callstacks should be collected for them.

The format is as follows:

set <tracepoint set> <on|off>

Pre-defined tracepoints are divided into sets. You can turn a whole set on or off by setting the final argument to "on" or "off". If a set is off, none of the tracepoints in that set are added.

Note that the "memory_events" set generates a very large number of trace records, so it is disabled by default.

tracepoint <tracepoint> <yes|no>

All uncommented tracepoints in an enabled group will be added. To suppress an individual tracepoint, comment it out with a #. The final yes/no argument indicates whether a callstack should be output in the tracefile every time the tracepoint is hit. Use this with care - although callstack generation is as fast as we can make it, it inevitably increases intrusiveness.

The final set, "user", allows additional tracepoints to be added. For example, to add a tracepoint on the symbol show_cpuinfo, simply add the following line to the set:

tracepoint show_cpuinfo no

No recompilation is required - the tracepoint will be used next time KPTrace is invoked.

User-defined tracepoints can also be used to start and stop the logging of trace records. The syntax is this:

stopon show_cpuinfo no
starton zoneinfo_show no

In this case, no trace records will be logged after the show_cpuinfo tracepoint is hit until zoneinfo_show is reached. Tracing then continues as normal. The most useful application of this is to put a stop-point on an error case in flight-recorder mode to ensure that the trace leading up to the error is preserved. It is not necessary to restart the logging unless this is specifically required.

Kernel API

A kernel API is provided to interface to KPTrace.

kpprintf(const char *format, ...)

provides a printf-style interface to the KPTrace trace buffer. The string has a timestamp and pid prepended to it, and is added to the trace output.

kptrace_write_record(const char *)

prepends a timestamp and pid to a constant string and adds it to the trace output.

kptrace_write_record_notime(const char *)

prepends a pid to the string and adds it to the trace output. Note that no timestamp is added; this allows this function to be used in some scheduler functions where a call to do_gettimeofday would cause a deadlock.

kptrace_pause(void)

stops the logging of trace records. This is equivalent to hitting a stop-point.

kptrace_restart(void)

restarts the logging of trace records. This is equivalent to hitting a start-point.

kptrace_mark(void)

indicates that this point in the code is significant. Marked points are indicated in the trace file, making it easier to find the most relevant parts of the trace. It is intended that marked points will be prominently indicated in a GUI.

All of these functions are defined in the file asm-sh/kptrace.h.

User space API

It's also possible to write printf()-style strings to the KPTrace trace buffer from a user space application. To do this, the following steps are required:

• Include the header file <kptrace.h>
• Link the application with the KPTrace library providing this function, with -lkptrace.

The function kpprintf() will then be available, which takes printf()-style arguments and will write directly into the trace buffer with a timestamp and process ID prepended.

kpprintf output is indicated in STWorkbench by yellow flags on the trace, and can be located by searching for the symbol kptrace_write_record.