Tracer Taxonomy

Jump to: navigation, search

Here is a matrix of information about various tracing systems for the Linux kernel.

This matrix was started in May, 2006 by Tim Bird, in an effort to understand the areas of overlap and differentiation between the major kernel tracing systems.

Attribute KFT LTTng LKST Systemtap Ad Hoc kernel tracer
Trace definition
tracepoint definition compiler generated in-source ?? externally defined with systemtap language in-source
is the kernel instrumented (y/n) yes yes yes yes
is user space instrumented (y/n) no ?? ?? yes
can a new trace point be statically compiled into the kernel? only function entry and exit are supported yes, with genevent helper yes no
what is the mechanism for statically declaring a new trace point automatically instruments functions trace_, with genevent?? ?? macro call
can new trace points be inserted dynamically? no no ?? yes
valid locations for dynamic tracepoints n/a n/a? n/a func entry, exit, other places within limits ??
what is the mechanism for dynamically inserting a new trace point? n/a ?? ?? insmod and kprobe
what language is used to define a trace point/trace handler? n/a ?? ?? scripting language
Trace generation
control interface procfs rw netlink ?? programmable, with possible run-time knobs (??trb) n/a
How is a trace started? via boot or trigger user action - lttctl ?? user action - stap n/a
when is the earliest a trace can start? at start_kernel, but often only get timestamps and time filtering after time_init after user space init ?? after user space init kernel boot
supported triggers function entry, exit, time none?? ?? programmable none
supported runtime filters function, duration, interrupt context none ?? programmable none
trace buffer static buffer relayfs ?? lookup tables static buffer
is the buffer allocated dynamically or statically? depends on trace mode dynamically ?? statically at script startup usually statically
is the buffer configurable in size? yes yes ?? yes
are events stored per-cpu? no yes ?? optionally
how is contention for the trace buffer handled? atomic slot reservation with fixed-size slots spinlocks ?? ??
transfer interface procfs read relayfs - lttd ?? /proc or relayfs streams
can buffer be read during trace runtime no yes ?? yes?? yes
viewer vi, cat, kd lttv, lttv-gui ?? text/binary streams, eclipse GUI
post-processing aggregators scripts/kd lttv ?? piped scripts if needed
post-trace binary data format none complex - see here ?? programmable
post-trace ascii data format see (KFT log output sample) ?? ?? programmable ad hoc
Architectures supported:
x86 yes yes yes yes
arm yes ?? ?? ??
ppc32 yes ?? ?? yes
ppc64 yes ?? ?? yes
mips yes ?? ?? ??
sh yes, but not tested recently ?? ?? ??
is there support for SMP? yes, but log is not per-cpu yes ?? yes
Attribute KFT LTTng LKST Systemtap Ad Hoc kernel tracer
instrumentation details:
what clock is used for timestamps? depends on arch, usually sched_clock() depends on arch, usually TSC ?? ??
what is the clock resolution? ?? ?? ?? ??
what is the timestamp size, in bits? ?? configurable (32 or 64) ?? ??
how is timestamp rollover handled? ignored uses heartbeats to support long timestamp ?? ??
event collection (logging):
are event structures fixed or variable in size? fixed variable ?? ??
trace-time aggregation:
trace run-time aggregation none none ?? programmable, with lots of different aggregation services provided programmed - ad hoc
configuration and control details:
how is the trace configuration specified? usually an ascii file ?? ?? ?? ascii probe definition which is then compiled
what is trace control user interface procfs rw command line programs (lttctl, lttd), can use gui (lttv) on local machine ?? command line programs (stap) n/a
can filters be modifed at trace time? no n/a ?? ?? n/a

Other stuff for table (not asked or organized yet)

  • what post-processing filters are available for trace data:
    • filter by pid?, uid?, cpuid?, time?, interrupt context?, event?, event type or class? other?
  • what is the format of trace data, during post-processing?
    • (still) binary?, ascii?, XML?
  • what post-processing presentations are available?
    • text?, GUI?, html/web format?, graphics (e.g. png), svg?
  • what post-processing aggregation is available?
    • event counts/summaries?
      • by uid
      • by pid
      • by event type
    • statistics over time?
      • call trees?
  • what data analysis operations are supported:
    • sorting
    • searching
    • visualization (drawing and scrolling)
  • performance
    • what is the overhead on the system when tracing is not configured or active? (I assume 0, but you never know)
    • ... when tracing is configured but not active?
    • ... when only some subset of configured tracing is active?
    • what is the overhead when tracing is active?
    • what is the cost per event to log the data?
    • what is the cost per event to transfer the data for post-processing?
  • size
    • what is the average or customary size of a trace buffer?
    • what is the average or customary size of a trace log?
    • what is the average or customary number of entries in a trace log?