Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kernel and Userspace Tracing with LTTng and Friends

Kernel and Userspace Tracing with LTTng and Friends

Tracing is an important tool for diagnosing performance issues in critical applications (kernel/userspace). Sometimes debugging is just not enough as it may distort the application's flow itself. This talk shall discuss the most important features of the commonly used tracing tools (LTTng, SystemTap, Dtrace) and some surprisingly cool features they have (such as fast tracing from inside GDB). This talks discusses everything from a Tracing 101 to some in-depth look at LTTng

Venue : FUDCon 2015, Pune

6bade386c277c9ce9bec3ae260951ec6?s=128

Suchakra Sharma

June 27, 2015
Tweet

Transcript

  1. Kernel and userspace tracing with LTTng and friends Suchakrapani Datt

    Sharma June 27, 2015 École Polytechnique de Montréal Laboratoire DORSAL
  2. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma whoami Suchakra • PhD

    student, Computer Engineering (Prof Michel Dagenais) DORSAL Lab, École Polytechnique de Montréal – UdeM • Works on debugging, tracing and trace analysis (LTTng), bytecode interpreters, JIT compilation, dynamic instrumentation • Loves wada-pao and poutine
  3. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Agenda Tracing 101 •

    Introduction to tracing • Building blocks of tracing Tracing Tools • ftrace, SystemTap, LTTng, GDB etc. • LTTng Kernel and Userspace Tracing • LTTng Internals, features and demo Trace Analysis • TraceCompass, LTTngTop
  4. tracing 101 POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

  5. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma I ♥

  6. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

  7. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Program

  8. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

  9. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

  10. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

  11. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

  12. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

  13. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Daya! something is WRONG

    with the tire!
  14. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Lets pause! Breakpoint

  15. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Analyzing... void set_tire_dim() {

    tire_dia = 26; tire_width 2; }
  16. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

  17. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Lets pause! Breakpoint

  18. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Analyzing... void set_tire_tread() {

    tread_depth = 0.5; tread_num = 4; }
  19. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Getting late! The wada-paos

    will finish!
  20. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Getting late! The wada-paos

    will finish! Result of program execution will be inaccurate
  21. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma #noवडडापडाओ

  22. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Ignore the problem ..and

    it will go away!* *Probably can get you fired as well
  23. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

  24. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

  25. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracepoints

  26. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

  27. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

  28. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Trace events being recorded

    on the road
  29. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma 400ms 220ms 200ms 100ms

    Timestamps
  30. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma PUNE वडडवडालड Reach in

    time Have a wada-pao right now, analyze the problem later!
  31. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Trace payload

  32. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

  33. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

  34. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma - --- --- -----

    ---- ---- ---- ------ - - ---- - - - -- ----- --------- - - -- ------ - --- - -- ------ - -- -- - ---- -- - -- ---- ------ - --- - -- ------ -
  35. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma - --- --- -----

    ---- -- ------ - - ---- - - -- ----- --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - ----- - - - - - ----------- - - - - - - - - - - - - - --- --- ----- ---- -- ------ - - ---- - - -- ----- --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - ----- - - - - - ----------- - - - - - - - - - - - -
  36. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma - --- --- -----

    ---- -- ------ - - ---- - - -- ----- --------- - - --- ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - -------- - -- - ---- --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - --- ----- - - - - - ----------- - - - - - - - - - - - - ------- -- - ----- - --- --- ----- ---- -- ------ - - ---- - - -- ----- --------- - - --- ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - -------- - -- - ---- --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - --
  37. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma - --- --- -----

    ---- -- ------ - - ---- - - -- ----- --------- - - --- ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - -------- - -- - ---- --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - --- ----- - - - - - ----------- - - - - - - - - - - - - ------- -- - ----- - --- --- ----- ---- -- ------ - - ---- - - -- ----- --------- - - --- ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - -------- - -- - ---- --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- DEAL WITH IT * *Hold on, we have awesome tools for that
  38. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing 101 Characterstics •

    It is fast, has low overhead • Minimal interference with the target • Can generate large amounts of data for analysis • High accuray Benefits • Can be used to detect 'heisenbugs', hard to detect errors • Relatively safe on production environments • Useful to detect anomalies over long runs
  39. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing 101 Classification •

    Kernelspace vs Userspace • Static vs Dynamic Building Blocks • Debugger style: ptrace() syscall • In-kernel • The TRACE_EVENT macro • Kprobes, kretprobes, jprobes etc.
  40. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing 101 TRACE_EVENT macro

    • Automagically adds callbacks to the tracepoint calls in the kernel to which that tracers can hook onto. • Used by Ftrace, LTTng, SystemTap etc. TRACE_EVENT(sched_switch, TP_PROTO(struct task_struct *prev, struct task_struct *next), TP_ARGS(prev, next), TP_STRUCT__entry( __array( char, prev_comm, TASK_COMM_LEN ) __field( pid_t, prev_pid ) __field( int, prev_prio ) __field( long, prev_state ) ... TP_fast_assign( ... ); Event Name include/trace/events/sched.h Trace fucntion prototype How to store data in tracer's ringbuffer Fill the fields above
  41. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing 101 More MACROs

    #undef TRACE_SYSTEM #define TRACE_SYSTEM sched #if !defined(_TRACE_SCHED_H) || defined(TRACE_HEADER_MULTI_READ) #define _TRACE_SCHED_H Using Tracepoints in Kernel #define CREATE_TRACE_POINTS #include <trace/events/sched.h> .. .. prepare_task_switch(struct rq *rq, struct task_struct *prev, struct task_struct *next) { trace_sched_switch(prev, next); sched_info_switch(rq, prev, next); perf_event_task_sched_out(prev, next); .. } kernel/sched/core.c
  42. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing 101 Hooking on

    to Tracepoints - ftrace int ftrace_event_reg(struct ftrace_event_call *call, enum trace_reg type, void *data) { struct ftrace_event_file *file = data; WARN_ON(!(call->flags & TRACE_EVENT_FL_TRACEPOINT)); switch (type) { case TRACE_REG_REGISTER: return tracepoint_probe_register(call->tp, call->class->probe, file); case TRACE_REG_UNREGISTER: tracepoint_probe_unregister(call->tp, call->class->probe, File); .. } kernel/trace/trace_events.c
  43. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing 101 kprobes, jprobes,

    kretprobes • kprobes– Trap into (almost) any kernel function, any instruction, anywhere. • Target instruction is patched with int3 • A handler executed at breakpoint • Can be Jump optimized (use jmp instead of int3) • jprobe – kprobe at function entry, saves function argument, stack etc. • kretprobe – kprobe at function entry, upon hit, a nop at the return address. Kprobe registered at the nop trampoline at boot.
  44. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing 101 kprobes, jprobes,

    kretprobes • kprobes– Trap into (almost) any kernel function, any instruction, anywhere. • Target instruction is patched with int3 • A handler executed at breakpoint • Can be Jump optimized (use jmp instead of int3) • jprobe – kprobe at function entry, saves function argument, stack etc. • kretprobe – kprobe at function entry, upon hit, a nop at the return address. Kprobe registered at the nop trampoline at boot. That's enough! :-/ SHOW ME WHAT I CAN DO
  45. tracing TOOLS POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

  46. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing Tools ftrace #

    cd /sys/kernel/debug/tracing # cat available_tracers blk function_graph wakeup_dl wakeup_rt wakeup function nop tinyfunctrace.sh !#/bin/bash echo $$ > /sys/kernel/debug/tracing/set_ftrace_pid echo function_graph > /sys/kernel/debug/tracing/current_tracer echo 1 > /sys/kernel/debug/tracing/tracing_on exec $* echo 0 > /sys/kernel/debug/tracing/tracing_on # ./tinyfunctrace whoami root
  47. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma ftrace # cat /sys/kernel/debug/tracing/trace

    | head -20 # tracer: function_graph # # CPU DURATION FUNCTION CALLS # | | | | | | | 3) 0.034 us | } /* __tlb_remove_page */ 3) 0.039 us | sync_mm_rss(); 3) 0.035 us | _cond_resched(); 3) 0.036 us | mem_cgroup_uncharge_end(); 3) + 74.216 us | } /* unmap_single_vma */ 3) + 74.474 us | } /* unmap_vmas */ 3) | free_pgtables() { 3) 0.048 us | unlink_anon_vmas(); 3) | unlink_file_vma() { 3) | mutex_lock() { 3) 0.032 us | _cond_resched(); 3) 0.341 us | } 3) | __remove_shared_vm_struct() { 3) 0.047 us | vma_interval_tree_remove(); 3) 0.289 us | } 3) 0.032 us | mutex_unlock(); Tracing Tools
  48. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma ftrace • It is

    a powerful in-kernel tracing system, loose coupling • Very tough to use vanilla. Use trace-cmd, a better interface for ftrace or kernelshark, a GUI interface # trace-cmd record -e sched_switch -F whoami # trace-cmd report version = 6 CPU 0 is empty CPU 1 is empty cpus=4 whoami-4428 [002] 86924.515608: sched_switch: trace-cmd:4428 [120] R ==> migration/2:25 [0] <idle>-0 [003] 86924.515629: sched_switch: swapper/3:0 [120] R ==> trace-cmd:4428 [120] whoami-4428 [003] 86924.516767: sched_switch: whoami:4428 [120] S ==> kworker/3:1:4421 [120] Tracing Tools
  49. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma SystemTap • Fedora favorite,

    very powerful, full of features! • Script driven, easy to use flexible tracing • Stap script → Parse to 'C' form → convert to kernel modules → Register static probes/kprobes → Send trace output back • Very versatile tool, but does not scale quite well. global tim; probe syscall.read.return { tim <<< gettimeofday_ns() - @entry(gettimeofday_ns()); } probe end { print(@hist_log(tim)); } Tracing Tools Example Script
  50. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma SystemTap value |-------------------------------------------------- count

    64 | 0 128 | 0 256 |@@@@@@@@@@@@@@@@@@@@@ 322 512 |@@@@ 63 1024 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 472 2048 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 678 4096 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 732 8192 |@@@@@@ 95 16384 | 10 32768 | 1 65536 | 0 131072 | 0 kernel.trace(“sched_switch”) kernel.function("sys_open") timer.ns(42424242) process("foo").statement("*@main.c:42") Tracing Tools Output Some more probe targets
  51. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma GDB Tracepoints! • Normal

    and Fast Tracepoints • Save or view trace frames • Dynamic tracing or static (trace markers) (gdb) trace foo (gdb) actions Enter actions for tracepoint #1, one per line. > collect $regs,$locals > while-stepping 9 > collect $regs > end > end (gdb) tstart [program executes/continues...] (gdb) tstop Tracing Tools Sample Run
  52. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma GDB Tracepoints! (gdb) tfind

    0 Found trace frame 0, tracepoint 1 54 bar = (bar & 0xFF000000) >> 24; (gdb) tdump Data collected at tracepoint 1, trace frame 0: rax 0x2000028 33554472 rbx 0x0 0 rcx 0x33402e76b0 220120118960 rdx 0x1 1 rsi 0x33405bca30 220123089456 rdi 0x2000028 33554472 . . rip 0x4006f1 0x4006f1 <foo+7> [and so on...] (gdb) tfind 4 Found trace frame 4, tracepoint 1 0x0000000000400700 55 r1 = (bar & 0x00F00000) >> 20; (gdb) p $rip $1 = (void (*)()) 0x400700 <foo+22> Tracing Tools Trace Analysis
  53. LTTng LINUX TRACE TOOLKIT next genetration POLYTECHNIQUE MONTREAL – Suchakrapani

    Datt Sharma
  54. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma LTTng • Low overhead,

    scalable kernel and userspace tracer • Output data in unified format (CTF) • One of the fastest tracers out there – and has minimal impact on the target • lttng-modules : kernel tracer modules • lttng-ust : in-process userspace tracing library $ lttng create fudcon $ lttng enable-event -k sched_switch $ lttng start .. Do something .. $ lttng stop $ lttng view $ lttng destroy Tracing Tools Example kernel trace session
  55. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma CTF Trace . .

    [22:19:34.773054343] (+0.000386207) isengard sched_switch: { cpu_id = 2 }, { prev_comm = "swapper/2", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "gnome-shell", next_tid = 1848, next_prio = 20 } [22:19:34.773112001] (+0.000057658) isengard sched_switch: { cpu_id = 1 }, { prev_comm = "Xorg", prev_tid = 920, prev_prio = 20, prev_state = 1, next_comm = "kworker/1:0", next_tid = 11144, next_prio = 20 } [22:19:34.773124663] (+0.000012662) isengard sched_switch: { cpu_id = 1 }, { prev_comm = "kworker/1:0", prev_tid = 11144, prev_prio = 20, prev_state = 1, next_comm = "swapper/1", next_tid = 0, next_prio = 20 } [22:19:34.773341466] (+0.000216803) isengard sched_switch: { cpu_id = 2 }, { prev_comm = "gnome-shell", prev_tid = 1848, prev_prio = 20, prev_state = 1, next_comm = "swapper/2", next_tid = 0, next_prio = 20 } . . Tracing Tools
  56. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma CTF Trace . .

    [22:19:34.773054343] (+0.000386207) isengard sched_switch: { cpu_id = 2 }, { prev_comm = "swapper/2", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "gnome-shell", next_tid = 1848, next_prio = 20 } [22:19:34.773112001] (+0.000057658) isengard sched_switch: { cpu_id = 1 }, { prev_comm = "Xorg", prev_tid = 920, prev_prio = 20, prev_state = 1, next_comm = "kworker/1:0", next_tid = 11144, next_prio = 20 } [22:19:34.773124663] (+0.000012662) isengard sched_switch: { cpu_id = 1 }, { prev_comm = "kworker/1:0", prev_tid = 11144, prev_prio = 20, prev_state = 1, next_comm = "swapper/1", next_tid = 0, next_prio = 20 } [22:19:34.773341466] (+0.000216803) isengard sched_switch: { cpu_id = 2 }, { prev_comm = "gnome-shell", prev_tid = 1848, prev_prio = 20, prev_state = 1, next_comm = "swapper/2", next_tid = 0, next_prio = 20 } . . Tracing Tools Time Delta Event name Trace Payload Timestamp
  57. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma CTF Trace . .

    [22:19:34.773054343] (+0.000386207) isengard sched_switch: { cpu_id = 2 }, { prev_comm = "swapper/2", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "gnome-shell", next_tid = 1848, next_prio = 20 } [22:19:34.773112001] (+0.000057658) isengard sched_switch: { cpu_id = 1 }, { prev_comm = "Xorg", prev_tid = 920, prev_prio = 20, prev_state = 1, next_comm = "kworker/1:0", next_tid = 11144, next_prio = 20 } [22:19:34.773124663] (+0.000012662) isengard sched_switch: { cpu_id = 1 }, { prev_comm = "kworker/1:0", prev_tid = 11144, prev_prio = 20, prev_state = 1, next_comm = "swapper/1", next_tid = 0, next_prio = 20 } [22:19:34.773341466] (+0.000216803) isengard sched_switch: { cpu_id = 2 }, { prev_comm = "gnome-shell", prev_tid = 1848, prev_prio = 20, prev_state = 1, next_comm = "swapper/2", next_tid = 0, next_prio = 20 } . . Tracing Tools Time Delta Event name Trace Payload Timestamp DO NOT PANIC! We'll see some better CTF view and analysis tools soon!
  58. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing Tools CTF over

    TCP/UDP/SSH Userspace Applications (C/C++/Java) Kernel Tracepoint Tracepoint Probes LTTng Session Daemon LTTng Commands LTTng Consumer Daemon CTF Trace on disk Trace Viewing and Analysis TraceCompass LTTV Babeltrace Any other compatible tool What's going On inside?
  59. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing Tools UST Example

    LTTng Session Daemon LTTng Consumer Daemon
  60. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing Tools Instrumented Userspace

    Application UST listener thread UST Example LTTng Session Daemon LTTng Consumer Daemon
  61. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing Tools Instrumented Userspace

    Application UST listener thread UST Example LTTng Session Daemon LTTng Consumer Daemon Unix Socket Unix Socket
  62. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing Tools Instrumented Userspace

    Application UST listener thread UST Example LTTng Session Daemon LTTng Consumer Daemon Unix Socket Unix Socket SHM Ring buffer
  63. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing Tools Instrumented Userspace

    Application UST listener thread UST Example LTTng Session Daemon LTTng Consumer Daemon Unix Socket Unix Socket SHM Ring buffer CTF Trace
  64. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Q : This UST

    thing looks nice, how to instrument my app? Tracing Tools
  65. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Q : This UST

    thing looks nice, how to instrument my app? A : lttng.org/docs or buy me a wada-pao. Tracing Tools
  66. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma MORE COOL FEATURES of

    LTTng
  67. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma LTTng Trace Snapshots •

    Take a snapshot of trace buffers. No need of those huge multi-GB trace files • Create a flight-recorder session FTW! Tracing Tools lttng_snapshot_record() $ lttng snapshot record Trace buffer Snapshot Trace Data
  68. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Snapshot Usecase • Trigger

    record as an ABRT layer. Integrate with ABRT to generate snapshots on kenrel panics, system-wide bugs in Fedora/RHEL • Record on alerts by logging infra (Nagios etc) • Correlate logs with traces • Remote lightweight system monitoring • Flight-recorder mode. Overwrite snapshots periodically Tracing Tools
  69. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Live Trace View •

    LTTng Relay Daemon – extract and analyze traces as they are being created! • View local or remote and save as well Tracing Tools LTTng Consumer Daemon LTTng Relay Daemon Live Viewer CTF on disk events (TCP) (TCP) live events CTF
  70. Fedora Infra Servers POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Live

    Trace View Usecase Tracing Tools LTTng session LTTng Relay Daemon Live Viewer TCP TCP Fedora Infra Servers LTTng session Fedora Infra Servers LTTng session Fedora Infra Servers LTTng session TCP TCP TCP
  71. trace viewing and analysis POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

  72. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma - --- --- -----

    ---- -- ------ - - ---- - - -- ----- --------- - - --- ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - -------- - -- - ---- --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - --- ----- - - - - - ----------- - - - - - - - - - - - - ------- -- - ----- - --- --- ----- ---- -- ------ - - ---- - - -- ----- --------- - - --- ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - -------- - -- - ---- --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- DEAL WITH IT * *Hold on, we have awesome tools for that
  73. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Trace Analysis babeltrace •

    Text based CTF viewer • Default viewer for lttng view command TraceCompass • Most detailed and feature rich viewer • Eclipse IDE integration or RCP app • Live and remote trace views • Control flow view, histograms, event list, trace statistics, CPU/resource allocation • Mix kernel and userspace trace analysis
  74. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma DEMO Latest TraceCompass RCP

    • http://projects.eclipse.org/projects/tools.tracecompass/downloads RCP with Critical Flow View (Experimental) • http://secretaire.dorsal.polymtl.ca/~gbastien/TracingRCP/DorsalExperimental/
  75. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Trace Analysis Remember, we

    talked about live trace view?
  76. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Trace Analysis LTTngTop •

    top like viewer – but for live and recorded traces • Pause live traces, move forward or back in time • But you don't believe me, do you? Remember, we talked about live trace view?
  77. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma LTTngTop DEMO

  78. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma One more thing...

  79. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma *Thanks to Brendan Gregg

    for this image *
  80. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma More about Systems Performance?

    http://www.brendangregg.com/linuxperf.html
  81. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma References • Graphics and

    text on slide 63, 67, 70 have been adapted from David Goulet's talk at FOSDEM '14. • Graphics on slide 69 have been adapted from http://lttng.org/docs • Graphics on slide 79 are from http://brendangregg.com/linuxperf.html All the images in this presentation drawn by the author are released under Creative Commons. All other graphics have been taken from OpenClipArt and are under public domain.
  82. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Acknowledgments Thanks to EfficiOS,

    Ericsson Montréal and DORSAL Lab, Polytechnique Montreal for the awesome work on LTTng/UST, TraceCompass and LttngTop. Thanks to Fedora community and Red Hat for this opportunity.
  83. lttng.org/docs POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

  84. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Questions? suchakrapani.sharma@polymtl.ca suchakra on

    #lttng (irc.oftc.net) @tuxology http://suchakra.in