Kernel and Userspace Tracing with LTTng and Friends

Tracing is an important tool for diagnosing performance issues in critical applications (kernel/userspace). Sometimes debugging is just not enough as it may distort the application's flow itself. This talk shall discuss the most important features of the commonly used tracing tools (LTTng, SystemTap, Dtrace) and some surprisingly cool features they have (such as fast tracing from inside GDB). This talks discusses everything from a Tracing 101 to some in-depth look at LTTng

Venue : FUDCon 2015, Pune

Transcript

1. Kernel and userspace tracing with LTTng and friends Suchakrapani Datt Sharma June 27, 2015 École Polytechnique de Montréal Laboratoire DORSAL
2. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma whoami Suchakra • PhD student, Computer Engineering (Prof Michel Dagenais) DORSAL Lab, École Polytechnique de Montréal – UdeM • Works on debugging, tracing and trace analysis (LTTng), bytecode interpreters, JIT compilation, dynamic instrumentation • Loves wada-pao and poutine
3. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Agenda Tracing 101 • Introduction to tracing • Building blocks of tracing Tracing Tools • ftrace, SystemTap, LTTng, GDB etc. • LTTng Kernel and Userspace Tracing • LTTng Internals, features and demo Trace Analysis • TraceCompass, LTTngTop
4. tracing 101 POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma
5. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma I ♥
6. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma
7. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Program
8. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma
9. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma
10. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma
11. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma
12. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma
13. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Daya! something is WRONG with the tire!
14. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Lets pause! Breakpoint
15. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Analyzing... void set_tire_dim() { tire_dia = 26; tire_width 2; }
16. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma
17. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Lets pause! Breakpoint
18. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Analyzing... void set_tire_tread() { tread_depth = 0.5; tread_num = 4; }
19. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Getting late! The wada-paos will finish!
20. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Getting late! The wada-paos will finish! Result of program execution will be inaccurate
21. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma #noवडडापडाओ
22. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Ignore the problem ..and it will go away!* *Probably can get you fired as well
23. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma
24. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma
25. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracepoints
26. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma
27. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma
28. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Trace events being recorded on the road
29. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma 400ms 220ms 200ms 100ms Timestamps
30. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma PUNE वडडवडालड Reach in time Have a wada-pao right now, analyze the problem later!
31. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Trace payload
32. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma
33. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma
34. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma - --- --- ----- ---- ---- ---- ------ - - ---- - - - -- ----- --------- - - -- ------ - --- - -- ------ - -- -- - ---- -- - -- ---- ------ - --- - -- ------ -
35. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma - --- --- ----- ---- -- ------ - - ---- - - -- ----- --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - ----- - - - - - ----------- - - - - - - - - - - - - - --- --- ----- ---- -- ------ - - ---- - - -- ----- --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - ----- - - - - - ----------- - - - - - - - - - - - -
36. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma - --- --- ----- ---- -- ------ - - ---- - - -- ----- --------- - - --- ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - -------- - -- - ---- --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - --- ----- - - - - - ----------- - - - - - - - - - - - - ------- -- - ----- - --- --- ----- ---- -- ------ - - ---- - - -- ----- --------- - - --- ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - -------- - -- - ---- --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - --
37. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma - --- --- ----- ---- -- ------ - - ---- - - -- ----- --------- - - --- ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - -------- - -- - ---- --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - --- ----- - - - - - ----------- - - - - - - - - - - - - ------- -- - ----- - --- --- ----- ---- -- ------ - - ---- - - -- ----- --------- - - --- ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - -------- - -- - ---- --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- DEAL WITH IT * *Hold on, we have awesome tools for that
38. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing 101 Characterstics • It is fast, has low overhead • Minimal interference with the target • Can generate large amounts of data for analysis • High accuray Benefits • Can be used to detect 'heisenbugs', hard to detect errors • Relatively safe on production environments • Useful to detect anomalies over long runs
39. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing 101 Classification • Kernelspace vs Userspace • Static vs Dynamic Building Blocks • Debugger style: ptrace() syscall • In-kernel • The TRACE_EVENT macro • Kprobes, kretprobes, jprobes etc.
40. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing 101 TRACE_EVENT macro • Automagically adds callbacks to the tracepoint calls in the kernel to which that tracers can hook onto. • Used by Ftrace, LTTng, SystemTap etc. TRACE_EVENT(sched_switch, TP_PROTO(struct task_struct *prev, struct task_struct *next), TP_ARGS(prev, next), TP_STRUCT__entry( __array( char, prev_comm, TASK_COMM_LEN ) __field( pid_t, prev_pid ) __field( int, prev_prio ) __field( long, prev_state ) ... TP_fast_assign( ... ); Event Name include/trace/events/sched.h Trace fucntion prototype How to store data in tracer's ringbuffer Fill the fields above
41. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing 101 More MACROs #undef TRACE_SYSTEM #define TRACE_SYSTEM sched #if !defined(_TRACE_SCHED_H) || defined(TRACE_HEADER_MULTI_READ) #define _TRACE_SCHED_H Using Tracepoints in Kernel #define CREATE_TRACE_POINTS #include <trace/events/sched.h> .. .. prepare_task_switch(struct rq *rq, struct task_struct *prev, struct task_struct *next) { trace_sched_switch(prev, next); sched_info_switch(rq, prev, next); perf_event_task_sched_out(prev, next); .. } kernel/sched/core.c
42. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing 101 Hooking on to Tracepoints - ftrace int ftrace_event_reg(struct ftrace_event_call *call, enum trace_reg type, void *data) { struct ftrace_event_file *file = data; WARN_ON(!(call->flags & TRACE_EVENT_FL_TRACEPOINT)); switch (type) { case TRACE_REG_REGISTER: return tracepoint_probe_register(call->tp, call->class->probe, file); case TRACE_REG_UNREGISTER: tracepoint_probe_unregister(call->tp, call->class->probe, File); .. } kernel/trace/trace_events.c
43. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing 101 kprobes, jprobes, kretprobes • kprobes– Trap into (almost) any kernel function, any instruction, anywhere. • Target instruction is patched with int3 • A handler executed at breakpoint • Can be Jump optimized (use jmp instead of int3) • jprobe – kprobe at function entry, saves function argument, stack etc. • kretprobe – kprobe at function entry, upon hit, a nop at the return address. Kprobe registered at the nop trampoline at boot.
44. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing 101 kprobes, jprobes, kretprobes • kprobes– Trap into (almost) any kernel function, any instruction, anywhere. • Target instruction is patched with int3 • A handler executed at breakpoint • Can be Jump optimized (use jmp instead of int3) • jprobe – kprobe at function entry, saves function argument, stack etc. • kretprobe – kprobe at function entry, upon hit, a nop at the return address. Kprobe registered at the nop trampoline at boot. That's enough! :-/ SHOW ME WHAT I CAN DO
45. tracing TOOLS POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma
46. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing Tools ftrace # cd /sys/kernel/debug/tracing # cat available_tracers blk function_graph wakeup_dl wakeup_rt wakeup function nop tinyfunctrace.sh !#/bin/bash echo $$ > /sys/kernel/debug/tracing/set_ftrace_pid echo function_graph > /sys/kernel/debug/tracing/current_tracer echo 1 > /sys/kernel/debug/tracing/tracing_on exec $* echo 0 > /sys/kernel/debug/tracing/tracing_on # ./tinyfunctrace whoami root
47. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma ftrace # cat /sys/kernel/debug/tracing/trace | head -20 # tracer: function_graph # # CPU DURATION FUNCTION CALLS # | | | | | | | 3) 0.034 us | } /* __tlb_remove_page */ 3) 0.039 us | sync_mm_rss(); 3) 0.035 us | _cond_resched(); 3) 0.036 us | mem_cgroup_uncharge_end(); 3) + 74.216 us | } /* unmap_single_vma */ 3) + 74.474 us | } /* unmap_vmas */ 3) | free_pgtables() { 3) 0.048 us | unlink_anon_vmas(); 3) | unlink_file_vma() { 3) | mutex_lock() { 3) 0.032 us | _cond_resched(); 3) 0.341 us | } 3) | __remove_shared_vm_struct() { 3) 0.047 us | vma_interval_tree_remove(); 3) 0.289 us | } 3) 0.032 us | mutex_unlock(); Tracing Tools
48. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma ftrace • It is a powerful in-kernel tracing system, loose coupling • Very tough to use vanilla. Use trace-cmd, a better interface for ftrace or kernelshark, a GUI interface # trace-cmd record -e sched_switch -F whoami # trace-cmd report version = 6 CPU 0 is empty CPU 1 is empty cpus=4 whoami-4428 [002] 86924.515608: sched_switch: trace-cmd:4428 [120] R ==> migration/2:25 [0] <idle>-0 [003] 86924.515629: sched_switch: swapper/3:0 [120] R ==> trace-cmd:4428 [120] whoami-4428 [003] 86924.516767: sched_switch: whoami:4428 [120] S ==> kworker/3:1:4421 [120] Tracing Tools
49. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma SystemTap • Fedora favorite, very powerful, full of features! • Script driven, easy to use flexible tracing • Stap script → Parse to 'C' form → convert to kernel modules → Register static probes/kprobes → Send trace output back • Very versatile tool, but does not scale quite well. global tim; probe syscall.read.return { tim <<< gettimeofday_ns() - @entry(gettimeofday_ns()); } probe end { print(@hist_log(tim)); } Tracing Tools Example Script
50. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma SystemTap value |-------------------------------------------------- count 64 | 0 128 | 0 256 |@@@@@@@@@@@@@@@@@@@@@ 322 512 |@@@@ 63 1024 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 472 2048 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 678 4096 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 732 8192 |@@@@@@ 95 16384 | 10 32768 | 1 65536 | 0 131072 | 0 kernel.trace(“sched_switch”) kernel.function("sys_open") timer.ns(42424242) process("foo").statement("*@main.c:42") Tracing Tools Output Some more probe targets
51. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma GDB Tracepoints! • Normal and Fast Tracepoints • Save or view trace frames • Dynamic tracing or static (trace markers) (gdb) trace foo (gdb) actions Enter actions for tracepoint #1, one per line. > collect $regs,$locals > while-stepping 9 > collect $regs > end > end (gdb) tstart [program executes/continues...] (gdb) tstop Tracing Tools Sample Run
52. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma GDB Tracepoints! (gdb) tfind 0 Found trace frame 0, tracepoint 1 54 bar = (bar & 0xFF000000) >> 24; (gdb) tdump Data collected at tracepoint 1, trace frame 0: rax 0x2000028 33554472 rbx 0x0 0 rcx 0x33402e76b0 220120118960 rdx 0x1 1 rsi 0x33405bca30 220123089456 rdi 0x2000028 33554472 . . rip 0x4006f1 0x4006f1 <foo+7> [and so on...] (gdb) tfind 4 Found trace frame 4, tracepoint 1 0x0000000000400700 55 r1 = (bar & 0x00F00000) >> 20; (gdb) p $rip $1 = (void (*)()) 0x400700 <foo+22> Tracing Tools Trace Analysis
53. LTTng LINUX TRACE TOOLKIT next genetration POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma
54. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma LTTng • Low overhead, scalable kernel and userspace tracer • Output data in unified format (CTF) • One of the fastest tracers out there – and has minimal impact on the target • lttng-modules : kernel tracer modules • lttng-ust : in-process userspace tracing library $ lttng create fudcon $ lttng enable-event -k sched_switch $ lttng start .. Do something .. $ lttng stop $ lttng view $ lttng destroy Tracing Tools Example kernel trace session
55. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma CTF Trace . . [22:19:34.773054343] (+0.000386207) isengard sched_switch: { cpu_id = 2 }, { prev_comm = "swapper/2", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "gnome-shell", next_tid = 1848, next_prio = 20 } [22:19:34.773112001] (+0.000057658) isengard sched_switch: { cpu_id = 1 }, { prev_comm = "Xorg", prev_tid = 920, prev_prio = 20, prev_state = 1, next_comm = "kworker/1:0", next_tid = 11144, next_prio = 20 } [22:19:34.773124663] (+0.000012662) isengard sched_switch: { cpu_id = 1 }, { prev_comm = "kworker/1:0", prev_tid = 11144, prev_prio = 20, prev_state = 1, next_comm = "swapper/1", next_tid = 0, next_prio = 20 } [22:19:34.773341466] (+0.000216803) isengard sched_switch: { cpu_id = 2 }, { prev_comm = "gnome-shell", prev_tid = 1848, prev_prio = 20, prev_state = 1, next_comm = "swapper/2", next_tid = 0, next_prio = 20 } . . Tracing Tools
56. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma CTF Trace . . [22:19:34.773054343] (+0.000386207) isengard sched_switch: { cpu_id = 2 }, { prev_comm = "swapper/2", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "gnome-shell", next_tid = 1848, next_prio = 20 } [22:19:34.773112001] (+0.000057658) isengard sched_switch: { cpu_id = 1 }, { prev_comm = "Xorg", prev_tid = 920, prev_prio = 20, prev_state = 1, next_comm = "kworker/1:0", next_tid = 11144, next_prio = 20 } [22:19:34.773124663] (+0.000012662) isengard sched_switch: { cpu_id = 1 }, { prev_comm = "kworker/1:0", prev_tid = 11144, prev_prio = 20, prev_state = 1, next_comm = "swapper/1", next_tid = 0, next_prio = 20 } [22:19:34.773341466] (+0.000216803) isengard sched_switch: { cpu_id = 2 }, { prev_comm = "gnome-shell", prev_tid = 1848, prev_prio = 20, prev_state = 1, next_comm = "swapper/2", next_tid = 0, next_prio = 20 } . . Tracing Tools Time Delta Event name Trace Payload Timestamp
57. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma CTF Trace . . [22:19:34.773054343] (+0.000386207) isengard sched_switch: { cpu_id = 2 }, { prev_comm = "swapper/2", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "gnome-shell", next_tid = 1848, next_prio = 20 } [22:19:34.773112001] (+0.000057658) isengard sched_switch: { cpu_id = 1 }, { prev_comm = "Xorg", prev_tid = 920, prev_prio = 20, prev_state = 1, next_comm = "kworker/1:0", next_tid = 11144, next_prio = 20 } [22:19:34.773124663] (+0.000012662) isengard sched_switch: { cpu_id = 1 }, { prev_comm = "kworker/1:0", prev_tid = 11144, prev_prio = 20, prev_state = 1, next_comm = "swapper/1", next_tid = 0, next_prio = 20 } [22:19:34.773341466] (+0.000216803) isengard sched_switch: { cpu_id = 2 }, { prev_comm = "gnome-shell", prev_tid = 1848, prev_prio = 20, prev_state = 1, next_comm = "swapper/2", next_tid = 0, next_prio = 20 } . . Tracing Tools Time Delta Event name Trace Payload Timestamp DO NOT PANIC! We'll see some better CTF view and analysis tools soon!
58. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing Tools CTF over TCP/UDP/SSH Userspace Applications (C/C++/Java) Kernel Tracepoint Tracepoint Probes LTTng Session Daemon LTTng Commands LTTng Consumer Daemon CTF Trace on disk Trace Viewing and Analysis TraceCompass LTTV Babeltrace Any other compatible tool What's going On inside?
59. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing Tools UST Example LTTng Session Daemon LTTng Consumer Daemon
60. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing Tools Instrumented Userspace Application UST listener thread UST Example LTTng Session Daemon LTTng Consumer Daemon
61. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing Tools Instrumented Userspace Application UST listener thread UST Example LTTng Session Daemon LTTng Consumer Daemon Unix Socket Unix Socket
62. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing Tools Instrumented Userspace Application UST listener thread UST Example LTTng Session Daemon LTTng Consumer Daemon Unix Socket Unix Socket SHM Ring buffer
63. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing Tools Instrumented Userspace Application UST listener thread UST Example LTTng Session Daemon LTTng Consumer Daemon Unix Socket Unix Socket SHM Ring buffer CTF Trace
64. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Q : This UST thing looks nice, how to instrument my app? Tracing Tools
65. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Q : This UST thing looks nice, how to instrument my app? A : lttng.org/docs or buy me a wada-pao. Tracing Tools
66. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma MORE COOL FEATURES of LTTng
67. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma LTTng Trace Snapshots • Take a snapshot of trace buffers. No need of those huge multi-GB trace files • Create a flight-recorder session FTW! Tracing Tools lttng_snapshot_record() $ lttng snapshot record Trace buffer Snapshot Trace Data
68. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Snapshot Usecase • Trigger record as an ABRT layer. Integrate with ABRT to generate snapshots on kenrel panics, system-wide bugs in Fedora/RHEL • Record on alerts by logging infra (Nagios etc) • Correlate logs with traces • Remote lightweight system monitoring • Flight-recorder mode. Overwrite snapshots periodically Tracing Tools
69. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Live Trace View • LTTng Relay Daemon – extract and analyze traces as they are being created! • View local or remote and save as well Tracing Tools LTTng Consumer Daemon LTTng Relay Daemon Live Viewer CTF on disk events (TCP) (TCP) live events CTF
70. Fedora Infra Servers POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Live Trace View Usecase Tracing Tools LTTng session LTTng Relay Daemon Live Viewer TCP TCP Fedora Infra Servers LTTng session Fedora Infra Servers LTTng session Fedora Infra Servers LTTng session TCP TCP TCP
71. trace viewing and analysis POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma
72. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma - --- --- ----- ---- -- ------ - - ---- - - -- ----- --------- - - --- ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - -------- - -- - ---- --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - --- ----- - - - - - ----------- - - - - - - - - - - - - ------- -- - ----- - --- --- ----- ---- -- ------ - - ---- - - -- ----- --------- - - --- ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - -------- - -- - ---- --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- DEAL WITH IT * *Hold on, we have awesome tools for that
73. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Trace Analysis babeltrace • Text based CTF viewer • Default viewer for lttng view command TraceCompass • Most detailed and feature rich viewer • Eclipse IDE integration or RCP app • Live and remote trace views • Control flow view, histograms, event list, trace statistics, CPU/resource allocation • Mix kernel and userspace trace analysis
74. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma DEMO Latest TraceCompass RCP • http://projects.eclipse.org/projects/tools.tracecompass/downloads RCP with Critical Flow View (Experimental) • http://secretaire.dorsal.polymtl.ca/~gbastien/TracingRCP/DorsalExperimental/
75. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Trace Analysis Remember, we talked about live trace view?
76. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Trace Analysis LTTngTop • top like viewer – but for live and recorded traces • Pause live traces, move forward or back in time • But you don't believe me, do you? Remember, we talked about live trace view?
77. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma LTTngTop DEMO
78. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma One more thing...
79. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma *Thanks to Brendan Gregg for this image *
80. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma More about Systems Performance? http://www.brendangregg.com/linuxperf.html
81. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma References • Graphics and text on slide 63, 67, 70 have been adapted from David Goulet's talk at FOSDEM '14. • Graphics on slide 69 have been adapted from http://lttng.org/docs • Graphics on slide 79 are from http://brendangregg.com/linuxperf.html All the images in this presentation drawn by the author are released under Creative Commons. All other graphics have been taken from OpenClipArt and are under public domain.
82. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Acknowledgments Thanks to EfficiOS, Ericsson Montréal and DORSAL Lab, Polytechnique Montreal for the awesome work on LTTng/UST, TraceCompass and LttngTop. Thanks to Fedora community and Red Hat for this opportunity.
83. lttng.org/docs POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma
84. POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Questions? suchakrapani.sharma@polymtl.ca suchakra on #lttng (irc.oftc.net) @tuxology http://suchakra.in