Kernel and Userspace Tracing with LTTng and Friends

Slide 1

Slide 1 text

Kernel and userspace tracing with LTTng and friends Suchakrapani Datt Sharma June 27, 2015 École Polytechnique de Montréal Laboratoire DORSAL

Slide 2

Slide 2 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma whoami Suchakra ● PhD student, Computer Engineering (Prof Michel Dagenais) DORSAL Lab, École Polytechnique de Montréal – UdeM ● Works on debugging, tracing and trace analysis (LTTng), bytecode interpreters, JIT compilation, dynamic instrumentation ● Loves wada-pao and poutine

Slide 3

Slide 3 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Agenda Tracing 101 ● Introduction to tracing ● Building blocks of tracing Tracing Tools ● ftrace, SystemTap, LTTng, GDB etc. ● LTTng Kernel and Userspace Tracing ● LTTng Internals, features and demo Trace Analysis ● TraceCompass, LTTngTop

Slide 4

Slide 4 text

tracing 101 POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

Slide 5

Slide 5 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma I ♥

Slide 6

Slide 6 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

Slide 7

Slide 7 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Program

Slide 8

Slide 8 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

Slide 9

Slide 9 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

Slide 10

Slide 10 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

Slide 11

Slide 11 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

Slide 12

Slide 12 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

Slide 13

Slide 13 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Daya! something is WRONG with the tire!

Slide 14

Slide 14 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Lets pause! Breakpoint

Slide 15

Slide 15 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Analyzing... void set_tire_dim() { tire_dia = 26; tire_width 2; }

Slide 16

Slide 16 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

Slide 17

Slide 17 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Lets pause! Breakpoint

Slide 18

Slide 18 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Analyzing... void set_tire_tread() { tread_depth = 0.5; tread_num = 4; }

Slide 19

Slide 19 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Getting late! The wada-paos will finish!

Slide 20

Slide 20 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Getting late! The wada-paos will finish! Result of program execution will be inaccurate

Slide 21

Slide 21 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma #noवडडापडाओ

Slide 22

Slide 22 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Ignore the problem ..and it will go away!* *Probably can get you fired as well

Slide 23

Slide 23 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

Slide 24

Slide 24 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

Slide 25

Slide 25 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracepoints

Slide 26

Slide 26 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

Slide 27

Slide 27 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

Slide 28

Slide 28 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Trace events being recorded on the road

Slide 29

Slide 29 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma 400ms 220ms 200ms 100ms Timestamps

Slide 30

Slide 30 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma PUNE वडडवडालड Reach in time Have a wada-pao right now, analyze the problem later!

Slide 31

Slide 31 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Trace payload

Slide 32

Slide 32 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

Slide 33

Slide 33 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

Slide 34

Slide 34 text

Slide 35

Slide 35 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma - --- --- ----- ---- -- ------ - - ---- - - -- ----- --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - ----- - - - - - ----------- - - - - - - - - - - - - - --- --- ----- ---- -- ------ - - ---- - - -- ----- --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - ----- - - - - - ----------- - - - - - - - - - - - -

Slide 36

Slide 36 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma - --- --- ----- ---- -- ------ - - ---- - - -- ----- --------- - - --- ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - -------- - -- - ---- --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - --- ----- - - - - - ----------- - - - - - - - - - - - - ------- -- - ----- - --- --- ----- ---- -- ------ - - ---- - - -- ----- --------- - - --- ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - -------- - -- - ---- --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - --

Slide 37

Slide 37 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma - --- --- ----- ---- -- ------ - - ---- - - -- ----- --------- - - --- ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - -------- - -- - ---- --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - --- ----- - - - - - ----------- - - - - - - - - - - - - ------- -- - ----- - --- --- ----- ---- -- ------ - - ---- - - -- ----- --------- - - --- ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - -------- - -- - ---- --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- DEAL WITH IT * *Hold on, we have awesome tools for that

Slide 38

Slide 38 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing 101 Characterstics ● It is fast, has low overhead ● Minimal interference with the target ● Can generate large amounts of data for analysis ● High accuray Benefits ● Can be used to detect 'heisenbugs', hard to detect errors ● Relatively safe on production environments ● Useful to detect anomalies over long runs

Slide 39

Slide 39 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing 101 Classification ● Kernelspace vs Userspace ● Static vs Dynamic Building Blocks ● Debugger style: ptrace() syscall ● In-kernel ● The TRACE_EVENT macro ● Kprobes, kretprobes, jprobes etc.

Slide 40

Slide 40 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing 101 TRACE_EVENT macro ● Automagically adds callbacks to the tracepoint calls in the kernel to which that tracers can hook onto. ● Used by Ftrace, LTTng, SystemTap etc. TRACE_EVENT(sched_switch, TP_PROTO(struct task_struct *prev, struct task_struct *next), TP_ARGS(prev, next), TP_STRUCT__entry( __array( char, prev_comm, TASK_COMM_LEN ) __field( pid_t, prev_pid ) __field( int, prev_prio ) __field( long, prev_state ) ... TP_fast_assign( ... ); Event Name include/trace/events/sched.h Trace fucntion prototype How to store data in tracer's ringbuffer Fill the fields above

Slide 41

Slide 41 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing 101 More MACROs #undef TRACE_SYSTEM #define TRACE_SYSTEM sched #if !defined(_TRACE_SCHED_H) || defined(TRACE_HEADER_MULTI_READ) #define _TRACE_SCHED_H Using Tracepoints in Kernel #define CREATE_TRACE_POINTS #include .. .. prepare_task_switch(struct rq *rq, struct task_struct *prev, struct task_struct *next) { trace_sched_switch(prev, next); sched_info_switch(rq, prev, next); perf_event_task_sched_out(prev, next); .. } kernel/sched/core.c

Slide 42

Slide 42 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing 101 Hooking on to Tracepoints - ftrace int ftrace_event_reg(struct ftrace_event_call *call, enum trace_reg type, void *data) { struct ftrace_event_file *file = data; WARN_ON(!(call->flags & TRACE_EVENT_FL_TRACEPOINT)); switch (type) { case TRACE_REG_REGISTER: return tracepoint_probe_register(call->tp, call->class->probe, file); case TRACE_REG_UNREGISTER: tracepoint_probe_unregister(call->tp, call->class->probe, File); .. } kernel/trace/trace_events.c

Slide 43

Slide 43 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing 101 kprobes, jprobes, kretprobes ● kprobes– Trap into (almost) any kernel function, any instruction, anywhere. ● Target instruction is patched with int3 ● A handler executed at breakpoint ● Can be Jump optimized (use jmp instead of int3) ● jprobe – kprobe at function entry, saves function argument, stack etc. ● kretprobe – kprobe at function entry, upon hit, a nop at the return address. Kprobe registered at the nop trampoline at boot.

Slide 44

Slide 44 text

Slide 45

Slide 45 text

tracing TOOLS POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

Slide 46

Slide 46 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing Tools ftrace # cd /sys/kernel/debug/tracing # cat available_tracers blk function_graph wakeup_dl wakeup_rt wakeup function nop tinyfunctrace.sh !#/bin/bash echo $$ > /sys/kernel/debug/tracing/set_ftrace_pid echo function_graph > /sys/kernel/debug/tracing/current_tracer echo 1 > /sys/kernel/debug/tracing/tracing_on exec $* echo 0 > /sys/kernel/debug/tracing/tracing_on # ./tinyfunctrace whoami root

Slide 47

Slide 47 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma ftrace # cat /sys/kernel/debug/tracing/trace | head -20 # tracer: function_graph # # CPU DURATION FUNCTION CALLS # | | | | | | | 3) 0.034 us | } /* __tlb_remove_page */ 3) 0.039 us | sync_mm_rss(); 3) 0.035 us | _cond_resched(); 3) 0.036 us | mem_cgroup_uncharge_end(); 3) + 74.216 us | } /* unmap_single_vma */ 3) + 74.474 us | } /* unmap_vmas */ 3) | free_pgtables() { 3) 0.048 us | unlink_anon_vmas(); 3) | unlink_file_vma() { 3) | mutex_lock() { 3) 0.032 us | _cond_resched(); 3) 0.341 us | } 3) | __remove_shared_vm_struct() { 3) 0.047 us | vma_interval_tree_remove(); 3) 0.289 us | } 3) 0.032 us | mutex_unlock(); Tracing Tools

Slide 48

Slide 48 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma ftrace ● It is a powerful in-kernel tracing system, loose coupling ● Very tough to use vanilla. Use trace-cmd, a better interface for ftrace or kernelshark, a GUI interface # trace-cmd record -e sched_switch -F whoami # trace-cmd report version = 6 CPU 0 is empty CPU 1 is empty cpus=4 whoami-4428 [002] 86924.515608: sched_switch: trace-cmd:4428 [120] R ==> migration/2:25 [0] -0 [003] 86924.515629: sched_switch: swapper/3:0 [120] R ==> trace-cmd:4428 [120] whoami-4428 [003] 86924.516767: sched_switch: whoami:4428 [120] S ==> kworker/3:1:4421 [120] Tracing Tools

Slide 49

Slide 49 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma SystemTap ● Fedora favorite, very powerful, full of features! ● Script driven, easy to use flexible tracing ● Stap script → Parse to 'C' form → convert to kernel modules → Register static probes/kprobes → Send trace output back ● Very versatile tool, but does not scale quite well. global tim; probe syscall.read.return { tim <<< gettimeofday_ns() - @entry(gettimeofday_ns()); } probe end { print(@hist_log(tim)); } Tracing Tools Example Script

Slide 50

Slide 50 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma SystemTap value |-------------------------------------------------- count 64 | 0 128 | 0 256 |@@@@@@@@@@@@@@@@@@@@@ 322 512 |@@@@ 63 1024 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 472 2048 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 678 4096 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 732 8192 |@@@@@@ 95 16384 | 10 32768 | 1 65536 | 0 131072 | 0 kernel.trace(“sched_switch”) kernel.function("sys_open") timer.ns(42424242) process("foo").statement("*@main.c:42") Tracing Tools Output Some more probe targets

Slide 51

Slide 51 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma GDB Tracepoints! ● Normal and Fast Tracepoints ● Save or view trace frames ● Dynamic tracing or static (trace markers) (gdb) trace foo (gdb) actions Enter actions for tracepoint #1, one per line. > collect $regs,$locals > while-stepping 9 > collect $regs > end > end (gdb) tstart [program executes/continues...] (gdb) tstop Tracing Tools Sample Run

Slide 52

Slide 52 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma GDB Tracepoints! (gdb) tfind 0 Found trace frame 0, tracepoint 1 54 bar = (bar & 0xFF000000) >> 24; (gdb) tdump Data collected at tracepoint 1, trace frame 0: rax 0x2000028 33554472 rbx 0x0 0 rcx 0x33402e76b0 220120118960 rdx 0x1 1 rsi 0x33405bca30 220123089456 rdi 0x2000028 33554472 . . rip 0x4006f1 0x4006f1 [and so on...] (gdb) tfind 4 Found trace frame 4, tracepoint 1 0x0000000000400700 55 r1 = (bar & 0x00F00000) >> 20; (gdb) p $rip $1 = (void (*)()) 0x400700 Tracing Tools Trace Analysis

Slide 53

Slide 53 text

LTTng LINUX TRACE TOOLKIT next genetration POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

Slide 54

Slide 54 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma LTTng ● Low overhead, scalable kernel and userspace tracer ● Output data in unified format (CTF) ● One of the fastest tracers out there – and has minimal impact on the target ● lttng-modules : kernel tracer modules ● lttng-ust : in-process userspace tracing library $ lttng create fudcon $ lttng enable-event -k sched_switch $ lttng start .. Do something .. $ lttng stop $ lttng view $ lttng destroy Tracing Tools Example kernel trace session

Slide 55

Slide 55 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma CTF Trace . . [22:19:34.773054343] (+0.000386207) isengard sched_switch: { cpu_id = 2 }, { prev_comm = "swapper/2", prev_tid = 0, prev_prio = 20, prev_state = 0, next_comm = "gnome-shell", next_tid = 1848, next_prio = 20 } [22:19:34.773112001] (+0.000057658) isengard sched_switch: { cpu_id = 1 }, { prev_comm = "Xorg", prev_tid = 920, prev_prio = 20, prev_state = 1, next_comm = "kworker/1:0", next_tid = 11144, next_prio = 20 } [22:19:34.773124663] (+0.000012662) isengard sched_switch: { cpu_id = 1 }, { prev_comm = "kworker/1:0", prev_tid = 11144, prev_prio = 20, prev_state = 1, next_comm = "swapper/1", next_tid = 0, next_prio = 20 } [22:19:34.773341466] (+0.000216803) isengard sched_switch: { cpu_id = 2 }, { prev_comm = "gnome-shell", prev_tid = 1848, prev_prio = 20, prev_state = 1, next_comm = "swapper/2", next_tid = 0, next_prio = 20 } . . Tracing Tools

Slide 56

Slide 56 text

Slide 57

Slide 57 text

Slide 58

Slide 58 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing Tools CTF over TCP/UDP/SSH Userspace Applications (C/C++/Java) Kernel Tracepoint Tracepoint Probes LTTng Session Daemon LTTng Commands LTTng Consumer Daemon CTF Trace on disk Trace Viewing and Analysis TraceCompass LTTV Babeltrace Any other compatible tool What's going On inside?

Slide 59

Slide 59 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing Tools UST Example LTTng Session Daemon LTTng Consumer Daemon

Slide 60

Slide 60 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing Tools Instrumented Userspace Application UST listener thread UST Example LTTng Session Daemon LTTng Consumer Daemon

Slide 61

Slide 61 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing Tools Instrumented Userspace Application UST listener thread UST Example LTTng Session Daemon LTTng Consumer Daemon Unix Socket Unix Socket

Slide 62

Slide 62 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Tracing Tools Instrumented Userspace Application UST listener thread UST Example LTTng Session Daemon LTTng Consumer Daemon Unix Socket Unix Socket SHM Ring buffer

Slide 63

Slide 63 text

Slide 64

Slide 64 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Q : This UST thing looks nice, how to instrument my app? Tracing Tools

Slide 65

Slide 65 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Q : This UST thing looks nice, how to instrument my app? A : lttng.org/docs or buy me a wada-pao. Tracing Tools

Slide 66

Slide 66 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma MORE COOL FEATURES of LTTng

Slide 67

Slide 67 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma LTTng Trace Snapshots ● Take a snapshot of trace buffers. No need of those huge multi-GB trace files ● Create a flight-recorder session FTW! Tracing Tools lttng_snapshot_record() $ lttng snapshot record Trace buffer Snapshot Trace Data

Slide 68

Slide 68 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Snapshot Usecase ● Trigger record as an ABRT layer. Integrate with ABRT to generate snapshots on kenrel panics, system-wide bugs in Fedora/RHEL ● Record on alerts by logging infra (Nagios etc) ● Correlate logs with traces ● Remote lightweight system monitoring ● Flight-recorder mode. Overwrite snapshots periodically Tracing Tools

Slide 69

Slide 69 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Live Trace View ● LTTng Relay Daemon – extract and analyze traces as they are being created! ● View local or remote and save as well Tracing Tools LTTng Consumer Daemon LTTng Relay Daemon Live Viewer CTF on disk events (TCP) (TCP) live events CTF

Slide 70

Slide 70 text

Fedora Infra Servers POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Live Trace View Usecase Tracing Tools LTTng session LTTng Relay Daemon Live Viewer TCP TCP Fedora Infra Servers LTTng session Fedora Infra Servers LTTng session Fedora Infra Servers LTTng session TCP TCP TCP

Slide 71

Slide 71 text

trace viewing and analysis POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

Slide 72

Slide 72 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma - --- --- ----- ---- -- ------ - - ---- - - -- ----- --------- - - --- ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - -------- - -- - ---- --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - --- ----- - - - - - ----------- - - - - - - - - - - - - ------- -- - ----- - --- --- ----- ---- -- ------ - - ---- - - -- ----- --------- - - --- ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- -- - - - - - - - --------- - -- - - - - -- - - - ---------- - - - -------- - -- - ---- --------- - - - ------ - --- - -- ------ - -- -- - ---- -- - -- ---- - - - ------ - - - - - -- --- - - - - - - -- -- - - -- - - - -- - -- DEAL WITH IT * *Hold on, we have awesome tools for that

Slide 73

Slide 73 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Trace Analysis babeltrace ● Text based CTF viewer ● Default viewer for lttng view command TraceCompass ● Most detailed and feature rich viewer ● Eclipse IDE integration or RCP app ● Live and remote trace views ● Control flow view, histograms, event list, trace statistics, CPU/resource allocation ● Mix kernel and userspace trace analysis

Slide 74

Slide 74 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma DEMO Latest TraceCompass RCP ● http://projects.eclipse.org/projects/tools.tracecompass/downloads RCP with Critical Flow View (Experimental) ● http://secretaire.dorsal.polymtl.ca/~gbastien/TracingRCP/DorsalExperimental/

Slide 75

Slide 75 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Trace Analysis Remember, we talked about live trace view?

Slide 76

Slide 76 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Trace Analysis LTTngTop ● top like viewer – but for live and recorded traces ● Pause live traces, move forward or back in time ● But you don't believe me, do you? Remember, we talked about live trace view?

Slide 77

Slide 77 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma LTTngTop DEMO

Slide 78

Slide 78 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma One more thing...

Slide 79

Slide 79 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma *Thanks to Brendan Gregg for this image *

Slide 80

Slide 80 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma More about Systems Performance? http://www.brendangregg.com/linuxperf.html

Slide 81

Slide 81 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma References ● Graphics and text on slide 63, 67, 70 have been adapted from David Goulet's talk at FOSDEM '14. ● Graphics on slide 69 have been adapted from http://lttng.org/docs ● Graphics on slide 79 are from http://brendangregg.com/linuxperf.html All the images in this presentation drawn by the author are released under Creative Commons. All other graphics have been taken from OpenClipArt and are under public domain.

Slide 82

Slide 82 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Acknowledgments Thanks to EfficiOS, Ericsson Montréal and DORSAL Lab, Polytechnique Montreal for the awesome work on LTTng/UST, TraceCompass and LttngTop. Thanks to Fedora community and Red Hat for this opportunity.

Slide 83

Slide 83 text

lttng.org/docs POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma

Slide 84

Slide 84 text

POLYTECHNIQUE MONTREAL – Suchakrapani Datt Sharma Questions? suchakrapani.sharma@polymtl.ca suchakra on #lttng (irc.oftc.net) @tuxology http://suchakra.in