Tweet
Tweet

Slide 1

Slide 1 text

Now You See Me Too Visual Tooling for Advanced Systems Analysis Suchakra Sharma 2nd Nov 2017 - San Francisco, CA

Slide 2

Slide 2 text

whoami 1 Suchakrapani Datt Sharma Suchakra - Staff Scientist, ShiftLeft Inc. - PhD, DORSAL Lab, École Polytechnique de Montréal (University of Montréal) - Loves tracing, security, performance analysis, low-level hardware stuff, poutine and samosas - @tuxology

Slide 3

Slide 3 text

whoami 1 Suchakrapani Datt Sharma Suchakra - Staff Scientist, ShiftLeft Inc. - PhD, DORSAL Lab, École Polytechnique de Montréal (University of Montréal) - Loves tracing, security, performance analysis, low-level hardware stuff, poutine and samosas - @tuxology

Slide 4

Slide 4 text

whoami 1 Suchakrapani Datt Sharma Suchakra - Staff Scientist, ShiftLeft Inc. - PhD, DORSAL Lab, École Polytechnique de Montréal (University of Montréal) - Loves tracing, security, performance analysis, low-level hardware stuff, poutine and samosas - @tuxology

Slide 5

Slide 5 text

whoami 1 Suchakrapani Datt Sharma Suchakra - Staff Scientist, ShiftLeft Inc. - PhD, DORSAL Lab, École Polytechnique de Montréal (University of Montréal) - Loves tracing, security, performance analysis, low-level hardware stuff, poutine and samosas - @tuxology Potatoes

Slide 6

Slide 6 text

whoami 1 Suchakrapani Datt Sharma Suchakra - Staff Scientist, ShiftLeft Inc. - PhD, DORSAL Lab, École Polytechnique de Montréal (University of Montréal) - Loves tracing, security, performance analysis, low-level hardware stuff, poutine and samosas - @tuxology Potatoes Visualization & Inference Data Set {P,S}

Slide 7

Slide 7 text

Agenda Humans and Perception Software Systems Visualization - Analysis Taxonomy - Gathering Systems Data - Store and Visualize - Examples - Role of Colors Sample Visuals - FlameGraphs/FlameCharts, Timelines, Heatmaps Demos Future Stuff 2 Suchakrapani Datt Sharma

Slide 8

Slide 8 text

Get Inspired

Slide 9

Slide 9 text

Humans and Perception 3 Suchakrapani Datt Sharma https://www.smithsonianmag.com/science-nature/rockart-ages-indonesian-cave-paintings-are-40000-years-old-180952970/ Hand stencils from a cave in Sulawesi, Indonesia (~40,000 years old)

Slide 10

Slide 10 text

Humans and Perception 4 Suchakrapani Datt Sharma http://atlas.cern/updates/physics-briefing/photon-tagged-jet-quenching-quark-gluon-plasma Event display of a lead-lead collision with a large transverse momentum photon (2017, CERN)

Slide 11

Slide 11 text

Visualizing Data Brief Historical Perspective 5 Suchakrapani Datt Sharma A Brief History of Data Visualization Michael Friendly : http://www.datavis.ca/papers/hbook.pdf Planetary movements shown as cyclic Inclinations over time Y axis is inclination, X axis is time. (Unknown, 10th Century) Earliest form of time-series - data is statistically less rigorous

Slide 12

Slide 12 text

Visualizing Data Brief Historical Perspective 6 Suchakrapani Datt Sharma Early prototype of bar Graphs, - The Latitude of Forms (Nicole Orseme, 1350) Velocity of an object with time Bar Charts before they were cool!

Slide 13

Slide 13 text

Visualizing Data Brief Historical Perspective 7 Suchakrapani Datt Sharma A Brief History of Data Visualization Michael Friendly : http://www.datavis.ca/papers/hbook.pdf A New and Correct Chart Showing the Variations of the Compass in the Western & Southern Oceans (Edmund Halley, c. 1749) Spatial Data – Shipping and cartography pushed the limits!

Slide 14

Slide 14 text

Visualizing Data Brief Historical Perspective 8 Suchakrapani Datt Sharma A New Chart of History (Joseph Priestly - 1769) Time and Space, development of Timelines, colors and their significance

Slide 15

Slide 15 text

Visualizing Data Brief Historical Perspective 9 Suchakrapani Datt Sharma Imports and Exports from America in 18th Century (William Playfair) Modern Time-series and line graphs, statistical rigor

Slide 16

Slide 16 text

This is not a presentation about Data Viz

Slide 17

Slide 17 text

Visualizing Data Stats & Data 101 - OpenIntro Statistics & Advanced High School Statistics - https://www.openintro.org/stat/ Classification and Process 101 - Angela Zoss - https://guides.library.duke.edu/c.php?g=289678&p=1930713 - https://www.slideshare.net/amzoss/everything-data - Udemy Course - https://www.udemy.com/introduction-to-data-visualization/ 10 Suchakrapani Datt Sharma

Slide 18

Slide 18 text

Software and Systems Visualizations

Slide 19

Slide 19 text

Get Inspired

Slide 20

Slide 20 text

Get Inspired ...again

Slide 21

Slide 21 text

[00:43:52.346539963] (+0.000001999) ubuntu-lisa kmem_cache_alloc: { cpu_id = 1 }, { call_site = 0xFFFFFFFF81211A3C, ptr = 0xFFFF880036C5BB00, bytes_req = 256, bytes_alloc = 256, gfp_flags = 37781696 } [00:43:52.346540662] (+0.000000699) ubuntu-lisa kmem_kmalloc: { cpu_id = 1 }, { call_site = 0xFFFFFFFF8139460C, ptr = 0xFFFF88003117DCC0, bytes_req = 24, bytes_alloc = 32, gfp_flags = 37781696 } [00:43:52.346546536] (+0.000005874) ubuntu-lisa kmem_cache_free: { cpu_id = 1 }, { call_site = 0xFFFFFFFF8121FCF4, ptr = 0xFFFF88007ADC6000 } [00:43:52.346547182] (+0.000000646) ubuntu-lisa syscall_exit_open: { cpu_id = 1 }, { ret = 3 } [00:43:52.346555162] (+0.000007980) ubuntu-lisa syscall_entry_newfstat: { cpu_id = 1 }, { fd = 3 } [00:43:52.346556962] (+0.000001800) ubuntu-lisa syscall_exit_newfstat: { cpu_id = 1 }, { ret = 0, statbuf = 140730433466352 } [00:43:52.346559844] (+0.000002882) ubuntu-lisa kmem_mm_page_alloc: { cpu_id = 1 }, { page = 0xFFFFEA0000B7A040, pfn = 188033, order = 0, gfp_flags = 37912778, migratetype = 1 } [00:43:52.346560899] (+0.000001055) ubuntu-lisa syscall_entry_read: { cpu_id = 1 }, { fd = 3, count = 4096 } [00:43:52.346563739] (+0.000002840) ubuntu-lisa syscall_exit_read: { cpu_id = 1 }, { ret = 19, buf = 94392247288688 } [00:43:52.346569483] (+0.000005744) ubuntu-lisa syscall_entry_read: { cpu_id = 1 }, { fd = 3, count = 4096 } [00:43:52.346570064] (+0.000000581) ubuntu-lisa syscall_exit_read: { cpu_id = 1 }, { ret = 0, buf = 94392247288688 } [00:43:52.346571514] (+0.000001450) ubuntu-lisa syscall_entry_close: { cpu_id = 1 }, { fd = 3 } [00:43:52.346572986] (+0.000001472) ubuntu-lisa syscall_exit_close: { cpu_id = 1 }, { ret = 0 } [00:43:52.346573738] (+0.000000752) ubuntu-lisa kmem_kfree: { cpu_id = 1 }, { call_site = 0xFFFFFFFF811B51FD, ptr = 0xFFFF88003117DCC0 } [00:43:52.346577658] (+0.000003920) ubuntu-lisa kmem_mm_page_alloc: { cpu_id = 1 }, { page = 0xFFFFEA0001DFA600, pfn = 491160, order = 0, gfp_flags = 37912778, migratetype = 1 } [00:43:52.346580365] (+0.000002707) ubuntu-lisa kmem_mm_page_alloc: { cpu_id = 1 }, { page = 0xFFFFEA0001443E40, pfn = 332025, order = 0, gfp_flags = 37912778, migratetype = 1 } [00:43:52.346583219] (+0.000002854) ubuntu-lisa syscall_entry_getuid: { cpu_id = 1 }, { } [00:43:52.346583541] (+0.000000322) ubuntu-lisa syscall_exit_getuid: { cpu_id = 1 }, { ret = 0 }

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

Systems Analysis A Brief Taxonomy of Analysis 14 Suchakrapani Datt Sharma

Slide 24

Slide 24 text

Systems Analysis A Brief Taxonomy of Analysis 14 Suchakrapani Datt Sharma Conception Structural Analysis of software and systems eg. Static analysis of kernel source code for program flow and dependency graphs

Slide 25

Slide 25 text

Systems Analysis A Brief Taxonomy of Analysis 14 Suchakrapani Datt Sharma Conception Live Structural Analysis of software and systems eg. Static analysis of kernel source code for program flow and dependency graphs Behavioral and Execution Analysis of software systems Dynamic analysis by instrumenting applications, buffer the events and display results live or at short samples eg. Live traces, sampled resource usage (CPU, memory)

Slide 26

Slide 26 text

Systems Analysis A Brief Taxonomy of Analysis 14 Suchakrapani Datt Sharma Conception Live Post-Mortem Structural Analysis of software and systems eg. Static analysis of kernel source code for program flow and dependency graphs Behavioral and Execution Analysis of software systems Dynamic analysis by instrumenting applications, buffer the events and display results live or at short samples eg. Live traces, sampled resource usage (CPU, memory) Behavioral, Execution and Root Cause Analysis of software systems Gather as much data with as much granularity. Characterize, visualize and infer. eg. Root cause analysis of abnormal interrupt latency, software/hardware traces

Slide 27

Slide 27 text

Visualizing Systems Data From Problem to Insight 15 Suchakrapani Datt Sharma

Slide 28

Slide 28 text

Visualizing Systems Data From Problem to Insight 15 Suchakrapani Datt Sharma Question Program is making a bunch of syscalls, some are slow. Which? Why? Where?

Slide 29

Slide 29 text

Visualizing Systems Data From Problem to Insight 15 Suchakrapani Datt Sharma Question Gather Program is making a bunch of syscalls, some are slow. Which? Why? Where? Instrument syscalls with timestamps and observe them as they occur in an execution Record stack of all functions called inside the syscall and compare executions Sample or Trace

Slide 30

Slide 30 text

Visualizing Systems Data From Problem to Insight 15 Suchakrapani Datt Sharma Question Gather Characterize Program is making a bunch of syscalls, some are slow. Which? Why? Where? Instrument syscalls with timestamps and observe them as they occur in an execution Record stack of all functions called inside the syscall and compare executions Sample or Trace Frequency Analysis Latency Analysis Multi-modal Analysis (Hierarchy + Time)

Slide 31

Slide 31 text

Visualizing Systems Data From Problem to Insight 15 Suchakrapani Datt Sharma Question Gather Characterize Visualize Program is making a bunch of syscalls, some are slow. Which? Why? Where? Instrument syscalls with timestamps and observe them as they occur in an execution Record stack of all functions called inside the syscall and compare executions Sample or Trace Frequency Analysis Latency Analysis Multi-modal Analysis (Hierarchy + Time) Histograms, Scatter- plots Heat-maps Graphs, Timelines, Flamecharts, Flamegraphs

Slide 32

Slide 32 text

Visualizing Systems Data Case Study - 16 Suchakrapani Datt Sharma How would the code flow in my web-server? What are terminal sources and sinks? Conception thttpd.c thttpd.c.212r.expand Callgraph Extraction From RTL CFLAGS=-fdump-rtl-expand Flow Classification clear_connection linger_clear_connection really_clear_connection handle_term shut_down logstats idle handle_usr1 handle_usr2 main occasional value_required.part.2 no_value_required.part.1 e_strdup.part.4 show_stats update_throttles handle_newconnect handle_alrm handle_hup wakeup_connection usage handle_chld Questions handle request_pool new_connect wakeup_connect process shutdown Gather Characterize Visualize

Slide 33

Slide 33 text

Visualizing Systems Data Case Study - 17 Suchakrapani Datt Sharma How do we observe disk I/O latencies? Live eBPF Program req_start/stop HISTOGRAM MAP kprobe/kretprobes Questions Gather Characterize Visualize usecs : count distribution 0 -> 1 : 0 | | 2 -> 3 : 0 | | 4 -> 7 : 0 | | 8 -> 15 : 0 | | 16 -> 31 : 115 |** | 32 -> 63 : 553 |************** | 64 -> 127 : 1087 |*************************** | 128 -> 255 : 1572 |****************************************| 256 -> 511 : 1040 |************************** | 512 -> 1023 : 54 |* |

Slide 34

Slide 34 text

Visualizing Systems Data Case Study - 18 Suchakrapani Datt Sharma What were most frequent calls made? What did those call-stacks looked like? How did they interact with the underlying system? SimpleWebServer.jar perf-.data Callstack Extraction (count + hierarchy) Perf Questions Gather Characterize Visualize Post-Mortem Search java/i.. pe.. __pe.. _.. _.. C.. pthr.. d.. p.. JavaThread::run __per.. do_.. page_fault java_start x86_p.. x86_.. fut.. d.. finis.. thread_entry java/io/.. s.. su.. rwsem_.. fute.. native_w.. pth.. sys.. [u.. o.. intel_pm.. Th.. sche.. su.. perf_pmu.. j.. call_r.. do_page.. jni_.. __mmap64 Java_.. inte.. nati.. ja.. s.. j.. j.. do_f.. sys_mmap perf.. f.. ent.. perf_.. rwsem_.. p.. ja.. fute.. [unknown] pth.. vm_mmap.. in.. J.. java.. entry_S.. perf_.. entr.. x.. na.. JNU_.. __perf_e.. down_w.. x86_.. sched.. Int.. j.. ja.. JavaCalls::call_virtual x8.. __do_pa.. sys_.. _.. inte.. x8.. i.. s.. start_thread fut.. c.. java/.. __.. p.. __s.. sch.. JV.. fin.. __sche.. __sc.. JN.. JavaThread::thread_main_inner __sch.. d.. fini.. d.. Monitor::no.. na.. down_r.. __p.. p.. ret_from_.. jav.. [u.. os::start_t.. [unknown] fut.. finish_ta.. finish.. d.. JavaThread::~Jav.. org/jibble/simplewebserver/RequestThread:::run __per.. fi.. Java.. nati.. sch.. Jav.. in.. ja.. intel.. call_stub java jni.. nati.. JavaCalls::call_virtual schedule M.. _.. r.. java/net/PlainSo.. pe.. schedule_.. ja.. x86_pmu_.. ca.. sys.. call_r.. [unknown] ent.. j.. do_.. sys_mma.. fut.. __clone _.. JavaCalls::call_helper s.. __s.. Ja.. perf-map-agent

Slide 35

Slide 35 text

Gathering Systems Data

Slide 36

Slide 36 text

Observability Layers 19 Suchakrapani Datt Sharma

Slide 37

Slide 37 text

Observability Layers Hardware Host OS Kernel functions, hardware perf counters 20 Suchakrapani Datt Sharma

Slide 38

Slide 38 text

Observability Layers Hardware Virtualization Containerization Host OS Guest OS Kernel functions, hardware perf counters Kernel functions, Hypervisor, Custom APIs 21 Suchakrapani Datt Sharma

Slide 39

Slide 39 text

Observability Layers Applications Applications Libs Libs Kernel functions, hardware perf counters Kernel functions, Hypervisor, Custom APIs Library functions, Syscalls 22 Suchakrapani Datt Sharma Hardware Virtualization Containerization Host OS Guest OS

Slide 40

Slide 40 text

Observability Layers Functions Kernel functions, hardware perf counters Kernel functions, Hypervisor, Custom APIs Library functions, Syscalls Application functions 23 Suchakrapani Datt Sharma Applications Applications Libs Libs Hardware Virtualization Containerization Host OS Guest OS

Slide 41

Slide 41 text

Observability Layers Instructions Kernel functions, hardware perf counters Kernel functions, Hypervisor, Custom APIs Library functions, Syscalls Application functions Branches, calls 24 Suchakrapani Datt Sharma Functions Applications Applications Libs Libs Hardware Virtualization Containerization Host OS Guest OS

Slide 42

Slide 42 text

Observability Layers Logic Subatomic Particles Kernel functions, hardware perf counters Kernel functions, Hypervisor, Custom APIs Library functions, Syscalls Application functions Branches, calls CPU pins EM waves ?? 25 Suchakrapani Datt Sharma Instructions Functions Applications Applications Libs Libs Hardware Virtualization Containerization Host OS Guest OS

Slide 43

Slide 43 text

Data Sources System - Functions: Static Tracing (Tracepoints, Ftrace) - Functions: Dynamic Tracing (Kprobes, Kretprobes) - PMU Events (Perf) - Hardware Counters (Perf) - Hardware Tracing Blocks (Intel PT, ARM CoreSight) - Kernel Logs (LSM Audit logs) Applications - Application Logs/Traces (JUL) - Userspace tracing (eg. Uftrace, compiler instrumentation) 26 Suchakrapani Datt Sharma

Slide 44

Slide 44 text

Deep Dive Tracing

Slide 45

Slide 45 text

Observing Functions bar() foo() baz() call_me_maybe() call_me_maybe() Fill Buffer Collect Data Event Event Event Program Flow, Arguments, Latency, Data Flow Observation Function 27 Suchakrapani Datt Sharma

Slide 46

Slide 46 text

Observing Functions Kernel Example - Dynamically instrument with Kprobes do_sys_open() jmp Restore registers Save registers CALL pre_handler ORIG INSN CALL pre_handler Custom handler eBPF program JMP back Patched Instruction Supported by eBPF (Stored in Maps Perf → Data), LTTng (CTF) 28 Suchakrapani Datt Sharma

Slide 47

Slide 47 text

Observing Functions bar() __cyg_profile_func_enter Generate Event Timestamp Event Event Event -finstrument-functions Userspace Example - Compiler-assisted instrumentation __cyg_profile_func_exit CTF Supported by LTTng $ lttng create $ lttng enable-event -a -u $ lttng add-context -u -t vpid -t vtid -t procname $ lttng start $ LD_PRELOAD=/usr/lib/liblttng-ust-cyg-profile.so ./thttpd $ lttng stop 29 Suchakrapani Datt Sharma

Slide 48

Slide 48 text

Observing Functions CTF Trace Events [17:43:26.548312332] (+0.000000098) ubuntu-lisa lttng_ust_cyg_profile:func_entry: { cpu_id = 0 }, { vpid = 2293, vtid = 2293, procname = "thttpd" }, { addr = 0x410B50, call_site = 0x411414 } [17:43:26.548312980] (+0.000000104) ubuntu-lisa lttng_ust_cyg_profile:func_entry: { cpu_id = 0 }, { vpid = 2293, vtid = 2293, procname = "thttpd" }, { addr = 0x410AE0, call_site = 0x410B86 } [17:43:26.548313787] (+0.000000103) ubuntu-lisa lttng_ust_cyg_profile:func_exit: { cpu_id = 0 }, { vpid = 2293, vtid = 2293, procname = "thttpd" }, { addr = 0x410AE0, call_site = 0x410B86 } [17:43:26.548314427] (+0.000000103) ubuntu-lisa lttng_ust_cyg_profile:func_exit: { cpu_id = 0 }, { vpid = 2293, vtid = 2293, procname = "thttpd" }, { addr = 0x410B50, call_site = 0x411414 } 29 Suchakrapani Datt Sharma Time Delta Event name Trace Payload Timestamp

Slide 49

Slide 49 text

Observing Functions eBPF + Kprobes BPF Code Kprobe Kernel Function trace.bpf LLVM/Clang Perf Buffer bpf() bpf() 30 Suchakrapani Datt Sharma

Slide 50

Slide 50 text

Observing Functions Kernel Example - Dynamically instrument with Kprobes do_sys_open() jmp Restore registers Save registers CALL pre_handler ORIG INSN CALL pre_handler Custom handler eBPF program JMP back Patched Instruction Supported by eBPF (Stored in Maps Perf → Data), LTTng (CTF) 28 Suchakrapani Datt Sharma

Slide 51

Slide 51 text

Observing Functions eBPF + Kprobes BPF Code Kprobe Kernel Function trace.bpf LLVM/Clang Perf Buffer bpf() bpf() 30 Suchakrapani Datt Sharma

Slide 52

Slide 52 text

eBPF + Kprobes - IOVisor BCC – Python, C++, Lua, Go (gobpf) APIs - Compile BPF programs directly via LLVM interface - Helper functions to manage maps, buffers, probes from bcc import BPF prog = """ int hello(void *ctx) { bpf_trace_printk("Hello, World!\\n"); return 0; } """ b = BPF(text=prog) b.attach_kprobe(event="sys_clone", fn_name="hello") print "PID MESSAGE" b.trace_print(fmt="{1} {5}") Attach to Kprobe event prog compiled to BPF bytecode Print trace pipe Complete Program trace_fields.py Observing Functions 31 Suchakrapani Datt Sharma

Slide 53

Slide 53 text

Some Formats Formats and Storage - Scaled Storage (Time-series): OpenTSDB, Graphite, InfluxDB, Prometheus - Compact Storage (Binary formats): CTF, Perf - Graph Storage (Source Analysis): CPG (Titan/JanusDB) - Custom - JSON (Uftrace) - XML Traces 32 Suchakrapani Datt Sharma

Slide 54

Slide 54 text

Characterization and Visualization of Data

Slide 55

Slide 55 text

Popular Means From Events to Visuals - Latency/Density (Syscalls) → Histograms → Heatmaps - Latency (Syscalls) vs Time → Scatter Plots - Throughput → Line Charts - Resource Consumption → Line Charts - Per-Entity (Process/Host/VM) Events → Timeline/Gantt - Event Summaries → Pie, Donut, Bar Charts - Function Graphs (Time) → Flame Charts - Function Graphs (Count) → Flame Graphs - Callgraphs : Graphs/TreeMap/Sunburst 33 Suchakrapani Datt Sharma

Slide 56

Slide 56 text

Histograms Latency Histograms - Periodic snapshots in simple ASCII 35 Suchakrapani Datt Sharma $ sudo ./biolatency.py usecs : count distribution 0 -> 1 : 0 | | 2 -> 3 : 0 | | 4 -> 7 : 0 | | 8 -> 15 : 0 | | 16 -> 31 : 115 |** | 32 -> 63 : 553 |************** | 64 -> 127 : 1087 |*************************** | 128 -> 255 : 1572 |****************************************| 256 -> 511 : 1040 |************************** | 512 -> 1023 : 54 |* | eBPF-BCC Tools

Slide 57

Slide 57 text

Histograms Latency Histograms - Periodic snapshots in simple ASCII 36 Suchakrapani Datt Sharma $ sudo ./biolatency.py usecs : count distribution 0 -> 1 : 0 | | 2 -> 3 : 0 | | 4 -> 7 : 0 | | 8 -> 15 : 0 | | 16 -> 31 : 115 |** | 32 -> 63 : 553 |************** | 64 -> 127 : 1087 |*************************** | 128 -> 255 : 1572 |****************************************| 256 -> 511 : 1040 |************************** | 512 -> 1023 : 54 |* |

Slide 58

Slide 58 text

Heatmaps Latency Heatmaps 37 Suchakrapani Datt Sharma Count Latency (us) 2 8 32 553 128 512

Slide 59

Slide 59 text

Heatmaps Latency Heatmaps 38 Suchakrapani Datt Sharma Time

Slide 60

Slide 60 text

Heatmaps Latency Heatmaps 38 Suchakrapani Datt Sharma Bi-modal Outlier Time

Slide 61

Slide 61 text

Heatmaps Latency Heatmaps 38 Suchakrapani Datt Sharma Bi-modal Outlier Time Averages don’t do justice

Slide 62

Slide 62 text

Heatmaps Latency Heatmaps 39 Suchakrapani Datt Sharma Time

Slide 63

Slide 63 text

Heatmaps Latency Heatmaps 40 Suchakrapani Datt Sharma Time Encode Counts as colors

Slide 64

Slide 64 text

Heatmaps Latency Heatmaps 41 Suchakrapani Datt Sharma Time Latency Outliers

Slide 65

Slide 65 text

Heatmaps Latency Heatmaps 42 Suchakrapani Datt Sharma Btw, this is really derived from an eBPF tool

Slide 66

Slide 66 text

Heatmaps Moaaaarr! Heatmaps 43 Suchakrapani Datt Sharma Luca Canali from CERN uses this!

Slide 67

Slide 67 text

Heatmaps Moaaaarr! Heatmaps 44 Suchakrapani Datt Sharma ..and apparently Honeycomb users as well.. Netflix as well! https://honeycomb.io/blog/2017/09/heatmaps-are-the-new-hotness/

Slide 68

Slide 68 text

More Stuff More Latency Visualizations - http://www.brendangregg.com/HeatMaps/latency.html - http://queue.acm.org/detail.cfm?id=1809426 - https://honeycomb.io/blog/2017/09/heatmaps-are-the-new-hotness/ - https://github.com/LucaCanali/PyLatencyMap Extra Reading: Utilization Heatmaps - http://www.brendangregg.com/HeatMaps/utilization.html 45 Suchakrapani Datt Sharma

Slide 69

Slide 69 text

Flames Function Graph (Uftrace) 46 Suchakrapani Datt Sharma [ 8898] | tmr_init() { 0.667 us [ 8898] | tmr_init(); 1.782 us [ 8898] | } /* tmr_init */ [ 8898] | httpd_initialize() { [ 8898] | httpd_initialize() { [ 8898] | check_options() { 0.435 us [ 8898] | check_options(); 1.501 us [ 8898] | } /* check_options */ 1.342 us [ 8898] | malloc(); [ 8898] | gethostname() { 4.893 us [ 8898] | gethostname(); 6.201 us [ 8898] | } /* gethostname */ 0.910 us [ 8898] | __strdup(); 0.744 us [ 8898] | __strdup(); 0.572 us [ 8898] | __strdup(); [ 8898] | httpd_set_logfp() { 0.448 us [ 8898] | httpd_set_logfp(); 1.393 us [ 8898] | } /* httpd_set_logfp */ 6.198 ms [ 8898] | } /* poll_init */ 6.203 ms [ 8898] | } /* poll_init */ 8.656 ms [ 8898] | } /* fdwatch_get_nfiles */ 8.657 ms [ 8898] | } /* fdwatch_get_nfiles */ 18.768 us [ 8898] | chroot(); 2.505 us [ 8898] | strncmp(); [ 8898] | syslog() { 727.596 us [ 8898] | __syslog_chk(); 730.243 us [ 8898] | } /* syslog */ [ 8898] | fprintf() { 1.194 us [ 8898] | __fprintf_chk(); 2.664 us [ 8898] | } /* fprintf */ 0.430 us [ 8898] | strcpy(); 4.179 us [ 8898] | chdir(); 8.285 us [ 8898] | sigset(); 1.497 us [ 8898] | sigset(); 1.385 us [ 8898] | sigset(); 6.322 us [ 8898] | alarm(); Snippet (thttpd request-response)

Slide 70

Slide 70 text

Flames Flame Chart (Uftrace) 47 Suchakrapani Datt Sharma Time Stack Depth Handle New Connection (137us) Service First Request https://github.com/namhyung/uftrace

Slide 71

Slide 71 text

Flames Flame Graph (Java mixed-mode) 48 Suchakrapani Datt Sharma Java Application System (Kernel) Search java/i.. pe.. __pe.. _.. _.. C.. pthr.. d.. p.. JavaThread::run __per.. do_.. page_fault java_start x86_p.. x86_.. fut.. d.. finis.. thread_entry java/io/.. s.. su.. rwsem_.. fute.. native_w.. pth.. sys.. [u.. o.. intel_pm.. Th.. sche.. su.. perf_pmu.. j.. call_r.. do_page.. jni_.. __mmap64 Java_.. inte.. nati.. ja.. s.. j.. j.. do_f.. sys_mmap perf.. f.. ent.. perf_.. rwsem_.. p.. ja.. fute.. [unknown] pth.. vm_mmap.. in.. J.. java.. entry_S.. perf_.. entr.. x.. na.. JNU_.. __perf_e.. down_w.. x86_.. sched.. Int.. j.. ja.. JavaCalls::call_virtual x8.. __do_pa.. sys_.. _.. inte.. x8.. i.. s.. start_thread fut.. c.. java/.. __.. p.. __s.. sch.. JV.. fin.. __sche.. __sc.. JN.. JavaThread::thread_main_inner __sch.. d.. fini.. d.. Monitor::no.. na.. down_r.. __p.. p.. ret_from_.. jav.. [u.. os::start_t.. [unknown] fut.. finish_ta.. finish.. d.. JavaThread::~Jav.. org/jibble/simplewebserver/RequestThread:::run __per.. fi.. Java.. nati.. sch.. Jav.. in.. ja.. intel.. call_stub java jni.. nati.. JavaCalls::call_virtual schedule M.. _.. r.. java/net/PlainSo.. pe.. schedule_.. ja.. x86_pmu_.. ca.. sys.. call_r.. [unknown] ent.. j.. do_.. sys_mma.. fut.. __clone _.. JavaCalls::call_helper s.. __s.. Ja.. JVM Single Request (SimpleWebServer.jar)

Slide 72

Slide 72 text

Old Flames Moooaaaar! Flaming Stuff 49 Suchakrapani Datt Sharma Java App GC HotSpot Flame Charts.. before they were cool! (Java) http://www.oracle.com/technetwork/server-storage/solaris/perftools-141913.html

Slide 73

Slide 73 text

Old Flames Moooaaaar! Flaming Stuff 50 Suchakrapani Datt Sharma CPUs DTrace + Solaris Kernel (CPU Stacks) http://www.oracle.com/technetwork/server-storage/solaris/perftools-141913.html

Slide 74

Slide 74 text

Flames Flame Charts - Function Graphs (Time): Keeps on running. Horizontal-axis is time. - Sourced from Chrome traces, LTTng traces, Uftrace Flame Graphs - Function Graphs (Count): Callstacks visualized during execution with population on horizontal-axis and depth of stack on vertical-axis - Sourced from perf events, eBPF stack traces, LTTng traces 51 Suchakrapani Datt Sharma

Slide 75

Slide 75 text

Callgraphs, Treemaps & Sunbursts 52 Suchakrapani Datt Sharma https://stackoverflow.com/a/6617588 Valgrind + Callgrind Baobab (Disk Usage)

Slide 76

Slide 76 text

Timelines Control Flow (LTTng/Custom Trace) 53 Suchakrapani Datt Sharma Usermode Syscall wget CPU0 Usermode Interrupted Usermode Syscall Usermode SoftIRQ 3 States Time IRQ 3 (Network) 2.6us IRQ raised 1.96 "

Slide 77

Slide 77 text

Timelines Critical Flow (LTTng) Identify the critical path of execution by following a process’ execution and its relation with other processes 54 Suchakrapani Datt Sharma Preempted wget bash Running Running Running Network

Slide 78

Slide 78 text

Tooling Demo

Slide 79

Slide 79 text

tracecompass.org

Slide 80

Slide 80 text

Short Note on Colors

Slide 81

Slide 81 text

Colors Diverging Trends - Think of it as temperature (heatmaps) Sequential Trend - Related but changing (heatmaps, flamegraphs) Qualitative Trend - Distinct entities (process states, timelines) 55 Suchakrapani Datt Sharma http://colorbrewer2.org

Slide 82

Slide 82 text

What Next? Wishlist for an Ultimate Analysis Pipeline - Pluggable sources - Distributed systems support - Diffs of executions - Time-synchronized views - Summarization of events and observations - Intuitive UIs - Filtering - Hardware Traces! Lolwut? 56 Suchakrapani Datt Sharma

Slide 83

Slide 83 text

Future Fun Hardware Trace Visualizations - Targeted Intel Processor Trace (PT) snapshot of mmap() - Sunburst of syscall latency (insn). Tracerception! 57 Suchakrapani Datt Sharma 173 ns and 917 instructions more

Slide 84

Slide 84 text

Future Fun Data Driven Visuals (Intel PT) 58 Suchakrapani Datt Sharma 32423545 1 243241334 2342353646 perf.data PTParse + VMPT vCPUs Consumption Process Control Flow Perf PT traces

Slide 85

Slide 85 text

- Linux Performance/Tracing - http://www.brendangregg.com/linuxperf.html - https://jvns.ca/blog/2017/07/05/linux-tracing-systems/ - https://www.kernel.org/doc/Documentation/trace/ftrace.txt - https://github.com/namhyung/uftrace - http://lttng.org/docs - http://tracecompass.org/#docs - https://www.kernel.org/doc/Documentation/kprobes.txt - https://github.com/jvm-profiling-tools/perf-map-agent - https://github.com/goldshtn/linux-tracing-workshop More References 60 Suchakrapani Datt Sharma

Slide 86

Slide 86 text

- Visualizations - A Tour Through the Visualization Zoo http://queue.acm.org/detail.cfm?id=1805128 - https://www.slideshare.net/brendangregg/javaone-2015-java-mixedmode- flame-graphs More References 60 Suchakrapani Datt Sharma

Slide 87

Slide 87 text

- Research and Future [Begel et al. 1999] BPF+: exploiting global data-flow optimization in a generalized packet filter architecture, ACM SIGCOMM ‘99 [Desnoyers et al. 2009] The LTTng tracer: A low impact performance and behavior monitor for GNU/Linux, Ottawa Linux Symposium 2009 [Desnoyers et al.] Low-Impact Operating Systems Tracing, PhD Thesis (polymtl.ca) [Clement 2016] Linux Kernel packet transmission performance in high-speed networks Masters Thesis (2016), KTH, Stockholm [Borkmann 2016] Advanced programmability and recent updates with tc’s cls_bpf NetDev 1.2 (2016) Tokyo More References 60 Suchakrapani Datt Sharma

Slide 88

Slide 88 text

- Research and Future [Philippe Moret et al. 2010] Visualizing and exploring profiles with calling context ring charts, Sofware: Practice and Experience, Wiley [J. Trümper et al.] Multiscale Visual Comparison of Execution Traces. Proceedings of the International Conference on Program Comprehension, IEEE Computer Society, 2013. [Johannes Bohnet et. al.] Analyzing Feature Implementation by Visual Exploration of Architecturally-Embedded Call-Graphs. 4th International Workshop on Dynamic Analysis, ACM, 2006. [Sharma 2017] Low-Impact System Performance Analysis Using Hardware Assisted Tracing Techniques, PhD Thesis (polymtl.ca) More References 60 Suchakrapani Datt Sharma

Slide 89

Slide 89 text

- LTTng developers - eBPF and BCC developers - Trace Compass developers - Geneviève Bastien, Mohamad Gebai, Francis Giraldeau and Hani Nemati! - Whole ShiftLeft Inc. team! - DORSAL Lab @ Polytechnique Montreal Ack LTTng 61 Suchakrapani Datt Sharma

Slide 90

Slide 90 text

Fin suchakra@shiftleft.io suchakrapani.sharma@polymtl.ca @tuxology