A Support Engineer Walkthrough on pt-stalk Marcos Albe Principal Support Engineer - Percona Marcelo Altmann Senior Support Engineer - Percona 

2 Thank You Sponsors! 

3 SAVE THE DATE! CALL FOR PAPERS OPENING SOON! www.perconalive.com April 23-25, 2018 Santa Clara Convention Center 

Who is speaking ? 

5 Who is Speaking ? • Marcelo Altmann - Senior Support Engineer @ Percona - Resident specialist on Surf & Brazil leisure •Marcos Albé - Principal Support Engineer @ Percona - Chief Barbeque Officer & Designated Beer Analyst 

Agenda 

7 Agenda • What do we USE • Who’s stalkin’? • How pt-stalk can be USEd • Walkthrough 

What do we USE 

9 We use one of these... 

10 Utilization 

11 Utilization Saturation 

12 Utilization Saturation Errors 

13 

Who’s stalkin’? 

15 check out those resources! 

16 Who’s stalkin’? •Aspersa stalk + Aspersa collect in one! (thanks Daniel & Baron) •# The name 'stalk' is because 'watch' is already taken, and 'stalk' is fun. •Now part of pt-toolkit •Monitors variable or a script’s single-valued output •Triggers capture when set threshold is reached - files from /proc: diskstats, interrupts, meminfo, slabinfo, procstat, procvmstat. - vmstat, iostat, top, netstat, pmap, mpstat, sysctl, df, hostname, mysqladmin ext, mysqladmin debug, lsof). - SHOW ENGINE INNODB STATUS, SHOW ENGINE INNODB MUTEX, SHOW GLOBAL VARIABLES, SHOW FULL PROCESSLIST and SHOW OPEN TABLES. •Production safe! •Custom triggers and custom captures •Companion tools (pt-sift, pt-mext, pt-diskstats, pt-summary); Bash dissection 

17 Who’s stalkin’: common parameters •Stalking part - --daemonize | --no-stalk - --function=status - --interval=1 - --variable=Threads_running - --threshold=25 - --cycles=5 •Collecting part - --run-time=30 - --sleep=300 - --iterations= - --plugin= - --dest=/var/lib/pt-stalk • Values shown are the defaults for each (where given) 

18 Who’s stalkin’: work flow 

19 Who’s stalkin’: working examples I Collect data during 60 seconds for 3 consecutive times, starting immediately. pt-stalk --no-stalk \ --dest=$HOME/percona \ --user=root \ --ask-pass \ --run-time=60 --sleep=60 --iterations=3 

20 Who’s stalkin’: working examples II Run as a daemon and trigger collection if InnoDB has more than 5 pending fsyncs. pt-stalk --daemonize \ --function=status \ --variable=Innodb_data_pending_fsyncs \ --threshold=5 \ --cycles=2 \ --interval=1 \ --iterations=5 \ --plugin=/usr/local/bin/raid_card_status.sh 

21 Who’s stalkin’: custom triggers #!/bin/bash # Save this bash snippet as /root/delayed_slave_trg.sh; trg_plugin() { mysql $EXT_ARGV -Ee "SHOW SLAVE STATUS" | \ grep "Seconds_Behind_Master" | \ awk "{ print $2 }"; } # use like pt-stalk --function=/root/delayed_slave_trg.sh --etc --etc 

22 Who’s stalkin’: custom capture #!/bin/bash # save as /root/collect_perf.sh before_collect() { # run perf stat; match sleep length to --run-time /usr/bin/perf stat --all-cpus \ --output=${d}/${p}-stat_a \ -- sleep ${OPT_RUN_TIME}; } # use like pt-stalk --plugin=/root/collect_perf.sh --etc --etc 

How pt-stalk can be USE’d 

24 How pt-stalk can be USE’d Linux Kernel CPU CPU Interconnect DRAM I/O Bridge I/O Controller Network Controller I/O Bus Expander Interconnect Disk Disk Swap Port Port Applications System libraries System Call Interface VFS Sockets Scheduler File Systems TCP/UDP Volume Manager IP Virtual Memory Block Device Interface Ethernet Device Drivers Memory Bus Interface Transports 

25 Linux Kernel CPU CPU Interconnect DRAM I/O Bridge I/O Controller Network Controller I/O Bus Expander Interconnect Disk Disk Swap Port Port Applications (MySQL!) System libraries System Call Interface VFS Sockets Scheduler File Systems TCP/UDP Volume Manager IP Virtual Memory Block Device Interface Ethernet Device Drivers Memory Bus Interface Transports ps df /proc/diskstats SHOW ENGINE INNODB STATUS /proc/interrupts iostat lsof /proc/meminfo mpstat SHOW ENGINE INNODB MUTEX mysqladmin ext SHOW OPEN TABLES pmap /proc/procstat SHOW FULL PROCESSLIST /proc/vmstat /proc/slabinfo sysctl SHOW ENGINE TOKUDB STATUS top SHOW GLOBAL VARIABLES vmstat SHOW SLAVE STATUS netstat strace tcpdump gdb (backtraces) oprofile 

26 strace tcpdump gdb (backtraces) oprofile How pt-stalk can be USE’d: Here Be Dragons! 

A walkthrough 

28 A walkthrough Using pt-stalk to solve problems with... ●...High IO ●...High CPU ●...Networking 

29 Useful links ●grep commands - https://gist.github.com/altmannmarcelo/eec40a54568299c2cc11b6cf585 9203d ●Check List - http://www.brendangregg.com/USEmethod/use-linux.html 

Questions ? Marcos Albe [email protected] Marcelo Altmann @altmannmarcelo [email protected]