A Percona Support Engineer Walkthrough on pt-stalk

This presentation shows the approach we use to diagnostic performance issues on the customer. We explain the USE method and how you can relate it with pt-stalk

Transcript

1. A Support Engineer Walkthrough on pt-stalk Marcos Albe Principal Support Engineer - Percona Marcelo Altmann Senior Support Engineer - Percona
2. 2 Thank You Sponsors!
3. 3 SAVE THE DATE! CALL FOR PAPERS OPENING SOON! www.perconalive.com April 23-25, 2018 Santa Clara Convention Center
4. Who is speaking ?
5. 5 Who is Speaking ? • Marcelo Altmann - Senior Support Engineer @ Percona - Resident specialist on Surf & Brazil leisure •Marcos Albé - Principal Support Engineer @ Percona - Chief Barbeque Officer & Designated Beer Analyst
6. Agenda
7. 7 Agenda • What do we USE • Who’s stalkin’? • How pt-stalk can be USEd • Walkthrough
8. What do we USE
9. 9 We use one of these...
10. 10 Utilization
11. 11 Utilization Saturation
12. 12 Utilization Saturation Errors
14. Who’s stalkin’?
15. 15 check out those resources!
16. 16 Who’s stalkin’? •Aspersa stalk + Aspersa collect in one! (thanks Daniel & Baron) •# The name 'stalk' is because 'watch' is already taken, and 'stalk' is fun. •Now part of pt-toolkit •Monitors variable or a script’s single-valued output •Triggers capture when set threshold is reached - files from /proc: diskstats, interrupts, meminfo, slabinfo, procstat, procvmstat. - vmstat, iostat, top, netstat, pmap, mpstat, sysctl, df, hostname, mysqladmin ext, mysqladmin debug, lsof). - SHOW ENGINE INNODB STATUS, SHOW ENGINE INNODB MUTEX, SHOW GLOBAL VARIABLES, SHOW FULL PROCESSLIST and SHOW OPEN TABLES. •Production safe! •Custom triggers and custom captures •Companion tools (pt-sift, pt-mext, pt-diskstats, pt-summary); Bash dissection
17. 17 Who’s stalkin’: common parameters •Stalking part - --daemonize | --no-stalk - --function=status - --interval=1 - --variable=Threads_running - --threshold=25 - --cycles=5 •Collecting part - --run-time=30 - --sleep=300 - --iterations= - --plugin= - --dest=/var/lib/pt-stalk • Values shown are the defaults for each (where given)
18. 18 Who’s stalkin’: work flow
19. 19 Who’s stalkin’: working examples I Collect data during 60 seconds for 3 consecutive times, starting immediately. pt-stalk --no-stalk \ --dest=$HOME/percona \ --user=root \ --ask-pass \ --run-time=60 --sleep=60 --iterations=3
20. 20 Who’s stalkin’: working examples II Run as a daemon and trigger collection if InnoDB has more than 5 pending fsyncs. pt-stalk --daemonize \ --function=status \ --variable=Innodb_data_pending_fsyncs \ --threshold=5 \ --cycles=2 \ --interval=1 \ --iterations=5 \ --plugin=/usr/local/bin/raid_card_status.sh
21. 21 Who’s stalkin’: custom triggers #!/bin/bash # Save this bash snippet as /root/delayed_slave_trg.sh; trg_plugin() { mysql $EXT_ARGV -Ee "SHOW SLAVE STATUS" | \ grep "Seconds_Behind_Master" | \ awk "{ print $2 }"; } # use like pt-stalk --function=/root/delayed_slave_trg.sh --etc --etc
22. 22 Who’s stalkin’: custom capture #!/bin/bash # save as /root/collect_perf.sh before_collect() { # run perf stat; match sleep length to --run-time /usr/bin/perf stat --all-cpus \ --output=${d}/${p}-stat_a \ -- sleep ${OPT_RUN_TIME}; } # use like pt-stalk --plugin=/root/collect_perf.sh --etc --etc
23. How pt-stalk can be USE’d
24. 24 How pt-stalk can be USE’d Linux Kernel CPU CPU Interconnect DRAM I/O Bridge I/O Controller Network Controller I/O Bus Expander Interconnect Disk Disk Swap Port Port Applications System libraries System Call Interface VFS Sockets Scheduler File Systems TCP/UDP Volume Manager IP Virtual Memory Block Device Interface Ethernet Device Drivers Memory Bus Interface Transports
25. 25 Linux Kernel CPU CPU Interconnect DRAM I/O Bridge I/O Controller Network Controller I/O Bus Expander Interconnect Disk Disk Swap Port Port Applications (MySQL!) System libraries System Call Interface VFS Sockets Scheduler File Systems TCP/UDP Volume Manager IP Virtual Memory Block Device Interface Ethernet Device Drivers Memory Bus Interface Transports ps df /proc/diskstats SHOW ENGINE INNODB STATUS /proc/interrupts iostat lsof /proc/meminfo mpstat SHOW ENGINE INNODB MUTEX mysqladmin ext SHOW OPEN TABLES pmap /proc/procstat SHOW FULL PROCESSLIST /proc/vmstat /proc/slabinfo sysctl SHOW ENGINE TOKUDB STATUS top SHOW GLOBAL VARIABLES vmstat SHOW SLAVE STATUS netstat strace tcpdump gdb (backtraces) oprofile
26. 26 strace tcpdump gdb (backtraces) oprofile How pt-stalk can be USE’d: Here Be Dragons!
27. A walkthrough
28. 28 A walkthrough Using pt-stalk to solve problems with... •...High IO •...High CPU •...Networking
29. 29 Useful links •grep commands - https://gist.github.com/altmannmarcelo/eec40a54568299c2cc11b6cf585 9203d •Check List - http://www.brendangregg.com/USEmethod/use-linux.html
30. Questions ? Marcos Albe marcos.albe@percona.com Marcelo Altmann @altmannmarcelo marcelo.altmann@percona.com