Don't reboot, debug! - DPC15

Transcript

1. 1 Don't reboot, debug! A medic first aid course in debugging your server Joshua Thijssen @JayTaph
2. 2 Joshua Thijssen Consultant and trainer Author of the Symfony Rainbow Series http://www.symfony-rainbow.com Blog: http://adayinthelifeof.nl Email: jthijssen@noxlogic.nl Twitter: @jaytaph Tech nalyze WWW.TECHANALYZE.IO
4. 3 It's not
5. 3 It is It's not
7. 4 Our website is not working anymore!!!
8. Have you tried turning it off and on again? 5
9. 6 Manager Not a suit
10. 6 Fix it! Every minute we’re losing money! Manager Not a suit
12. 8 Deal now or deal later?
14. 9 ➡ Is it reproducible later? Probably not.
15. 9 ➡ Is it reproducible later? Probably not. ➡ Are you solving the problem, or desperately trying to remove a symptom?
16. 9 ➡ Is it reproducible later? Probably not. ➡ Are you solving the problem, or desperately trying to remove a symptom? ➡ Short term relieve vs long term solution
18. ➡ Actually analyze, maybe fix the problem. 10
19. ➡ Actually analyze, maybe fix the problem. ➡ It will cost less to analyze/fix it now, than to fix it later. 10
20. ➡ Actually analyze, maybe fix the problem. ➡ It will cost less to analyze/fix it now, than to fix it later. 10
24. 12 ➡ We reboot our system every night!
25. 12 ➡ We reboot our system every night! ➡ Why? Memory leaks? Just crappy code?
26. 12 ➡ We reboot our system every night! ➡ Why? Memory leaks? Just crappy code? ➡ There is some state not handled correctly!
27. 12 ➡ We reboot our system every night! ➡ Why? Memory leaks? Just crappy code? ➡ There is some state not handled correctly! ➡ What happens when # users increase with 200%? Restart every 12 hours?
28. 12 ➡ We reboot our system every night! ➡ Why? Memory leaks? Just crappy code? ➡ There is some state not handled correctly! ➡ What happens when # users increase with 200%? Restart every 12 hours? ➡ Let’s hope you won’t get many visitors!
29. Find the culprit 13
30. Bottleneck Troubleshooting Flowchart (BTF) 14
31. Site is slow or not responding. It’s your DB Bottleneck Troubleshooting Flowchart (BTF) 14
33. ➡ Apache / PHP 15
34. ➡ Apache / PHP ➡ Monitoring / backup 15
35. ➡ Apache / PHP ➡ Monitoring / backup ➡ Hanging cron jobs & runaway tools 15
36. ➡ Apache / PHP ➡ Monitoring / backup ➡ Hanging cron jobs & runaway tools ➡ Connectivity / DNS problems 15
37. 16 Linux 101
38. 17 Processes
40. 18 ➡ Isolated userspace.
41. 18 ➡ Isolated userspace. ➡ PID (process id) and state.
42. 18 ➡ Isolated userspace. ➡ PID (process id) and state. ➡ Kernel “preempts”, or process yields.
43. 18 ➡ Isolated userspace. ➡ PID (process id) and state. ➡ Kernel “preempts”, or process yields. ➡ Multitasking.
46. 20 ➡ R Running or runnable
47. 20 ➡ R Running or runnable ➡ S Interruptible sleep
48. 20 ➡ R Running or runnable ➡ S Interruptible sleep ➡ D Uninterruptible sleep
49. 20 ➡ R Running or runnable ➡ S Interruptible sleep ➡ D Uninterruptible sleep ➡ T Stopped
50. 20 ➡ R Running or runnable ➡ S Interruptible sleep ➡ D Uninterruptible sleep ➡ T Stopped ➡ Z Defunct process (zombies)
52. 21 ➡ Most processes are sleeping.
53. 21 ➡ Most processes are sleeping. ➡ External processes (and the kernel) can “wake up” a process at any time by sending “signals”.
54. 21 ➡ Most processes are sleeping. ➡ External processes (and the kernel) can “wake up” a process at any time by sending “signals”. ➡ Fire signals with “kill”.
56. 22 ➡ Uninterruptible means it won’t handle signals (directly), but waits on its task to finish (it must wake up by itself).
57. 22 ➡ Uninterruptible means it won’t handle signals (directly), but waits on its task to finish (it must wake up by itself). ➡ Used for high-performance loops that needs to focus (like I/O).
58. 22 ➡ Uninterruptible means it won’t handle signals (directly), but waits on its task to finish (it must wake up by itself). ➡ Used for high-performance loops that needs to focus (like I/O). ➡ Still can be preempted by the scheduler!
60. 23 ➡ Zombies aren’t bad.
61. 23 ➡ Zombies aren’t bad. ➡ It’s just bad programming or administration that creates zombies.
62. 23 ➡ Zombies aren’t bad. ➡ It’s just bad programming or administration that creates zombies. ➡ They will not eat brains (at least not much).
63. 23 ➡ Zombies aren’t bad. ➡ It’s just bad programming or administration that creates zombies. ➡ They will not eat brains (at least not much). ➡ But there shouldn’t be many.
64. 24 Load average
66. 25 ➡ 1 minute, 5 minutes, 15 minutes averages
67. 25 ➡ 1 minute, 5 minutes, 15 minutes averages ➡ Calculated as the number of runnable processes.
68. 25 ➡ 1 minute, 5 minutes, 15 minutes averages ➡ Calculated as the number of runnable processes. ➡ Depends on number of CPU’s!
69. 26 14:57:22 up 35 days, 18:57, 1 user, load average: 1.52, 0.66, 0.27
70. 26 14:57:22 up 35 days, 18:57, 1 user, load average: 1.52, 0.66, 0.27 ➡ 1.52 average runnable (or blocking) processes in the last minute.
71. 26 14:57:22 up 35 days, 18:57, 1 user, load average: 1.52, 0.66, 0.27 ➡ 1.52 average runnable (or blocking) processes in the last minute. ➡ 0.66 average in 5 minutes
72. 26 14:57:22 up 35 days, 18:57, 1 user, load average: 1.52, 0.66, 0.27 ➡ 1.52 average runnable (or blocking) processes in the last minute. ➡ 0.66 average in 5 minutes ➡ 0.27 average in 15 minutes.
73. 26 14:57:22 up 35 days, 18:57, 1 user, load average: 1.52, 0.66, 0.27 ➡ 1.52 average runnable (or blocking) processes in the last minute. ➡ 0.66 average in 5 minutes ➡ 0.27 average in 15 minutes. ➡ Single CPU: 52% more than it can handle.
74. 26 14:57:22 up 35 days, 18:57, 1 user, load average: 1.52, 0.66, 0.27 ➡ 1.52 average runnable (or blocking) processes in the last minute. ➡ 0.66 average in 5 minutes ➡ 0.27 average in 15 minutes. ➡ Single CPU: 52% more than it can handle. ➡ quad core system: not doing very much
75. 27 Memory
76. 28 Q: How much memory does this process use? This is REALLY hard question to answer! It depends on many factors!
77. 29 ➡ Virtual memory ➡ Shared memory ➡ Resident memory ➡ Swapped memory
79. 30 ➡ 4GB memory space, even if you have less memory installed.
80. 30 ➡ 4GB memory space, even if you have less memory installed. ➡ 1GB is reserved for kernel.
81. 30 ➡ 4GB memory space, even if you have less memory installed. ➡ 1GB is reserved for kernel. ➡ Kernel can swap out memory.
82. 30 ➡ 4GB memory space, even if you have less memory installed. ➡ 1GB is reserved for kernel. ➡ Kernel can swap out memory. ➡ CPU pagefaults and loads back pages.
84. 31 ➡ Process can allocate memory, but does not necessary use it (for instance: preallocation)
85. 31 ➡ Process can allocate memory, but does not necessary use it (for instance: preallocation) ➡ VIRT will increase!
87. 33 Q: How much free memory does this system have? This is an easier, but still hard question to answer!
88. 34 $ free -m total used free shared buffers cached Mem: 375 349 25 0 111 94 -/+ buffers/cache: 143 231 Swap: 400 7 392
89. 35 Monitoring
90. 36 ➡ Monitor everything ➡ System / infra monitoring ➡ Application monitoring
93. 39 Logging Logging
94. 40 ➡ Log EVERYTHING from all sources. ➡ Filter later.
95. 41 $ php composer.phar require monolog/monolog ➡ syslog ➡ files ➡ mail ➡ slack / hipchat / irc ➡ logstash
96. 42 System tools
97. TAIL 43
99. 44 ➡ Most daemons will log into /var/log/*
100. 44 ➡ Most daemons will log into /var/log/* ➡ tail -f /var/log/messages
101. 44 ➡ Most daemons will log into /var/log/* ➡ tail -f /var/log/messages ➡ Many times, this is ALL you need!
103. 45 ➡ Know your tools (top, htop, vmstat, iostat, ps)
104. 45 ➡ Know your tools (top, htop, vmstat, iostat, ps) ➡ Know the /proc filesystem
105. 45 ➡ Know your tools (top, htop, vmstat, iostat, ps) ➡ Know the /proc filesystem ➡ sniff around with tcpdump, netstat, nc etc...
106. 45 ➡ Know your tools (top, htop, vmstat, iostat, ps) ➡ Know the /proc filesystem ➡ sniff around with tcpdump, netstat, nc etc... ➡ man <keyword>
107. 46 strace
108. 47 ➡ strace displays system calls and signals ➡ Communication between applications and the kernel.
109. 48 $ strace -ff -p <pid> .... socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 20 fcntl(20, F_GETFL) = 0x2 (flags O_RDWR) fcntl(20, F_SETFL, O_RDWR|O_NONBLOCK) = 0 connect(20, {sa_family=AF_INET, sin_port=htons(11211), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress) poll([{fd=20, events=POLLOUT}], 1, -1) = 1 ([{fd=20, revents=POLLOUT}]) write(20, "get ez_client1/acls/"..., 44) = 44 read(20, "END\r\n", 8196) = 5 write(20, "get ez_client1/acl/g"..., 40) = 40 read(20, "END\r\n", 8196) = 5 write(20, "quit\r\n", 6) = 6 shutdown(20, 2 /* send and receive */) = 0 close(20) = 0 mkdir("/tmp/smarty", 0777) = -1 EEXIST (File exists) chmod("/tmp/smarty", 0777) = 0 mkdir("/tmp/smarty", 0777) = -1 EEXIST (File exists) chmod("/tmp/smarty", 0777) = 0 access("/userdata/client1/user/templates/nl/block-right-last.tpl", F_OK) = -1 ENOENT (No such file or directory) access("/userdata/client1/user/templates/block-right-last.tpl", F_OK) = -1 ENOENT (No such file or directory) access("/userdata/client1/theme/templates/nl/block-right-last.tpl", F_OK) = -1 ENOENT (No such file or directory) access("/userdata/client1/theme/templates/block-right-last.tpl", F_OK) = -1 ENOENT (No such file or directory) access("/etc/noxlogic/root/themes/ezshopping/templates/nl/block-right-last.tpl", F_OK) = -1 ENOENT (No such file or directory) access("/etc/noxlogic/root/themes/ezshopping/templates/block-right-last.tpl", F_OK) = -1 ENOENT (No such file or directory) mkdir("/tmp/smarty", 0777) = -1 EEXIST (File exists) chmod("/tmp/smarty", 0777) = 0 access("/userdata/client1/user/templates/nl/block-right.tpl", F_OK) = -1 ENOENT (No such file or directory) access("/userdata/client1/user/templates/block-right.tpl", F_OK) = -1 ENOENT (No such file or directory) access("/userdata/client1/theme/templates/nl/block-right.tpl", F_OK) = -1 ENOENT (No such file or directory) access("/userdata/client1/theme/templates/block-right.tpl", F_OK) = -1 ENOENT (No such file or directory) access("/etc/noxlogic/root/themes/ezshopping/templates/nl/block-right.tpl", F_OK) = -1 ENOENT (No such file or directory) access("/etc/noxlogic/root/themes/ezshopping/templates/block-right.tpl", F_OK) = -1 ENOENT (No such file or directory) mkdir("/tmp/smarty", 0777) = -1 EEXIST (File exists)
110. 49 $ strace ping www.google.com .... mprotect(0xb757f000, 4096, PROT_READ) = 0 munmap(0xb76d8000, 44104) = 0 stat64("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=59, ...}) = 0 socket(PF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 3 connect(3, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("192.168.178.4")}, 16) = 0 gettimeofday({1347446161, 382120}, NULL) = 0 poll([{fd=3, events=POLLOUT}], 1, 0) = 1 ([{fd=3, revents=POLLOUT}]) send(3, "u\205\1\0\0\1\0\0\0\0\0\0\3www\6google\3com\0\0\1\0\1", 32, MSG_NOSIGNAL) = 32 poll([{fd=3, events=POLLIN}], 1, 5000
111. ➡ strace -e trace=open ➡ strace -ff -p <pid> 50
112. 51 vmstat
113. 52 $ vmstat 1 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 896 17208 124784 121964 0 0 0 0 73 159 0 1 99 0 2 0 896 14968 125628 121976 0 0 844 0 6857 29935 5 58 35 2 2 0 896 14472 125628 121964 0 0 0 0 10691 48526 10 90 0 0 2 0 896 13976 126120 121952 0 0 476 252 10430 49144 7 91 0 2 0 0 896 12744 127016 121968 0 0 896 0 7799 38732 3 71 23 3 0 0 896 12744 127016 121964 0 0 0 0 30 93 0 0 100 0 0 0 896 12760 127016 121964 0 0 0 0 30 92 0 0 100 0 0 0 896 12760 127016 121964 0 0 0 0 29 99 0 1 99 0 0 0 896 12760 127016 121964 0 0 0 0 32 110 0 0 100 0 0 0 896 12752 127024 121964 0 0 0 324 33 108 0 0 99 1 0 0 896 12752 127024 121964 0 0 0 0 30 103 0 0 100 0 0 0 896 12752 127032 121964 0 0 0 12 34 108 0 0 100 0 0 0 896 12752 127032 121964 0 0 0 0 30 105 0 0 100 0 0 0 896 12752 127032 121964 0 0 0 236 80 101 0 1 99 0
114. 53 System tap / dtrace System TAP (dtrace)
115. 54 ➡ Unobtrusive probes inside the kernel ➡ Scripts written in D language. ➡ SUN / Solaris only (licensing)
116. 55 ➡ “GPL” version of dtrace ➡ Awesome, but complex ➡ But you need / want debug info packages
117. probe syscall.open { printf(“%s(%d) open (%s)\n”, execname(), pid(), argstr); } 56 stap syscall.stp
118. 57 ➡ There are some “providers” in the PHP core (zend_dtrace.{c,h,d})
119. 58 ➡ Valgrind ➡ GDB ➡ XDebug / profiler ➡ MySQL proxy
120. 59 Think about your app / infra BEFORE going live...
121. Make a plan 60
123. 61 ➡ Design for (vertical) scalability.
124. 61 ➡ Design for (vertical) scalability. ➡ Remove SPOFs.
125. 61 ➡ Design for (vertical) scalability. ➡ Remove SPOFs. ➡ Horizontal scalability is easier, but more restrictive.
126. 61 ➡ Design for (vertical) scalability. ➡ Remove SPOFs. ➡ Horizontal scalability is easier, but more restrictive. ➡ Configuration is key.
127. 61 ➡ Design for (vertical) scalability. ➡ Remove SPOFs. ➡ Horizontal scalability is easier, but more restrictive. ➡ Configuration is key. ➡ Don’t run on full capacity. Have a contingency buffer for peaks and know how to scale out
129. 62 ➡ One machine for one purpose (app / mail / cron / db / etc).
130. 62 ➡ One machine for one purpose (app / mail / cron / db / etc). ➡ Virtual machines are easy to setup and maintain (puppet / ansible) and are cheap.
131. 62 ➡ One machine for one purpose (app / mail / cron / db / etc). ➡ Virtual machines are easy to setup and maintain (puppet / ansible) and are cheap. ➡ Try to async as much as possible.
132. 62 ➡ One machine for one purpose (app / mail / cron / db / etc). ➡ Virtual machines are easy to setup and maintain (puppet / ansible) and are cheap. ➡ Try to async as much as possible. ➡ Message queues are easy to implement (gearman / *MQ etc).
133. 63 Recap Conclusion
136. 65 ➡ Don’t reboot, debug!
137. 65 ➡ Don’t reboot, debug! ➡ Analyze what’s going on,
138. 65 ➡ Don’t reboot, debug! ➡ Analyze what’s going on, ➡ find and isolate the culprit.
139. 65 ➡ Don’t reboot, debug! ➡ Analyze what’s going on, ➡ find and isolate the culprit. ➡ Threat the problem, not the symptoms.
141. 66 ➡ There are many tools out there to analyze your system realtime.
142. 66 ➡ There are many tools out there to analyze your system realtime. ➡ Know your running environment (even it’s “not your business”).
143. 66 ➡ There are many tools out there to analyze your system realtime. ➡ Know your running environment (even it’s “not your business”). ➡ Ask 3rd party help if needed.
145. Find me on twitter: @jaytaph Find me for development and training: www.noxlogic.nl Find me on email: jthijssen@noxlogic.nl Find me for blogs: www.adayinthelifeof.nl Thank You! https://joind.in/14205