Debugging & Profiling of AWS Lambda: ServerlessConf - IOpipe

Exploring the debugging and profiling tools for AWS Lambda. We use the v8-profiler for NodeJS with IOpipe to see deep into a serverless application and trace function calls. https://www.iopipe.com

Transcript

1. ServerlessConf NYC’17 Profiling & debugging AWS Lambda Erica Windisch CTO, Founder IOpipe, Inc.
2. Erica Windisch CTO & Founder IOpipe, Inc. Former maintainer of Docker & OpenStack Building cloud, ops, and infrastructure tooling for over 15 years.
3. (Thanks & credit to Katie Bayes!)
4. Traditional Instrumentation
5. Kernel Hardware & Virtualization Application Detail Accuracy ❖ Step-debugging ❖ CPU Profiling ❖ Heap snapshots &
 allocation profiles
6. Kernel Hardware & Virtualization Application Detail Accuracy ❖ Step-debugging ❖ CPU Profiling ❖ Heap snapshots &
 allocation profiles
7. Virtualization - Valgrind https://www.cprogramming.com/debugging/valgrind.html
8. Kernel Hardware & Virtualization Application Detail Accuracy ❖ Step-debugging ❖ CPU Profiling ❖ Heap snapshots &
 allocation profiles
9. Kernel-level tooling
10. Linux perf tools https://github.com/brendangregg/perf-tools
11. — Brendan Gregg
12. Almost everything you need to know about kernel-level performance tracing & beyond http://www.brendangregg.com/
13. Kernel Hardware & Virtualization Application Detail Accuracy ❖ Step-debugging ❖ CPU Profiling ❖ Heap snapshots &
 allocation profiles
14. ptrace (i.e. gdb, strace, et al) https://sysdig.com/blog/sysdig-vs-dtrace-vs-strace-a-technical-discussion/
15. There’s an excessive
 number of tools (and they don’t work on Lambda)
16. Instrumenting Lambda
17. Things that don’t work: Linux eBPF! $ iosnoop Tracing block I/O. Ctrl-C to end. /usr/bin/iosnoop: line 172: cd: /sys/kernel/debug/tracing: Permission denied ERROR: accessing tracing. Root user? Kernel has FTRACE? debugfs mounted? (mount -t debugfs debugfs /sys/kernel/debug)
18. Things that don’t work: Linux eBPF! λ$ ls -l /sys ⨯ Error: function response: Command failed: ls -l /sys ls: cannot access /sys: No such file or directory
19. Things that don’t work: Linux perf
20. Things that don’t work: Linux perf λ$ cat /proc/sys/kernel/perf_event_paranoid 2
21. Kernel-level instrumentation
 is out of reach
22. HW+Virt instrumentation
 is (mostly) out of reach.
23. Application instrumentation
 is hard(er) on Lambda
24. Lambda Limitations ✤ Event-loop is out of our direct control ✤ Process command-line flags inaccessible ✤ Ptrace disabled! ✤ Linux kernel perf features not available on Lambda. ✤ Local dev+test is an evolving story ✤ Many languages, requiring different implementations ✤ Debugger breakpoints in prod aren’t generally viable
25. Local Debugging
 & Instrumentation
27. (Thanks Paul Maddox, Sanath Kumar Ramesh, & AWS!)
28. Debugging Lambda
 in the cloud github.com/trek10inc/
 aws-lambda-debugger
29. Profiling Lambda
 in the cloud (To be released soon!)
30. https://github.com/node-inspector/v8-profiler
33. (Thanks Pam Selle!)
38. iopipe.com