Profiling in Production (Alexander Else 2019)

Transcript

1. Alexander Else | Principal Engineer, Observability | Atlassian Profiling a Go Service in Production
2. This is a really challenging topic… because it’s already been covered so well.
3. Using pprof to diagnose a production issue *Identify and fix a specific performance problem we had *What I learned about go from this experience Setting yourself up for success with pprof *Why you might want to profile your production services *How you should prepare to do it
4. I’d like to tell you a story.
8. Great, where do we start?
9. How can we dance when our [CPU fan] is turning? How can we sleep when our [CPU cores] are burning? — Midnight Oil
11. Live Demo What could possibly go wrong?
13. Profiling in Production
14. Everyone Tests in Production
15. What’s different? * OS, hardware, software * Configuration * Dependencies * Workload * Users
16. You’ve decided to Profile in Production 1. Make it easy 2. Make it safe
17. Don’t use http.DefaultServeMux • Magically exposes things you don’t know about • https://mmcloughlin.com/posts/your-pprof-is-showing • Anything can register a route! Instead • Start with a clean mux and server mux !:= http.NewServeMux() srv !:= &http.Server{Addr: “8080”, Handler: mux} • Import net/http/pprof and expose what you need, or use middleware to do the lifting • eg. https://github.com/go-chi/chi/blob/master/middleware/profiler.go
18. Make it private • Reveals a lot about your code and internal state • Increased surface area Instead • Restrict access to it • Put your pprof endpoints behind auth • Run on a non-public IP and port srv !:= &http.Server{Addr: “127.0.0.1:9090”, Handler: mux} • Respond to a signal $ kill -USR1 $PID
19. Performance impact • There’s non-zero performance impact to your running process • Negligible in my cases - what about yours? Instead • Look at the perf impact under controlled conditions • Gain confidence that you can use it when you need it
20. Convenience • People don’t like cumbersome tools and processes • More than 1 command is too much Instead • curl (with authentication!) • SSM • chatops
21. Muscle memory • Tools are only useful if you can use them when you need to • You don’t want to read StackOverflow at 3am Instead • Regularly use your tools. It’s a dress rehearsal • Use your alerts and incident runbooks to tell you what to do • Do it the same way across all your systems
22. Alexander Else | Principal Engineer, Observability | Atlassian Profiling a Go Service in Production