DIY Performance Tuning When All You Have is Python

We wanted to know where our backend was spending its time and where we could make it faster. We knew it would take months to get an open source tool like Zipkin to work in our system, so we instead used our understanding of networking and Python to find a solution that met most of our needs with a fraction of the development time. In this talk, I'll show how we can learn to feel empowered to tackle big problems with a little bit of Python on our side.

Transcript

1. DIY Performance Tuning When All You Have is Python Amy Nguyen @amyngyn amynguyen.net March 22, 2017
2. Hi! I'm Amy. • Software engineer at Pinterest since 2015. • What I've been doing on Visibility Team: distributed tracing, writing a cache, a whole "D3 is everything" phase, a UX design overhaul of our internal tools • Find me on Twitter at @amyngyn!
3. Distributed Tracing
4. Does my function call this service?
5. Does my function call this service? How much time is my function spending on calling each service?
6. Does my function call this service? How much time is my function spending on calling each service? What functions do each of those backend services call?
7. Why does my function take so long? Does my function call this service? How much time is my function spending on calling each service? What functions do each of those backend services call?
8. Why does my function take so long? Does my function call this service? How much time is my function spending on calling each service? What functions do each of those backend services call? ? ? ? ? ?
9. What if we could record (or trace) data about every network request going through our system and analyze the data?
17. my_function urllib ??? memcache ??? mysql ???
18. my_function urllib ??? socket memcache ??? mysql ???
19. Monkey patching
20. my_function urllib ??? socket memcache ??? mysql ??? ! ! !
21. my_function urllib ??? my_socket memcache ??? mysql ??? ! ! !
22. What can I do now that I went through all of this work?
28. 20% decrease in latency!!!!
30. Thanks!