DIY Performance Tuning When All You Have is Python

We wanted to know where our backend was spending its time and where we could make it faster. We knew it would take months to get an open source tool like Zipkin to work in our system, so we instead used our understanding of networking and Python to find a solution that met most of our needs with a fraction of the development time. In this talk, I'll show how we can learn to feel empowered to tackle big problems with a little bit of Python on our side.

Transcript

1. DIY Performance Tuning
   When All You Have is Python
   Amy Nguyen
   @amyngyn
   amynguyen.net
   March 22, 2017
   

   View Slide

2. Hi! I'm Amy.
   ● Software engineer at Pinterest since 2015.
   ● What I've been doing on Visibility Team:
   distributed tracing, writing a cache,
   a whole "D3 is everything" phase,
   a UX design overhaul of our internal tools
   ● Find me on Twitter at @amyngyn!
   

   View Slide

3. Distributed Tracing
   

   View Slide

4. Does my function
   call this service?
   

   View Slide

5. Does my function
   call this service?
   How much time is my function
   spending on calling each service?
   

   View Slide

6. Does my function
   call this service?
   How much time is my function
   spending on calling each service?
   What functions do each of those
   backend services call?
   

   View Slide

7. Why does my
   function take
   so long?
   Does my function
   call this service?
   How much time is my function
   spending on calling each service?
   What functions do each of those
   backend services call?
   

   View Slide

8. Why does my
   function take
   so long?
   Does my function
   call this service?
   How much time is my function
   spending on calling each service?
   What functions do each of those
   backend services call?
   ?
   ?
   ?
   ?
   ?
   

   View Slide

9. What if we could record (or trace)
   data about every network request
   going through our system
   and analyze the data?
   

   View Slide

10. View Slide

11. View Slide

12. View Slide

13. View Slide

14. View Slide

15. View Slide

16. View Slide

17. my_function urllib ???
    memcache ???
    mysql ???
    

    View Slide

18. my_function urllib ???
    socket
    memcache ???
    mysql ???
    

    View Slide

19. Monkey patching
    

    View Slide

20. my_function urllib ???
    socket
    memcache ???
    mysql ???
    !
    !
    !
    

    View Slide

21. my_function urllib ???
    my_socket
    memcache ???
    mysql ???
    !
    !
    !
    

    View Slide

22. What can I do now that I went
    through all of this work?
    

    View Slide

23. View Slide

24. View Slide

25. View Slide

26. View Slide

27. View Slide

28. 20% decrease in latency!!!!
    

    View Slide

29. View Slide

30. Thanks!
    

    View Slide