Debugging (Varnish)

An overview of some of the research done in the practice and pedagogy of debugging, as well as some examples of ways that Fastly has addressed issues with debugging Varnish "at scale". A video of this talk is available at https://www.infoq.com/fr/presentations/varnishcon-devon-odell-debugging

Transcript

1. Debugging (Varnish)
   Devon H. O’Dell | @dhobsd | Fastly | VarnishCon 2016
   

   View Slide

2. Who am I?
   Tech Lead, Varnish PFID @ Fastly
   Personal interest in performance,
   debugging, concurrency, and
   introspection.
   Usually confused about something.
   

   View Slide

3. View Slide

4. Why does debugging matter?
   

   View Slide

5. The Root of the Problem
   

   View Slide

6. “Everyone knows that debugging is
   twice as hard as writing a program in
   the first place. So if you're as clever as
   you can be when you write it, how will
   you ever debug it?” (Brian Kernighan,
   TEOPS, 1974)
   

   View Slide

7. View Slide

8. View Slide

9. View Slide

10. View Slide

11. View Slide

12. View Slide

13. View Slide

14. View Slide

15. View Slide

16. “The relationships between debugging success
    and individual factors … have not been
    researched… Given that debugging presents
    challenges for many novices, exploring the
    influence of self theories on student approaches
    to debugging appears worthwhile.”
    

    View Slide

17. View Slide

18. View Slide

19. View Slide

20. View Slide

21. View Slide

22. sub vcl_recv {
    if (req.http.answer ~ “^42) {
    log “enlightened client”;
    }
    return vcl_fetch;
    }
    

    View Slide

23. View Slide

24. View Slide

25. View Slide

26. View Slide

27. View Slide

28. View Slide

29. View Slide

30. View Slide

31. View Slide

32. Varnish Debugging Tools
    • varnishlog / varnishtop / varnishncsa / varnishreplay / varnishadm
    • vcl tracing
    • crash traces
    • gdb / lldb
    • perf / hwpmc
    

    View Slide

33. View Slide

34. View Slide

35. View Slide

36. View Slide

37. locking_function : 25276 threads, 2 locations: cache_foo.c:220 (8);
    cache_foo.c:210 (25268)
    

    View Slide

38. View Slide

39. View Slide

40. View Slide

41. Thank you!
    Devon H. O’Dell | https://9vx.org/ | @dhobsd | https://www.fastly.com
    

    View Slide