Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Failures In Implementing Reliability

Failures In Implementing Reliability

Stories on trying to do something reliably but failing. Presented at SRE Finland meetup.

Jaakko Pallari

January 16, 2020
Tweet

More Decks by Jaakko Pallari

Other Decks in Programming

Transcript

  1. Welcome to SRE Finland!

    View Slide

  2. Slack channel
    Sign up @ devopsfinland.org
    Join #sre-finland

    View Slide

  3. View Slide

  4. The best DevOps company *
    * according to us

    View Slide

  5. The best DevOps company *
    * according to us
    we also do SRE, Dataops
    and DevSecOps

    View Slide

  6. Jaakko Pallari
    SRE Finland co-organiser
    Lead SRE Consultant @ Polar Squad
    Background in SW dev and DevOps

    View Slide

  7. FAILURES IN
    IMPLEMENTING
    RELIABILITY

    View Slide

  8. Disclaimer
    No SRE involved

    View Slide

  9. View Slide

  10. View Slide

  11. SCALE!

    View Slide

  12. View Slide

  13. View Slide

  14. View Slide

  15. SCALE?

    View Slide

  16. SCALE?
    SCALE!

    View Slide

  17. SCALE!

    View Slide

  18. SCAL
    SCALE!

    View Slide

  19. View Slide

  20. “use the right
    tool for the
    right job lol”

    View Slide

  21. View Slide

  22. Mission:
    Zero downtime
    upgrades

    View Slide

  23. Azure
    Ansible
    Kubernetes
    Go
    PostgreSQL
    Monorepo

    View Slide

  24. View Slide

  25. Azure
    Kubernetes
    Go
    PostgreSQL
    Monorepo

    View Slide

  26. Azure ARM
    Ansible
    Kubernetes
    Go
    PostgreSQL
    Monorepo

    View Slide

  27. View Slide

  28. View Slide

  29. What are we
    even installing
    here?

    View Slide

  30. What is even
    installed in
    production?

    View Slide

  31. View Slide

  32. 1.0.4, 1.0.5 ... 1.6.1, 1.6.2
    in prod latest

    View Slide

  33. PRODUCTION
    PRODUCTION v2

    View Slide

  34. View Slide

  35. THE INFRA

    View Slide

  36. THE INFRA

    View Slide

  37. EMPOWER AND TRUST

    View Slide

  38. THE INFRA

    View Slide

  39. THE INFRA

    View Slide

  40. What was
    learned

    View Slide

  41. It’s OK to trust the
    tools you know

    View Slide

  42. Focus on
    the right issue

    View Slide

  43. EMPOWER AND TRUST
    the teams you work with

    View Slide

  44. thank you

    View Slide