Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Shipping Sentry

Shipping Sentry

How we ship Sentry to production and on premises.

Armin Ronacher

November 15, 2016
Tweet

More Decks by Armin Ronacher

Other Decks in Programming

Transcript

  1. Armin Ronacher
    Shipping Sentry

    View full-size slide

  2. Armin Ronacher
    @mitsuhiko
    Flask / Sentry / Lektor

    View full-size slide

  3. Find the Slides at
    lucumr.pocoo.org/talks

    View full-size slide

  4. reach out to me!
    I want to talk :)

    View full-size slide

  5. 「 THE TWO PRODUCTS 」

    View full-size slide

  6. sentry vs ‘getsentry’

    View full-size slide

  7. sentry open source repo
    on-premise
    monthly releases

    View full-size slide

  8. ‘getsentry’ billing & quotas
    depends on sentry
    hourly deploys

    View full-size slide

  9. 「 GROWING THE TEAM 」

    View full-size slide

  10. 2.5 to 25 more process
    keep processes light
    keep developers happy

    View full-size slide

  11. 2 locations process in code
    natural for us because of
    the Open Source nature

    View full-size slide

  12. 「 THE GOALS 」

    View full-size slide

  13. deploy in seconds
    be unable to screw up
    and if you do: instant rollbacks

    View full-size slide

  14. tag a release once a month

    View full-size slide

  15. 「 WORKFLOW 」

    View full-size slide

  16. commit
    review
    integration
    deploy

    View full-size slide

  17. requires good test coverage
    requires good local setup
    makes it easier for newcomers

    View full-size slide

  18. 「 COMMITTING 」

    View full-size slide

  19. lint on commit!

    View full-size slide

  20. 1 Release / Month
    5 Deployments / Day
    On Prem:
    Hosted:

    View full-size slide

  21. master is stable

    View full-size slide

  22. 1. branch off master
    2. pull request
    3. merge

    View full-size slide

  23. all the pull requests

    View full-size slide

  24. !! AVOID DOWNTIME !!

    View full-size slide

  25. postgres <3
    transactional ddl, concurrent
    indexes, cheap alter table add
    nullable columns

    View full-size slide

  26. bidirectional compatibility

    View full-size slide

  27. separation of state and connections

    View full-size slide

  28. 「 CONTINUOUS TESTING 」

    View full-size slide

  29. sentry travis-ci.org
    test all the code

    View full-size slide

  30. ‘getsentry’ travis-ci.com
    test code relevant for us

    View full-size slide

  31. 「 CONTINUOUS DELIVERY 」

    View full-size slide

  32. FREIGHT
    wait for travis > build > ship

    View full-size slide

  33. bidirectional
    communication with
    the main slack channel

    View full-size slide

  34. dev never matches prod :(

    View full-size slide

  35. thus: fast rollbacks!
    (backwards + forwards compatibility)

    View full-size slide

  36. 「 CODE STRUCTURE 」

    View full-size slide

  37. large systems are organisms

    View full-size slide

  38. not all things will run the
    same code at the same time

    View full-size slide

  39. data schema ~ code behavior

    View full-size slide

  40. break up larger features

    View full-size slide

  41. feature flag it!
    (we shipped some code to on-prem we backed out)

    View full-size slide

  42. 「 REPO STRUCTURE 」

    View full-size slide

  43. move towards “monorepos”
    (but within what is possible with our tools)
    {not as mono as we would like}

    View full-size slide

  44. 「 MOVING PARTS 」

    View full-size slide

  45. keep dev basic: fewer parts

    View full-size slide

  46. do not diverge dev from prod
    too much

    View full-size slide

  47. virtual machines and docker
    are not an acceptable dev
    environment

    View full-size slide

  48. 「 REPRODUCIBLE BUILDS 」

    View full-size slide

  49. pip freeze / yarn

    View full-size slide

  50. nothing is more frustrating than a failed deploy
    because a dependency of a dependency of a
    dependency of a dependency pushed out a
    broken release

    View full-size slide

  51. build once > ship to many

    View full-size slide

  52. 「 BINARY DEPS 」

    View full-size slide

  53. OS X & “manylinux”

    View full-size slide

  54. C/C++/Rust Modules for Python

    View full-size slide

  55. Build in Docker on old CentOS

    View full-size slide

  56. Debian / RHEL / Ubuntu

    View full-size slide

  57. 「 MONITOR FAILURES 」

    View full-size slide

  58. associate failures to users

    View full-size slide

  59. map support requests to failures

    View full-size slide

  60. use sentry :-)

    View full-size slide

  61. 「 FRIENDLY ROBOTS 」

    View full-size slide

  62. replace yourself!

    View full-size slide

  63. bots and webhooks

    View full-size slide

  64. github hooks

    View full-size slide

  65. notify to communication hub

    View full-size slide

  66. testing danger.systems

    View full-size slide

  67. 「 BETTER CLIMATE 」

    View full-size slide

  68. the more robots,
    the better the integration,
    the smaller the fear of doing damage

    View full-size slide

  69. If you can launch a feature on your first
    day of work that's motivating

    View full-size slide

  70. also: happy customers

    View full-size slide

  71. Q&A
    reach out to me!
    I want to talk :)

    View full-size slide