Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Shipping Sentry

Shipping Sentry

How we ship Sentry to production and on premises.

Armin Ronacher

November 15, 2016
Tweet

More Decks by Armin Ronacher

Other Decks in Programming

Transcript

  1. Armin Ronacher
    Shipping Sentry

    View Slide

  2. Armin Ronacher
    @mitsuhiko
    Flask / Sentry / Lektor

    View Slide

  3. Find the Slides at
    lucumr.pocoo.org/talks

    View Slide

  4. reach out to me!
    I want to talk :)

    View Slide

  5. sentry.io

    View Slide

  6. View Slide

  7. View Slide

  8. View Slide

  9. View Slide

  10. View Slide

  11. 「 THE TWO PRODUCTS 」

    View Slide

  12. sentry vs ‘getsentry’

    View Slide

  13. sentry open source repo
    on-premise
    monthly releases

    View Slide

  14. ‘getsentry’ billing & quotas
    depends on sentry
    hourly deploys

    View Slide

  15. 「 GROWING THE TEAM 」

    View Slide

  16. 2.5 to 25 more process
    keep processes light
    keep developers happy

    View Slide

  17. 2 locations process in code
    natural for us because of
    the Open Source nature

    View Slide

  18. 「 THE GOALS 」

    View Slide

  19. deploy in seconds
    be unable to screw up
    and if you do: instant rollbacks

    View Slide

  20. tag a release once a month

    View Slide

  21. 「 WORKFLOW 」

    View Slide

  22. commit
    review
    integration
    deploy

    View Slide

  23. requires good test coverage
    requires good local setup
    makes it easier for newcomers

    View Slide

  24. 「 COMMITTING 」

    View Slide

  25. lint on commit!

    View Slide

  26. 1 Release / Month
    5 Deployments / Day
    On Prem:
    Hosted:

    View Slide

  27. View Slide

  28. master is stable

    View Slide

  29. 1. branch off master
    2. pull request
    3. merge

    View Slide

  30. all the pull requests

    View Slide

  31. !! AVOID DOWNTIME !!

    View Slide

  32. postgres <3
    transactional ddl, concurrent
    indexes, cheap alter table add
    nullable columns

    View Slide

  33. bidirectional compatibility

    View Slide

  34. separation of state and connections

    View Slide

  35. 「 CONTINUOUS TESTING 」

    View Slide

  36. sentry travis-ci.org
    test all the code

    View Slide

  37. ‘getsentry’ travis-ci.com
    test code relevant for us

    View Slide

  38. 「 CONTINUOUS DELIVERY 」

    View Slide

  39. FREIGHT
    wait for travis > build > ship

    View Slide

  40. bidirectional
    communication with
    the main slack channel

    View Slide

  41. dev never matches prod :(

    View Slide

  42. thus: fast rollbacks!
    (backwards + forwards compatibility)

    View Slide

  43. 「 CODE STRUCTURE 」

    View Slide

  44. large systems are organisms

    View Slide

  45. not all things will run the
    same code at the same time

    View Slide

  46. data schema ~ code behavior

    View Slide

  47. break up larger features

    View Slide

  48. feature flag it!
    (we shipped some code to on-prem we backed out)

    View Slide

  49. 「 REPO STRUCTURE 」

    View Slide

  50. move towards “monorepos”
    (but within what is possible with our tools)
    {not as mono as we would like}

    View Slide

  51. 「 MOVING PARTS 」

    View Slide

  52. keep dev basic: fewer parts

    View Slide

  53. do not diverge dev from prod
    too much

    View Slide

  54. virtual machines and docker
    are not an acceptable dev
    environment

    View Slide

  55. 「 REPRODUCIBLE BUILDS 」

    View Slide

  56. pip freeze / yarn

    View Slide

  57. nothing is more frustrating than a failed deploy
    because a dependency of a dependency of a
    dependency of a dependency pushed out a
    broken release

    View Slide

  58. build once > ship to many

    View Slide

  59. 「 BINARY DEPS 」

    View Slide

  60. OS X & “manylinux”

    View Slide

  61. C/C++/Rust Modules for Python

    View Slide

  62. Build in Docker on old CentOS

    View Slide

  63. Debian / RHEL / Ubuntu

    View Slide

  64. 「 MONITOR FAILURES 」

    View Slide

  65. associate failures to users

    View Slide

  66. map support requests to failures

    View Slide

  67. use sentry :-)

    View Slide

  68. 「 FRIENDLY ROBOTS 」

    View Slide

  69. replace yourself!

    View Slide

  70. bots and webhooks

    View Slide

  71. github hooks

    View Slide

  72. notify to communication hub

    View Slide

  73. testing danger.systems

    View Slide

  74. 「 BETTER CLIMATE 」

    View Slide

  75. the more robots,
    the better the integration,
    the smaller the fear of doing damage

    View Slide

  76. If you can launch a feature on your first
    day of work that's motivating

    View Slide

  77. also: happy customers

    View Slide

  78. Q&A
    reach out to me!
    I want to talk :)

    View Slide