$30 off During Our Annual Pro Sale. View Details »

Addicted to Stable

Addicted to Stable

Closing keynote for RubyConf Brazil.

John Nunemaker
PRO

September 01, 2012
Tweet

More Decks by John Nunemaker

Other Decks in Programming

Transcript

  1. Addicted to Stable
    a.k.a how to sleep like an octocat

    View Slide

  2. View Slide

  3. View Slide

  4. View Slide

  5. View Slide

  6. View Slide

  7. View Slide

  8. View Slide

  9. Hundreds of Sites

    View Slide

  10. Hundreds of Sites
    Thousands of Decks

    View Slide

  11. Hundreds of Sites
    Thousands of Decks
    Millions of People

    View Slide

  12. Hundreds of Sites
    Thousands of Decks
    Millions of People
    Billions of Views

    View Slide


  13. People want features,
    but need stability.

    View Slide

  14. Sta·ble
    Not likely to change or fail

    View Slide

  15. App
    Code and Servers

    View Slide

  16. Testing
    Automate

    View Slide

  17. ?

    View Slide

  18. ?
    test/unit

    View Slide

  19. ?
    rspec
    test/unit

    View Slide

  20. ?
    rspec
    test/unit minitest

    View Slide

  21. YES

    View Slide

  22. design YES

    View Slide

  23. design YES change

    View Slide

  24. design YES change
    regression

    View Slide

  25. View Slide

  26. View Slide

  27. Work by me

    View Slide

  28. Work by me
    Pushed code fix

    View Slide

  29. View Slide

  30. Work by robots

    View Slide

  31. Work by robots
    Closed issue

    View Slide

  32. Work by robots
    Closed issue
    Ran test suite

    View Slide

  33. Work by robots
    Closed issue
    Ran test suite
    Deployed code

    View Slide

  34. Backup and Restore
    Automate

    View Slide

  35. A Single Server
    Never trust

    View Slide

  36. View Slide

  37. Replicate

    View Slide

  38. Replicate
    Promote

    View Slide

  39. Replicate
    Promote
    Practice

    View Slide


  40. Me, at 8pm on a Saturday
    I’m confirming the steps
    and order of commands.

    View Slide

  41. View Slide

  42. Deleted
    Everything
    I accidentally
    Sincerely,
    Your Employee

    View Slide

  43. Deleted
    Everything
    I accidentally
    Sincerely,
    Your Customer

    View Slide

  44. users
    users.archived
    gauges
    gauges.archived

    View Slide

  45. hubot restore

    View Slide

  46. Deleted
    Everything
    I accidentally
    Sincerely,
    You

    View Slide

  47. View Slide

  48. Deployment
    Automate

    View Slide

  49. View Slide

  50. hubot deploy github

    View Slide

  51. hubot deploy speakerdeck

    View Slide

  52. hubot deploy gauges

    View Slide

  53. hubot deploy gauges/my-branch

    View Slide

  54. puppet
    https://github.com/puppetlabs/puppet

    View Slide

  55. moonshine
    https://github.com/railsmachine/moonshine

    View Slide

  56. HOSTFILTER=‘app7.foo.com’ \
    cap production deploy:setup

    View Slide

  57. HOSTFILTER=‘app7.foo.com’ \
    cap production deploy

    View Slide

  58. Exceptions
    Collect and Fix

    View Slide

  59. View Slide

  60. View Slide

  61. < 1 exception
    per million page views (gauges)

    View Slide

  62. Metrics
    Collect and Graph

    View Slide

  63. View Slide

  64. View Slide

  65. View Slide

  66. View Slide

  67. View Slide

  68. View Slide

  69. Alerts
    Email, Push and SMS

    View Slide

  70. View Slide

  71. View Slide

  72. View Slide

  73. Nagios

    View Slide

  74. Pingdom
    Nagios

    View Slide

  75. Pager Duty
    Pingdom
    Nagios

    View Slide

  76. Failover
    Automate

    View Slide

  77. web1 web2
    haproxy/heartbeat

    View Slide

  78. Tracking
    app3 app4 app5 app6
    web1 web2

    View Slide

  79. Tracking
    app3 app4 app5 app6
    web1 web2
    - option httpchk GET /pulse/tracking

    View Slide

  80. Tracking
    app3 app4 app5 app6
    web1 web2
    class PulseApp < Sinatra::Base
    get '/pulse/tracking' do
    pulse :kestrel
    end
    end

    View Slide

  81. Tracking
    app3 app4 app5 app6
    web1 web2
    app1 app2
    Reporting

    View Slide

  82. Tracking
    app3 app4 app5 app6
    web1 web2
    app1 app2
    Reporting
    - option httpchk GET /pulse/reporting

    View Slide

  83. Tracking
    app3 app4 app5 app6
    web1 web2
    app1 app2
    Reporting
    class PulseApp < Sinatra::Base
    get '/pulse/reporting' do
    pulse :mongo
    end
    end

    View Slide

  84. Tracking
    app3 app4 app5 app6
    web1 web2
    q1 q2
    app1 app2
    Reporting

    View Slide

  85. Tracking
    app3 app4 app5 app6
    web1 web2
    q1 q2
    Workers
    db1
    db2 db3
    app1 app2
    Reporting

    View Slide

  86. Team
    Collaboration and Freedom

    View Slide

  87. 46 hubbers
    When I started in December

    View Slide

  88. 115 hubbers
    As of August 31, 2012 at 10:43am EST

    View Slide

  89. How is that stable???

    View Slide

  90. Collaborate
    Always

    View Slide

  91. Pull Requests
    Discussion + ❤

    View Slide

  92. View Slide

  93. View Slide

  94. Chat
    Campfire to your phone

    View Slide

  95. View Slide

  96. Video
    Face to face and Pairing

    View Slide

  97. View Slide

  98. Internal Tools
    Ideas, Hiring, Music and more!

    View Slide

  99. View Slide

  100. View Slide

  101. 300+ ideas

    View Slide

  102. 300+ ideas
    3000+ comments

    View Slide

  103. 300+ ideas
    3000+ comments
    100+ people

    View Slide

  104. Communicate
    Always

    View Slide

  105. View Slide

  106. View Slide

  107. 4300+ statuses

    View Slide

  108. 4300+ statuses
    4200+ comments

    View Slide

  109. 4300+ statuses
    4200+ comments
    100+ people

    View Slide

  110. View Slide

  111. View Slide

  112. 170 talks

    View Slide

  113. 170 talks
    112 people

    View Slide

  114. 170 talks
    112 people
    2 summits/year

    View Slide

  115. View Slide

  116. Document
    Always

    View Slide

  117. View Slide

  118. View Slide

  119. Empower
    Always

    View Slide

  120. View Slide

  121. Anyone can view

    View Slide

  122. Anyone can view
    Anyone can commit

    View Slide

  123. Anyone can view
    Anyone can commit
    Anyone can deploy

    View Slide

  124. View Slide

  125. 296 branches

    View Slide

  126. 296 branches
    2980 commits

    View Slide

  127. 296 branches
    2980 commits
    307 issues

    View Slide

  128. IN ONE MONTH

    View Slide

  129. ONLY THE WEB APP

    View Slide

  130. Brand
    Surprise and Delight

    View Slide

  131. Feedback
    Stupid Easy

    View Slide

  132. View Slide

  133. View Slide

  134. View Slide

  135. Support
    Fast and Friendly

    View Slide

  136. View Slide

  137. View Slide

  138. View Slide

  139. View Slide

  140. Documentation
    Useful and Thorough

    View Slide

  141. View Slide

  142. View Slide

  143. View Slide

  144. View Slide

  145. Communication
    Honest and Open

    View Slide

  146. View Slide

  147. View Slide

  148. View Slide

  149. If you present,
    GitHub covers the bill for you.

    View Slide

  150. If you present,
    GitHub covers the bill for you and a buddy.

    View Slide

  151. View Slide


  152. On Saturday, June 2nd, we had a
    perfect storm of cascading failures
    resulting in around four hours of
    lost tracking data.

    View Slide


  153. We have commissioned a new
    database cluster with automatic
    failover that we hope to be using
    sometime next week.

    View Slide


  154. We hate that we lost your data.
    Know that we have learned from
    this and will rise from the ashes of
    your data loss like a GLORIOUS
    PHOENIX.

    View Slide

  155. Celebration
    Public and Genuine

    View Slide

  156. View Slide

  157. View Slide

  158. View Slide

  159. View Slide

  160. automate testing, backing
    up, restoring, deploying,
    collect exceptions, metrics,
    send alerts, fail gracefully,
    collaborate, communicate,
    document all the things,
    empower, get feedback,
    provide awesome support,
    communicate, celebrate

    View Slide

  161. The End.

    View Slide