Refactoring with Science

Using data and Science to refactor with greater confidence.

Transcript

1. Refactoring GitHub
   with Science
   !
   #BigRubyConf
   

   View Slide

2. !
   Hello, my name is Wynn.
   

   View Slide

3. !
   Folks call me @pengwynn.
   

   View Slide

4. !
   http://wynn.fm
   

   View Slide

5. Changing code is EASY!
   !
   

   View Slide

6. Changing code with CONFIDENCE isn't.
   !
   

   View Slide

7. How do we gain CONFIDENCE?
   !
   

   View Slide

8. Write lots of TESTS.
   !
   ❯ testrb test/integration/api/repos_test.rb
   Loading suite in dotcom mode ...
   Run options:
   # Running tests:
   Finished tests in 30.638101s, 4.7979 tests/s, 64.7886 assertions/s.
   147 tests, 1985 assertions, 0 failures, 0 errors, 0 skips
   ruby -v: ruby 2.1.0p0-github (development) [x86_64-darwin13.0]
   

   View Slide

9. Use data to inform decisions.
   !
   

   View Slide

10. !
    /graph me -1hr @api.queries.serialized
    >
    

    View Slide

11. The Graph Store !
    !
    

    View Slide

12. The Graph Store !
    !
    There's a graph for that™
    

    View Slide

13. The Graph Store !
    !
    There's a graph for that™
    …if not, make one.
    

    View Slide

14. !
    Built on Graphite.
    

    View Slide

15. !
    

    View Slide

16. !
    

    View Slide

17. !
    # counter
    GitHub.stats.increment("the-thing.the-action")
    !
    # gauge
    GitHub.stats.gauge("the-thing.the-gauge", thing.current_reading)
    !
    # timing
    GitHub.stats.time("the-thing.the-operation") { the_operation }
    

    View Slide

18. !
    # counter
    GitHub.stats.increment("the-thing.the-action")
    !
    # gauge
    GitHub.stats.gauge("the-thing.the-gauge", thing.current_reading)
    !
    # timing
    GitHub.stats.time("the-thing.the-operation") { the_operation }
    

    View Slide

19. !
    # counter
    GitHub.stats.increment("the-thing.the-action")
    !
    # gauge
    GitHub.stats.gauge("the-thing.the-gauge", thing.current_reading)
    !
    # timing
    GitHub.stats.time("the-thing.the-operation") { the_operation }
    

    View Slide

20. Deploy with confidence. "
    !
    

    View Slide

21. How# has "'d over the years.
    !
    

    View Slide

22. 1989
    10 PRINT "WYNN"
    20 GOTO 10
    

    View Slide

23. View Slide

24. 1999
    !
    

    View Slide

25. 2004
    !
    

    View Slide

26. 2009
    !
    # General configuration settings, required for all recipes!
    set :application, "project-name"
    set :domain, "project-domain"
    set :extra_domains, %w() # Add something like www.project-domain.com
    role :app, domain
    role :web, domain
    role :db, domain, :primary => true
    !
    # set :user, "user"
    !
    # Deployment Settings
    set :repository, "repository url"
    set :deploy_to, "/absolute/path/to/location/your/server"
    set :deploy_via, :checkout
    set :config_files, %w()
    !
    # SSH Keys for caching (you must generate these first.)
    ssh_options[:keys] = %w(~/.ssh/mykey1 ~/.ssh/mykey2)
    !
    # Change this to :thin if you want to use Thin instead.
    set :app_server, :mongrel
    !
    # Change this to :merb if you want to use Merb instead. (experimental)
    set :app_framework, :rails
    !
    # =============================================================
    # Application Server Settings (Thin or Mongrel)
    # =============================================================
    set :app_servers, 1
    set :app_server_port, 7007
    set :app_environment, 'production'
    set :app_server_address, '127.0.0.1'
    set :app_server_conf, "#{shared_path}/config/thin.yml"
    !
    # =============================================================
    # Nginx Settings
    

    View Slide

27. Today
    !
    

    View Slide

28. !
    The GitHub Flow
    

    View Slide

29. !
    Start with a Pull Request.
    

    View Slide

30. !
    

    View Slide

31. !
    Create a branch.
    

    View Slide

32. !
    Add commits.
    

    View Slide

33. !
    Open a pull request.
    

    View Slide

34. !
    Discuss and review.
    

    View Slide

35. !
    Merge and deploy.
    

    View Slide

36. !
    Or deploy and merge. $
    

    View Slide

37. !
    %
    

    View Slide

38. !
    

    View Slide

39. !
    The following deployments are based on actual events. Branch names
    and SHAs have been changed to protect the innocent.
    

    View Slide

40. !
    hubot where can I deploy?
    >
    Environments for github
    -----------------------------------------------------
    production: unlocked
    lab-one: locked 6 hours ago by jasonrudolph
    lab-two: unlocked
    sekret-lab: unlocked
    playground: unlocked
    

    View Slide

41. !
    hubot deploy github/api-my-feature to production
    >
    

    View Slide

42. !
    hubot deploy github/api-my-feature to production
    >
    project
    

    View Slide

43. !
    hubot deploy github/api-my-feature to production
    >
    branch
    

    View Slide

44. !
    hubot deploy github/api-my-feature to production
    >
    environment
    

    View Slide

45. !
    hubot deploy github/api-my-feature to production
    >
    Merge master into api-my-feature for pengwynn - hubot github/github@deadbeef
    

    View Slide

46. !
    Get up to date with master. ⏳
    

    View Slide

47. !
    Wait on the CI server. '
    

    View Slide

48. !
    

    View Slide

49. !
    hubot tv me
    >
    

    View Slide

50. !
    hubot deploy github/api-my-feature to production
    >
    Build #8675309 (deadbeef) of github/api-my-feature
    was successful (203s)
    

    View Slide

51. !
    Release the kraken! (
    

    View Slide

52. !
    hubot deploy github/api-my-feature to production
    >
    pengwynn is deploying github/api-my-feature (beefdead..deadbeef) to production
    pengwynn's production deployment of github is NOW )))! (22s)
    

    View Slide

53. "The first transport is away!"
    

    View Slide

54. ## TODO
    - [x] Deploy "
    - [ ] Observe * !
    

    View Slide

55. !
    

    View Slide

56. !
    

    View Slide

57. !
    

    View Slide

58. !
    

    View Slide

59. !
    Problem? ABORT. +
    

    View Slide

60. !
    hubot deploy! github
    >
    pengwynn is deploying github/master to production
    

    View Slide

61. !
    ✨Everything -? MERGE.✨
    

    View Slide

62. !
    

    View Slide

63. !
    >
    pengwynn: it looks like you merged "api-my-feature" branch
    into master, so I've unlocked production.
    

    View Slide

64. !
    >
    pengwynn is deploying github/master to production
    pengwynn is deploying github/master to lab-one
    pengwynn is deploying github/master to lab-two
    pengwynn is deploying github/master to sekret-lab
    

    View Slide

65. Prune with confidence. ✂️
    !
    

    View Slide

66. !
    “Are we even calling this anywhere?”
    

    View Slide

67. !
    Find and remove vestigial code.
    

    View Slide

68. !
    GitHub::Backscatter
    

    View Slide

69. !
    GitHub::Backscatter
    not open source,
    but you can roll your own
    

    View Slide

70. !
    *69 for Ruby methods
    

    View Slide

71. class Foo
    include GitHub::Backscatter
    !
    def the_method(args)
    backscatter_measure
    # ...
    end
    end
    

    View Slide

72. class Foo
    include GitHub::Backscatter
    !
    def the_method(args)
    backscatter_measure
    # ...
    end
    end
    

    View Slide

73. class Foo
    include GitHub::Backscatter
    !
    def the_method(args)
    backscatter_measure
    # ...
    end
    end
    

    View Slide

74. View Slide

75. The calls are
    coming from
    INSIDE the APP.
    0
    

    View Slide

76. class Foo
    include GitHub::Backscatter
    !
    def the_method(args)
    backscatter_measure 1000
    # ...
    end
    end
    

    View Slide

77. class Foo
    include GitHub::Backscatter
    !
    def the_method(args)
    backscatter_measure 1000
    # ...
    end
    end
    trace 1 out of every 1000 calls
    

    View Slide

78. Experiment with confidence. 1
    !
    

    View Slide

79. !
    1
    

    View Slide

80. !
    github/dat-science
    

    View Slide

81. !
    @jbarnette
    

    View Slide

82. !
    @rick
    

    View Slide

83. !
    

    View Slide

84. !
    

    View Slide

85. !
    Safely observe alternate code paths.
    

    View Slide

86. require "dat/science"
    !
    class MyApp::Widget
    def allows?(user)
    experiment = Dat::Science::Experiment.new \
    "widget-permissions" do |e|
    !
    e.control { old_method } # old way
    e.candidate { new_method } # new way
    end
    !
    experiment.run
    end
    end
    

    View Slide

87. require "dat/science"
    !
    class MyApp::Widget
    def allows?(user)
    experiment = Dat::Science::Experiment.new \
    "widget-permissions" do |e|
    !
    e.control { old_method } # old way
    e.candidate { new_method } # new way
    end
    !
    experiment.run
    end
    end
    

    View Slide

88. require "dat/science"
    !
    class MyApp::Widget
    def allows?(user)
    experiment = Dat::Science::Experiment.new \
    "widget-permissions" do |e|
    !
    e.control { old_method } # old way
    e.candidate { new_method } # new way
    end
    !
    experiment.run
    end
    end
    

    View Slide

89. require "dat/science"
    !
    class MyApp::Widget
    def allows?(user)
    experiment = Dat::Science::Experiment.new \
    "widget-permissions" do |e|
    !
    e.control { old_method } # old way
    e.candidate { new_method } # new way
    end
    !
    experiment.run
    end
    end
    

    View Slide

90. require "dat/science"
    !
    class MyApp::Widget
    def allows?(user)
    experiment = Dat::Science::Experiment.new \
    "widget-permissions" do |e|
    !
    e.control { old_method } # old way
    e.candidate { new_method } # new way
    end
    !
    experiment.run
    end
    end
    

    View Slide

91. require "dat/science"
    !
    class MyApp::Widget
    def allows?(user)
    experiment = Dat::Science::Experiment.new \
    "widget-permissions" do |e|
    !
    e.control { old_method } # old way
    e.candidate { new_method } # new way
    end
    !
    experiment.run
    end
    end
    

    View Slide

92. !
    Fancy a DSL?
    

    View Slide

93. require "dat/science"
    !
    class MyApp::Widget
    def allows?(user)
    science "widget-permissions" do |e|
    e.control { model.check_user(user).valid? }
    e.candidate { user.can? :read, model }
    end
    end
    end
    

    View Slide

94. require "dat/science"
    !
    class MyApp::Widget
    def allows?(user)
    science "widget-permissions" do |e|
    e.control { model.check_user(user).valid? }
    e.candidate { user.can? :read, model }
    end
    end
    end
    

    View Slide

95. !
    Create and run.
    

    View Slide

96. require "dat/science"
    !
    class Blog::Post
    def permalink(user)
    science "permalink" do |e|
    e.control { post[:permalink] }
    e.candidate { post.slug }
    e.comparator {|a, b| a.downcase == b.downcase }
    end
    end
    end
    

    View Slide

97. require "dat/science"
    !
    class Blog::Post
    def permalink(user)
    science "permalink" do |e|
    e.control { post[:permalink] }
    e.candidate { post.slug }
    e.comparator {|a, b| a.downcase == b.downcase }
    end
    end
    end
    

    View Slide

98. !
    Ramping up experiments.
    

    View Slide

99. require "dat/science"
    !
    module Myapp
    class Experiment < Dat::Science::Experiment
    def enabled?
    rand(100) < 10 # Run 10% of the time
    end
    end
    end
    

    View Slide

100. !
     jnunemaker/flipper 2
     

     View Slide

101. require "dat/science"
     !
     module Myapp
     class Experiment < Dat::Science::Experiment
     def enabled?
     MyApp.flipper[name].enabled?
     end
     end
     end
     

     View Slide

102. !
     Publishing results.
     

     View Slide

103. require "dat/science"
     !
     module Myapp
     class Experiment < Dat::Science::Experiment
     def publish
     MyApp.instrument "science.#{event}", payload
     end
     end
     end
     

     View Slide

104. require "dat/science"
     !
     module Myapp
     class Experiment < Dat::Science::Experiment
     def publish
     MyApp.instrument "science.#{event}", payload
     end
     end
     end
     

     View Slide

105. View Slide

106. View Slide

107. !
     Adding custom context.
     

     View Slide

108. require "dat/science"
     !
     class Blog::Post
     def permalink(user)
     science "permalink" do |e|
     e.context :post => post
     e.control { post[:permalink] }
     e.candidate { post.slug }
     e.comparator {|a, b| a.downcase == b.downcase }
     end
     end
     end
     

     View Slide

109. require "dat/science"
     !
     class Blog::Post
     def permalink(user)
     science "permalink" do |e|
     e.context :post => post
     e.control { post[:permalink] }
     e.candidate { post.slug }
     e.comparator {|a, b| a.downcase == b.downcase }
     end
     end
     end
     

     View Slide

110. Mad science.
     

     View Slide

111. module GitHub
     class Experiment < Dat::Science::Experiment
     class << self
     # Public: enable mad science mode: returns the candidate values by default
     # instead of the control.
     def mad_science=(val)
     @mad_science = val
     end
     !
     # Internal: whether or not to always run the candidate instead of the
     # control.
     def candidate_instead_of_control?
     @mad_science
     end
     end
     …
     end
     

     View Slide

112. module GitHub
     class Experiment < Dat::Science::Experiment
     class << self
     # Public: enable mad science mode: returns the candidate values by default
     # instead of the control.
     def mad_science=(val)
     @mad_science = val
     end
     !
     # Internal: whether or not to always run the candidate instead of the
     # control.
     def candidate_instead_of_control?
     @mad_science
     end
     end
     …
     end
     

     View Slide

113. !
     Analyzing results.
     

     View Slide

114. !
     github/dat-analysis
     

     View Slide

115. Thanks, Jim.
     

     View Slide

116. !
     EOF
     

     View Slide