Extending Gems - Patterns and Anti-Patterns of Making Your Gem Pluggable

One strength of the Ruby community is the simplicity of sharing code via gems. When a gem is popular enough, it can even develop an ecosystem of additional gems that build on it.

But extending a gem that wasn't built with that flexibility in mind isn't always easy. This talk highlights tips and techniques for making your gem simpler to plug into, and avoid mistakes that will have devs cursing under their breath.

We'll cover the highs and lows of interacting with others gems, from configuration to documentation and everywhere in between.

http://rubyconf13.multifaceted.io/talks/extending-gems.html

Transcript

1. Extending Gems
   Patterns and Anti-Patterns of
   Making Your Gem Pluggable
   Jason Clark
   @jasonrclark
   Ruby Agent Engineer
   Monday, November 11, 13
   

   View Slide

2. 2
   Storytime!
   Monday, November 11, 13
   

   View Slide

3. 3
   newrelic_rpm
   module NewRelic::Agent::BrowserMonitoring
   def browser_monitoring_queue_time
   queue_time = current_transaction.queue_time
   millis = queue_time.to_f * 1000.0
   clamp_to_positive(millis.round)
   end
   def footer_js_string(config)
   "... #{browser_monitoring_queue_time}"
   end
   end
   Monday, November 11, 13
   

   View Slide

4. 4
   module NewRelic::Agent::BrowserMonitoring
   def current_timings
   TransactionTimings.new
   end
   def footer_js_string(config)
   "... #{current_timings.queue_time_in_millis}"
   end
   end
   newrelic_rpm
   Monday, November 11, 13
   

   View Slide

5. 5
   Meanwhile...
   Elsewhere At New Relic
   http://flic.kr/p/9xEUug
   Monday, November 11, 13
   

   View Slide

6. 6
   module NewRelic::Agent::BrowserMonitoring
   # MONKEY PATCHED!
   def footer_js_string(config)
   "... #{browser_monitoring_queue_time}"
   end
   end
   rpm_site
   Monday, November 11, 13
   

   View Slide

7. 7
   http://pixabay.com/en/elephant-foot-ten-foot-elephant-52536/
   Monday, November 11, 13
   

   View Slide

8. 8
   Monday, November 11, 13
   

   View Slide

9. 9
   Reskiq
   It forks then it threads!
   Monday, November 11, 13
   

   View Slide

10. 10
    gem install
    my_awesome_code
    Monday, November 11, 13
    

    View Slide

11. Where We're Going
    11
    • Pass It On
    • Events
    • Middleware
    • Lifecycle
    • Names and Paths
    • Config
    • Docs
    Monday, November 11, 13
    

    View Slide

12. Pass It On
    12
    Monday, November 11, 13
    

    View Slide

13. Loggers
    13
    Monday, November 11, 13
    

    View Slide

14. activerecord
    14
    ActiveRecord::Base.logger =
    ::Logger.new("my_special.log")
    Monday, November 11, 13
    

    View Slide

15. 15 http://flic.kr/p/7GDRue
    Dependency Injection
    Monday, November 11, 13
    

    View Slide

16. 16
    http://flic.kr/p/gzxjKu
    Duck Typing
    Monday, November 11, 13
    

    View Slide

17. Instrumentors
    17
    Monday, November 11, 13
    

    View Slide

18. excon
    18
    connection = Excon.new(
    'http://nerd.jasonrclark.com',
    :instrumentor => SimpleInstrumentor
    )
    Monday, November 11, 13
    

    View Slide

19. excon
    19
    class SimpleInstrumentor
    class << self
    attr_accessor :events
    def instrument(name, params = {}, &blk)
    puts "#{name} just happened."
    yield if block_given?
    end
    end
    end
    Monday, November 11, 13
    

    View Slide

20. 20
    http://pixabay.com/en/elephant-foot-ten-foot-elephant-52536/
    Monday, November 11, 13
    

    View Slide

21. Backends
    21
    Monday, November 11, 13
    

    View Slide

22. Delayed::Job
    22
    # without delayed_job
    @user.activate!(@device)
    # with delayed_job
    @user.delay.activate!(@device)
    Monday, November 11, 13
    

    View Slide

23. Delayed::Job
    23
    Monday, November 11, 13
    

    View Slide

24. Delayed::Job
    24
    ::Delayed.const_set(:Job, backend)
    Monday, November 11, 13
    

    View Slide

25. 25
    newrelic_rpm
    def failed_jobs
    Delayed::Job.count(
    :conditions => 'failed_at is not NULL')
    end
    Monday, November 11, 13
    

    View Slide

26. Gem-Specific
    26
    Monday, November 11, 13
    

    View Slide

27. Resque
    27
    class SendMessageJob
    def self.perform(id, text)
    user = User.find(id)
    user.send_message(text)
    end
    end
    Resque.enqueue(SendMessageJob, 42, "Yo")
    Monday, November 11, 13
    

    View Slide

28. 28
    Resque
    https://github.com/resque/resque/blob/master/docs/HOOKS.md
    class SendMessageJob
    def self.before_perform_log_it(*args)
    logger.debug("Messaging #{args}")
    end
    ...
    end
    Monday, November 11, 13
    

    View Slide

29. Where We've Been
    29
    • Pass It On
    • Events
    • Middleware
    • Lifecycle
    • Names and Paths
    • Config
    • Docs
    Monday, November 11, 13
    

    View Slide

30. Events
    30
    Monday, November 11, 13
    

    View Slide

31. What's Important?
    31
    Monday, November 11, 13
    

    View Slide

32. 32
    Monday, November 11, 13
    

    View Slide

33. Surfacing Events
    33
    Monday, November 11, 13
    

    View Slide

34. 34
    newrelic_rpm
    events = NewRelic::Agent.instance.events
    events.subscribe(:start_transaction) do
    set_transaction_custom_parameters
    end
    Monday, November 11, 13
    

    View Slide

35. 35
    Resque
    Resque.after_fork do |job|
    ActiveRecord::Base.establish_connection
    end
    Monday, November 11, 13
    

    View Slide

36. 36
    http://pixabay.com/en/elephant-foot-ten-foot-elephant-52536/
    Monday, November 11, 13
    

    View Slide

37. Ordering?
    37
    Monday, November 11, 13
    

    View Slide

38. ActiveSupport::Notifications
    38
    activesupport
    Monday, November 11, 13
    

    View Slide

39. Where We've Been
    39
    • Pass It On
    • Events
    • Middleware
    • Lifecycle
    • Names and Paths
    • Config
    • Docs
    Monday, November 11, 13
    

    View Slide

40. Middleware
    40
    Monday, November 11, 13
    

    View Slide

41. 41
    Rack::Cache
    ActionDispatch::Cookies
    YourApplication
    Web Request
    Monday, November 11, 13
    

    View Slide

42. 42
    # [status, headers, response]
    def call(env)
    # ... before
    result = @app.call(env)
    # ... after
    result
    end
    Monday, November 11, 13
    

    View Slide

43. 43
    # [status, headers, response]
    def call(env)
    # ... before
    result = @app.call(env)
    # ... after
    result
    end
    Monday, November 11, 13
    

    View Slide

44. http://flic.kr/p/SSGmF
    ActionDispatch::Static
    Rack::Lock
    
    Rack::Runtime
    Rack::MethodOverride
    ActionDispatch::RequestId
    Rails::Rack::Logger
    ActionDispatch::ShowExceptions
    ActionDispatch::DebugExceptions
    ActionDispatch::RemoteIp
    ActionDispatch::Reloader
    ActionDispatch::Callbacks
    ActiveRecord::ConnectionAdapters
    ActiveRecord::QueryCache
    ActionDispatch::Cookies
    ActionDispatch::Session::CookieStore
    ActionDispatch::Flash
    ActionDispatch::ParamsParser
    Rack::Head
    Rack::ConditionalGet
    Rack::ETag
    AgentSnoop::Middleware
    RpmTestApp::Application.routes
    Monday, November 11, 13
    

    View Slide

45. Who's There?
    45
    Monday, November 11, 13
    

    View Slide

46. 46
    # [status, headers, response]
    def call(env)
    # ... before
    result = @app.call(env)
    # ... after
    result
    end
    Monday, November 11, 13
    

    View Slide

47. 47
    class MyMiddleware
    def initialize(options=nil)
    # options == { :foo => 1 }
    end
    def call(worker, msg, queue)
    yield
    end
    end
    Monday, November 11, 13
    

    View Slide

48. 48
    Sidekiq.configure_server do |config|
    config.server_middleware do |chain|
    chain.add MyMiddleware, :foo => 1
    end
    end
    Monday, November 11, 13
    

    View Slide

49. Where We've Been
    49
    • Pass It On
    • Events
    • Middleware
    • Lifecycle
    • Names and Paths
    • Config
    • Docs
    Monday, November 11, 13
    

    View Slide

50. Lifecycle
    50
    Monday, November 11, 13
    

    View Slide

51. Forks and Daemons
    51
    http://flic.kr/p/79tE5Q
    Monday, November 11, 13
    

    View Slide

52. Forking
    52
    fork()
    state
    Monday, November 11, 13
    

    View Slide

53. fork()
    state
    fork()
    Forking
    53
    state
    Monday, November 11, 13
    

    View Slide

54. before_fork
    after_fork
    54
    Monday, November 11, 13
    

    View Slide

55. Add Threads
    55
    fork()
    Monday, November 11, 13
    

    View Slide

56. fork()
    Add Threads
    56
    Monday, November 11, 13
    

    View Slide

57. Enter the Lock
    57
    fork()
    @lock
    Monday, November 11, 13
    

    View Slide

58. @loick
    fork()
    Enter the Lock
    58
    @lock
    Monday, November 11, 13
    

    View Slide

59. 59
    https://github.com/resque/resque/issues/1101
    Forks +
    Threads + ==
    Locks
    Resque
    Monday, November 11, 13
    

    View Slide

60. Load Up!
    60
    Monday, November 11, 13
    

    View Slide

61. unicorn
    61
    ♥ὑὑ~/myapp:unicorn_rails -c ./unicorn.rb
    unicorn_rails
    master
    unicorn_rails
    worker[0]
    unicorn_rails
    worker[1]
    unicorn_rails
    worker[2]
    unicorn_rails
    worker[3]
    Monday, November 11, 13
    

    View Slide

62. 62
    loads gems
    forks
    after_fork
    starts loop
    forks
    after_fork
    starts loop
    loads gems
    preload_app true preload_app false
    unicorn
    Monday, November 11, 13
    

    View Slide

63. 63
    loads gems
    forks
    after_fork
    starts loop
    forks
    after_fork
    starts loop
    loads gems
    preload_app true preload_app false
    unicorn
    Monday, November 11, 13
    

    View Slide

64. Where We've Been
    64
    • Pass It On
    • Events
    • Middleware
    • Lifecycle
    • Names and Paths
    • Config
    • Docs
    Monday, November 11, 13
    

    View Slide

65. Names and
    Paths
    65
    Monday, November 11, 13
    

    View Slide

66. Forced
    66
    Monday, November 11, 13
    

    View Slide

67. 67
    Sequel.plugin :newrelic_instrumentation
    Monday, November 11, 13
    

    View Slide

68. 68
    Sequel.plugin :newrelic_instrumentation
    =>
    require 'sequel/plugins/newrelic_instrumentation'
    Monday, November 11, 13
    

    View Slide

69. 69
    ~/source/newrelic/ruby_agent/lib: ls -la
    drwxr-xr-x Sep 19 14:46 .
    drwxr-xr-x Oct 10 11:19 ..
    drwxr-xr-x Oct 10 11:20 new_relic
    -rw-r--r-- Sep 19 14:46 newrelic_rpm.rb
    drwxr-xr-x Sep 19 14:46 sequel
    drwxr-xr-x Oct 7 08:53 tasks
    Monday, November 11, 13
    

    View Slide

70. 70
    ~/source/newrelic/ruby_agent/lib: ls -la
    drwxr-xr-x Sep 19 14:46 .
    drwxr-xr-x Oct 10 11:19 ..
    drwxr-xr-x Oct 10 11:20 new_relic
    -rw-r--r-- Sep 19 14:46 newrelic_rpm.rb
    drwxr-xr-x Sep 19 14:46 sequel
    drwxr-xr-x Oct 7 08:53 tasks
    Monday, November 11, 13
    

    View Slide

71. 71
    Sequel.plugin NewRelic::Sequel::Instrumentation
    Monday, November 11, 13
    

    View Slide

72. Generic
    72
    Monday, November 11, 13
    

    View Slide

73. 73
    Monday, November 11, 13
    

    View Slide

74. 73
    ActiveRecord
    Monday, November 11, 13
    

    View Slide

75. 73
    ActiveRecord
    DataMapper
    Monday, November 11, 13
    

    View Slide

76. 73
    ActiveRecord
    DataMapper
    HTTPClient
    Monday, November 11, 13
    

    View Slide

77. Qualify Yourself
    74
    Monday, November 11, 13
    

    View Slide

78. 75
    newrelic_rpm
    module NewRelic
    module Agent
    class Thread < ::Thread
    ...
    end
    end
    end
    Monday, November 11, 13
    

    View Slide

79. elsewhere...
    76
    class Thread
    # Re-opened NewRelic::Agent::Thread
    # D'oh!
    end
    Monday, November 11, 13
    

    View Slide

80. 77
    newrelic_plugin
    module NewRelic
    class Logger
    ...
    end
    end
    Monday, November 11, 13
    

    View Slide

81. 78
    newrelic_rpm
    module NewRelic
    module Agent
    class AgentLogger
    LOG_LEVELS = {
    "debug" => Logger::DEBUG,
    ...
    }
    end
    end
    end
    Monday, November 11, 13
    

    View Slide

82. 79
    newrelic_rpm
    module NewRelic
    module Agent
    class AgentLogger
    LOG_LEVELS = {
    "debug" => Logger::DEBUG,
    ...
    }
    end
    end
    end
    Monday, November 11, 13
    

    View Slide

83. 80
    newrelic_rpm
    module NewRelic
    module Agent
    class AgentLogger
    LOG_LEVELS = {
    "debug" => ::Logger::DEBUG,
    ...
    }
    end
    end
    end
    Monday, November 11, 13
    

    View Slide

84. Where We've Been
    81
    • Pass It On
    • Events
    • Middleware
    • Lifecycle
    • Names and Paths
    • Config
    • Docs
    Monday, November 11, 13
    

    View Slide

85. Config
    82
    Monday, November 11, 13
    

    View Slide

86. yml
    83
    Monday, November 11, 13
    

    View Slide

87. yml + ERB
    84
    Monday, November 11, 13
    

    View Slide

88. yml + ERB
    85
    file = File.read("~/config/awesome_gem.yml")
    # Locals available in ERB
    default_awesome_key = "YEAH!"
    erb = ERB.new(file).result(binding)
    config = YAML.load(erb)
    Monday, November 11, 13
    

    View Slide

89. yml + ERB
    86
    ~/config/awesome_gem.yml
    development:
    name: <%= default_awesome_key %>
    license_key: <%= ENV["LICENSE_KEY"] %>
    Monday, November 11, 13
    

    View Slide

90. Ruby
    87
    Monday, November 11, 13
    

    View Slide

91. 88
    unicorn
    worker_processes 5
    preload_app true
    timeout 30
    after_fork do |server, worker|
    defined?(ActiveRecord::Base) and
    ActiveRecord::Base.establish_connection
    end
    Monday, November 11, 13
    

    View Slide

92. 89
    on_worker_boot do
    ActiveSupport.on_load(:active_record) do
    ActiveRecord::Base.establish_connection
    end
    end
    Monday, November 11, 13
    

    View Slide

93. Great for Apps?
    What about Gems?
    90
    Monday, November 11, 13
    

    View Slide

94. 91
    newrelic_rpm
    if defined?(::Puma) &&
    ::Puma.respond_to?(:cli_config)
    config = ::Puma.cli_config
    config.options[:worker_boot] << Proc.new do
    NewRelic::Agent.after_fork(...)
    end
    end
    Monday, November 11, 13
    

    View Slide

95. 92
    newrelic_rpm
    if defined?(::Puma) &&
    ::Puma.respond_to?(:cli_config)
    config = ::Puma.cli_config
    config.options[:worker_boot] << Proc.new do
    NewRelic::Agent.after_fork(...)
    end
    end
    Monday, November 11, 13
    

    View Slide

96. Where We've Been
    93
    • Pass It On
    • Events
    • Middleware
    • Lifecycle
    • Names and Paths
    • Config
    • Docs
    Monday, November 11, 13
    

    View Slide

97. Docs
    94
    Monday, November 11, 13
    

    View Slide

98. README
    95
    Monday, November 11, 13
    

    View Slide

99. sinatra
    96
    Monday, November 11, 13
    

    View Slide

100. 97
     Monday, November 11, 13
     

     View Slide

101. 98
     Monday, November 11, 13
     

     View Slide

102. 99
     Monday, November 11, 13
     

     View Slide

103. Expected
     Extensions?
     100
     Monday, November 11, 13
     

     View Slide

104. 101
     Monday, November 11, 13
     

     View Slide

105. Don't Bury It
     102
     Monday, November 11, 13
     

     View Slide

106. 103
     Monday, November 11, 13
     

     View Slide

107. 104
     Monday, November 11, 13
     

     View Slide

108. Versioning
     105
     Monday, November 11, 13
     

     View Slide

109. 106
     Monday, November 11, 13
     

     View Slide

110. Where We've Been
     107
     • Pass It On
     • Events
     • Middleware
     • Lifecycle
     • Names and Paths
     • Config
     • Docs
     Monday, November 11, 13
     

     View Slide

111. Where We've Been
     108
     Monday, November 11, 13
     

     View Slide

112. Where We've Been
     108
     • Pass It On
     • Events
     • Middleware
     • Lifecycle
     • Names and Paths
     • Config
     • Docs
     Monday, November 11, 13
     

     View Slide

113. Where We've Been
     108
     • Pass It On
     • Events
     • Middleware
     • Lifecycle
     • Names and Paths
     • Config
     • Docs
     Jason Clark
     @jasonrclark
     Ruby Agent Engineer
     Monday, November 11, 13
     

     View Slide