Extending Gems (OS Bridge 2014)

One strength of the Ruby community is the simplicity of sharing code via gems. When a gem is popular enough, it can even develop an ecosystem of additional gems that build on it.

But extending a gem that wasn’t built with that flexibility in mind isn’t always easy. This talk highlights tips and techniques for making your gem simpler to plug into, and avoid mistakes that will have other devs cursing under their breath.

Based on experience working on New Relic’s Ruby agent (aka the newrelic_rpm gem), we’ll cover the highs and lows of interacting with others gems, from configuration to documentation and everywhere in between.

Transcript

1. Extending Gems
   Jason Clark
   @jasonrclark
   Ruby Agent Engineer
   Patterns and Anti-Patterns of
   Pluggable Gems
   Thursday, June 26, 14
   

   View Slide

2. Storytime!
   Thursday, June 26, 14
   

   View Slide

3. newrelic_rpm
   module NewRelic::Agent::BrowserMonitoring
   def browser_monitoring_queue_time
   queue_time = current_transaction.queue_time
   millis = queue_time.to_f * 1000.0
   clamp_to_positive(millis.round)
   end
   def footer_js_string(config)
   "...#{browser_monitoring_queue_time}"<br/>end<br/>end<br/>Thursday, June 26, 14<br/>

   View Slide

4. module NewRelic::Agent::BrowserMonitoring
   def current_timings
   TransactionTimings.new
   end
   def footer_js_string(config)
   "...#{current_timings.queue_time_in_millis}"<br/>end<br/>end<br/>newrelic_rpm<br/>Thursday, June 26, 14<br/>

   View Slide

5. Meanwhile...
   Elsewhere At New Relic
   http://flic.kr/p/9xEUug
   Thursday, June 26, 14
   

   View Slide

6. module NewRelic::Agent::BrowserMonitoring
   # MONKEY PATCH!
   def footer_js_string(config)
   "...#{browser_monitoring_queue_time}"<br/>end<br/>end<br/>rpm_site<br/>Thursday, June 26, 14<br/>

   View Slide

7. http://pixabay.com/en/elephant-foot-ten-foot-elephant-52536/
   Thursday, June 26, 14
   

   View Slide

8. Thursday, June 26, 14
   

   View Slide

9. Reskiq
   It forks then it threads!
   Thursday, June 26, 14
   

   View Slide

10. gem install
    my_awesome_code
    Thursday, June 26, 14
    

    View Slide

11. Where We're Going
    • Pass It On
    • Events
    • Middleware
    • Lifecycle
    • Names and Paths
    • Config
    • Docs
    Thursday, June 26, 14
    

    View Slide

12. Pass It On
    Thursday, June 26, 14
    

    View Slide

13. Loggers
    Thursday, June 26, 14
    

    View Slide

14. activerecord
    ActiveRecord::Base.logger =
    ::Logger.new("my_special.log")
    Thursday, June 26, 14
    

    View Slide

15. http://flic.kr/p/7GDRue
    Dependency Injection
    Thursday, June 26, 14
    

    View Slide

16. http://flic.kr/p/gzxjKu
    Duck Typing
    Thursday, June 26, 14
    

    View Slide

17. Instrumentors
    Thursday, June 26, 14
    

    View Slide

18. excon
    connection = Excon.new(
    'http://nerd.jasonrclark.com',
    :instrumentor => SimpleInstrumentor
    )
    Thursday, June 26, 14
    

    View Slide

19. excon
    class SimpleInstrumentor
    class << self
    def instrument(name, params = {}, &blk)
    puts "#{name} just happened."
    yield if block_given?
    end
    end
    end
    Thursday, June 26, 14
    

    View Slide

20. http://pixabay.com/en/elephant-foot-ten-foot-elephant-52536/
    Thursday, June 26, 14
    

    View Slide

21. Backends
    Thursday, June 26, 14
    

    View Slide

22. Delayed::Job
    # without delayed_job
    @user.activate!(@device)
    # with delayed_job
    @user.delay.activate!(@device)
    Thursday, June 26, 14
    

    View Slide

23. Delayed::Job
    Thursday, June 26, 14
    

    View Slide

24. Delayed::Job
    ::Delayed.const_set(:Job, backend)
    Thursday, June 26, 14
    

    View Slide

25. newrelic_rpm
    def failed_jobs
    Delayed::Job.count(
    :conditions => 'failed_at is not NULL')
    end
    Thursday, June 26, 14
    

    View Slide

26. Gem-Specific
    Thursday, June 26, 14
    

    View Slide

27. Resque
    class SendMessageJob
    def self.perform(id, text)
    user = User.find(id)
    user.send_message(text)
    end
    end
    Resque.enqueue(SendMessageJob, 42, "Yo")
    Thursday, June 26, 14
    

    View Slide

28. Resque
    https://github.com/resque/resque/blob/master/docs/HOOKS.md
    class SendMessageJob
    def self.before_perform_log_it(*args)
    logger.debug("Messaging #{args}")
    end
    ...
    end
    Thursday, June 26, 14
    

    View Slide

29. Where We've Been
    • Pass It On
    • Events
    • Middleware
    • Lifecycle
    • Names and Paths
    • Config
    • Docs
    Thursday, June 26, 14
    

    View Slide

30. Events
    Thursday, June 26, 14
    

    View Slide

31. http://www.confreaks.com/videos/3327-railsconf-
    make-an-event-of-it
    Thursday, June 26, 14
    

    View Slide

32. Resque
    Resque.after_fork do |job|
    ActiveRecord::Base.establish_connection
    end
    Thursday, June 26, 14
    

    View Slide

33. http://pixabay.com/en/elephant-foot-ten-foot-elephant-52536/
    Thursday, June 26, 14
    

    View Slide

34. Where We've Been
    • Pass It On
    • Events
    • Middleware
    • Lifecycle
    • Names and Paths
    • Config
    • Docs
    Thursday, June 26, 14
    

    View Slide

35. Middleware
    Thursday, June 26, 14
    

    View Slide

36. Rack::Cache
    ActionDispatch::Cookies
    YourApplication
    Web Request
    Thursday, June 26, 14
    

    View Slide

37. # [status, headers, response]
    def call(env)
    # ... before
    result = @app.call(env)
    # ... after
    result
    end
    Thursday, June 26, 14
    

    View Slide

38. # [status, headers, response]
    def call(env)
    # ... before
    result = @app.call(env)
    # ... after
    result
    end
    Thursday, June 26, 14
    

    View Slide

39. http://flic.kr/p/SSGmF
    ActionDispatch::Static
    Rack::Lock
    
    Rack::Runtime
    Rack::MethodOverride
    ActionDispatch::RequestId
    Rails::Rack::Logger
    ActionDispatch::ShowExceptions
    ActionDispatch::DebugExceptions
    ActionDispatch::RemoteIp
    ActionDispatch::Reloader
    ActionDispatch::Callbacks
    ActiveRecord::ConnectionAdapters
    ActiveRecord::QueryCache
    ActionDispatch::Cookies
    ActionDispatch::Session::CookieStore
    ActionDispatch::Flash
    ActionDispatch::ParamsParser
    Rack::Head
    Rack::ConditionalGet
    Rack::ETag
    AgentSnoop::Middleware
    RpmTestApp::Application.routes
    Thursday, June 26, 14
    

    View Slide

40. Who's There?
    Thursday, June 26, 14
    

    View Slide

41. # [status, headers, response]
    def call(env)
    # ... before
    result = @app.call(env)
    # ... after
    result
    end
    Thursday, June 26, 14
    

    View Slide

42. class MyMiddleware
    def initialize(options={})
    # options == { :foo => 1 }
    end
    def call(worker, msg, queue)
    yield
    end
    end
    Thursday, June 26, 14
    

    View Slide

43. Sidekiq.configure_server do |config|
    config.server_middleware do |chain|
    chain.add MyMiddleware, :foo => 1
    end
    end
    Thursday, June 26, 14
    

    View Slide

44. Where We've Been
    • Pass It On
    • Events
    • Middleware
    • Lifecycle
    • Names and Paths
    • Config
    • Docs
    Thursday, June 26, 14
    

    View Slide

45. Lifecycle
    Thursday, June 26, 14
    

    View Slide

46. Forks and Daemons
    http://flic.kr/p/79tE5Q
    Thursday, June 26, 14
    

    View Slide

47. Forking
    fork()
    state
    Thursday, June 26, 14
    

    View Slide

48. fork()
    state
    fork()
    Forking
    state
    Thursday, June 26, 14
    

    View Slide

49. before_fork
    after_fork
    Thursday, June 26, 14
    

    View Slide

50. Add Threads
    fork()
    Thursday, June 26, 14
    

    View Slide

51. fork()
    Add Threads
    Thursday, June 26, 14
    

    View Slide

52. Enter the Lock
    fork()
    @lock
    Thursday, June 26, 14
    

    View Slide

53. @loick
    fork()
    Enter the Lock
    @lock
    Thursday, June 26, 14
    

    View Slide

54. https://github.com/resque/resque/issues/1101
    Forks +
    Threads + ==
    Locks
    Resque
    Thursday, June 26, 14
    

    View Slide

55. Load Up!
    Thursday, June 26, 14
    

    View Slide

56. unicorn
    ♥ὑὑ~/myapp:unicorn_rails -c ./unicorn.rb
    unicorn_rails
    master
    unicorn_rails
    worker[0]
    unicorn_rails
    worker[1]
    unicorn_rails
    worker[2]
    unicorn_rails
    worker[3]
    Thursday, June 26, 14
    

    View Slide

57. loads gems
    forks
    after_fork
    starts loop
    forks
    after_fork
    starts loop
    loads gems
    preload_app true preload_app false
    unicorn
    Thursday, June 26, 14
    

    View Slide

58. loads gems
    forks
    after_fork
    starts loop
    forks
    after_fork
    starts loop
    loads gems
    preload_app true preload_app false
    unicorn
    Thursday, June 26, 14
    

    View Slide

59. loads gems
    forks
    after_fork
    starts loop
    forks
    after_fork
    starts loop
    loads gems
    preload_app true preload_app false
    unicorn
    Thursday, June 26, 14
    

    View Slide

60. Where We've Been
    • Pass It On
    • Events
    • Middleware
    • Lifecycle
    • Names and Paths
    • Config
    • Docs
    Thursday, June 26, 14
    

    View Slide

61. Names and
    Paths
    Thursday, June 26, 14
    

    View Slide

62. Forced
    Thursday, June 26, 14
    

    View Slide

63. Sequel.plugin :newrelic_instrumentation
    Thursday, June 26, 14
    

    View Slide

64. Sequel.plugin :newrelic_instrumentation
    =>
    require 'sequel/plugins/newrelic_instrumentation'
    Thursday, June 26, 14
    

    View Slide

65. ~/source/newrelic/ruby_agent/lib: ls -la
    drwxr-xr-x Sep 19 14:46 .
    drwxr-xr-x Oct 10 11:19 ..
    drwxr-xr-x Oct 10 11:20 new_relic
    -rw-r--r-- Sep 19 14:46 newrelic_rpm.rb
    drwxr-xr-x Sep 19 14:46 sequel
    drwxr-xr-x Oct 7 08:53 tasks
    Thursday, June 26, 14
    

    View Slide

66. ~/source/newrelic/ruby_agent/lib: ls -la
    drwxr-xr-x Sep 19 14:46 .
    drwxr-xr-x Oct 10 11:19 ..
    drwxr-xr-x Oct 10 11:20 new_relic
    -rw-r--r-- Sep 19 14:46 newrelic_rpm.rb
    drwxr-xr-x Sep 19 14:46 sequel
    drwxr-xr-x Oct 7 08:53 tasks
    Thursday, June 26, 14
    

    View Slide

67. Sequel.plugin NewRelic::Sequel::Instrumentation
    Thursday, June 26, 14
    

    View Slide

68. Generic
    Thursday, June 26, 14
    

    View Slide

69. Thursday, June 26, 14
    

    View Slide

70. ActiveRecord
    Thursday, June 26, 14
    

    View Slide

71. ActiveRecord
    DataMapper
    Thursday, June 26, 14
    

    View Slide

72. ActiveRecord
    DataMapper
    HTTPClient
    Thursday, June 26, 14
    

    View Slide

73. Qualify Yourself
    Thursday, June 26, 14
    

    View Slide

74. newrelic_rpm
    module NewRelic
    module Agent
    class Thread < ::Thread
    ...
    end
    end
    end
    Thursday, June 26, 14
    

    View Slide

75. elsewhere...
    class Thread
    # Re-opens NewRelic::Agent::Thread
    # D'oh!
    end
    Thursday, June 26, 14
    

    View Slide

76. newrelic_plugin
    module NewRelic
    class Logger
    ...
    end
    end
    Thursday, June 26, 14
    

    View Slide

77. newrelic_rpm
    module NewRelic
    module Agent
    class AgentLogger
    LOG_LEVELS = {
    "debug" => Logger::DEBUG,
    ...
    }
    end
    end
    end
    Thursday, June 26, 14
    

    View Slide

78. newrelic_rpm
    module NewRelic
    module Agent
    class AgentLogger
    LOG_LEVELS = {
    "debug" => Logger::DEBUG,
    ...
    }
    end
    end
    end
    Thursday, June 26, 14
    

    View Slide

79. newrelic_rpm
    module NewRelic
    module Agent
    class AgentLogger
    LOG_LEVELS = {
    "debug" => ::Logger::DEBUG,
    ...
    }
    end
    end
    end
    Thursday, June 26, 14
    

    View Slide

80. Where We've Been
    • Pass It On
    • Events
    • Middleware
    • Lifecycle
    • Names and Paths
    • Config
    • Docs
    Thursday, June 26, 14
    

    View Slide

81. Config
    Thursday, June 26, 14
    

    View Slide

82. yml
    Thursday, June 26, 14
    

    View Slide

83. yml + ERB
    Thursday, June 26, 14
    

    View Slide

84. yml + ERB
    file = File.read("./config/awesome_gem.yml")
    # Locals available in ERB
    default_awesome_key = "YEAH!"
    erb = ERB.new(file).result(binding)
    config = YAML.load(erb)
    Thursday, June 26, 14
    

    View Slide

85. yml + ERB
    ~/config/awesome_gem.yml
    development:
    name: <%= default_awesome_key %>
    license_key: <%= ENV["LICENSE_KEY"] %>
    Thursday, June 26, 14
    

    View Slide

86. Ruby
    Thursday, June 26, 14
    

    View Slide

87. unicorn
    worker_processes 5
    preload_app true
    timeout 30
    after_fork do |server, worker|
    defined?(ActiveRecord::Base) and
    ActiveRecord::Base.establish_connection
    end
    Thursday, June 26, 14
    

    View Slide

88. on_worker_boot do
    ActiveSupport.on_load(:active_record) do
    ActiveRecord::Base.establish_connection
    end
    end
    Thursday, June 26, 14
    

    View Slide

89. Great for Apps?
    What about Gems?
    Thursday, June 26, 14
    

    View Slide

90. newrelic_rpm
    if defined?(::Puma) &&
    ::Puma.respond_to?(:cli_config)
    config = ::Puma.cli_config
    config.options[:worker_boot] << Proc.new do
    NewRelic::Agent.after_fork(...)
    end
    end
    Thursday, June 26, 14
    

    View Slide

91. newrelic_rpm
    if defined?(::Puma) &&
    ::Puma.respond_to?(:cli_config)
    config = ::Puma.cli_config
    config.options[:worker_boot] << Proc.new do
    NewRelic::Agent.after_fork(...)
    end
    end
    Thursday, June 26, 14
    

    View Slide

92. Where We've Been
    • Pass It On
    • Events
    • Middleware
    • Lifecycle
    • Names and Paths
    • Config
    • Docs
    Thursday, June 26, 14
    

    View Slide

93. Docs
    Thursday, June 26, 14
    

    View Slide

94. README
    Thursday, June 26, 14
    

    View Slide

95. sinatra
    Thursday, June 26, 14
    

    View Slide

96. Thursday, June 26, 14
    

    View Slide

97. Thursday, June 26, 14
    

    View Slide

98. Thursday, June 26, 14
    

    View Slide

99. Expected
    Extensions?
    Thursday, June 26, 14
    

    View Slide

100. Thursday, June 26, 14
     

     View Slide

101. Don't Bury It
     Thursday, June 26, 14
     

     View Slide

102. Thursday, June 26, 14
     

     View Slide

103. Thursday, June 26, 14
     

     View Slide

104. Versioning
     Thursday, June 26, 14
     

     View Slide

105. Thursday, June 26, 14
     

     View Slide

106. Where We've Been
     • Pass It On
     • Events
     • Middleware
     • Lifecycle
     • Names and Paths
     • Config
     • Docs
     Thursday, June 26, 14
     

     View Slide

107. Where We've Been
     Thursday, June 26, 14
     

     View Slide

108. Where We've Been
     • Pass It On
     • Events
     • Middleware
     • Lifecycle
     • Names and Paths
     • Config
     • Docs
     Thursday, June 26, 14
     

     View Slide

109. Where We've Been
     • Pass It On
     • Events
     • Middleware
     • Lifecycle
     • Names and Paths
     • Config
     • Docs
     Jason Clark
     @jasonrclark
     Ruby Agent Engineer
     Thursday, June 26, 14
     

     View Slide