Patterns of Pluggable Gems (ConFoo 2014)

Patterns of
Pluggable Gems
Jason Clark
@jasonrclark
Ruby Agent Engineer
Wednesday, February 26, 14

View Slide

Storytime!

View Slide

newrelic_rpm
module NewRelic::Agent::BrowserMonitoring
def browser_monitoring_queue_time
queue_time = current_transaction.queue_time
millis = queue_time.to_f * 1000.0
clamp_to_positive(millis.round)
end
def footer_js_string(config)
"...#{browser_monitoring_queue_time}" end end Wednesday, February 26, 14 

View Slide

def current_timings
TransactionTimings.new
end
"...#{current_timings.queue_time_in_millis}" end end newrelic_rpm Wednesday, February 26, 14 

View Slide

Meanwhile...
Elsewhere At New Relic
http://ﬂic.kr/p/9xEUug

View Slide

# MONKEY PATCH!
"...#{browser_monitoring_queue_time}" end end rpm_site Wednesday, February 26, 14 

View Slide

http://pixabay.com/en/elephant-foot-ten-foot-elephant-52536/

View Slide


View Slide

Reskiq
It forks then it threads!

View Slide

gem install
my_awesome_code

View Slide

Where We're Going
• Pass It On
• Events
• Middleware
• Lifecycle
• Names and Paths
• Conﬁg
• Docs

View Slide

Pass It On

View Slide

Loggers

View Slide

activerecord
ActiveRecord::Base.logger =
::Logger.new("my_special.log")

View Slide

http://ﬂic.kr/p/7GDRue
Dependency Injection

View Slide

http://ﬂic.kr/p/gzxjKu
Duck Typing

View Slide

Instrumentors

View Slide

excon
connection = Excon.new(
'http://nerd.jasonrclark.com',
:instrumentor => SimpleInstrumentor
)

View Slide

excon
class SimpleInstrumentor
class << self
def instrument(name, params = {}, &blk)
puts "#{name} just happened."
yield if block_given?
end
end
end

View Slide


View Slide

Backends

View Slide

Delayed::Job
# without delayed_job
@user.activate!(@device)
# with delayed_job
@user.delay.activate!(@device)

View Slide

Delayed::Job

View Slide

Delayed::Job
::Delayed.const_set(:Job, backend)

View Slide

newrelic_rpm
def failed_jobs
Delayed::Job.count(
:conditions => 'failed_at is not NULL')
end

View Slide

Gem-Speciﬁc

View Slide

Resque
class SendMessageJob
def self.perform(id, text)
user = User.find(id)
user.send_message(text)
end
end
Resque.enqueue(SendMessageJob, 42, "Yo")

View Slide

Resque
https://github.com/resque/resque/blob/master/docs/HOOKS.md
class SendMessageJob
def self.before_perform_log_it(*args)
logger.debug("Messaging #{args}")
end
...
end

View Slide

Where We've Been
• Pass It On
• Events
• Middleware
• Lifecycle
• Names and Paths
• Conﬁg
• Docs

View Slide

Events

View Slide

See my talk
tomorrow at 3!

View Slide

Resque
Resque.after_fork do |job|
ActiveRecord::Base.establish_connection
end

View Slide


View Slide

Where We've Been
• Pass It On
• Events
• Middleware
• Lifecycle
• Names and Paths
• Conﬁg
• Docs

View Slide

Middleware

View Slide

Rack::Cache
ActionDispatch::Cookies
YourApplication
Web Request

View Slide

# [status, headers, response]
def call(env)
# ... before
result = @app.call(env)
# ... after
result
end

View Slide

http://ﬂic.kr/p/SSGmF
ActionDispatch::Static
Rack::Lock

Rack::Runtime
Rack::MethodOverride
ActionDispatch::RequestId
Rails::Rack::Logger
ActionDispatch::ShowExceptions
ActionDispatch::DebugExceptions
ActionDispatch::RemoteIp
ActionDispatch::Reloader
ActionDispatch::Callbacks
ActiveRecord::ConnectionAdapters
ActiveRecord::QueryCache
ActionDispatch::Cookies
ActionDispatch::Session::CookieStore
ActionDispatch::Flash
ActionDispatch::ParamsParser
Rack::Head
Rack::ConditionalGet
Rack::ETag
AgentSnoop::Middleware
RpmTestApp::Application.routes

View Slide

Who's There?

View Slide

# [status, headers, response]
def call(env)
# ... before
result = @app.call(env)
# ... after
result
end

View Slide

class MyMiddleware
def initialize(options={})
# options == { :foo => 1 }
end
def call(worker, msg, queue)
yield
end
end

View Slide

Sidekiq.configure_server do |config|
config.server_middleware do |chain|
chain.add MyMiddleware, :foo => 1
end
end

View Slide

Where We've Been
• Pass It On
• Events
• Middleware
• Lifecycle
• Names and Paths
• Conﬁg
• Docs

View Slide

Lifecycle

View Slide

Forks and Daemons
http://ﬂic.kr/p/79tE5Q

View Slide

Forking
fork()
state

View Slide

fork()
state
fork()
Forking
state

View Slide

before_fork
after_fork

View Slide

Add Threads
fork()

View Slide

fork()
Add Threads

View Slide

Enter the Lock
fork()
@lock

View Slide

@loick
fork()
Enter the Lock
@lock

View Slide

https://github.com/resque/resque/issues/1101
Forks +
Threads + ==
Locks
Resque

View Slide

Load Up!

View Slide

unicorn
♥ὑὑ~/myapp:unicorn_rails -c ./unicorn.rb
unicorn_rails
master
unicorn_rails
worker[0]
unicorn_rails
worker[1]
unicorn_rails
worker[2]
unicorn_rails
worker[3]

View Slide

loads gems
forks
after_fork
starts loop
forks
after_fork
starts loop
loads gems
preload_app true preload_app false
unicorn

View Slide

Where We've Been
• Pass It On
• Events
• Middleware
• Lifecycle
• Names and Paths
• Conﬁg
• Docs

View Slide

Names and
Paths

View Slide

Forced

View Slide

Sequel.plugin :newrelic_instrumentation

View Slide

Sequel.plugin :newrelic_instrumentation
=>
require 'sequel/plugins/newrelic_instrumentation'

View Slide

~/source/newrelic/ruby_agent/lib: ls -la
drwxr-xr-x Sep 19 14:46 .
drwxr-xr-x Oct 10 11:19 ..
drwxr-xr-x Oct 10 11:20 new_relic
-rw-r--r-- Sep 19 14:46 newrelic_rpm.rb
drwxr-xr-x Sep 19 14:46 sequel
drwxr-xr-x Oct 7 08:53 tasks

View Slide

Sequel.plugin NewRelic::Sequel::Instrumentation

View Slide

Generic

View Slide


View Slide

ActiveRecord

View Slide

ActiveRecord
DataMapper

View Slide

ActiveRecord
DataMapper
HTTPClient

View Slide

Qualify Yourself

View Slide

newrelic_rpm
module NewRelic
module Agent
class Thread < ::Thread
...
end
end
end

View Slide

elsewhere...
class Thread
# Re-opens NewRelic::Agent::Thread
# D'oh!
end

View Slide

newrelic_plugin
module NewRelic
class Logger
...
end
end

View Slide

newrelic_rpm
module NewRelic
module Agent
class AgentLogger
LOG_LEVELS = {
"debug" => Logger::DEBUG,
...
}
end
end
end

View Slide

newrelic_rpm
module NewRelic
module Agent
class AgentLogger
LOG_LEVELS = {
"debug" => ::Logger::DEBUG,
...
}
end
end
end

View Slide

Where We've Been
• Pass It On
• Events
• Middleware
• Lifecycle
• Names and Paths
• Conﬁg
• Docs

View Slide

Conﬁg

View Slide

yml

View Slide

yml + ERB

View Slide

yml + ERB
file = File.read("./config/awesome_gem.yml")
# Locals available in ERB
default_awesome_key = "YEAH!"
erb = ERB.new(file).result(binding)
config = YAML.load(erb)

View Slide

yml + ERB
~/config/awesome_gem.yml
development:
name: <%= default_awesome_key %>
license_key: <%= ENV["LICENSE_KEY"] %>

View Slide

Ruby

View Slide

unicorn
worker_processes 5
preload_app true
timeout 30
after_fork do |server, worker|
defined?(ActiveRecord::Base) and
end

View Slide

on_worker_boot do
ActiveSupport.on_load(:active_record) do
end
end

View Slide

Great for Apps?
What about Gems?

View Slide

newrelic_rpm
if defined?(::Puma) &&
::Puma.respond_to?(:cli_config)
config = ::Puma.cli_config
config.options[:worker_boot] << Proc.new do
NewRelic::Agent.after_fork(...)
end
end

View Slide

Where We've Been
• Pass It On
• Events
• Middleware
• Lifecycle
• Names and Paths
• Conﬁg
• Docs

View Slide

Docs

View Slide

README

View Slide

sinatra

View Slide


View Slide

Expected
Extensions?

View Slide


View Slide

Don't Bury It

View Slide


View Slide

Versioning

View Slide


View Slide

Where We've Been
• Pass It On
• Events
• Middleware
• Lifecycle
• Names and Paths
• Conﬁg
• Docs

View Slide

Where We've Been

View Slide

Where We've Been
• Pass It On
• Events
• Middleware
• Lifecycle
• Names and Paths
• Conﬁg
• Docs

View Slide

Where We've Been
• Pass It On
• Events
• Middleware
• Lifecycle
• Names and Paths
• Conﬁg
• Docs
Jason Clark
@jasonrclark
Ruby Agent Engineer

View Slide

Patterns of Pluggable Gems (ConFoo 2014)

Patterns of Pluggable Gems (ConFoo 2014)

Jason R Clark

More Decks by Jason R Clark

Other Decks in Technology

Featured

Transcript