Tweet
Tweet

Slide 1

Slide 1 text

Ruby Can Too Scale: Highly Performant Microservices in Ruby Tim Krajcar

Slide 2

Slide 2 text

April 2015

Slide 3

Slide 3 text

TimKrajcar \

Slide 4

Slide 4 text

TimKrajcar This document and the information herein (including any information that may be incorporated by reference) is provided for informational purposes only and should not be construed as an offer, commitment, promise or obligation on behalf of New Relic, Inc. (“New Relic”) to sell securities or deliver any product, material, code, functionality, or other feature. Any information provided hereby is proprietary to New Relic and may not be replicated or disclosed without New Relic’s express written permission. Such information may contain forward-looking statements within the meaning of federal securities laws. Any statement that is not a historical fact or refers to expectations, projections, future plans, objectives, estimates, goals, or other characterizations of future events is a forward-looking statement. These forward-looking statements can often be identified as such because the context of the statement will include words such as “believes,” “anticipates,” “expects” or words of similar import. Actual results may differ materially from those expressed in these forward-looking statements, which speak only as of the date hereof, and are subject to change at any time without notice. Existing and prospective investors, customers and other third parties transacting business with New Relic are cautioned not to place undue reliance on this forward-looking information. The achievement or success of the matters covered by such forward-looking statements are based on New Relic’s current assumptions, expectations, and beliefs and are subject to substantial risks, uncertainties, assumptions, and changes in circumstances that may cause the actual results, performance, or achievements to differ materially from those expressed or implied in any forward-looking statement. Further information on factors that could affect such forward-looking statements is included in the filings we make with the SEC from time to time. Copies of these documents may be obtained by visiting New Relic’s Investor Relations website at ir.newrelic.com or the SEC’s website at www.sec.gov. New Relic assumes no obligation and does not intend to update these forward-looking statements, except as required by law. New Relic makes no warranties, expressed or implied, in this document or otherwise, with respect to the information provided.

Slide 5

Slide 5 text

TimKrajcar

Slide 6

Slide 6 text

TimKrajcar why this talk?

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

TimKrajcar "ruby can't scale"

Slide 12

Slide 12 text

–Johnny Appleseed

Slide 13

Slide 13 text

–Johnny Appleseed

Slide 14

Slide 14 text

–Johnny Appleseed

Slide 15

Slide 15 text

TimKrajcar "ruby can't scale"

Slide 16

Slide 16 text

TimKrajcar "rails can't scale"

Slide 17

Slide 17 text

TimKrajcar second system effect

Slide 18

Slide 18 text

–Johnny Appleseed

Slide 19

Slide 19 text

–Johnny Appleseed and, even more importantly, stack

Slide 20

Slide 20 text

TimKrajcar why ruby?

Slide 21

Slide 21 text

TimKrajcar expressive programming language = fun to write code = I want to write more of it

Slide 22

Slide 22 text

TimKrajcar expressive programming language = fun to write code = I want to write more of it productivity!!

Slide 23

Slide 23 text

TimKrajcar things not covered

Slide 24

Slide 24 text

TimKrajcar 1. database tuning

Slide 25

Slide 25 text

TimKrajcar 1. database tuning 2. caching

Slide 26

Slide 26 text

TimKrajcar 1. database tuning 2. caching 3. network optimization

Slide 27

Slide 27 text

TimKrajcar 1. database tuning 2. caching 3. network optimization 4. browser-y things

Slide 28

Slide 28 text

TimKrajcar microservices at New Relic

Slide 29

Slide 29 text

TimKrajcar ~60 total services

Slide 30

Slide 30 text

TimKrajcar some not so micro

Slide 31

Slide 31 text

TimKrajcar Ruby: common for prototypes

Slide 32

Slide 32 text

TimKrajcar templates and auto- generation

Slide 33

Slide 33 text

–Johnny Appleseed

Slide 34

Slide 34 text

TimKrajcar stereotypical example: customer permissions service

Slide 35

Slide 35 text

–Johnny Appleseed

Slide 36

Slide 36 text

–Johnny Appleseed

Slide 37

Slide 37 text

PERFORMANCE FROM THE FIRST COMMIT

Slide 38

Slide 38 text

–Johnny Appleseed Structured programming with 'go to' statements Donald E. Knuth COMPUTING SURVEYS, DECEMBER 1974

Slide 39

Slide 39 text

–Johnny Appleseed We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil.

Slide 40

Slide 40 text

–Johnny Appleseed Yet we should not pass up our opportunities in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, [and] will be wise to look carefully at the critical code.

Slide 41

Slide 41 text

TimKrajcar benchmarking methodologies matter

Slide 42

Slide 42 text

TimKrajcar app server time vs request queuing

Slide 43

Slide 43 text

–Johnny Appleseed

Slide 44

Slide 44 text

–Johnny Appleseed

Slide 45

Slide 45 text

TimKrajcar X-Request-Start HTTP header

Slide 46

Slide 46 text

TimKrajcar test suite tools

Slide 47

Slide 47 text

TimKrajcar minitest-perf https://github.com/mrsimo/minitest-perf

Slide 48

Slide 48 text

> bundle exec minitest-perf Slowest individual tests 1210.00ms | NewRelic::Agent::AgentTest#test_after_fork_should_ 1190.78ms | LicenseTest#test_for_scary_license_terms 1034.58ms | NewRelic::Agent::Agent::ConnectTest#test_connect_g 1034.20ms | NewRelic::Agent::Agent::ConnectTest#test_logging_c 1032.60ms | NewRelic::Agent::Agent::ConnectTest#test_environme 1031.61ms | NewRelic::Agent::Agent::ConnectTest#test_connect_s 1030.49ms | NewRelic::Agent::Agent::ConnectTest#test_connect_s 730.88ms | OrphanedConfigTest#test_all_default_source_config_ 455.05ms | NewRelic::Agent::MetricStatsTest#test_record_scope 434.18ms | NewRelicServiceTest#test_json_marshaller_should_ha

Slide 49

Slide 49 text

Slowest test suites 638.87ms | 2 | LicenseTest 293.86ms | 3 | OrphanedConfigTest 193.41ms | 27 | NewRelic::Agent::Agent::ConnectTest 191.36ms | 1 | NewRelic::Agent::Threading::BacktraceServic 130.64ms | 14 | NewRelic::Agent::PipeChannelManagerTest 118.76ms | 14 | PipeServiceTest 111.14ms | 4 | DispatcherTest 32.82ms | 63 | NewRelic::Agent::AgentTest 25.50ms | 20 | NewRelic::Agent::MetricStatsTest 19.37ms | 12 | NewRelic::Agent::UtilizationDataTest

Slide 50

Slide 50 text

TimKrajcar framework choices

Slide 51

Slide 51 text

TimKrajcar framework choices reductions

Slide 52

Slide 52 text

–Johnny Appleseed

Slide 53

Slide 53 text

TimKrajcar example!!

Slide 54

Slide 54 text

TimKrajcar a typical Ruby microservice: REST DB-to-JSON API https://github.com/tkrajcar/ruby-can-too-scale

Slide 55

Slide 55 text

TimKrajcar a typical Ruby microservice: REST DB-to-JSON API https://github.com/tkrajcar/ruby-can-too-scale !! warning: YOLOCODE !!

Slide 56

Slide 56 text

TimKrajcar •Rails 4.2.6 straight outta rails new + Puma + nginx •JSON only, using scaffolded controller & model •testing #index (~25kb) and #show (<1 kb) •read-only •Dedicated EC2 c3.large instances (2 vCPUs) •Postgres on its own dedicated RDS instance •30 users as fast as possible (separate instance)

Slide 57

Slide 57 text

TimKrajcar best metric: client requests per minute

Slide 58

Slide 58 text

TimKrajcar #index /articles.json #show /articles/1.json Stock Rails scaffolding 4,211 18,011

Slide 59

Slide 59 text

TimKrajcar #index /articles.json #show /articles/1.json Stock Rails scaffolding 4,211 18,011 No jbuilder templates 4,561 +8.3% 18,909 +4.9%

Slide 60

Slide 60 text

TimKrajcar #index /articles.json #show /articles/1.json Stock Rails scaffolding 4,211 18,011 No jbuilder templates 4,561 +8.3% 18,909 +4.9% oj gem 5,548 +21.6% 19,714 +9.4%

Slide 61

Slide 61 text

TimKrajcar #index /articles.json #show /articles/1.json Stock Rails scaffolding 4,211 18,011 No jbuilder templates 4,561 +8.3% 18,909 +4.9% oj gem 5,548 +21.6% 19,714 +9.4% Sequel instead of ActiveRecord 7,035 +26.8% 22,378 +13.5%

Slide 62

Slide 62 text

TimKrajcar #index /articles.json #show /articles/1.json Stock Rails scaffolding 4,211 18,011 No jbuilder templates 4,561 +8.3% 18,909 +4.9% oj gem 5,548 +21.6% 19,714 +9.4% Sequel instead of ActiveRecord 7,035 +26.8% 22,378 +13.5% Sinatra (with Sequel and oj) 13,245 85,017

Slide 63

Slide 63 text

TimKrajcar #index /articles.json #show /articles/1.json Stock Rails scaffolding 4,211 18,011 No jbuilder templates 4,561 +8.3% 18,909 +4.9% oj gem 5,548 +21.6% 19,714 +9.4% Sequel instead of ActiveRecord 7,035 +26.8% 22,378 +13.5% Sinatra (with Sequel and oj) 13,245 85,017

Slide 64

Slide 64 text

TimKrajcar tl;dr: use Sinatra and Sequel

Slide 65

Slide 65 text

DIVING DEEPER

Slide 66

Slide 66 text

TimKrajcar Derailed Benchmarks https://github.com/schneems/derailed_benchmarks

Slide 67

Slide 67 text

> bundle exec derailed bundle:mem TOP: 46.5625 MiB activeadmin: 17.5156 MiB active_admin: 17.5039 MiB meta_search: 6.2773 MiB meta_search/searches/active_record: 1.707 MiB meta_search/builder: 1.5938 MiB meta_search/where: 1.0977 MiB polyamorous: 0.4688 MiB active_admin/comments: 3.6055 MiB active_admin/comments/views: 1.375 MiB active_admin/views: 1.2305 MiB (Also required by: active_admin/comments/views/active_admin_comments) /Users/tkrajcar/.rbenv/versions/2.1.7/lib/ruby/gems/ 2.1.0/gems/activeadmin-0.6.3/lib/active_admin/views/components/ paginated_collection.rb: 0.3125 MiB bourbon: 3.3438 MiB

Slide 68

Slide 68 text

> bundle exec derailed bundle:objects Measuring objects created by gems in groups [:default, "production"] Total allocated: 72414462 bytes (425465 objects) Total retained: 24736698 bytes (44790 objects) allocated memory by gem ----------------------------------- 27407703 activesupport-3.2.22.2 9240545 erubis-2.7.0 5722758 actionpack-3.2.22.2 3287857 activerecord-3.2.22.2 3176375 haml-4.0.5 1728840 meta_search-1.1.3 1300638 devise-1.5.4 1216006 2.1.7/lib 1016914 formtastic-2.2.1 993225 activeadmin-0.6.3

Slide 69

Slide 69 text

TimKrajcar Stackprof https://github.com/tmm1/stackprof

Slide 70

Slide 70 text

–Johnny Appleseed

Slide 71

Slide 71 text

TimKrajcar other interpreters

Slide 72

Slide 72 text

DESIGNING TO AVOID CASCADING FAILURES

Slide 73

Slide 73 text

TimKrajcar timeouts

Slide 74

Slide 74 text

–Johnny Appleseed

Slide 75

Slide 75 text

No content

Slide 76

Slide 76 text

TimKrajcar Timeout

Slide 77

Slide 77 text

require 'timeout' result = begin Timeout::timeout(0.5) { do_a_possibly_slow_thing } rescue Timeout::Error # log/notify about the timeout 0 end

Slide 78

Slide 78 text

No content

Slide 79

Slide 79 text

No content

Slide 80

Slide 80 text

No content

Slide 81

Slide 81 text

TimKrajcar circuit breaker pattern

Slide 82

Slide 82 text

–Johnny Appleseed http://martinfowler.com/bliki/CircuitBreaker.html

Slide 83

Slide 83 text

TimKrajcar cb2 https://github.com/pedro/cb2

Slide 84

Slide 84 text

require 'cb2' breaker = CB2::Breaker.new( service: "cashwings", # identify each circuit breaker individually duration: 60, # in this amount of seconds... threshold: 5, # ...this % of errors will open the breaker reenable_after: 600, # once opened, how long until we try again redis: Redis.new) # uses Redis to keep track by service name

Slide 85

Slide 85 text

require 'cb2' breaker = CB2::Breaker.new( service: "cashwings", # identify each circuit breaker individually duration: 60, # in this amount of seconds... threshold: 5, # ...this % of errors will open the breaker reenable_after: 600, # once opened, how long until we try again redis: Redis.new) # uses Redis to keep track by service name

Slide 86

Slide 86 text

begin breaker.run do some_api_request() end rescue CB2::BreakerOpen # log/notify about the breaker opening! alternate_response() end

Slide 87

Slide 87 text

TimKrajcar parallelize

Slide 88

Slide 88 text

TimKrajcar parallelize everything especially HTTP

Slide 89

Slide 89 text

TimKrajcar Typhoeus::Hydra https://github.com/typhoeus/typhoeus

Slide 90

Slide 90 text

require 'typhoeus' request1 = Typhoeus::Request.new("http://example.com/1.json") request2 = Typhoeus::Request.new("http://example.com/2.json") hydra = Typhoeus::Hydra.new hydra.queue(request1) hydra.queue(request2) hydra.run # blocks here until queue finishes puts "Request 1 status #{request1.response.code}" puts "Request 2 status #{request2.response.code}"

Slide 91

Slide 91 text

TimKrajcar background everything* *unless you can't

Slide 92

Slide 92 text

TimKrajcar https://github.com/mperham/sidekiq

Slide 93

Slide 93 text

# app/workers/welcome_emailer.rb class WelcomeEmailer include Sidekiq::Worker def perform(email) send_email_to(email, "Welcome!") end end # app/controllers/welcome_controller.rb def welcome # blah blah create a @user WelcomeEmailer.perform_async(@user.email) end

Slide 94

Slide 94 text

# app/workers/welcome_emailer.rb class WelcomeEmailer include Sidekiq::Worker def perform(email) send_email_to(email, "Welcome!") end end # app/controllers/welcome_controller.rb def welcome # blah blah create a @user WelcomeEmailer.perform_async(@user.email) end doesn't block

Slide 95

Slide 95 text

TimKrajcar rate limit by client

Slide 96

Slide 96 text

TimKrajcar Rack::Throttle https://github.com/bendiken/rack-throttle

Slide 97

Slide 97 text

# config/application.rb require 'rack/throttle' class Application < Rails::Application config.middleware.use Rack::Throttle::Hourly, max: 100 end

Slide 98

Slide 98 text

# config/application.rb require 'rack/throttle' class Application < Rails::Application config.middleware.use Rack::Throttle::Hourly, max: 100 end client = by IP or you can define custom client IDs

Slide 99

Slide 99 text

TimKrajcar monitoring & alerting

Slide 100

Slide 100 text

TimKrajcar low traffic? drive synthetic traffic

Slide 101

Slide 101 text

TimKrajcar synthetic-monitor https://github.com/johnboyes/synthetic-monitor

Slide 102

Slide 102 text

No content

Slide 103

Slide 103 text

TimKrajcar RECAP!!

Slide 104

Slide 104 text

TimKrajcar 1. use Ruby to write services, because it's fun! 2. monitor both request queuing time and app server time 3. make informed framework decisions (use sinatra) 4. set real timeouts and use circuit breakers 5. parallelize everything 6. background everything 7. rate limit by client, even for internal clients 8. drive synthetic traffic to help catch problems right away

Slide 105

Slide 105 text

TimKrajcar references

Slide 106

Slide 106 text

TimKrajcar Nate Berkopec: nateberkopec.com High Scalability: highscalability.com Improved Production Stability With Circuit Breakers: blog.heroku.com

Slide 107

Slide 107 text

TimKrajcar thanks!