Cache is King: Get the Most Bang for Your Buck From Ruby

Slide 1

Slide 1 text

Cache is King Get the Most Bang for Your Buck From Ruby

Slide 2

Slide 2 text

Site Reliability Engineer

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Adding Indexes Using SELECT statements Batch Processing

Slide 5

Slide 5 text

Elasticsearch::Transport::Errors::GatewayTimeout 504 { "statusCode": 200, "took": "100ms" }

Slide 6

Slide 6 text

Resque

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

Demo Time!

Slide 9

Slide 9 text

Quantity of Datastore Hits

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

The average company has... 60 thousand assets 24 million vulnerabilities ?

Slide 12

Slide 12 text

MySQL Elasticsearch Cluster

Slide 13

Slide 13 text

Serialization

Slide 14

Slide 14 text

MySQL Elasticsearch Cluster ActiveModelSerializers

Slide 15

Slide 15 text

module Beehive module Serializers class Vulnerability < ActiveModel::Serializer attributes :id, :client_id, :created_at, :updated_at, :priority, :details, :notes, :asset_id, :solution_id, :owner_id, :ticket_id end end end

Slide 16

Slide 16 text

200 MILLION

Slide 17

Slide 17 text

11 hours and counting...

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

(1.6ms) (0.9ms) (4.1ms) (5.2ms) (5.2ms) (1.3ms) (3.1ms) (2.9ms) (2.2ms) (4.9ms) (6.0ms) (0.3ms) (1.6ms) (0.9ms) (2.2ms) (3.0ms) (2.1ms) (1.3ms) (2.1ms) (8.1ms) (1.4ms)

Slide 21

Slide 21 text

MySQL

Slide 22

Slide 22 text

Bulk Serialization

Slide 23

Slide 23 text

class BulkVulnerabilityCache attr_accessor :vulnerabilities, :client, :vulnerability_ids def initialize(vulns, client) self.vulnerabilities = vulns self.vulnerability_ids = vulns.map(&:id) self.client = client end # MySQL Lookups end

Slide 24

Slide 24 text

module Serializers class Vulnerability attr_accessor :vulnerability, :cache def initialize(vuln, bulk_cache) self.cache = bulk_cache self.vulnerability = vuln end end end self.cache = bulk_cache

Slide 25

Slide 25 text

class Vulnerability has_many :custom_fields end

Slide 26

Slide 26 text

CustomField.where(:vulnerability_id => vuln.id) cache.fetch('custom_fields', vuln.id)

Slide 27

Slide 27 text

The Result... (pry)> vulns = Vulnerability.limit(300); (pry)> Benchmark.realtime { vulns.each(&:serialize) } => 6.022452222998254 (pry)> Benchmark.realtime do > BulkVulnerability.new(vulns, [], client).serialize > end => 0.7267019419959979

Slide 28

Slide 28 text

Decrease in database hits Individual Serialization: Bulk Serialization: 2,100 7

Slide 29

Slide 29 text

1k vulns 1k vulns 1k vulns Vulnerability Batches

Slide 30

Slide 30 text

1k vulns 1k vulns 1k vulns Vulnerability Batches 7k 7

Slide 31

Slide 31 text

MySQL Queries Bulk Serialization Deployed

Slide 32

Slide 32 text

Bulk Serialization Deployed RDS CPU Utilization

Slide 33

Slide 33 text

Process in Bulk

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

Elasticsearch Cluster + Redis MySQL Vulnerabilities

Slide 37

Slide 37 text

Redis.get Client 1 Index Client 2 Index Client 3 & 4 Index

Slide 38

Slide 38 text

indexing_hashes = vulnerability_hashes.map do |hash| { :_index => Redis.get(“elasticsearch_index_#{hash[:client_id]}”) :_type => hash[:doc_type], :_id => hash[:id], :data => hash[:data] } end

Slide 39

Slide 39 text

indexing_hashes = vulnerability_hashes.map do |hash| { :_index => Redis.get(“elasticsearch_index_#{hash[:client_id]}”) :_type => hash[:doc_type], :_id => hash[:id], :data => hash[:data] } end

Slide 40

Slide 40 text

(pry)> index_name = Redis.get(“elasticsearch_index_#{client_id}”) DEBUG -- : [Redis] command=GET args="elasticsearch_index_1234" DEBUG -- : [Redis] call_time=1.07 ms GET

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

client_indexes = Hash.new do |h, client_id| h[client_id] = Redis.get(“elasticsearch_index_#{client_id}”) end

Slide 44

Slide 44 text

indexing_hashes = vuln_hashes.map do |hash| { :_index => Redis.get(“elasticsearch_index_#{client_id}”) :_type => hash[:doc_type], :_id => hash[:id], :data => hash[:data] } end client_indexes[hash[:client_id]],

Slide 45

Slide 45 text

1 + 1 + 1 Client 1 Client 2 Client 3 1k 1k 1k

Slide 46

Slide 46 text

1000x

Slide 47

Slide 47 text

65% job speed up

Slide 48

Slide 48 text

Local Cache

Slide 49

Slide 49 text

Redis

Slide 50

Slide 50 text

Process in Bulk Hash Cache

Slide 51

Slide 51 text

Sharded Databases CLIENT 1 CLIENT 2 CLIENT 3

Slide 52

Slide 52 text

Asset.with_shard(client_id).find(1)

Slide 53

Slide 53 text

{ 'client_123' => 'shard_123', 'client_456' => 'shard_456', 'client_789' => 'shard_789' } Sharding Configuration

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

Sharding Configuration Size 20 bytes 1kb 13kb

Slide 56

Slide 56 text

285 Workers

Slide 57

Slide 57 text

7.8 MB/second

Slide 58

Slide 58 text

ActiveRecord::Base.connection

Slide 59

Slide 59 text

(pry)> ActiveRecord::Base.connection => #

Slide 60

Slide 60 text

module Octopus class Proxy attr_accessor :proxy_config delegate :current_shard, :current_shard=, :current_slave_group, :current_slave_group=, :shard_names, :shards_for_group, :shards, :sharded, :config, :initialize_shards, :shard_name, to: :proxy_config, prefix: false end end

Slide 61

Slide 61 text

Know your gems

Slide 62

Slide 62 text

Process in Bulk Framework Cache Hash Cache

Slide 63

Slide 63 text

Avoid making datastore hits you don’t need

Slide 64

Slide 64 text

User.where(:id => user_ids).each do |user| # Lots of user processing end

Slide 65

Slide 65 text

FALSE

Slide 66

Slide 66 text

(pry)> User.where(:id => []) User Load (1.0ms) SELECT `users`.* FROM `users` WHERE 1=0 => []

Slide 67

Slide 67 text

No content

Slide 68

Slide 68 text

return unless user_ids.any? User.where(:id => user_ids).each do |user| # Lots of user processing end

Slide 69

Slide 69 text

(pry)> Benchmark.realtime do > 10_000.times { User.where(:id => []) } > end => 0.5508159045130014 (pry)> Benchmark.realtime do > 10_000.times do > next unless ids.any? > User.where(:id => []) > end > end => 0.0006368421018123627

Slide 70

Slide 70 text

(pry)> Benchmark.realtime do > 10_000.times { User.where(:id => []) } > end => 0.5508159045130014 “Ruby is slow” Hitting the database is slow!

Slide 71

Slide 71 text

User.where(:id => user_ids).each do |user| # Lots of user processing end

Slide 72

Slide 72 text

User.where(:id => user_ids).each do |user| # Lots of user processing end users = User.where(:id => user_ids).active.short.single

Slide 73

Slide 73 text

.none

Slide 74

Slide 74 text

(pry)> User.where(:id => []).active.tall.single User Load (0.7ms) SELECT ùsers`.* FROM ùsers` WHERE 1=0 AND ùsers`.àctive` = 1 AND ùsers`.`short` = 0 AND ùsers`.`single` = 1 => [] (pry)> User.none.active.tall.single => [] .none in action...

Slide 75

Slide 75 text

No content

Slide 76

Slide 76 text

No content

Slide 77

Slide 77 text

Logging pry(main)> Rails.logger.level = 0 $ redis-cli monitor > commands-redis-2018-10-01.txt pry(main)> Search.connection.transport.logger = Logger.new(STDOUT)

Slide 78

Slide 78 text

Preventing useless datastore hits

Slide 79

Slide 79 text

No content

Slide 80

Slide 80 text

Report Elasticsearch MySQL Redis

Slide 81

Slide 81 text

(pry)> Report.blank_reports.count => 10805 (pry)> Report.active.count => 25842 (pry)> Report.average_asset_count => 1657 Investigating Existing Reports

Slide 82

Slide 82 text

Report Elasticsearch MySQL Redis

Slide 83

Slide 83 text

No content

Slide 84

Slide 84 text

10+ hrs

Slide 85

Slide 85 text

3 hrs

Slide 86

Slide 86 text

Process in Bulk Framework Cache Database Guards Hash Cache

Slide 87

Slide 87 text

Resque Workers Redis

Slide 88

Slide 88 text

45 workers 45 workers 45 workers

Slide 89

Slide 89 text

70 workers 70 workers 70 workers

Slide 90

Slide 90 text

No content

Slide 91

Slide 91 text

No content

Slide 92

Slide 92 text

No content

Slide 93

Slide 93 text

48 MB 16 MB

Slide 94

Slide 94 text

Redis Requests 70 workers 100k 200k

Slide 95

Slide 95 text

No content

Slide 96

Slide 96 text

Slide 97

Slide 97 text

Resque Throttling

Slide 98

Slide 98 text

Redis Requests 100k 200k

Slide 99

Slide 99 text

Redis Network Traffic 48MB 16MB

Slide 100

Slide 100 text

No content

Slide 101

Slide 101 text

Process in Bulk Framework Cache Database Guards Remove Datastore Hits Hash Cache

Slide 102

Slide 102 text

Every datastore hit COUNTS

Slide 103

Slide 103 text

No content

Slide 104

Slide 104 text

Questions

Slide 105

Slide 105 text

Contact https://www.linkedin.com/in/mollystruve/ https://github.com/mstruve @molly_struve [email protected]