Cache is King: Get the Most Bang for Your Buck From Ruby

Sometimes your fastest queries can cause the most problems. I will take you beyond the slow query optimization and instead zero in on the performance impacts surrounding the quantity of your datastore hits. Using real world examples dealing with datastores such as Elasticsearch, MySQL, and Redis, I will demonstrate how many fast queries can wreak just as much havoc as a few big slow ones. With each example I will make use of the simple tools available in Ruby to decrease and eliminate the need for these fast and seemingly innocuous datastore hits.

Transcript

1. Cache is King Get the Most Bang for Your Buck From Ruby
2. Site Reliability Engineer
4. Adding Indexes Using SELECT statements Batch Processing
5. Elasticsearch::Transport::Errors::GatewayTimeout 504 { "statusCode": 200, "took": "100ms" }
6. Resque
8. Demo Time!
9. Quantity of Datastore Hits
11. The average company has... 60 thousand assets 24 million vulnerabilities ?
12. MySQL Elasticsearch Cluster
13. Serialization
14. MySQL Elasticsearch Cluster ActiveModelSerializers
15. module Beehive module Serializers class Vulnerability < ActiveModel::Serializer attributes :id, :client_id, :created_at, :updated_at, :priority, :details, :notes, :asset_id, :solution_id, :owner_id, :ticket_id end end end
16. 200 MILLION
17. 11 hours and counting...
20. (1.6ms) (0.9ms) (4.1ms) (5.2ms) (5.2ms) (1.3ms) (3.1ms) (2.9ms) (2.2ms) (4.9ms) (6.0ms) (0.3ms) (1.6ms) (0.9ms) (2.2ms) (3.0ms) (2.1ms) (1.3ms) (2.1ms) (8.1ms) (1.4ms)
21. MySQL
22. Bulk Serialization
23. class BulkVulnerabilityCache attr_accessor :vulnerabilities, :client, :vulnerability_ids def initialize(vulns, client) self.vulnerabilities = vulns self.vulnerability_ids = vulns.map(&:id) self.client = client end # MySQL Lookups end
24. module Serializers class Vulnerability attr_accessor :vulnerability, :cache def initialize(vuln, bulk_cache) self.cache = bulk_cache self.vulnerability = vuln end end end self.cache = bulk_cache
25. class Vulnerability has_many :custom_fields end
26. CustomField.where(:vulnerability_id => vuln.id) cache.fetch('custom_fields', vuln.id)
27. The Result... (pry)> vulns = Vulnerability.limit(300); (pry)> Benchmark.realtime { vulns.each(&:serialize) } => 6.022452222998254 (pry)> Benchmark.realtime do > BulkVulnerability.new(vulns, [], client).serialize > end => 0.7267019419959979
28. Decrease in database hits Individual Serialization: Bulk Serialization: 2,100 7
29. 1k vulns 1k vulns 1k vulns Vulnerability Batches
30. 1k vulns 1k vulns 1k vulns Vulnerability Batches 7k 7
31. MySQL Queries Bulk Serialization Deployed
32. Bulk Serialization Deployed RDS CPU Utilization
33. Process in Bulk
36. Elasticsearch Cluster + Redis MySQL Vulnerabilities
37. Redis.get Client 1 Index Client 2 Index Client 3 & 4 Index
38. indexing_hashes = vulnerability_hashes.map do |hash| { :_index => Redis.get(“elasticsearch_index_#{hash[:client_id]}”) :_type => hash[:doc_type], :_id => hash[:id], :data => hash[:data] } end
39. indexing_hashes = vulnerability_hashes.map do |hash| { :_index => Redis.get(“elasticsearch_index_#{hash[:client_id]}”) :_type => hash[:doc_type], :_id => hash[:id], :data => hash[:data] } end
40. (pry)> index_name = Redis.get(“elasticsearch_index_#{client_id}”) DEBUG -- : [Redis] command=GET args="elasticsearch_index_1234" DEBUG -- : [Redis] call_time=1.07 ms GET
43. client_indexes = Hash.new do |h, client_id| h[client_id] = Redis.get(“elasticsearch_index_#{client_id}”) end
44. indexing_hashes = vuln_hashes.map do |hash| { :_index => Redis.get(“elasticsearch_index_#{client_id}”) :_type => hash[:doc_type], :_id => hash[:id], :data => hash[:data] } end client_indexes[hash[:client_id]],
45. 1 + 1 + 1 Client 1 Client 2 Client 3 1k 1k 1k
46. 1000x
47. 65% job speed up
48. Local Cache
49. Redis
50. Process in Bulk Hash Cache
51. Sharded Databases CLIENT 1 CLIENT 2 CLIENT 3
52. Asset.with_shard(client_id).find(1)
53. { 'client_123' => 'shard_123', 'client_456' => 'shard_456', 'client_789' => 'shard_789' } Sharding Configuration
55. Sharding Configuration Size 20 bytes 1kb 13kb
56. 285 Workers
57. 7.8 MB/second
58. ActiveRecord::Base.connection
59. (pry)> ActiveRecord::Base.connection => #<Octopus::Proxy:0x000055b38c697d10 @proxy_config= #<Octopus::ProxyConfig:0x000055b38c694ae8
60. module Octopus class Proxy attr_accessor :proxy_config delegate :current_shard, :current_shard=, :current_slave_group, :current_slave_group=, :shard_names, :shards_for_group, :shards, :sharded, :config, :initialize_shards, :shard_name, to: :proxy_config, prefix: false end end
61. Know your gems
62. Process in Bulk Framework Cache Hash Cache
63. Avoid making datastore hits you don’t need
64. User.where(:id => user_ids).each do |user| # Lots of user processing end
65. FALSE
66. (pry)> User.where(:id => []) User Load (1.0ms) SELECT `users`.* FROM `users` WHERE 1=0 => []
68. return unless user_ids.any? User.where(:id => user_ids).each do |user| # Lots of user processing end
69. (pry)> Benchmark.realtime do > 10_000.times { User.where(:id => []) } > end => 0.5508159045130014 (pry)> Benchmark.realtime do > 10_000.times do > next unless ids.any? > User.where(:id => []) > end > end => 0.0006368421018123627
70. (pry)> Benchmark.realtime do > 10_000.times { User.where(:id => []) } > end => 0.5508159045130014 “Ruby is slow” Hitting the database is slow!
71. User.where(:id => user_ids).each do |user| # Lots of user processing end
72. User.where(:id => user_ids).each do |user| # Lots of user processing end users = User.where(:id => user_ids).active.short.single
73. .none
74. (pry)> User.where(:id => []).active.tall.single User Load (0.7ms) SELECT `users`.* FROM `users` WHERE 1=0 AND `users`.`active` = 1 AND `users`.`short` = 0 AND `users`.`single` = 1 => [] (pry)> User.none.active.tall.single => [] .none in action...
77. Logging pry(main)> Rails.logger.level = 0 $ redis-cli monitor > commands-redis-2018-10-01.txt pry(main)> Search.connection.transport.logger = Logger.new(STDOUT)
78. Preventing useless datastore hits
80. Report Elasticsearch MySQL Redis
81. (pry)> Report.blank_reports.count => 10805 (pry)> Report.active.count => 25842 (pry)> Report.average_asset_count => 1657 Investigating Existing Reports
82. Report Elasticsearch MySQL Redis
84. 10+ hrs
85. 3 hrs
86. Process in Bulk Framework Cache Database Guards Hash Cache
87. Resque Workers Redis
88. 45 workers 45 workers 45 workers
89. 70 workers 70 workers 70 workers
93. 48 MB 16 MB
94. Redis Requests 70 workers 100k 200k
96. ?
97. Resque Throttling
98. Redis Requests 100k 200k
99. Redis Network Traffic 48MB 16MB
101. Process in Bulk Framework Cache Database Guards Remove Datastore Hits Hash Cache
102. Every datastore hit COUNTS
104. Questions
105. Contact https://www.linkedin.com/in/mollystruve/ https://github.com/mstruve @molly_struve molly.struve@gmail.com