Tokyo RubyKaigi 12 - Scaling Ruby at GitHub

Transcript

1. Scaling Ruby @ @jhawthorn
2. John Hawthorn GitHub: @jhawthorn Twi tt er: @jhawthorn Bluesky: @jhawthorn.com Mastodon: @john@hawthorn.pub δϣϯ ϗʔιʔϯ Sta ff Software Engineer @ GitHub Ruby commi tt er Rails core team
3. • Joined GitHub in 2018 • "Ruby Architecture" Team • Developing & upgrading Ruby • Developing & upgrading Rails • Integrating Ruby/Rails into GitHub • Some database stu f
4. Scaling Ruby @ @jhawthorn
6. Scaling users Scaling teams Scaling
7. Scaling users Scaling teams Customers Tra ff ic Servers Developers Lines of code Features
8. GitHub's beginning + +
9. Ruby upgrades Ruby 1.8 Ruby 2.3 2.0 (fork) 2.1 (fork)
10. Rails upgrades Rails 1.2 Rails 2.3 2.3 (fork) 3.0 (fork) Rails 4.2 Rails 5.2 4.2
11. Today GitHub is running Rails 8.1.0.alpha in production!
12. We upgrade Rails most weeks
13. Scheduled GitHub action update Rails from main branch Open pull request
14. Today GitHub is running Ruby 3.4.1
15. GitHub runs with YJIT ~15-20% performance boost Since Ruby 3.3
16. We upgrade Ruby most weeks (in CI only, production runs stable Ruby)
17. Smaller changes Upgrading weekly Stay closer to upstream Encourages contributing upstream
18. It's easier to see the language and framework as part of our application
19. A mostly normal Ruby/Rails application "The Monolith"
20. "The Monolith" ~3.8 million lines of code ~2 million lines of tests >1000 developers 2500 pull requests / month
21. Problems: I can't read 3.8 million lines of code I can't review 2500 pull requests / month
22. CODEOWNERS h tt ps://github.blog/engineering/architecture-optimization/how-we-organize-and-get-things-done-with-serviceowners/ # CODEOWNERS app/models/issues*.rb @github/issues-team app/models/user.rb @github/users-team app/models/profile.rb @github/users-team
24. Error A tt ribution ActiveRecord::StatementInvalid active_record/lib/active_record/query.rb:123 in `query` app/models/user.rb:124:in 'Array#each' app/models/user.rb:123:in 'find_user' app/models/issue.rb:123:in 'some_user' app/controllers/issues_controller.rb:123:in 'show' lib/middleware/cache_middleware.rb:123:in 'call'
25. ActiveRecord::StatementInvalid active_record/lib/active_record/query.rb:123 in `query` app/models/user.rb:124:in 'Array#each' app/models/user.rb:123:in 'find_user' app/models/issue.rb:123:in 'some_user' app/controllers/issues_controller.rb:123:in 'show' lib/middleware/cache_middleware.rb:123:in 'call' Error A tt ribution
26. ActiveRecord::StatementInvalid active_record/lib/active_record/query.rb:123 in `query` app/models/user.rb:124:in 'Array#each' app/models/user.rb:123:in 'find_user' app/models/issue.rb:123:in 'some_user' app/controllers/issues_controller.rb:123:in 'show' lib/middleware/cache_middleware.rb:123:in 'call' user.rb "users" service @github/users-team Error A tt ribution
27. ActiveRecord::StatementInvalid active_record/lib/active_record/query.rb:123 in `query` app/models/user.rb:124:in 'Array#each' app/models/user.rb:123:in 'find_user' app/models/issue.rb:123:in 'some_user' app/controllers/issues_controller.rb:123:in 'show' lib/middleware/cache_middleware.rb:123:in 'call' issues_controller.rb "issues" service @github/issues-team Error A tt ribution
28. ActiveRecord::StatementInvalid active_record/lib/active_record/query.rb:123 in `query` app/models/user.rb:124:in 'Array#each' app/models/user.rb:123:in 'find_user' app/models/issue.rb:123:in 'some_user' app/controllers/issues_controller.rb:123:in 'show' lib/middleware/cache_middleware.rb:123:in 'call' service: users (@github/users-team) cause service: issues (@github/issues-team) Error A tt ribution
29. Not everything can be divided by file
30. Remove OpenStruct Goal:
31. # test/linters/openstruct_test.rb openstruct_count = grep(/OpenStruct/).count assert_equal 0, openstruct_count Remove OpenStruct # CODEOWNERS test/linters/openstruct_test.rb @github/ruby-architecture Our team has to approve any new OpenStruct 50
32. # test/linters/openstruct_test.rb openstruct_count = grep(/OpenStruct/).count if openstruct_count > 50 flunk "OpenStruct has performance issues." \ "Please use Struct or a plain Ruby class instead." elsif openstruct_count < 50 flunk "You removed an OpenStruct. Thank you!" \ "Please update the counter in #{__FILE__}" end "Ratchet" technique 🔧
33. 😬 Hard to review Likely to have merge con fl icts commit: "Remove OpenStruct everywhere" +1243 -1243
34. commit: "Remove OpenStruct everywhere" +1243 -1243 script/create-prs-by-codeowner commit: "Remove OpenStruct from issues" +30 -30 commit: "Remove OpenStruct from users" +20 -20 ...
35. Deploying https://github.blog/engineering/engineering-principles/how-github-uses-merge-queue-to-ship-hundreds-of-changes-every-day/ ~2500 pull requests / month ~20 deploys a day
36. Branch deploys https://github.blog/engineering/engineering-principles/deploying-branches-to-github-com/ 1. git branch some_branch 2. Commit and push work on some_branch 3. Deploy some_branch to production 4. Merge some_branch to main "main is always production-ready"
37. Branch deploys jhawthorn .deploy my_branch to github hubot <defunkt> is already deploying
38. Deploy queue jhawthorn .queue me to deploy my_branch hubot There are two PRs ahead of you
39. Merge queue https://github.blog/engineering/engineering-principles/how-github-uses-merge-queue-to-ship-hundreds-of-changes-every-day/
40. If one PR has a bug The whole merge group fails The next group is delayed (waiting for the rollback, waiting for CI, etc)
41. As your application grows the cost of a bug grows Users impacted Developers impacted
42. Feature fl ags
43. Feature fl ags if Flipper.enabled?(:some_feature) 100% enabled true Enabled 50% of the time rand() < 0.5 Gates "dark shipping" 0% enabled false
44. Feature fl ags if Flipper.enabled?(:some_feature, current_user) actor actor could be any type User Organization @jhawthorn @ruby Repository @ruby/ruby
45. Feature fl ags if Flipper.enabled?(:some_feature, current_user) actor Enable for 10% of users crc(actor.id) % 10 == 0 Enable "sta ff " group actor.staff? Gates
46. ENV "Feature fl ags" if ENV["ENABLE_YJIT"] == "1" RubyVM::YJIT.enable end
47. ENV "Feature fl ags" if rand() < ENV["ENABLE_YJIT_PCT"].to_f / 100.0 RubyVM::YJIT.enable end
48. ENV "Feature fl ags" hostname = Socket.gethostname hostname_rand = Zlib.crc32(hostname) / 2.0 ** 32 if hostname_rand < ENV["ENABLE_YJIT_PCT"].to_f / 100.0 RubyVM::YJIT.enable end
49. Scientist h tt ps://github.com/github/scientist Compare two implementations in production
50. Scientist users_orig = original_code(args) return users_orig
51. Scientist users_orig = original_code(args) users_new = new_code(args) return users_orig
52. Scientist users_orig = original_code(args) users_new = new_code(args) unless users_new == users_orig report_error "users didn't match" end return users_orig
53. Scientist h tt ps://github.com/github/scientist science "users" do |e| e.use { original_code(args) } e.try { new_code(args) } end Runs every time. Result returned Controlled by feature fl ag Result compared to "use" block Reports metrics of both! Mismatches recorded
56. Similar open source tool: rack-mini-profiler
57. Profilers Features SQL Vernier stackprof allocation_tracer
58. Trilogy τϦϩδʔ MySQL-compatible database client
60. Trilogy • MySQL-compatible database client • Fast! • No dependency on libmysqlclient/libmariadb • Automatic casting • Embedding friendly https://github.blog/open-source/maintainers/introducing-trilogy-a-new-database-adapter-for-ruby-on-rails/
61. Database Scaling 🔥 x 1000 app DB https://github.blog/engineering/infrastructure/partitioning-githubs-relational-databases-scale/ app app
62. DB Scaling - Replication
63. DB Scaling - Replication Primary Replicas read-only
64. DB Scaling - Replication Primary Replicas SELECT * SELECT * SELECT * INSERT ... read-only
65. DB Scaling - Replication Primary Replicas read-only May have stale data delay: < 1s when healthy
66. DB Scaling - Replication class ApplicationRecord self.abstract_class = true connects_to database: { writing: :primary, reading: :primary_replica } end
67. DB Scaling - Replication Queries the replica SELECT * FROM users WHERE id = 123 ActiveRecord::Base.connected_to(role: :reading) do User.find(123) end Available since Rails 6.0.0! https://github.com/rails/rails/pull/34052 by @eileencodes
68. DB Scaling - Replication Primary Replicas anon logged in
69. DB Scaling - Replication Primary Replicas anon logged in
70. DB Scaling - Replication Primary Replicas anon logged in 50% 50% 50% 50%
71. DB Scaling - Replication Primary Replicas 2x scale 50% 50%
72. Can we do be tt er? Primary Replicas logged in
73. Primary Replicas logged in < 1s delay Write in last 1s? yes no "Read your own write"
74. ❯ rails g active_record:multi_db create config/initializers/multi_db.rb # config/initializers/multi_db.rb Rails.application.configure do config.active_record.database_selector = { delay: 2.seconds } end
75. https://github.blog/engineering/infrastructure/mitigating-replication-lag-and-reducing-read-load-with-freno/ https://github.com/github/freno-client https://github.com/github/freno
76. Primary Replicas logged in Is data replicated? yes no "Read your own write" w/ freno 98% 2% 50x scale!
77. There's much more sharding proxies etc more queries to replicas
78. Database Retries
79. Replicas retry!
80. (mostly) Default in Rails 7.1+ # Hack: assume all queries are retryable def raw_execute(*args, **kwargs) super(*args, **kwargs.merge(allow_retry: true)) end
81. # database.yml production: adapter: trilogy connection_retries: 2 retry_deadline: 2 Stop retrying after 2 seconds 2 retries (default 1)
82. Caution! ⚠ Retries can hide unhealthy systems ⚠ Retries + scale + problem = self DoS
83. Circuit breakers errors more often but faster
84. circuit_breaker = CircuitBreaker.get("key") if circuit_breaker.allow_request? begin # do something expensive circuit_breaker.success rescue => e circuit_breaker.failure # do fallback end else # do fallback end After a certain number of failures Disallow requests
85. class ResilientTrilogy def query(...) if @circuit_breaker.allow_request? begin ret = @trilogy.query(...) @circuit_breaker.success ret rescue ConnectionError => e @circuit_breaker.failure raise end else raise CircuitOpenError end end end
86. class ResilientTrilogy def query(...) if @circuit_breaker.allow_request? begin ret = @trilogy.query(...) @circuit_breaker.success ret rescue ConnectionError => e @circuit_breaker.failure raise end else raise CircuitOpenError end end end No fallback, but allows systems to recover
87. A sample of Ruby at scale
88. Language, framework, and gems are a part of our application
89. Let's scale Ruby together
90. Thank you ͋Γ͕ͱ͏͍͟͝·͢