[RailsConf 2019] Terraforming legacy Rails applications

Video: https://www.youtube.com/watch?v=-NKpMn6XSjU
RailsConf: https://railsconf.com/program/sessions#session-832
GitHub: https://github.com/evilmartians/terraforming-rails

Rails has been around for (can you imagine!) about 15 years. Most Rails applications are no longer MVPs, but they grew from MVPs and usually contain a lot of legacy code that "just works."

And this legacy makes shipping new features harder and riskier: the new functionality have to co-exist with the code written years ago, and who knows what will blow up next?

I've been working on legacy codebases for the last few years, and I found turning legacy code into a legendary code to be a very exciting and challenging process.

I want to share the ideas and techniques I apply to make legacy codebases habitable and to prepare a breeding ground for the new features.

Transcript

1. TERRAFORMING LEGACY RAILS APPLICATIONS
2. palkan_tula palkan RailsConf 2019 2 @palkan @palkan_tula Vladimir Dementyev 933 722 406 504
3. palkan_tula palkan RailsConf 2019 3 evilmartians.com Brooklyn, NY Moscow, Russia
5. palkan_tula palkan RailsConf 2019 evl.ms/blog 5
6. palkan_tula palkan RailsConf 2019 evilmartians.com 6
7. palkan_tula palkan RailsConf 2019 FROM SCRATCH TWO TYPES OF PROJECTS 7 “Build us something cool!” SpaceX Star Hopper
8. palkan_tula palkan RailsConf 2019 FROM LEGACY TWO TYPES OF PROJECTS 8 “We’ve built something, make it cool!” Pepelats from Kin-dza-dza! (Soviet sci-fi) That’s my jam!
9. palkan_tula palkan RailsConf 2019 LEGACY 9 Jason Swett, RubyHACK 2019
10. palkan_tula palkan RailsConf 2019 10 TEAM LEAD Evaluate project Make ready for others to join Prepare guidelines Captain Zelyonyy (Green), The Mystery of the third planet
11. THIS TALK
12. palkan_tula palkan RailsConf 2019 12 #BoardGameGeek NOT ONLY A DEVELOPER
13. palkan_tula palkan RailsConf 2019 13 TERRAFORMING MARS “Make Mars habitable!” Jacob Fryxelius, 2016
14. palkan_tula palkan RailsConf 2019 TERRAFORMING MARS 14 Corporations are competing to transform Mars into a habitable planet by raising temperature, creating a breathable atmosphere, and making oceans of water. As terraforming progresses, more and more people will immigrate from Earth to live on the Red Planet.
15. palkan_tula palkan RailsConf 2019 TERRAFORMING CODE 15 Engineers are transforming the legacy code into a habitable code by optimizing development process, adding confidence with tests and linters and creating a codeable atmosphere. As terraforming progresses, more and more people could join the project and work on new features.
16. palkan_tula palkan RailsConf 2019 16 MARS LANDING Setting up the project PHASE #1
17. palkan_tula palkan RailsConf 2019 17 GOAL PHASE #1 “Developers should be able to run project with the least possible effort”
18. palkan_tula palkan RailsConf 2019 18 // Make running project locally as easy as running a few commands $ git clone project/repo.git $ cd repo $ <dev env setup and provision> $ rails c Loading development environment 2.6.2 (main):0> PHASE #1 GOAL
19. DEVELOPMENT ENVIRONMENT
20. palkan_tula palkan RailsConf 2019 “OLD SCHOOL” DEV ENV SETUP 20 Dev setup instructions from yet another Rails project
21. palkan_tula palkan RailsConf 2019 “NEW SCHOOL” 21
22. palkan_tula palkan RailsConf 2019 DOCKER FOR DEV 22 Repeatable and predictable setup Always in sync
23. palkan_tula palkan RailsConf 2019 KEEPING ’N SYNC 23
24. palkan_tula palkan RailsConf 2019 DOCKER FOR DEV 24 Repeatable and predictable setup Always in sync Painfully slow on Mac?
25. palkan_tula palkan RailsConf 2019 SLOW D4M 25 Use :cached folders Store assets/bundles in volumes Use NFS (if the first two are not enough) bit.ly/d4m-nfs
26. palkan_tula palkan RailsConf 2019 DOCKER FOR DEV 26 Repeatable and predictable setup Always in sync Painfully slow on Mac Windows "
27. palkan_tula palkan RailsConf 2019 # provision application $ dip provision # run web app $ dip rails s # simply launch bash within app directory $ dip bash # launch Postgres console $ dip psql BONUS: DIP 27 github.com/bibendi/dip
28. palkan_tula palkan RailsConf 2019 BONUS: DIP 28 # integrate with ZSH and # pretend you’re not using Docker at all :) $ dip console | source /dev/stdin # that runs the command within docker container… $ rails c # …but looks like you run it locally $ rspec spec/test_spec.rb:23 github.com/bibendi/dip
29. palkan_tula palkan RailsConf 2019 29 READY TO LAUNCH? $ rails c ???
30. palkan_tula palkan RailsConf 2019 30 $ rails c Traceback (most recent call last): 60: from bin/rails:9:in `<main>' // 57 lines we don’t care about 2: from /bundle/gems/activerecord-4.2.11.1/lib/ active_record/connection_adapters/postgresql_adapter.rb: 651:in `connect' 1: from /bundle/gems/activerecord-4.2.11.1/lib/ active_record/connection_adapters/postgresql_adapter.rb: 651:in `new' /bundle/gems/activerecord-4.2.11.1/lib/active_record/ connection_adapters/postgresql_adapter.rb:651:in `initialize': could not connect to server: No such file or directory (PG ::ConnectionBad)
31. palkan_tula palkan RailsConf 2019 31 $ rails c Traceback (most recent call last): 60: from bin/rails:9:in `<main>' // 56 lines we don’t care about 3: from /bundle/gems/aws-sdk-core-2.10.9/lib/ seahorse/client/base.rb:83:in `after_initialize' 2: from /bundle/gems/aws-sdk-core-2.10.9/lib/ seahorse/client/base.rb:83:in `each' 1: from /bundle/gems/aws-sdk-core-2.10.9/lib/ seahorse/client/base.rb:84:in `block in after_initialize' /bundle/gems/aws-sdk-core-2.10.9/lib/aws-sdk-core/plugins/ regional_endpoint.rb:34:in `after_initialize': missing region; use :region option or export region name to ENV['AWS_REGION'] (Aws ::Errors ::MissingRegionError)
32. palkan_tula palkan RailsConf 2019 32 WHY WE FAILED “The only host is localhost” Magic ENV Unexpected external deps Tha’s OK; Falcon 9 also failed
33. CONFIGURATION
34. palkan_tula palkan RailsConf 2019 CONFIGURATION 34 Provide sensible defaults Minimize the number of external deps Organize configuration
35. palkan_tula palkan RailsConf 2019 SENSIBLE DEFAULTS 35
36. palkan_tula palkan RailsConf 2019 Every required change could lead to “plz, help me” message in your Slack 36
37. palkan_tula palkan RailsConf 2019 SENSIBLE DEFAULTS 37 # database.yml default: &default url: <%= ENV.fetch(“DATABASE_URL”, nil) %> # cable.yml default: &default adapter: redis url: <%= ENV.fetch(“REDIS_URL”, “localhost”) %> # docker-compose.yml services: app: &app environment: - REDIS_URL=redis: //redis:6379/ - DATABASE_URL=postgres: //postgres:postgres@postgres:5432
38. palkan_tula palkan RailsConf 2019 OPTIONAL DEPS 38 # config/application.rb # Store uploaded files on the local file system # (unless AWS S3 is configured) config.active_storage.service = if AWSConfig.storage_configured? $stdout.puts "Using :amazon service for Active Storage" :amazon else :local end
39. palkan_tula palkan RailsConf 2019 OPTIONAL SERVICES 39 class TwilioNotifyClient def initialize(sid:) if sid.nil? @noop = true warn "Twilio Notify client initialized in the no-op mode.\n” \ "See docs/twilio.md on how to configure it in development" return end end def send_notification(*args) return if noop? # ... end end TwilioNotifyClient.new(sid: ENV["TWILIO_NOTIFY_SID"])
40. palkan_tula palkan RailsConf 2019 CONFIGURATION 40 Provide sensible defaults Minimize the number of external deps Organize configuration
41. palkan_tula palkan RailsConf 2019 ENV HELL 41 ENV across all over the codebase Rails.env.smth? checks
42. palkan_tula palkan RailsConf 2019 ENV HELL 42 Limit ENV usage to config/environments/ <env>.rb Replace Rails.env.smth? with custom configuration settings
43. palkan_tula palkan RailsConf 2019 43 # config/application.rb config.graphiql_enabled = false # config/development.rb config.graphiql_enabled = true # routes.rb - if Rails.env.development? + if Rails.application.config.graphiql_enabled mount Graphiql ::Rack.new, at: "/graphiql" end ENV HELL
44. palkan_tula palkan RailsConf 2019 LINT/ENV 44 bit.ly/lint-env-cop
45. palkan_tula palkan RailsConf 2019 ENV HELL 45 ENV across all over the codebase Rails.env.smth? checks .env with dozens of entries (and .env.sample is out-of-date)
46. palkan_tula palkan RailsConf 2019 ENV HELL 46
47. palkan_tula palkan RailsConf 2019 ENV HELL 47
48. palkan_tula palkan RailsConf 2019 HEROKU HELL 48 $ heroku config -a legacy-project | wc -l 131
49. NEW WORLD?
50. palkan_tula palkan RailsConf 2019 KEEPING CONFIGS 50 Store sensitive information in Rails Credentials (or similar) Keep non-sensitive information in named YAML files Allow overriding via ENV
51. palkan_tula palkan RailsConf 2019 RAILS 6 CREDENTIALS 51
52. palkan_tula palkan RailsConf 2019 KEEPING CONFIGS 52 github.com/palkan/anyway_config
53. palkan_tula palkan RailsConf 2019 KEEPING CONFIGS 53 github.com/palkan/anyway_config
54. palkan_tula palkan RailsConf 2019 54 PHASE #1 * We’re on Mars! MARS LANDING ✅ Repeatable dev env setup ✅ Transparent and zero-“changeme” configuration
55. palkan_tula palkan RailsConf 2019 55 BREATHABLE ATMOSPHERE Raising tests reliability level PHASE #2
56. palkan_tula palkan RailsConf 2019 TESTS ~ OXYGEN 56 Developing without tests is like breathing with no air Developing with slow tests is like breathing on the top of mt. Jomolungma Developing with unreliable (flaky) tests is like breathing in Chelyabinsk
57. CHELYABINSK
58. CHELYABINSK
59. palkan_tula palkan RailsConf 2019 59 GOAL PHASE #2 “Tests should be reliable and fast to not block the development” * Make oxygen level at least 21% to 2065!
60. palkan_tula palkan RailsConf 2019 BREATHABLE TESTS 60 Improve test suite speed
61. RAISE YOUR HAND IF YOUR TESTS ARE SLOW
62. palkan_tula palkan RailsConf 2019 github.com/palkan/test-prof TestProf 62 test-prof.evilmartians.io
63. palkan_tula palkan RailsConf 2019 TestProf 63 Tests specific profilers RSpec / Minitest extensions to speed up tests with as little refactoring as possible Custom Rubocop cops
64. palkan_tula palkan RailsConf 2019 64 3700 tests / 22 minutes = 170 TPM 9800 tests / 14 minutes = 700 TPM 4x faster! TestProf Born in “production”
65. palkan_tula palkan RailsConf 2019 HOW DID WE GET THERE? 65
66. palkan_tula palkan RailsConf 2019 99 PROBLEMS OF SLOW TESTS 66 bit.ly/test-prof-paris
67. palkan_tula palkan RailsConf 2019 FASTER TESTS: HIGHLIGHTS 67 Get rid of database_cleaner
68. palkan_tula palkan RailsConf 2019 RAILS 5.1 68 github.com/rails/rails/pull/28083
69. palkan_tula palkan RailsConf 2019 ActiveRecordSharedConnection 69 test-prof.evilmartians.io/#/active_record_shared_connection # Rails 5.1+ # connection is shared out-of-the-box # Rails <5.1 require “test_prof/recipes/active_record_one_love"
70. palkan_tula palkan RailsConf 2019 FASTER TESTS: HIGHLIGHTS 70 Get rid of database_cleaner Do not inline background jobs by default
71. palkan_tula palkan RailsConf 2019 SIDEKIQ SHAME 71 gist.github.com/nateberkopec/3932fce995c9feddd411417fc9bf33bf
72. palkan_tula palkan RailsConf 2019 SIDEKIQ SHAME 72 shared_context "sq:inline" do around(:each) do |ex| Sidekiq ::Testing.inline!(&ex) end end RSpec.configure do |config| config.include_context "sq:inline", sidekiq: :inline end
73. palkan_tula palkan RailsConf 2019 SIDEKIQ SHAME 73 evl.ms/blog
74. palkan_tula palkan RailsConf 2019 FASTER TESTS: HIGHLIGHTS 74 You (likely) don’t need database_cleaner Do not inline background jobs by default Take care of your factories
75. palkan_tula palkan RailsConf 2019 FACTORY CASCADE 75 factory :comment do answer author end factory :answer do question author end factory :question do author end create(:comment) # => creates 5 records
76. palkan_tula palkan RailsConf 2019 FACTORY THERAPY 76 evl.ms/blog
77. palkan_tula palkan RailsConf 2019 FASTER TESTS: HIGHLIGHTS 77 You (likely) don’t need database_cleaner Do not inline background jobs by default Take care of your factories …more
78. palkan_tula palkan RailsConf 2019 test-prof.evilmartians.io 78
79. palkan_tula palkan RailsConf 2019 79 Improve test suite speed Remove/reduce flakiness BREATHABLE TESTS
80. palkan_tula palkan RailsConf 2019 FIXING FLAKY TESTS 80
81. palkan_tula palkan RailsConf 2019 81 FIGHTING THE FLAKINESS bit.ly/flaky-fight
82. palkan_tula palkan RailsConf 2019 FACTORY LINTER 82 // Check factory definitions $ bundle exec rake factory_lint Factory lint detected the following errors: - :city should use a sequence for :name attribute, 'cause it has a uniqueness constraint bit.ly/factory-lint
83. palkan_tula palkan RailsConf 2019 FACTORY LINTER 83 # city.rb FactoryBot.define do factory :city do - name { Faker ::Address.city } + sequence(:name) { |n| Faker ::Address.city + " ( #{n})" } end end bit.ly/factory-lint
84. palkan_tula palkan RailsConf 2019 84 PHASE #2 BREATHABLE ATMOSPHERE ✅ Tests do not block the development
85. binding.cat
86. palkan_tula palkan RailsConf 2019 86 TURN ICE INTO WATER Bring project back to the healthy state PHASE #3 * Korolev crater
87. palkan_tula palkan RailsConf 2019 87 GOAL PHASE #3 “Project shouldn’t have significant security, consistency and performance issues“
88. palkan_tula palkan RailsConf 2019 SECURITY 88 Check dependencies (bundler-audit)
89. palkan_tula palkan RailsConf 2019 bundler-audit 89 $ bundle audit update && bundle audit check Name: nokogiri Version: 1.8.5 Advisory: CVE-2019-11068 Criticality: Unknown URL: https: //github.com/sparklemotion/nokogiri/issues/1892 Title: Nokogiri gem, via libxslt, is affected by improper access control vulnerability Solution: upgrade to >= 1.10.3 Vulnerabilities found! github.com/rubysec/bundler-audit
90. palkan_tula palkan RailsConf 2019 SECURITY 90 Check dependencies (bundler-audit) Check application code (brakeman, rubocop)
91. palkan_tula palkan RailsConf 2019 BRAKEMAN 91 brakemanscanner.org
92. palkan_tula palkan RailsConf 2019 RUBOCOP SECURITY 92 // Check potentially insecure code $ bundle exec rubocop --only Security Offenses: script.rb:33:21: C: Security/JSONLoad: Prefer JSON.parse over JSON.load. data = JSON.load(input.meta) ^^^^
93. palkan_tula palkan RailsConf 2019 CONSISTENCY 93 Models vs. DB user.update!(params.require(:user).permit(:name)) => PG ::StringDataRightTruncation: ERROR: value too long for type character varying(255)
94. palkan_tula palkan RailsConf 2019 database_consistency 94 github.com/djezzl/database_consistency
95. palkan_tula palkan RailsConf 2019 95 class Topic < ActiveRecord ::Base validates :description, presence: true end $ bundle exec database_consistency // description column is missing `null: false` constraint fail Topic description column should be required in the database database_consistency github.com/djezzl/database_consistency
96. palkan_tula palkan RailsConf 2019 database_validations 96 github.com/toptal/database_validations class Home < ActiveRecord ::Base # checks that a foreign key present db_belongs_to :city # checks that a unique index is present validates_db_uniqueness_of :name, case_sensitive: false end
97. palkan_tula palkan RailsConf 2019 CONSISTENCY 97 Models vs. DB Code style?
98. palkan_tula palkan RailsConf 2019 RUBOCOP 98 Strict config
99. palkan_tula palkan RailsConf 2019 .rubocop_strict.yml 99 AllCops: DisabledByDefault: true Security: Enabled: true Lint/Debugger: Enabled: true Lint/Syntax: Enabled: true Lint/Env: Enabled: true RSpec/Focus: Enabled: true
100. palkan_tula palkan RailsConf 2019 .rubocop_strict.yml 100 AllCops: DisabledByDefault: true Security: Enabled: true Lint/Debugger: Enabled: true Lint/Syntax: Enabled: true Lint/Env: Enabled: true RSpec/Focus: Enabled: true
101. palkan_tula palkan RailsConf 2019 .rubocop_strict.yml 101 AllCops: DisabledByDefault: true Security: Enabled: true Lint/Debugger: Enabled: true Lint/Syntax: Enabled: true Lint/Env: Enabled: true RSpec/Focus: Enabled: true
102. palkan_tula palkan RailsConf 2019 .rubocop_strict.yml 102 AllCops: DisabledByDefault: true Security: Enabled: true Lint/Debugger: Enabled: true Lint/Syntax: Enabled: true Lint/Env: Enabled: true RSpec/Focus: Enabled: true
103. palkan_tula palkan RailsConf 2019 .rubocop_strict.yml 103 AllCops: DisabledByDefault: true Security: Enabled: true Lint/Debugger: Enabled: true Lint/Syntax: Enabled: true Lint/Env: Enabled: true RSpec/Focus: Enabled: true
104. palkan_tula palkan RailsConf 2019 .rubocop_strict.yml + CI 104 # .circleci/config.yml rubocop_strict: executor: ruby steps: - attach_workspace: at: . - run: name: Strict Rubocop check command: | bundle exec rubocop -c .rubocop_strict.yml
105. palkan_tula palkan RailsConf 2019 RUBOCOP 105 Strict config Progressive enhancement
106. palkan_tula palkan RailsConf 2019 .rubocop.yml 106 require: - rubocop-rspec - standard/cop/semantic_blocks inherit_gem: standard: config/base.yml AllCops: Include: # explicitly specify terraformed code - "**/community/**/*.rb" Exclude: - "db/schema.rb" - "bin /*"
107. palkan_tula palkan RailsConf 2019 .rubocop.yml 107 require: - rubocop-rspec - standard/cop/semantic_blocks inherit_gem: standard: config/base.yml AllCops: Include: # explicitly specify terraformed code - "**/community/**/*.rb" Exclude: - "db/schema.rb" - "bin /*"
108. palkan_tula palkan RailsConf 2019 .rubocop.yml 108 require: - rubocop-rspec - standard/cop/semantic_blocks inherit_gem: standard: config/base.yml AllCops: Include: # explicitly specify terraformed code - "**/community/**/*.rb" Exclude: - "db/schema.rb" - "bin /*"
109. palkan_tula palkan RailsConf 2019 STANDARD 109 github.com/testdouble/standard
110. palkan_tula palkan RailsConf 2019 110 SIDE EFFECTS
111. palkan_tula palkan RailsConf 2019 SIDE EFFECTS 111 Non-atomic transactions class Post < ActiveRecord ::Base after_create :notify_author def notify_users # send email even if transaction failed UsersMailer.new_post(author, post.title).deliver_later end end
112. palkan_tula palkan RailsConf 2019 ISOLATOR 112 github.com/palkan/isolator
113. palkan_tula palkan RailsConf 2019 ISOLATOR 113 github.com/palkan/isolator $ bundle exec rspec 1) Create post example Failure/Error: job_or_instantiate(*args).enqueue Isolator ::BackgroundJobError: You are trying to enqueue background job inside db transaction. Details: ActionMailer ::DeliveryJob (UsersMailer, new_post)
114. palkan_tula palkan RailsConf 2019 114 DEAD CODE
115. palkan_tula palkan RailsConf 2019 DEAD CODE 115 Gems
116. palkan_tula palkan RailsConf 2019 DEAD GEMS 116 $ GEM_TRACK=1 bundle exec rspec Maybe unused gems: activerecord-postgres_enum-0.3.0 avatax-18.12.0 aws-sdk-2.10.9 rails-4.2.11.1 gibbon-2.2.5 bit.ly/track-gems
117. palkan_tula palkan RailsConf 2019 117 Gems Routes/controllers DEAD CODE
118. palkan_tula palkan RailsConf 2019 traceroute 118 github.com/amatsuda/traceroute $ bundle exec rake traceroute Unused routes (3): users#create users#new catalog#purchase Unreachable action methods (1): users#index2
119. palkan_tula palkan RailsConf 2019 119 Gems Routes/controllers Views DEAD CODE
120. palkan_tula palkan RailsConf 2019 TEMPLATES TRACKER 120 $ TT=1 bundle exec rspec ======== Unused Templates ========= /app/app/views/home/index.html.erb /app/app/views/housekeeping/scheduling/index.html.erb bit.ly/track-templates
121. palkan_tula palkan RailsConf 2019 121 Gems Routes/controllers Views Factories NEW! DEAD CODE
122. palkan_tula palkan RailsConf 2019 FACTORY TRACE 122 $ FB_TRACE=1 bundle exec rspec total number of unique used factories & traits: 1 total number of unique unused factories & traits: 3 unused trait 'with_phone' of factory 'user' unused factory 'special_user' unused global trait 'with_email' github.com/djezzzl/factory_trace
123. palkan_tula palkan RailsConf 2019 123 TURN ICE INTO WATER PHASE #3 ✅ Project is healthy enough to bring more people
124. palkan_tula palkan RailsConf 2019 124 ROBOTS Automating things BONUS
125. palkan_tula palkan RailsConf 2019 LEFT HOOK 125 github.com/Arkweid/lefthook $ git commit -m "chore: use lefthook" RUNNING HOOKS GROUP: pre-commit EXECUTE > rubocop 1 file inspected, 1 offense detected rubocop
126. palkan_tula palkan RailsConf 2019 LEFT HOOK 126 github.com/Arkweid/lefthook $ git commit -m "chore: use lefthook" RUNNING HOOKS GROUP: pre-commit EXECUTE > rubocop 1 file inspected, no offenses detected ✅ rubocop RUNNING HOOKS GROUP: prepare-commit-msg EXECUTE > jira_link.rb JIRA ticket ID (type 'N/n' to skip): 2019 ✅ jira_link.rb
127. palkan_tula palkan RailsConf 2019 DANGER 127 danger.systems
128. palkan_tula palkan RailsConf 2019 FINAL SCORING 128
129. palkan_tula palkan RailsConf 2019 github.com/evilmartians/terraforming-rails DO IT YOURSELF 129
130. palkan_tula palkan RailsConf 2019 …OR NOT 130 evilmartians.com
131. COMING SOON
132. SPECIAL THANKS 2061.su
133. palkan_tula palkan RailsConf 2019 133 THANK YOU! Vladimir Dementyev evilmartians.com @evilmartians