Pro Yearly is on sale from $80 to $50! »

[RailsConf 2019] Terraforming legacy Rails applications

[RailsConf 2019] Terraforming legacy Rails applications

Video: https://www.youtube.com/watch?v=-NKpMn6XSjU
RailsConf: https://railsconf.com/program/sessions#session-832
GitHub: https://github.com/evilmartians/terraforming-rails

Rails has been around for (can you imagine!) about 15 years. Most Rails applications are no longer MVPs, but they grew from MVPs and usually contain a lot of legacy code that "just works."

And this legacy makes shipping new features harder and riskier: the new functionality have to co-exist with the code written years ago, and who knows what will blow up next?

I've been working on legacy codebases for the last few years, and I found turning legacy code into a legendary code to be a very exciting and challenging process.

I want to share the ideas and techniques I apply to make legacy codebases habitable and to prepare a breeding ground for the new features.

52cc8a838bf44a589d2572833b2dd1b9?s=128

Vlad Dem

May 01, 2019
Tweet

Transcript

  1. TERRAFORMING LEGACY RAILS APPLICATIONS

  2. palkan_tula palkan RailsConf 2019 2 @palkan @palkan_tula Vladimir Dementyev 933

    722 406 504
  3. palkan_tula palkan RailsConf 2019 3 evilmartians.com Brooklyn, NY Moscow, Russia

  4. None
  5. palkan_tula palkan RailsConf 2019 evl.ms/blog 5

  6. palkan_tula palkan RailsConf 2019 evilmartians.com 6

  7. palkan_tula palkan RailsConf 2019 FROM SCRATCH TWO TYPES OF PROJECTS

    7 “Build us something cool!” SpaceX Star Hopper
  8. palkan_tula palkan RailsConf 2019 FROM LEGACY TWO TYPES OF PROJECTS

    8 “We’ve built something, make it cool!” Pepelats from Kin-dza-dza! (Soviet sci-fi) That’s my jam!
  9. palkan_tula palkan RailsConf 2019 LEGACY 9 Jason Swett, RubyHACK 2019

  10. palkan_tula palkan RailsConf 2019 10 TEAM LEAD Evaluate project Make

    ready for others to join Prepare guidelines Captain Zelyonyy (Green), The Mystery of the third planet
  11. THIS TALK

  12. palkan_tula palkan RailsConf 2019 12 #BoardGameGeek NOT ONLY A DEVELOPER

  13. palkan_tula palkan RailsConf 2019 13 TERRAFORMING MARS “Make Mars habitable!”

    Jacob Fryxelius, 2016
  14. palkan_tula palkan RailsConf 2019 TERRAFORMING MARS 14 Corporations are competing

    to transform Mars into a habitable planet by raising temperature, creating a breathable atmosphere, and making oceans of water. As terraforming progresses, more and more people will immigrate from Earth to live on the Red Planet.
  15. palkan_tula palkan RailsConf 2019 TERRAFORMING CODE 15 Engineers are transforming

    the legacy code into a habitable code by optimizing development process, adding confidence with tests and linters and creating a codeable atmosphere. As terraforming progresses, more and more people could join the project and work on new features.
  16. palkan_tula palkan RailsConf 2019 16 MARS LANDING Setting up the

    project PHASE #1
  17. palkan_tula palkan RailsConf 2019 17 GOAL PHASE #1 “Developers should

    be able to run project with the least possible effort”
  18. palkan_tula palkan RailsConf 2019 18 // Make running project locally

    as easy as running a few commands $ git clone project/repo.git $ cd repo $ <dev env setup and provision> $ rails c Loading development environment 2.6.2 (main):0> PHASE #1 GOAL
  19. DEVELOPMENT ENVIRONMENT

  20. palkan_tula palkan RailsConf 2019 “OLD SCHOOL” DEV ENV SETUP 20

    Dev setup instructions from yet another Rails project
  21. palkan_tula palkan RailsConf 2019 “NEW SCHOOL” 21

  22. palkan_tula palkan RailsConf 2019 DOCKER FOR DEV 22 Repeatable and

    predictable setup Always in sync
  23. palkan_tula palkan RailsConf 2019 KEEPING ’N SYNC 23

  24. palkan_tula palkan RailsConf 2019 DOCKER FOR DEV 24 Repeatable and

    predictable setup Always in sync Painfully slow on Mac?
  25. palkan_tula palkan RailsConf 2019 SLOW D4M 25 Use :cached folders

    Store assets/bundles in volumes Use NFS (if the first two are not enough) bit.ly/d4m-nfs
  26. palkan_tula palkan RailsConf 2019 DOCKER FOR DEV 26 Repeatable and

    predictable setup Always in sync Painfully slow on Mac Windows "
  27. palkan_tula palkan RailsConf 2019 # provision application $ dip provision

    # run web app $ dip rails s # simply launch bash within app directory $ dip bash # launch Postgres console $ dip psql BONUS: DIP 27 github.com/bibendi/dip
  28. palkan_tula palkan RailsConf 2019 BONUS: DIP 28 # integrate with

    ZSH and # pretend you’re not using Docker at all :) $ dip console | source /dev/stdin # that runs the command within docker container… $ rails c # …but looks like you run it locally $ rspec spec/test_spec.rb:23 github.com/bibendi/dip
  29. palkan_tula palkan RailsConf 2019 29 READY TO LAUNCH? $ rails

    c ???
  30. palkan_tula palkan RailsConf 2019 30 $ rails c Traceback (most

    recent call last): 60: from bin/rails:9:in `<main>' // 57 lines we don’t care about 2: from /bundle/gems/activerecord-4.2.11.1/lib/ active_record/connection_adapters/postgresql_adapter.rb: 651:in `connect' 1: from /bundle/gems/activerecord-4.2.11.1/lib/ active_record/connection_adapters/postgresql_adapter.rb: 651:in `new' /bundle/gems/activerecord-4.2.11.1/lib/active_record/ connection_adapters/postgresql_adapter.rb:651:in `initialize': could not connect to server: No such file or directory (PG ::ConnectionBad)
  31. palkan_tula palkan RailsConf 2019 31 $ rails c Traceback (most

    recent call last): 60: from bin/rails:9:in `<main>' // 56 lines we don’t care about 3: from /bundle/gems/aws-sdk-core-2.10.9/lib/ seahorse/client/base.rb:83:in `after_initialize' 2: from /bundle/gems/aws-sdk-core-2.10.9/lib/ seahorse/client/base.rb:83:in `each' 1: from /bundle/gems/aws-sdk-core-2.10.9/lib/ seahorse/client/base.rb:84:in `block in after_initialize' /bundle/gems/aws-sdk-core-2.10.9/lib/aws-sdk-core/plugins/ regional_endpoint.rb:34:in `after_initialize': missing region; use :region option or export region name to ENV['AWS_REGION'] (Aws ::Errors ::MissingRegionError)
  32. palkan_tula palkan RailsConf 2019 32 WHY WE FAILED “The only

    host is localhost” Magic ENV Unexpected external deps Tha’s OK; Falcon 9 also failed
  33. CONFIGURATION

  34. palkan_tula palkan RailsConf 2019 CONFIGURATION 34 Provide sensible defaults Minimize

    the number of external deps Organize configuration
  35. palkan_tula palkan RailsConf 2019 SENSIBLE DEFAULTS 35

  36. palkan_tula palkan RailsConf 2019 Every required change could lead to

    “plz, help me” message in your Slack 36
  37. palkan_tula palkan RailsConf 2019 SENSIBLE DEFAULTS 37 # database.yml default:

    &default url: <%= ENV.fetch(“DATABASE_URL”, nil) %> # cable.yml default: &default adapter: redis url: <%= ENV.fetch(“REDIS_URL”, “localhost”) %> # docker-compose.yml services: app: &app environment: - REDIS_URL=redis: //redis:6379/ - DATABASE_URL=postgres: //postgres:postgres@postgres:5432
  38. palkan_tula palkan RailsConf 2019 OPTIONAL DEPS 38 # config/application.rb #

    Store uploaded files on the local file system # (unless AWS S3 is configured) config.active_storage.service = if AWSConfig.storage_configured? $stdout.puts "Using :amazon service for Active Storage" :amazon else :local end
  39. palkan_tula palkan RailsConf 2019 OPTIONAL SERVICES 39 class TwilioNotifyClient def

    initialize(sid:) if sid.nil? @noop = true warn "Twilio Notify client initialized in the no-op mode.\n” \ "See docs/twilio.md on how to configure it in development" return end end def send_notification(*args) return if noop? # ... end end TwilioNotifyClient.new(sid: ENV["TWILIO_NOTIFY_SID"])
  40. palkan_tula palkan RailsConf 2019 CONFIGURATION 40 Provide sensible defaults Minimize

    the number of external deps Organize configuration
  41. palkan_tula palkan RailsConf 2019 ENV HELL 41 ENV across all

    over the codebase Rails.env.smth? checks
  42. palkan_tula palkan RailsConf 2019 ENV HELL 42 Limit ENV usage

    to config/environments/ <env>.rb Replace Rails.env.smth? with custom configuration settings
  43. palkan_tula palkan RailsConf 2019 43 # config/application.rb config.graphiql_enabled = false

    # config/development.rb config.graphiql_enabled = true # routes.rb - if Rails.env.development? + if Rails.application.config.graphiql_enabled mount Graphiql ::Rack.new, at: "/graphiql" end ENV HELL
  44. palkan_tula palkan RailsConf 2019 LINT/ENV 44 bit.ly/lint-env-cop

  45. palkan_tula palkan RailsConf 2019 ENV HELL 45 ENV across all

    over the codebase Rails.env.smth? checks .env with dozens of entries (and .env.sample is out-of-date)
  46. palkan_tula palkan RailsConf 2019 ENV HELL 46

  47. palkan_tula palkan RailsConf 2019 ENV HELL 47

  48. palkan_tula palkan RailsConf 2019 HEROKU HELL 48 $ heroku config

    -a legacy-project | wc -l 131
  49. NEW WORLD?

  50. palkan_tula palkan RailsConf 2019 KEEPING CONFIGS 50 Store sensitive information

    in Rails Credentials (or similar) Keep non-sensitive information in named YAML files Allow overriding via ENV
  51. palkan_tula palkan RailsConf 2019 RAILS 6 CREDENTIALS 51

  52. palkan_tula palkan RailsConf 2019 KEEPING CONFIGS 52 github.com/palkan/anyway_config

  53. palkan_tula palkan RailsConf 2019 KEEPING CONFIGS 53 github.com/palkan/anyway_config

  54. palkan_tula palkan RailsConf 2019 54 PHASE #1 * We’re on

    Mars! MARS LANDING ✅ Repeatable dev env setup ✅ Transparent and zero-“changeme” configuration
  55. palkan_tula palkan RailsConf 2019 55 BREATHABLE ATMOSPHERE Raising tests reliability

    level PHASE #2
  56. palkan_tula palkan RailsConf 2019 TESTS ~ OXYGEN 56 Developing without

    tests is like breathing with no air Developing with slow tests is like breathing on the top of mt. Jomolungma Developing with unreliable (flaky) tests is like breathing in Chelyabinsk
  57. CHELYABINSK

  58. CHELYABINSK

  59. palkan_tula palkan RailsConf 2019 59 GOAL PHASE #2 “Tests should

    be reliable and fast to not block the development” * Make oxygen level at least 21% to 2065!
  60. palkan_tula palkan RailsConf 2019 BREATHABLE TESTS 60 Improve test suite

    speed
  61. RAISE YOUR HAND IF YOUR TESTS ARE SLOW

  62. palkan_tula palkan RailsConf 2019 github.com/palkan/test-prof TestProf 62 test-prof.evilmartians.io

  63. palkan_tula palkan RailsConf 2019 TestProf 63 Tests specific profilers RSpec

    / Minitest extensions to speed up tests with as little refactoring as possible Custom Rubocop cops
  64. palkan_tula palkan RailsConf 2019 64 3700 tests / 22 minutes

    = 170 TPM 9800 tests / 14 minutes = 700 TPM 4x faster! TestProf Born in “production”
  65. palkan_tula palkan RailsConf 2019 HOW DID WE GET THERE? 65

  66. palkan_tula palkan RailsConf 2019 99 PROBLEMS OF SLOW TESTS 66

    bit.ly/test-prof-paris
  67. palkan_tula palkan RailsConf 2019 FASTER TESTS: HIGHLIGHTS 67 Get rid

    of database_cleaner
  68. palkan_tula palkan RailsConf 2019 RAILS 5.1 68 github.com/rails/rails/pull/28083

  69. palkan_tula palkan RailsConf 2019 ActiveRecordSharedConnection 69 test-prof.evilmartians.io/#/active_record_shared_connection # Rails 5.1+

    # connection is shared out-of-the-box # Rails <5.1 require “test_prof/recipes/active_record_one_love"
  70. palkan_tula palkan RailsConf 2019 FASTER TESTS: HIGHLIGHTS 70 Get rid

    of database_cleaner Do not inline background jobs by default
  71. palkan_tula palkan RailsConf 2019 SIDEKIQ SHAME 71 gist.github.com/nateberkopec/3932fce995c9feddd411417fc9bf33bf

  72. palkan_tula palkan RailsConf 2019 SIDEKIQ SHAME 72 shared_context "sq:inline" do

    around(:each) do |ex| Sidekiq ::Testing.inline!(&ex) end end RSpec.configure do |config| config.include_context "sq:inline", sidekiq: :inline end
  73. palkan_tula palkan RailsConf 2019 SIDEKIQ SHAME 73 evl.ms/blog

  74. palkan_tula palkan RailsConf 2019 FASTER TESTS: HIGHLIGHTS 74 You (likely)

    don’t need database_cleaner Do not inline background jobs by default Take care of your factories
  75. palkan_tula palkan RailsConf 2019 FACTORY CASCADE 75 factory :comment do

    answer author end factory :answer do question author end factory :question do author end create(:comment) # => creates 5 records
  76. palkan_tula palkan RailsConf 2019 FACTORY THERAPY 76 evl.ms/blog

  77. palkan_tula palkan RailsConf 2019 FASTER TESTS: HIGHLIGHTS 77 You (likely)

    don’t need database_cleaner Do not inline background jobs by default Take care of your factories …more
  78. palkan_tula palkan RailsConf 2019 test-prof.evilmartians.io 78

  79. palkan_tula palkan RailsConf 2019 79 Improve test suite speed Remove/reduce

    flakiness BREATHABLE TESTS
  80. palkan_tula palkan RailsConf 2019 FIXING FLAKY TESTS 80

  81. palkan_tula palkan RailsConf 2019 81 FIGHTING THE FLAKINESS bit.ly/flaky-fight

  82. palkan_tula palkan RailsConf 2019 FACTORY LINTER 82 // Check factory

    definitions $ bundle exec rake factory_lint Factory lint detected the following errors: - :city should use a sequence for :name attribute, 'cause it has a uniqueness constraint bit.ly/factory-lint
  83. palkan_tula palkan RailsConf 2019 FACTORY LINTER 83 # city.rb FactoryBot.define

    do factory :city do - name { Faker ::Address.city } + sequence(:name) { |n| Faker ::Address.city + " ( #{n})" } end end bit.ly/factory-lint
  84. palkan_tula palkan RailsConf 2019 84 PHASE #2 BREATHABLE ATMOSPHERE ✅

    Tests do not block the development
  85. binding.cat

  86. palkan_tula palkan RailsConf 2019 86 TURN ICE INTO WATER Bring

    project back to the healthy state PHASE #3 * Korolev crater
  87. palkan_tula palkan RailsConf 2019 87 GOAL PHASE #3 “Project shouldn’t

    have significant security, consistency and performance issues“
  88. palkan_tula palkan RailsConf 2019 SECURITY 88 Check dependencies (bundler-audit)

  89. palkan_tula palkan RailsConf 2019 bundler-audit 89 $ bundle audit update

    && bundle audit check Name: nokogiri Version: 1.8.5 Advisory: CVE-2019-11068 Criticality: Unknown URL: https: //github.com/sparklemotion/nokogiri/issues/1892 Title: Nokogiri gem, via libxslt, is affected by improper access control vulnerability Solution: upgrade to >= 1.10.3 Vulnerabilities found! github.com/rubysec/bundler-audit
  90. palkan_tula palkan RailsConf 2019 SECURITY 90 Check dependencies (bundler-audit) Check

    application code (brakeman, rubocop)
  91. palkan_tula palkan RailsConf 2019 BRAKEMAN 91 brakemanscanner.org

  92. palkan_tula palkan RailsConf 2019 RUBOCOP SECURITY 92 // Check potentially

    insecure code $ bundle exec rubocop --only Security Offenses: script.rb:33:21: C: Security/JSONLoad: Prefer JSON.parse over JSON.load. data = JSON.load(input.meta) ^^^^
  93. palkan_tula palkan RailsConf 2019 CONSISTENCY 93 Models vs. DB user.update!(params.require(:user).permit(:name))

    => PG ::StringDataRightTruncation: ERROR: value too long for type character varying(255)
  94. palkan_tula palkan RailsConf 2019 database_consistency 94 github.com/djezzl/database_consistency

  95. palkan_tula palkan RailsConf 2019 95 class Topic < ActiveRecord ::Base

    validates :description, presence: true end $ bundle exec database_consistency // description column is missing `null: false` constraint fail Topic description column should be required in the database database_consistency github.com/djezzl/database_consistency
  96. palkan_tula palkan RailsConf 2019 database_validations 96 github.com/toptal/database_validations class Home <

    ActiveRecord ::Base # checks that a foreign key present db_belongs_to :city # checks that a unique index is present validates_db_uniqueness_of :name, case_sensitive: false end
  97. palkan_tula palkan RailsConf 2019 CONSISTENCY 97 Models vs. DB Code

    style?
  98. palkan_tula palkan RailsConf 2019 RUBOCOP 98 Strict config

  99. palkan_tula palkan RailsConf 2019 .rubocop_strict.yml 99 AllCops: DisabledByDefault: true Security:

    Enabled: true Lint/Debugger: Enabled: true Lint/Syntax: Enabled: true Lint/Env: Enabled: true RSpec/Focus: Enabled: true
  100. palkan_tula palkan RailsConf 2019 .rubocop_strict.yml 100 AllCops: DisabledByDefault: true Security:

    Enabled: true Lint/Debugger: Enabled: true Lint/Syntax: Enabled: true Lint/Env: Enabled: true RSpec/Focus: Enabled: true
  101. palkan_tula palkan RailsConf 2019 .rubocop_strict.yml 101 AllCops: DisabledByDefault: true Security:

    Enabled: true Lint/Debugger: Enabled: true Lint/Syntax: Enabled: true Lint/Env: Enabled: true RSpec/Focus: Enabled: true
  102. palkan_tula palkan RailsConf 2019 .rubocop_strict.yml 102 AllCops: DisabledByDefault: true Security:

    Enabled: true Lint/Debugger: Enabled: true Lint/Syntax: Enabled: true Lint/Env: Enabled: true RSpec/Focus: Enabled: true
  103. palkan_tula palkan RailsConf 2019 .rubocop_strict.yml 103 AllCops: DisabledByDefault: true Security:

    Enabled: true Lint/Debugger: Enabled: true Lint/Syntax: Enabled: true Lint/Env: Enabled: true RSpec/Focus: Enabled: true
  104. palkan_tula palkan RailsConf 2019 .rubocop_strict.yml + CI 104 # .circleci/config.yml

    rubocop_strict: executor: ruby steps: - attach_workspace: at: . - run: name: Strict Rubocop check command: | bundle exec rubocop -c .rubocop_strict.yml
  105. palkan_tula palkan RailsConf 2019 RUBOCOP 105 Strict config Progressive enhancement

  106. palkan_tula palkan RailsConf 2019 .rubocop.yml 106 require: - rubocop-rspec -

    standard/cop/semantic_blocks inherit_gem: standard: config/base.yml AllCops: Include: # explicitly specify terraformed code - "**/community/**/*.rb" Exclude: - "db/schema.rb" - "bin /*"
  107. palkan_tula palkan RailsConf 2019 .rubocop.yml 107 require: - rubocop-rspec -

    standard/cop/semantic_blocks inherit_gem: standard: config/base.yml AllCops: Include: # explicitly specify terraformed code - "**/community/**/*.rb" Exclude: - "db/schema.rb" - "bin /*"
  108. palkan_tula palkan RailsConf 2019 .rubocop.yml 108 require: - rubocop-rspec -

    standard/cop/semantic_blocks inherit_gem: standard: config/base.yml AllCops: Include: # explicitly specify terraformed code - "**/community/**/*.rb" Exclude: - "db/schema.rb" - "bin /*"
  109. palkan_tula palkan RailsConf 2019 STANDARD 109 github.com/testdouble/standard

  110. palkan_tula palkan RailsConf 2019 110 SIDE EFFECTS

  111. palkan_tula palkan RailsConf 2019 SIDE EFFECTS 111 Non-atomic transactions class

    Post < ActiveRecord ::Base after_create :notify_author def notify_users # send email even if transaction failed UsersMailer.new_post(author, post.title).deliver_later end end
  112. palkan_tula palkan RailsConf 2019 ISOLATOR 112 github.com/palkan/isolator

  113. palkan_tula palkan RailsConf 2019 ISOLATOR 113 github.com/palkan/isolator $ bundle exec

    rspec 1) Create post example Failure/Error: job_or_instantiate(*args).enqueue Isolator ::BackgroundJobError: You are trying to enqueue background job inside db transaction. Details: ActionMailer ::DeliveryJob (UsersMailer, new_post)
  114. palkan_tula palkan RailsConf 2019 114 DEAD CODE

  115. palkan_tula palkan RailsConf 2019 DEAD CODE 115 Gems

  116. palkan_tula palkan RailsConf 2019 DEAD GEMS 116 $ GEM_TRACK=1 bundle

    exec rspec Maybe unused gems: activerecord-postgres_enum-0.3.0 avatax-18.12.0 aws-sdk-2.10.9 rails-4.2.11.1 gibbon-2.2.5 bit.ly/track-gems
  117. palkan_tula palkan RailsConf 2019 117 Gems Routes/controllers DEAD CODE

  118. palkan_tula palkan RailsConf 2019 traceroute 118 github.com/amatsuda/traceroute $ bundle exec

    rake traceroute Unused routes (3): users#create users#new catalog#purchase Unreachable action methods (1): users#index2
  119. palkan_tula palkan RailsConf 2019 119 Gems Routes/controllers Views DEAD CODE

  120. palkan_tula palkan RailsConf 2019 TEMPLATES TRACKER 120 $ TT=1 bundle

    exec rspec ======== Unused Templates ========= /app/app/views/home/index.html.erb /app/app/views/housekeeping/scheduling/index.html.erb bit.ly/track-templates
  121. palkan_tula palkan RailsConf 2019 121 Gems Routes/controllers Views Factories NEW!

    DEAD CODE
  122. palkan_tula palkan RailsConf 2019 FACTORY TRACE 122 $ FB_TRACE=1 bundle

    exec rspec total number of unique used factories & traits: 1 total number of unique unused factories & traits: 3 unused trait 'with_phone' of factory 'user' unused factory 'special_user' unused global trait 'with_email' github.com/djezzzl/factory_trace
  123. palkan_tula palkan RailsConf 2019 123 TURN ICE INTO WATER PHASE

    #3 ✅ Project is healthy enough to bring more people
  124. palkan_tula palkan RailsConf 2019 124 ROBOTS Automating things BONUS

  125. palkan_tula palkan RailsConf 2019 LEFT HOOK 125 github.com/Arkweid/lefthook $ git

    commit -m "chore: use lefthook" RUNNING HOOKS GROUP: pre-commit EXECUTE > rubocop 1 file inspected, 1 offense detected rubocop
  126. palkan_tula palkan RailsConf 2019 LEFT HOOK 126 github.com/Arkweid/lefthook $ git

    commit -m "chore: use lefthook" RUNNING HOOKS GROUP: pre-commit EXECUTE > rubocop 1 file inspected, no offenses detected ✅ rubocop RUNNING HOOKS GROUP: prepare-commit-msg EXECUTE > jira_link.rb JIRA ticket ID (type 'N/n' to skip): 2019 ✅ jira_link.rb
  127. palkan_tula palkan RailsConf 2019 DANGER 127 danger.systems

  128. palkan_tula palkan RailsConf 2019 FINAL SCORING 128

  129. palkan_tula palkan RailsConf 2019 github.com/evilmartians/terraforming-rails DO IT YOURSELF 129

  130. palkan_tula palkan RailsConf 2019 …OR NOT 130 evilmartians.com

  131. COMING SOON

  132. SPECIAL THANKS 2061.su

  133. palkan_tula palkan RailsConf 2019 133 THANK YOU! Vladimir Dementyev evilmartians.com

    @evilmartians