[RailsConf 2019] Terraforming legacy Rails applications

[RailsConf 2019] Terraforming legacy Rails applications

Video: https://www.youtube.com/watch?v=-NKpMn6XSjU
RailsConf: https://railsconf.com/program/sessions#session-832
GitHub: https://github.com/evilmartians/terraforming-rails

Rails has been around for (can you imagine!) about 15 years. Most Rails applications are no longer MVPs, but they grew from MVPs and usually contain a lot of legacy code that "just works."

And this legacy makes shipping new features harder and riskier: the new functionality have to co-exist with the code written years ago, and who knows what will blow up next?

I've been working on legacy codebases for the last few years, and I found turning legacy code into a legendary code to be a very exciting and challenging process.

I want to share the ideas and techniques I apply to make legacy codebases habitable and to prepare a breeding ground for the new features.

52cc8a838bf44a589d2572833b2dd1b9?s=128

Vlad Dem

May 01, 2019
Tweet

Transcript

  1. 4.
  2. 7.

    palkan_tula palkan RailsConf 2019 FROM SCRATCH TWO TYPES OF PROJECTS

    7 “Build us something cool!” SpaceX Star Hopper
  3. 8.

    palkan_tula palkan RailsConf 2019 FROM LEGACY TWO TYPES OF PROJECTS

    8 “We’ve built something, make it cool!” Pepelats from Kin-dza-dza! (Soviet sci-fi) That’s my jam!
  4. 10.

    palkan_tula palkan RailsConf 2019 10 TEAM LEAD Evaluate project Make

    ready for others to join Prepare guidelines Captain Zelyonyy (Green), The Mystery of the third planet
  5. 11.
  6. 14.

    palkan_tula palkan RailsConf 2019 TERRAFORMING MARS 14 Corporations are competing

    to transform Mars into a habitable planet by raising temperature, creating a breathable atmosphere, and making oceans of water. As terraforming progresses, more and more people will immigrate from Earth to live on the Red Planet.
  7. 15.

    palkan_tula palkan RailsConf 2019 TERRAFORMING CODE 15 Engineers are transforming

    the legacy code into a habitable code by optimizing development process, adding confidence with tests and linters and creating a codeable atmosphere. As terraforming progresses, more and more people could join the project and work on new features.
  8. 17.

    palkan_tula palkan RailsConf 2019 17 GOAL PHASE #1 “Developers should

    be able to run project with the least possible effort”
  9. 18.

    palkan_tula palkan RailsConf 2019 18 // Make running project locally

    as easy as running a few commands $ git clone project/repo.git $ cd repo $ <dev env setup and provision> $ rails c Loading development environment 2.6.2 (main):0> PHASE #1 GOAL
  10. 20.

    palkan_tula palkan RailsConf 2019 “OLD SCHOOL” DEV ENV SETUP 20

    Dev setup instructions from yet another Rails project
  11. 24.

    palkan_tula palkan RailsConf 2019 DOCKER FOR DEV 24 Repeatable and

    predictable setup Always in sync Painfully slow on Mac?
  12. 25.

    palkan_tula palkan RailsConf 2019 SLOW D4M 25 Use :cached folders

    Store assets/bundles in volumes Use NFS (if the first two are not enough) bit.ly/d4m-nfs
  13. 26.

    palkan_tula palkan RailsConf 2019 DOCKER FOR DEV 26 Repeatable and

    predictable setup Always in sync Painfully slow on Mac Windows "
  14. 27.

    palkan_tula palkan RailsConf 2019 # provision application $ dip provision

    # run web app $ dip rails s # simply launch bash within app directory $ dip bash # launch Postgres console $ dip psql BONUS: DIP 27 github.com/bibendi/dip
  15. 28.

    palkan_tula palkan RailsConf 2019 BONUS: DIP 28 # integrate with

    ZSH and # pretend you’re not using Docker at all :) $ dip console | source /dev/stdin # that runs the command within docker container… $ rails c # …but looks like you run it locally $ rspec spec/test_spec.rb:23 github.com/bibendi/dip
  16. 30.

    palkan_tula palkan RailsConf 2019 30 $ rails c Traceback (most

    recent call last): 60: from bin/rails:9:in `<main>' // 57 lines we don’t care about 2: from /bundle/gems/activerecord-4.2.11.1/lib/ active_record/connection_adapters/postgresql_adapter.rb: 651:in `connect' 1: from /bundle/gems/activerecord-4.2.11.1/lib/ active_record/connection_adapters/postgresql_adapter.rb: 651:in `new' /bundle/gems/activerecord-4.2.11.1/lib/active_record/ connection_adapters/postgresql_adapter.rb:651:in `initialize': could not connect to server: No such file or directory (PG ::ConnectionBad)
  17. 31.

    palkan_tula palkan RailsConf 2019 31 $ rails c Traceback (most

    recent call last): 60: from bin/rails:9:in `<main>' // 56 lines we don’t care about 3: from /bundle/gems/aws-sdk-core-2.10.9/lib/ seahorse/client/base.rb:83:in `after_initialize' 2: from /bundle/gems/aws-sdk-core-2.10.9/lib/ seahorse/client/base.rb:83:in `each' 1: from /bundle/gems/aws-sdk-core-2.10.9/lib/ seahorse/client/base.rb:84:in `block in after_initialize' /bundle/gems/aws-sdk-core-2.10.9/lib/aws-sdk-core/plugins/ regional_endpoint.rb:34:in `after_initialize': missing region; use :region option or export region name to ENV['AWS_REGION'] (Aws ::Errors ::MissingRegionError)
  18. 32.

    palkan_tula palkan RailsConf 2019 32 WHY WE FAILED “The only

    host is localhost” Magic ENV Unexpected external deps Tha’s OK; Falcon 9 also failed
  19. 34.
  20. 36.

    palkan_tula palkan RailsConf 2019 Every required change could lead to

    “plz, help me” message in your Slack 36
  21. 37.

    palkan_tula palkan RailsConf 2019 SENSIBLE DEFAULTS 37 # database.yml default:

    &default url: <%= ENV.fetch(“DATABASE_URL”, nil) %> # cable.yml default: &default adapter: redis url: <%= ENV.fetch(“REDIS_URL”, “localhost”) %> # docker-compose.yml services: app: &app environment: - REDIS_URL=redis: //redis:6379/ - DATABASE_URL=postgres: //postgres:postgres@postgres:5432
  22. 38.

    palkan_tula palkan RailsConf 2019 OPTIONAL DEPS 38 # config/application.rb #

    Store uploaded files on the local file system # (unless AWS S3 is configured) config.active_storage.service = if AWSConfig.storage_configured? $stdout.puts "Using :amazon service for Active Storage" :amazon else :local end
  23. 39.

    palkan_tula palkan RailsConf 2019 OPTIONAL SERVICES 39 class TwilioNotifyClient def

    initialize(sid:) if sid.nil? @noop = true warn "Twilio Notify client initialized in the no-op mode.\n” \ "See docs/twilio.md on how to configure it in development" return end end def send_notification(*args) return if noop? # ... end end TwilioNotifyClient.new(sid: ENV["TWILIO_NOTIFY_SID"])
  24. 40.
  25. 41.

    palkan_tula palkan RailsConf 2019 ENV HELL 41 ENV across all

    over the codebase Rails.env.smth? checks
  26. 42.

    palkan_tula palkan RailsConf 2019 ENV HELL 42 Limit ENV usage

    to config/environments/ <env>.rb Replace Rails.env.smth? with custom configuration settings
  27. 43.

    palkan_tula palkan RailsConf 2019 43 # config/application.rb config.graphiql_enabled = false

    # config/development.rb config.graphiql_enabled = true # routes.rb - if Rails.env.development? + if Rails.application.config.graphiql_enabled mount Graphiql ::Rack.new, at: "/graphiql" end ENV HELL
  28. 45.

    palkan_tula palkan RailsConf 2019 ENV HELL 45 ENV across all

    over the codebase Rails.env.smth? checks .env with dozens of entries (and .env.sample is out-of-date)
  29. 50.

    palkan_tula palkan RailsConf 2019 KEEPING CONFIGS 50 Store sensitive information

    in Rails Credentials (or similar) Keep non-sensitive information in named YAML files Allow overriding via ENV
  30. 54.

    palkan_tula palkan RailsConf 2019 54 PHASE #1 * We’re on

    Mars! MARS LANDING ✅ Repeatable dev env setup ✅ Transparent and zero-“changeme” configuration
  31. 56.

    palkan_tula palkan RailsConf 2019 TESTS ~ OXYGEN 56 Developing without

    tests is like breathing with no air Developing with slow tests is like breathing on the top of mt. Jomolungma Developing with unreliable (flaky) tests is like breathing in Chelyabinsk
  32. 59.

    palkan_tula palkan RailsConf 2019 59 GOAL PHASE #2 “Tests should

    be reliable and fast to not block the development” * Make oxygen level at least 21% to 2065!
  33. 63.

    palkan_tula palkan RailsConf 2019 TestProf 63 Tests specific profilers RSpec

    / Minitest extensions to speed up tests with as little refactoring as possible Custom Rubocop cops
  34. 64.

    palkan_tula palkan RailsConf 2019 64 3700 tests / 22 minutes

    = 170 TPM 9800 tests / 14 minutes = 700 TPM 4x faster! TestProf Born in “production”
  35. 69.

    palkan_tula palkan RailsConf 2019 ActiveRecordSharedConnection 69 test-prof.evilmartians.io/#/active_record_shared_connection # Rails 5.1+

    # connection is shared out-of-the-box # Rails <5.1 require “test_prof/recipes/active_record_one_love"
  36. 70.

    palkan_tula palkan RailsConf 2019 FASTER TESTS: HIGHLIGHTS 70 Get rid

    of database_cleaner Do not inline background jobs by default
  37. 72.

    palkan_tula palkan RailsConf 2019 SIDEKIQ SHAME 72 shared_context "sq:inline" do

    around(:each) do |ex| Sidekiq ::Testing.inline!(&ex) end end RSpec.configure do |config| config.include_context "sq:inline", sidekiq: :inline end
  38. 74.

    palkan_tula palkan RailsConf 2019 FASTER TESTS: HIGHLIGHTS 74 You (likely)

    don’t need database_cleaner Do not inline background jobs by default Take care of your factories
  39. 75.

    palkan_tula palkan RailsConf 2019 FACTORY CASCADE 75 factory :comment do

    answer author end factory :answer do question author end factory :question do author end create(:comment) # => creates 5 records
  40. 77.

    palkan_tula palkan RailsConf 2019 FASTER TESTS: HIGHLIGHTS 77 You (likely)

    don’t need database_cleaner Do not inline background jobs by default Take care of your factories …more
  41. 82.

    palkan_tula palkan RailsConf 2019 FACTORY LINTER 82 // Check factory

    definitions $ bundle exec rake factory_lint Factory lint detected the following errors: - :city should use a sequence for :name attribute, 'cause it has a uniqueness constraint bit.ly/factory-lint
  42. 83.

    palkan_tula palkan RailsConf 2019 FACTORY LINTER 83 # city.rb FactoryBot.define

    do factory :city do - name { Faker ::Address.city } + sequence(:name) { |n| Faker ::Address.city + " ( #{n})" } end end bit.ly/factory-lint
  43. 86.

    palkan_tula palkan RailsConf 2019 86 TURN ICE INTO WATER Bring

    project back to the healthy state PHASE #3 * Korolev crater
  44. 87.

    palkan_tula palkan RailsConf 2019 87 GOAL PHASE #3 “Project shouldn’t

    have significant security, consistency and performance issues“
  45. 89.

    palkan_tula palkan RailsConf 2019 bundler-audit 89 $ bundle audit update

    && bundle audit check Name: nokogiri Version: 1.8.5 Advisory: CVE-2019-11068 Criticality: Unknown URL: https: //github.com/sparklemotion/nokogiri/issues/1892 Title: Nokogiri gem, via libxslt, is affected by improper access control vulnerability Solution: upgrade to >= 1.10.3 Vulnerabilities found! github.com/rubysec/bundler-audit
  46. 92.

    palkan_tula palkan RailsConf 2019 RUBOCOP SECURITY 92 // Check potentially

    insecure code $ bundle exec rubocop --only Security Offenses: script.rb:33:21: C: Security/JSONLoad: Prefer JSON.parse over JSON.load. data = JSON.load(input.meta) ^^^^
  47. 93.

    palkan_tula palkan RailsConf 2019 CONSISTENCY 93 Models vs. DB user.update!(params.require(:user).permit(:name))

    => PG ::StringDataRightTruncation: ERROR: value too long for type character varying(255)
  48. 95.

    palkan_tula palkan RailsConf 2019 95 class Topic < ActiveRecord ::Base

    validates :description, presence: true end $ bundle exec database_consistency // description column is missing `null: false` constraint fail Topic description column should be required in the database database_consistency github.com/djezzl/database_consistency
  49. 96.

    palkan_tula palkan RailsConf 2019 database_validations 96 github.com/toptal/database_validations class Home <

    ActiveRecord ::Base # checks that a foreign key present db_belongs_to :city # checks that a unique index is present validates_db_uniqueness_of :name, case_sensitive: false end
  50. 99.

    palkan_tula palkan RailsConf 2019 .rubocop_strict.yml 99 AllCops: DisabledByDefault: true Security:

    Enabled: true Lint/Debugger: Enabled: true Lint/Syntax: Enabled: true Lint/Env: Enabled: true RSpec/Focus: Enabled: true
  51. 100.

    palkan_tula palkan RailsConf 2019 .rubocop_strict.yml 100 AllCops: DisabledByDefault: true Security:

    Enabled: true Lint/Debugger: Enabled: true Lint/Syntax: Enabled: true Lint/Env: Enabled: true RSpec/Focus: Enabled: true
  52. 101.

    palkan_tula palkan RailsConf 2019 .rubocop_strict.yml 101 AllCops: DisabledByDefault: true Security:

    Enabled: true Lint/Debugger: Enabled: true Lint/Syntax: Enabled: true Lint/Env: Enabled: true RSpec/Focus: Enabled: true
  53. 102.

    palkan_tula palkan RailsConf 2019 .rubocop_strict.yml 102 AllCops: DisabledByDefault: true Security:

    Enabled: true Lint/Debugger: Enabled: true Lint/Syntax: Enabled: true Lint/Env: Enabled: true RSpec/Focus: Enabled: true
  54. 103.

    palkan_tula palkan RailsConf 2019 .rubocop_strict.yml 103 AllCops: DisabledByDefault: true Security:

    Enabled: true Lint/Debugger: Enabled: true Lint/Syntax: Enabled: true Lint/Env: Enabled: true RSpec/Focus: Enabled: true
  55. 104.

    palkan_tula palkan RailsConf 2019 .rubocop_strict.yml + CI 104 # .circleci/config.yml

    rubocop_strict: executor: ruby steps: - attach_workspace: at: . - run: name: Strict Rubocop check command: | bundle exec rubocop -c .rubocop_strict.yml
  56. 106.

    palkan_tula palkan RailsConf 2019 .rubocop.yml 106 require: - rubocop-rspec -

    standard/cop/semantic_blocks inherit_gem: standard: config/base.yml AllCops: Include: # explicitly specify terraformed code - "**/community/**/*.rb" Exclude: - "db/schema.rb" - "bin /*"
  57. 107.

    palkan_tula palkan RailsConf 2019 .rubocop.yml 107 require: - rubocop-rspec -

    standard/cop/semantic_blocks inherit_gem: standard: config/base.yml AllCops: Include: # explicitly specify terraformed code - "**/community/**/*.rb" Exclude: - "db/schema.rb" - "bin /*"
  58. 108.

    palkan_tula palkan RailsConf 2019 .rubocop.yml 108 require: - rubocop-rspec -

    standard/cop/semantic_blocks inherit_gem: standard: config/base.yml AllCops: Include: # explicitly specify terraformed code - "**/community/**/*.rb" Exclude: - "db/schema.rb" - "bin /*"
  59. 111.

    palkan_tula palkan RailsConf 2019 SIDE EFFECTS 111 Non-atomic transactions class

    Post < ActiveRecord ::Base after_create :notify_author def notify_users # send email even if transaction failed UsersMailer.new_post(author, post.title).deliver_later end end
  60. 113.

    palkan_tula palkan RailsConf 2019 ISOLATOR 113 github.com/palkan/isolator $ bundle exec

    rspec 1) Create post example Failure/Error: job_or_instantiate(*args).enqueue Isolator ::BackgroundJobError: You are trying to enqueue background job inside db transaction. Details: ActionMailer ::DeliveryJob (UsersMailer, new_post)
  61. 116.

    palkan_tula palkan RailsConf 2019 DEAD GEMS 116 $ GEM_TRACK=1 bundle

    exec rspec Maybe unused gems: activerecord-postgres_enum-0.3.0 avatax-18.12.0 aws-sdk-2.10.9 rails-4.2.11.1 gibbon-2.2.5 bit.ly/track-gems
  62. 118.

    palkan_tula palkan RailsConf 2019 traceroute 118 github.com/amatsuda/traceroute $ bundle exec

    rake traceroute Unused routes (3): users#create users#new catalog#purchase Unreachable action methods (1): users#index2
  63. 120.

    palkan_tula palkan RailsConf 2019 TEMPLATES TRACKER 120 $ TT=1 bundle

    exec rspec ======== Unused Templates ========= /app/app/views/home/index.html.erb /app/app/views/housekeeping/scheduling/index.html.erb bit.ly/track-templates
  64. 122.

    palkan_tula palkan RailsConf 2019 FACTORY TRACE 122 $ FB_TRACE=1 bundle

    exec rspec total number of unique used factories & traits: 1 total number of unique unused factories & traits: 3 unused trait 'with_phone' of factory 'user' unused factory 'special_user' unused global trait 'with_email' github.com/djezzzl/factory_trace
  65. 123.

    palkan_tula palkan RailsConf 2019 123 TURN ICE INTO WATER PHASE

    #3 ✅ Project is healthy enough to bring more people
  66. 125.

    palkan_tula palkan RailsConf 2019 LEFT HOOK 125 github.com/Arkweid/lefthook $ git

    commit -m "chore: use lefthook" RUNNING HOOKS GROUP: pre-commit EXECUTE > rubocop 1 file inspected, 1 offense detected rubocop
  67. 126.

    palkan_tula palkan RailsConf 2019 LEFT HOOK 126 github.com/Arkweid/lefthook $ git

    commit -m "chore: use lefthook" RUNNING HOOKS GROUP: pre-commit EXECUTE > rubocop 1 file inspected, no offenses detected ✅ rubocop RUNNING HOOKS GROUP: prepare-commit-msg EXECUTE > jira_link.rb JIRA ticket ID (type 'N/n' to skip): 2019 ✅ jira_link.rb