Hunting bugs in the dark with git

Talk from the 2018 October WebDev Meetup Chur

Transcript

1. HUNTING BUGS IN THE DARK WITH GIT WebDev Meetup Chur 18.10.2018 Joel Ambass 1 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
2. STORYTIME 2 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
3. EN NORMALA TAG IM BÜRO WENN PLÖTZLICH... 3 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
4. He Joel, könntisch mol üsi Rails Version update? — CTO 4 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
5. Natürlich, das sött kei grossi Sach si. — optimistic me 5 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
6. EASY 6 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
7. UPDATE GEMFILE from gem 'rails', '5.1.0' to gem 'rails', '5.2.0' 7 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
8. INSTALL NEW VERSIONS $ bundle update rails 8 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
9. MIGRATE $ rails app:update Choose what you want identical config/boot.rb exist config conflict config/routes.rb Overwrite /myapp/config/routes.rb? (enter "h" for help) [Ynaqdh] force config/routes.rb conflict config/application.rb Overwrite /myapp/config/application.rb? (enter "h" for help) [Ynaqdh] force config/application.rb conflict config/environment.rb 9 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
10. RUN TESTS AND FINISH $ bundle exec rspec specs/ 10 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
11. RUN TESTS AND FINISH FAIL $ bundle exec rspec specs/ ! Finished in 8 minutes 58 seconds (files took 13.95 seconds to load) 1954 examples, 1 failure 11 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
12. SO.. THERE'S A BUG Failure/Error: open.order(created_at: :desc) ArgumentError: wrong number of arguments (given 0, expected 1+) in this code scope :open, -> { where(state: 'open') } scope :active, -> { open.order(created_at: :desc) } 12 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
13. BUT WHY? 13 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
14. WE DIDN'T CHANGE OUR OWN CODE... 14 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
15. SO COULD IT BE... 15 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
16. THAT THERE IS A BUG IN RAILS ! 16 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
17. BUT... 17 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
18. The First Rule of Programming: It's Always Your Fault — random dude on the internet 18 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
19. The First Rule of Programming: It's (Almost) Always Your Fault — another random dude on the internet (me) 19 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
20. STEP 1 VERIFY THAT YOU'RE NOT CRAZY 20 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
21. CREATE A MINIMAL WORKING EXAMPLE 21 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
22. SPECIFY YOUR VERSIONS gemfile(true) do source "https://rubygems.org" # Activate the gem you are reporting the issue against. gem "activerecord", '5.2.0' gem "sqlite3" end 22 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
23. SETUP A MINIMAL ENVIRONMENT ActiveRecord::Schema.define do create_table :businesses, force: true do |t| t.string :state end end class Business < ActiveRecord::Base scope :open, -> { where(state: "open") } scope :available, -> { open } end 23 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
24. SETUP A MINIMAL ENVIRONMENT ActiveRecord::Schema.define do create_table :businesses, force: true do |t| t.string :state end end class Business < ActiveRecord::Base scope :open, -> { where(state: "open") } scope :available, -> { open } end 23 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
25. WRITE A TEST CASE class BugTest < Minitest::Test def test_overwritten_open Business.create!(state: nil) Business.create!(state: :open) assert_equal Business.available, Business.open end end 24 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
26. PUT IT IN A FILE # frozen_string_literal: true begin require "bundler/inline" rescue LoadError => e $stderr.puts "Bundler version 1.10 or later is required. Please update your Bundler" raise e end gemfile(true) do source "https://rubygems.org" # Activate the gem you are reporting the issue against. gem "activerecord", '5.2.0' gem "sqlite3" end require "active_record" require "minitest/autorun" require "logger" # Ensure backward compatibility with minitest 4. Minitest::Test = MiniTest::Unit::TestCase unless defined?(Minitest::Test) # This connection will do for database-independent bug reports. ActiveRecord::Base.establish_connection(adapter: "sqlite3", database: ":memory:") ActiveRecord::Base.logger = Logger.new(STDOUT) ActiveRecord::Schema.define do create_table :businesses, force: true do |t| t.string :state end end class Business < ActiveRecord::Base scope :open, -> { where(state: "open") } scope :available, -> { open } end class BugTest < Minitest::Test def test_overwritten_open Business.create!(state: nil) Business.create!(state: :open) assert_equal Business.available, Business.open end end 25 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
27. RUN IT! $ ruby test.rb BugTest#test_overwritten_open: ArgumentError: wrong number of arguments (given 0, expected 1..3) 1 runs, 0 assertions, 0 failures, 1 errors, 0 skips 26 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
28. CHANGE TO LAST KNOWN WORKING VERSION gemfile(true) do source "https://rubygems.org" # Activate the gem you are reporting the issue against. gem "activerecord", '5.1.0' gem "sqlite3" end 27 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
29. RUN IT! $ ruby test.rb 1 runs, 1 assertions, 0 failures, 0 errors, 0 skips ! 28 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
30. STEP 2 FIND THE BUG 29 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
31. THE PROBLEMS 1. We don't know the codebase 2. The codebase is very big 3. We don't know where the bug is hiding 30 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
32. WHAT DO WE HAVE? A script that tells us whether the bug is present or not in a given version 31 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
33. WHAT DO WE HAVE? Every change ever made to rails in a git repository 32 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
34. WHAT DO WE HAVE? Version labels which tell us which commits were made between two versions 33 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
35. ! 34 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
36. ! 1. Checkout every commit made between rails 5.1.0 and 5.2.0 2. run the script for it 3. see if it fails 4. find the commit which introduced the bug 35 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
37. !" Nr of commits between rails 5.1.0 and 5.2.0: 580 Nr of seconds I wanna spend on this: 0 36 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
38. STEP 3 FIND THE BUG (IN A SANE WAY) 37 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
39. GIT BISECT TO THE RESCUE ▸ performs a binary search on all commits between two given. ▸ checks out each of those commits. ▸ tells us which commit introduced the bug. 38 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
40. WORKFLOW 1. specify good and bad commit 2. check if bug is present in checked out commit 3. mark commit as bad/good 4. continue with next commit 39 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
41. WORKFLOW 40 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
42. $ git checkout 5-2-stable $ git bisect start $ git bisect bad # Current version is bad $ git bisect good 5-1-stable # 5-1-stable is known to be good # run script $ ruby test.rb # mark as bad $ git bisect bad # or mark as good $ git bisect good # repeat from run script... 41 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
43. WE CAN DO BETTER.. 42 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
44. IT SUPPORTS SCRIPTS 1. specify good and bad commit. 2. specify script which should be run for each commit. 3. let git bisect do the work. 43 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
45. $ git checkout 5-2-stable $ git bisect start $ git bisect bad # Current version is bad $ git bisect good 5-1-stable # 5-1-stable is known to be good $ git bisect run ruby test.rb # run script 44 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
46. WE FOUND IT 4bdd86fb1e608d0031ee5e03af4ae0b13b41fea3 is the first bad commit commit 4bdd86fb1e608d0031ee5e03af4ae0b13b41fea3 Author: Ryuta Kamizono <kamipo@gmail.com> Date: Thu Jun 1 04:40:55 2017 +0900 The receiver in a scope should be a `relation` Currently the receiver in a scope is `klass`, not `relation`. I think it is a strange because the receiver in `default_scope` and a scope on association is `relation`. I fixed to the receiver is to be a `relation` properly for consistency. :040000 040000 72808c470ac5f21220d7706092fec572eed33bfe e172211262e36eefaa17c0c356b3889ad1d5f38a M activerecord bisect run success 45 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
47. COMMON PITFALLS 46 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
48. THE SCRIPT NEEDS TO USE THE CHECKED OUT COMMIT gemfile(true) do source "https://rubygems.org" gem "activerecord", path: "./activerecord/" gem "activesupport", path: "./activesupport/" gem "activemodel", path: "./activemodel/" gem "sqlite3" end 47 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
49. FRAGILE COMMITS ▸ External dependency is not available anymore. ▸ There's no way to figure out which version of a dependency is needed. 48 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
50. MANY USE CASES ▸ See which commit introduced a bug. ▸ See which commit improved performance in benchmarks. ▸ See which commit changed any kind of behaviour. 49 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass
51. HAPPY HUNTING! 50 — WebDev Meetup Chur - 18.10.2018 - Joel Ambass