Observing Change: A Gold Master Test in Practice

Tests try to observe change. But are some systems too big to observe them all? What if we need to test a function with a very complex output?

In this talk, we'll explore a Gold Master test– a special test for evaluating complicated legacy systems. We'll look at how this test takes an input, such as a production database, runs it through a transformative function, and then compares the output to an approved version of the output.

Testers of every experience level will leave this talk with a new technique for evaluating complex environments, and a broader conception of what a test can be.

Transcript

1. welcome 1
2. Any code of your own that you haven't looked at for six or more months might as well have been written by someone else. — Eagleson's Law 2
3. legacy code 3
4. old interfaces 4
5. vulnerabilities 5
6. dead code 6
7. profit & users 7
8. developing legacy code is inevitable 8
9. tests 9
10. untested legacy code 10
11. types of tests 1.unit 2.API 3.UI 11
13. ??? 13
14. observing change: a gold master test in practice jake @jwworth worth hashrocket 14
15. agenda 1.defining the gold master test 2.writing the test 3.working with the test 15
16. 1. defining the gold master test 16
18. In nearly every legacy system, what the system does is more important than what it's supposed to do. — Michael C. Feathers, Working Effectively With Legacy Code, p. 186 18
19. characterization test A characterization test is a test that characterizes the actual behavior of a piece of code.1 1 Michael C. Feathers, Working Effectively With Legacy Code, pg. 195. 19
20. characterization test process1 1.Use a piece of code in a test harness. 2.Write an assertion that you know will fail. 3.Let the failure tell you what the behavior is. 4.Change the test so that it expects the behavior the code produces. 1 Michael C. Feathers, Working Effectively With Legacy Code, pg. 195. 20
21. # some/crummy_spec.rb expect(something).to eq 2 => Failure/Error: expect(something).to eq 2 expected: 2 got: 1 expect(something).to eq 1 => 1 example, 0 failures 21
22. a heuristic for writing characterization tests1 (abridged) 1.Write tests where you will make changes. Write as many cases as you feel you need. 2.Look at the specific things you are going to change, and attempt to write tests for those. 3.Write additional tests on a case-by-case basis. 1 Michael C. Feathers, Working Effectively With Legacy Code, pg. 195. 22
23. gold master test A gold master test is a regression test for complex, untested systems that asserts a consistent macro-level behavior. 23
24. first run 24
25. subsequent runs 25
26. ideal application • mature • complex • we expect minimal change to output 26
27. benefits • rigorous development standards • exposes surprising behavior • useful for forensic analysis 27
28. gold master test A gold master test is a regression test for complex, untested systems that asserts a consistent macro-level behavior. 28
29. 2. writing the test 29
30. feature revisited • large production database • fed into a complex PostgreSQL function • output into a giant CSV file 30
31. testing writing phases i. preparation ii. test iii. evaluation 31
32. i. preparation 32
33. # lib/tasks/create_db.rake desc "Generate the gold master test database" task :create_db do destination = Rails.root.join('spec/fixtures/gold_master.sql') sh "pg_dump sanitized_utility_database <pg flags> #{destination}" end 33
34. ii. test 34
35. # spec/features/gold_master_spec.rb describe 'Fan.shred!' do it 'produces a consistent result' do end end 35
36. # spec/features/gold_master_spec.rb ApplicationRecord.connection.execute <<-SQL truncate schema_migrations; #{Rails.root.join('spec/fixtures/gold_master.sql').read} SQL 36
37. # spec/features/gold_master_spec.rb actual = Fan.shred! 37
38. testing strategy this test can do two things: 1.first run: generate gold master 2.subsequent runs: compare current result to gold master 38
39. # spec/features/gold_master_spec.rb gold_master_file = Rails.root.join('spec/fixtures/gold_master.txt') if !gold_master_file.exist? gold_master_file.write(actual) ... 39
40. # spec/features/gold_master_spec.rb gold_master_file = Rails.root.join('spec/fixtures/gold_master.txt') if !gold_master_file.exist? gold_master_file.write(actual) else gold_master = gold_master_file.read if gold_master != actual gold_master_file.write(actual) end ... 40
41. # spec/features/gold_master_spec.rb gold_master_file = Rails.root.join('spec/fixtures/gold_master.txt') if !gold_master_file.exist? gold_master_file.write(actual) else gold_master = gold_master_file.read if gold_master != actual gold_master_file.write(actual) end expect(actual).to eq(gold_master) end 41
42. # spec/features/gold_master_spec.rb describe 'Fan.shred!' do it 'produces a consistent result' do ApplicationRecord.connection.execute <<-SQL truncate schema_migrations; #{Rails.root.join('spec/fixtures/gold_master.sql').read} SQL actual = Fan.shred! gold_master_file = Rails.root.join('spec/fixtures/gold_master.txt') if !gold_master_file.exist? gold_master_file.write(actual) else gold_master = gold_master_file.read if gold_master != actual gold_master_file.write(actual) end expect(actual).to eq(gold_master) end end end 42
43. iii. evaluation 43
44. what happens when it fails? 44
45. 3. working with the test 45
47. ideal application • [x] mature • [x] complex • [x] we expect minimal change to output 47
48. assertion The homepage, given the same data, should not change, without us knowing why. https://github.com/hashrocket/hr-til/tree/gold- master-demo 48
49. preparation revisited • get a production database dump • scrub sensitive information • dump as plaintext sql • check in sql 49
50. scrub data -- db/sanitize_production.sql -- Sanitize devs update developers set username = concat('rocketeer', id), email = concat('rocketeer', id, '@hashrocket.com'), twitter_handle = concat('hashrocket', id), admin = false, slack_name = concat('rocketeer', id); -- Clear sessions delete from authem_sessions; -- Limit the posts delete from posts where id > 200; 50
51. # spec/features/gold_master_spec.rb describe 'gold master' do before do ApplicationRecord.connection.execute <<-SQL truncate schema_migrations; #{Rails.root.join('spec/fixtures/gold_master.sql').read} SQL end end 51
52. # spec/features/gold_master_spec.rb visit root_path 52
53. # spec/features/gold_master_spec.rb page_html = page.html 53
54. # spec/features/gold_master_spec.rb describe 'gold master' do before do ApplicationRecord.connection.execute <<-SQL truncate schema_migrations; #{Rails.root.join('spec/fixtures/gold_master.sql').read} SQL visit root_path end specify 'the homepage does not change' do page_html = page.html gold_master_file = Rails.root.join('spec/fixtures/gold_master.txt') if !gold_master_file.exist? gold_master_file.write(page_html) else gold_master = gold_master_file.read if gold_master != page_html gold_master_file.write(page_html) end expect(page_html).to eq(gold_master) end end end 54
57. challenges • maintenance -> it's an investment • slower -> we can optimize data • implies correctness -> teaching opportunity 57
58. future plans 58
60. conclusion 60
61. thank you! jake @jwworth worth hashrocket 61
62. further reading • Slides: https://speakerdeck.com/jwworth • Code samples: https://github.com/hashrocket/hr-til/tree/gold-master-demo • Image Source: Watermelon Kaboom!!, KT King, Flickr.com, https://www.flickr.com/photos/xtrah/5005443977, Accessed 18 Mar 2017. 1. Change Code Without Fear, Nada daVeiga, Dr. Dobb's, http://www.drdobbs.com/tools/change-code-without-fear/206105233, February 6, 2008. Accessed 26 February, 2017. 2. Detecting behavioural changes when refactoring a web-based legacy system, Peter Spegel, May 21, 2015. 3. Gold Master Testing, Bryan Helmkamp, Code Climate Blog, http://blog.codeclimate.com/blog/2014/02/20/gold-master-testing/, February 20, 2014. Accessed February 26, 2017. 4. Gold Master Testing, Jake Worth, Hashrocket Blog, https://hashrocket.com/blog/posts/gold-master-testing, November 8, 2016. 5. How I run Legacy Code Retreat, J. B. Rainsberger, Legacy Code Retreat, http://legacycoderetreat.typepad.com/blog/2011/11/how-i-run-legacy-code- retreat.html, November 21, 2011. Accessed 26 February, 2017. 6. How Not To Write Golden Master Tests, J. B. Rainsberger, The Code Whisperer, http://blog.thecodewhisperer.com/permalink/how-not-to-write-golden-master- tests. Accessed 22 March, 2017. 7. Using the Golden Master technique to test legacy code, Chris Melinn, https://chrismelinn.wordpress.com/2013/04/12/using-the-golden-master-technique-to- test-legacy-code/, 12 April 2013. Accessed 26 February, 2017. 8. Working Effectively With Characterization Tests - Part 2, Alberto Savoia, http://www.artima.com/weblogs/viewpost.jsp?thread=198674, March 13, 2007. Accessed February 26, 2017. 9. Working Effectively With Characterization Tests, Alberto Savoia, artima developer, http://www.artima.com/weblogs/viewpost.jsp?thread=198296, March 9, 2007. Accessed February 26, 2017. 10.Working Effectively With Legacy Code, Michael C. Feathers, 2005. 62