Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Observing Change: A Gold Master Test in Practice

Observing Change: A Gold Master Test in Practice

Tests try to observe change. But are some systems too big to observe them all? What if we need to test a function with a very complex output?

In this talk, we'll explore a Gold Master test– a special test for evaluating complicated legacy systems. We'll look at how this test takes an input, such as a production database, runs it through a transformative function, and then compares the output to an approved version of the output.

Testers of every experience level will leave this talk with a new technique for evaluating complex environments, and a broader conception of what a test can be.

A9689bb4f705b8bede56deddf66c87ef?s=128

Jake Worth

April 25, 2017
Tweet

Transcript

  1. welcome 1

  2. Any code of your own that you haven't looked at

    for six or more months might as well have been written by someone else. — Eagleson's Law 2
  3. legacy code 3

  4. old interfaces 4

  5. vulnerabilities 5

  6. dead code 6

  7. profit & users 7

  8. developing legacy code is inevitable 8

  9. tests 9

  10. untested legacy code 10

  11. types of tests 1.unit 2.API 3.UI 11

  12. 12

  13. ??? 13

  14. observing change: a gold master test in practice jake @jwworth

    worth hashrocket 14
  15. agenda 1.defining the gold master test 2.writing the test 3.working

    with the test 15
  16. 1. defining the gold master test 16

  17. 17

  18. In nearly every legacy system, what the system does is

    more important than what it's supposed to do. — Michael C. Feathers, Working Effectively With Legacy Code, p. 186 18
  19. characterization test A characterization test is a test that characterizes

    the actual behavior of a piece of code.1 1 Michael C. Feathers, Working Effectively With Legacy Code, pg. 195. 19
  20. characterization test process1 1.Use a piece of code in a

    test harness. 2.Write an assertion that you know will fail. 3.Let the failure tell you what the behavior is. 4.Change the test so that it expects the behavior the code produces. 1 Michael C. Feathers, Working Effectively With Legacy Code, pg. 195. 20
  21. # some/crummy_spec.rb expect(something).to eq 2 => Failure/Error: expect(something).to eq 2

    expected: 2 got: 1 expect(something).to eq 1 => 1 example, 0 failures 21
  22. a heuristic for writing characterization tests1 (abridged) 1.Write tests where

    you will make changes. Write as many cases as you feel you need. 2.Look at the specific things you are going to change, and attempt to write tests for those. 3.Write additional tests on a case-by-case basis. 1 Michael C. Feathers, Working Effectively With Legacy Code, pg. 195. 22
  23. gold master test A gold master test is a regression

    test for complex, untested systems that asserts a consistent macro-level behavior. 23
  24. first run 24

  25. subsequent runs 25

  26. ideal application • mature • complex • we expect minimal

    change to output 26
  27. benefits • rigorous development standards • exposes surprising behavior •

    useful for forensic analysis 27
  28. gold master test A gold master test is a regression

    test for complex, untested systems that asserts a consistent macro-level behavior. 28
  29. 2. writing the test 29

  30. feature revisited • large production database • fed into a

    complex PostgreSQL function • output into a giant CSV file 30
  31. testing writing phases i. preparation ii. test iii. evaluation 31

  32. i. preparation 32

  33. # lib/tasks/create_db.rake desc "Generate the gold master test database" task

    :create_db do destination = Rails.root.join('spec/fixtures/gold_master.sql') sh "pg_dump sanitized_utility_database <pg flags> #{destination}" end 33
  34. ii. test 34

  35. # spec/features/gold_master_spec.rb describe 'Fan.shred!' do it 'produces a consistent result'

    do end end 35
  36. # spec/features/gold_master_spec.rb ApplicationRecord.connection.execute <<-SQL truncate schema_migrations; #{Rails.root.join('spec/fixtures/gold_master.sql').read} SQL 36

  37. # spec/features/gold_master_spec.rb actual = Fan.shred! 37

  38. testing strategy this test can do two things: 1.first run:

    generate gold master 2.subsequent runs: compare current result to gold master 38
  39. # spec/features/gold_master_spec.rb gold_master_file = Rails.root.join('spec/fixtures/gold_master.txt') if !gold_master_file.exist? gold_master_file.write(actual) ... 39

  40. # spec/features/gold_master_spec.rb gold_master_file = Rails.root.join('spec/fixtures/gold_master.txt') if !gold_master_file.exist? gold_master_file.write(actual) else gold_master

    = gold_master_file.read if gold_master != actual gold_master_file.write(actual) end ... 40
  41. # spec/features/gold_master_spec.rb gold_master_file = Rails.root.join('spec/fixtures/gold_master.txt') if !gold_master_file.exist? gold_master_file.write(actual) else gold_master

    = gold_master_file.read if gold_master != actual gold_master_file.write(actual) end expect(actual).to eq(gold_master) end 41
  42. # spec/features/gold_master_spec.rb describe 'Fan.shred!' do it 'produces a consistent result'

    do ApplicationRecord.connection.execute <<-SQL truncate schema_migrations; #{Rails.root.join('spec/fixtures/gold_master.sql').read} SQL actual = Fan.shred! gold_master_file = Rails.root.join('spec/fixtures/gold_master.txt') if !gold_master_file.exist? gold_master_file.write(actual) else gold_master = gold_master_file.read if gold_master != actual gold_master_file.write(actual) end expect(actual).to eq(gold_master) end end end 42
  43. iii. evaluation 43

  44. what happens when it fails? 44

  45. 3. working with the test 45

  46. 46

  47. ideal application • [x] mature • [x] complex • [x]

    we expect minimal change to output 47
  48. assertion The homepage, given the same data, should not change,

    without us knowing why. https://github.com/hashrocket/hr-til/tree/gold- master-demo 48
  49. preparation revisited • get a production database dump • scrub

    sensitive information • dump as plaintext sql • check in sql 49
  50. scrub data -- db/sanitize_production.sql -- Sanitize devs update developers set

    username = concat('rocketeer', id), email = concat('rocketeer', id, '@hashrocket.com'), twitter_handle = concat('hashrocket', id), admin = false, slack_name = concat('rocketeer', id); -- Clear sessions delete from authem_sessions; -- Limit the posts delete from posts where id > 200; 50
  51. # spec/features/gold_master_spec.rb describe 'gold master' do before do ApplicationRecord.connection.execute <<-SQL

    truncate schema_migrations; #{Rails.root.join('spec/fixtures/gold_master.sql').read} SQL end end 51
  52. # spec/features/gold_master_spec.rb visit root_path 52

  53. # spec/features/gold_master_spec.rb page_html = page.html 53

  54. # spec/features/gold_master_spec.rb describe 'gold master' do before do ApplicationRecord.connection.execute <<-SQL

    truncate schema_migrations; #{Rails.root.join('spec/fixtures/gold_master.sql').read} SQL visit root_path end specify 'the homepage does not change' do page_html = page.html gold_master_file = Rails.root.join('spec/fixtures/gold_master.txt') if !gold_master_file.exist? gold_master_file.write(page_html) else gold_master = gold_master_file.read if gold_master != page_html gold_master_file.write(page_html) end expect(page_html).to eq(gold_master) end end end 54
  55. 55

  56. 56

  57. challenges • maintenance -> it's an investment • slower ->

    we can optimize data • implies correctness -> teaching opportunity 57
  58. future plans 58

  59. 59

  60. conclusion 60

  61. thank you! jake @jwworth worth hashrocket 61

  62. further reading • Slides: https://speakerdeck.com/jwworth • Code samples: https://github.com/hashrocket/hr-til/tree/gold-master-demo •

    Image Source: Watermelon Kaboom!!, KT King, Flickr.com, https://www.flickr.com/photos/xtrah/5005443977, Accessed 18 Mar 2017. 1. Change Code Without Fear, Nada daVeiga, Dr. Dobb's, http://www.drdobbs.com/tools/change-code-without-fear/206105233, February 6, 2008. Accessed 26 February, 2017. 2. Detecting behavioural changes when refactoring a web-based legacy system, Peter Spegel, May 21, 2015. 3. Gold Master Testing, Bryan Helmkamp, Code Climate Blog, http://blog.codeclimate.com/blog/2014/02/20/gold-master-testing/, February 20, 2014. Accessed February 26, 2017. 4. Gold Master Testing, Jake Worth, Hashrocket Blog, https://hashrocket.com/blog/posts/gold-master-testing, November 8, 2016. 5. How I run Legacy Code Retreat, J. B. Rainsberger, Legacy Code Retreat, http://legacycoderetreat.typepad.com/blog/2011/11/how-i-run-legacy-code- retreat.html, November 21, 2011. Accessed 26 February, 2017. 6. How Not To Write Golden Master Tests, J. B. Rainsberger, The Code Whisperer, http://blog.thecodewhisperer.com/permalink/how-not-to-write-golden-master- tests. Accessed 22 March, 2017. 7. Using the Golden Master technique to test legacy code, Chris Melinn, https://chrismelinn.wordpress.com/2013/04/12/using-the-golden-master-technique-to- test-legacy-code/, 12 April 2013. Accessed 26 February, 2017. 8. Working Effectively With Characterization Tests - Part 2, Alberto Savoia, http://www.artima.com/weblogs/viewpost.jsp?thread=198674, March 13, 2007. Accessed February 26, 2017. 9. Working Effectively With Characterization Tests, Alberto Savoia, artima developer, http://www.artima.com/weblogs/viewpost.jsp?thread=198296, March 9, 2007. Accessed February 26, 2017. 10.Working Effectively With Legacy Code, Michael C. Feathers, 2005. 62