Behat for Characterization Tests

Once an API ships it doesn't matter how it should behave—how it actually behaves is the important part. Users depend on the existing behavior, and we need a way to ensure it doesn't change. Behat is a tool that was built to help design software, but it’s actually a great tool for capturing existing behavior too. We’ve used these tools to gain confidence to refactor 5+-year-old apps by capturing the existing behavior before making changes. I want to share the secrets we learned with you.

Transcript

1. @mheap #phpworld for characterization tests
2. @mheap #phpworld Imagine this…
3. @mheap #phpworld $$$
4. @mheap #phpworld $$$ $$$ $$$
5. @mheap #phpworld $$$ Billing Rating Auth Report Masking Encoding $$$ $$$
6. @mheap #phpworld $$$ Billing Rating Auth Report Masking Encoding $$$ $$$
7. @mheap #phpworld $$$ $$$ $$$ Billing Rating Auth Report Masking Encoding public function provides() { return [Client ::class]; }
8. @mheap #phpworld
9. @mheap #phpworld $ ls -l test total 0
10. @mheap https://twitter.com/brianwisti/status/503987766032494592 YOU ARE IN A LEGACY CODEBASE > RUN TESTS YOU HAVE NO TESTS > READ SPEC YOU HAVE NO SPEC > WRITE FIX YOU ARE EATEN BY AN ELDER CODE HACK.
11. @mheap #phpworld Imagine
12. @mheap #phpworld Hello, I’m Michael
13. @mheap #phpworld @mheap
14. @mheap #phpworld
16. @mheap #phpworld Today
17. @mheap #phpworld What is legacy code?
18. @mheap Michael Feathers - Working Effectively with Legacy Code “code without tests” Legacy Code is
19. @mheap Legacy Code is “valuable code that we feel afraid to change” JB Rainsberger - Surviving Legacy Code with Golden Master and Sampling
20. @mheap #phpworld What if I make a change and break something?
21. @mheap #phpworld What if I make a change and change something?
22. @mheap #phpworld Intended or Unintended
23. @mheap #phpworld Intended or Unintended
24. @mheap #phpworld Intended or Unintended
25. @mheap #phpworld But there aren’t any tests!
26. @mheap #phpworld class SalesUtil { const BQ = 1000.0; const BCR = 0.20; const OQM1 = 1.5; const OQM2 = self ::OQM1 * 2; static public function calculate($tSales) { if ($tSales <= static ::BQ) { return $tSales * static ::BCR; } else if ($tSales <= static ::BQ*2) { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1; } else { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1 + ($tSales - static ::BQ * 2) * static ::BCR * static ::OQM2; } } }
27. @mheap #phpworld class SalesUtil { const BQ = 1000.0; const BCR = 0.20; const OQM1 = 1.5; const OQM2 = self ::OQM1 * 2; static public function calculate($tSales) { if ($tSales <= static ::BQ) { return $tSales * static ::BCR; } else if ($tSales <= static ::BQ*2) { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1; } else { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1 + ($tSales - static ::BQ * 2) * static ::BCR * static ::OQM2; } } }
28. @mheap #phpworld class SalesUtil { const BQ = 1000.0; const BCR = 0.20; const OQM1 = 1.5; const OQM2 = self ::OQM1 * 2; static public function calculate($tSales) { if ($tSales <= static ::BQ) { return $tSales * static ::BCR; } else if ($tSales <= static ::BQ*2) { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1; } else { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1 + ($tSales - static ::BQ * 2) * static ::BCR * static ::OQM2; } } }
29. @mheap #phpworld class SalesUtil { const BQ = 1000.0; const BCR = 0.20; const OQM1 = 1.5; const OQM2 = self ::OQM1 * 2; static public function calculate($tSales) { if ($tSales <= static ::BQ) { return $tSales * static ::BCR; } else if ($tSales <= static ::BQ*2) { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1; } else { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1 + ($tSales - static ::BQ * 2) * static ::BCR * static ::OQM2; } } }
30. @mheap #phpworld use PHPUnit\Framework\TestCase; class SalesUtilTest extends TestCase { public function testX() { $this ->assertEquals(null, SalesUtil ::calculate(1000)); } }
31. @mheap #phpworld use PHPUnit\Framework\TestCase; class SalesUtilTest extends TestCase { public function testX() { $this ->assertEquals(null, SalesUtil ::calculate(999)); } }
32. @mheap #phpworld use PHPUnit\Framework\TestCase; class SalesUtilTest extends TestCase { public function testX() { $this ->assertEquals(null, SalesUtil ::calculate(1000)); } }
33. @mheap #phpworld There was 1 failure: 1) SalesUtilTest ::testX Failed asserting that 200 matches expected null. /Users/michael/development/oss/characterisation-tests-examples/sales-util/test/SalesUtilTest.php:8 FAILURES! Tests: 1, Assertions: 1, Failures: 1.
34. @mheap #phpworld use PHPUnit\Framework\TestCase; class SalesUtilTest extends TestCase { public function testSalesLTE1000Pay20PercentCommission() { $this ->assertEquals(200, SalesUtil ::calculate(1000)); } }
35. @mheap #phpworld class SalesUtil { const BQ = 1000.0; const BCR = 0.20; const OQM1 = 1.5; const OQM2 = self ::OQM1 * 2; static public function calculate($tSales) { if ($tSales <= static ::BQ) { return $tSales * static ::BCR; } else if ($tSales <= static ::BQ*2) { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1; } else { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1 + ($tSales - static ::BQ * 2) * static ::BCR * static ::OQM2; } } }
36. @mheap #phpworld There was 1 failure: 1) SalesUtilTest ::testX Failed asserting that 500.0 matches expected null. /Users/michael/development/oss/characterisation-tests-examples/sales-util/test/SalesUtilTest.php:13 FAILURES! Tests: 2, Assertions: 2, Failures: 1.
37. @mheap #phpworld public function test2000Gets500Commission() { $this ->assertEquals(500, SalesUtil ::calculate(2000)); } public function testX() { $this ->assertEquals(null, SalesUtil ::calculate(1600)); }
38. @mheap #phpworld There was 1 failure: 1) SalesUtilTest ::testX Failed asserting that 380.0 matches expected null. /Users/michael/development/oss/characterisation-tests-examples/sales-util/test/SalesUtilTest.php:19 FAILURES! Tests: 3, Assertions: 3, Failures: 1.
39. @mheap #phpworld class SalesUtil { const BQ = 1000.0; const BCR = 0.20; const OQM1 = 1.5; const OQM2 = self ::OQM1 * 2; static public function calculate($tSales) { if ($tSales <= static ::BQ) { return $tSales * static ::BCR; } else if ($tSales <= static ::BQ*2) { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1; } else { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1 + ($tSales - static ::BQ * 2) * static ::BCR * static ::OQM2; } } }
40. @mheap #phpworld Input Expected 0 0 10 2 500 100 1000 200 1001 200.3 1500 350 1982 494.6 1999.9 499.97 2000 500 2001 500.9 5000 3200 7890 5801 10000 7700
41. @mheap #phpworld public function testX() { $this ->assertEquals(null, SalesUtil ::calculate(-123)); }
42. @mheap #phpworld public function testNegativeSalesGivesCommissionIs() { $this ->assertEquals(-24.6, SalesUtil ::calculate(-123)); }
43. @mheap #phpworld Do not change the code
44. @mheap #phpworld Do not change the code
45. @mheap #phpworld Do NOT change the code
46. @mheap https://en.wikipedia.org/wiki/Equivalence_partitioning What’s the minimum number of tests?
47. @mheap #phpworld We’re half way! (and we’ve barely mentioned Behat)
48. @mheap #phpworld Choosing your pinch points
49. @mheap #phpworld Choose HTTP
50. @mheap #phpworld Choose Behat
51. @mheap #phpworld Given I have 1 "Red Bucket" in my basket And I have 2 "Large Spades" in my basket When I add 1 "Red Bucket" in my basket Then I should see 2 "Red Buckets" in my basket And I should see 2 "Large Spades" in my basket
52. @mheap #phpworld Given I have 1 "Red Bucket" in my basket And I have 2 "Large Spades" in my basket When I add 1 "Red Bucket" in my basket Then I should see 2 "Red Buckets" in my basket And I should see 2 "Large Spades" in my basket
53. @mheap #phpworld Given I have 1 "Red Bucket" in my basket And I have 2 "Large Spades" in my basket When I add 1 "Red Bucket" in my basket Then I should see 2 "Red Buckets" in my basket And I should see 2 "Large Spades" in my basket
54. @mheap #phpworld Given I have 1 "Red Bucket" in my basket And I have 2 "Large Spades" in my basket When I add 1 "Red Bucket" in my basket Then I should see 2 "Red Buckets" in my basket And I should see 2 "Large Spades" in my basket
55. @mheap #phpworld Given I have 1 "Red Bucket" in my basket And I have 2 "Large Spades" in my basket When I add 1 "Red Bucket" in my basket Then I should see 2 "Red Buckets" in my basket And I should see 2 "Large Spades" in my basket
56. @mheap #phpworld Be pragmatic
57. @mheap #phpworld composer require behat/behat --dev vendor/bin/behat --init
58. @mheap #phpworld Testing a HTTP API
59. @mheap #phpworld $username = $_GET['username'] ?? error('username is required'); $key = $_GET['key'] ?? error('key is required'); $validUsers = [ "michael" => "kangar00s", "oscar" => "phparch17" ]; $validKey = $validUsers[strtolower($username)] ?? error('could not find user', 404); if ($key !== $validKey){ error('invalid apikey'); } output([ 'success' => 'true', 'user' => ['name' => $username] ]);
60. @mheap #phpworld $username = $_GET['username'] ?? error('username is required'); $key = $_GET['key'] ?? error('key is required'); $validUsers = [ "michael" => "kangar00s", "oscar" => "phparch17" ]; $validKey = $validUsers[strtolower($username)] ?? error('could not find user', 404); if ($key !== $validKey){ error('invalid apikey'); } output([ 'success' => 'true', 'user' => ['name' => $username] ]);
61. @mheap #phpworld $username = $_GET['username'] ?? error('username is required'); $key = $_GET['key'] ?? error('key is required'); $validUsers = [ "michael" => "kangar00s", "oscar" => "phparch17" ]; $validKey = $validUsers[strtolower($username)] ?? error('could not find user', 404); if ($key !== $validKey){ error('invalid apikey'); } output([ 'success' => 'true', 'user' => ['name' => $username] ]);
62. @mheap #phpworld error('username is required'); error('key is required'); error('could not find user', 404); error('invalid apikey'); output([ 'success' => 'true', 'user' => ['name' => $username] ]); 1. Missing username 2. Missing key 3. Invalid username 4. Invalid key 5. Successful authentication
63. @mheap #phpworld Scenario: No Username When I make a "GET" request to "/" Then the response status code should be "400" And the "error" property equals "username is required"
64. @mheap #phpworld Scenario: No API Key When I make a "GET" request to "/?username=foo" Then the response status code should be "400" And the "error" property equals "key is required”
65. @mheap #phpworld Scenario: Invalid Username When I make a "GET" request to "/?username=bananas&key=foo" Then the response status code should be "404" And the "error" property equals "could not find user” Scenario: Invalid API Key When I make a "GET" request to "/?username=michael&key=foo" Then the response status code should be "400" And the "error" property equals "invalid apikey"
66. @mheap #phpworld Scenario: Successful login When I make a "GET" request to "/?username=michael&key=kangar00s" Then the response status code should be "200" And the "success" property equals "true" And the "user.name" property equals "michael”
67. @mheap #phpworld Scenario Outline: Successful login When I make a "GET" request to "/?username=<name>&key=<key>" Then the response status code should be "200" And the "success" property equals "true" And the "user.name" property equals "<name>" Examples: | name | key | | michael | kangar00s | | oscar | phparch16 |
68. @mheap #phpworld More complex tests
69. @mheap #phpworld Given that header property "X-CustomAuthKey" is "Secret123" And that header property "Accept" is "application/json+vnd.v2" And that the request body is valid JSON ''' { "alpha":"beta", "count":3, "collection":["a","b","c"] } ''' When I make a "POST" request to "/account/balance" Then the response status code should be "200" And the "X-Remaining" header property equals "18" And the "balance.usd" property equals"1832.54" And the "balance.gbp" property equals "13.99" And the "balance.eur" property equals "19422.18"
70. @mheap #phpworld When I make a "POST" request to "/account/balance" Then the response status code should be “200" And the "X-Remaining" header property equals "18" And the response body contains the JSON data ''' { "in_credit": true, "balance": { "usd": 1832.54, "gbp": 13.99, "eur": 19422.18 } } '''
71. @mheap #phpworld When I make a "POST" request to "/account/balance" Then the response status code should be "200" And the "X-Remaining" header property equals "18" And the value of the "balance.usd" property matches the pattern “/^\d{1,3}\. \d{2}$/" And the value of the “last_updated” property matches the pattern "/^[0-9]{4} [\-][0-9]{2}[\-][0-9]{2} [0-9]{2}[:][0-9]{2}[:][0-9]{2}$/"
72. @mheap #phpworld composer require datasift/testrest-extension
73. @mheap #phpworld # behat.yml default: extensions: DataSift\BehatExtension: base_url: "http: //localhost:8080/" suites: default: contexts: - 'DataSift\BehatExtension\Context\RestContext'
74. @mheap #phpworld HTTP Mocks
75. @mheap #phpworld function get_balance($username) { return $client ->get( “http: //localhost:88/billing/check_balance/".$username ) ->getBody(); } $balance = get_balance($username); if ($username != 'michael') { error('wrong username'); } if (!$balance['has_credit']) { error('no credit remaining'); } success(['data' => 'valid account'])
76. @mheap #phpworld Mountebank
77. @mheap #phpworld npm install -g mountebank
78. @mheap #phpworld Given Mountebank is running And a mock exists at "/billing/check_balance" it should return "200" with the body: ''' { "has_credit": false, "credit_limit": 0 } ''' And the mocks are created When I make a "GET" request to "/auth?username=michael" Then the response status code should be "403" And the response is JSON And the response body JSON equals ''' { "error": "no credit remaining" } '''
79. @mheap #phpworld Given Mountebank is running And a mock exists at "/billing/check_balance" it should return "200" with the body: ''' { "has_credit": true, "credit_limit": 10000 } ''' And the mocks are created When I make a "GET" request to "/auth?username=michael" Then the response status code should be "200" And the response is JSON And the response body JSON equals ''' { "data": "valid account" } '''
80. @mheap #phpworld Automating a browser
81. @mheap #phpworld $ composer require --dev behat/mink-extension $ composer require --dev behat/mink-selenium2-driver $ java -Dwebdriver.chrome.driver=chromedriver -jar selenium- server-standalone-3.7.1.jar
82. @mheap #phpworld # behat.yml default: extensions: Behat\MinkExtension: base_url: 'https: //wikipedia.org' sessions: default: selenium2: browser: "chrome" suites: my_suite: contexts: - \Behat\MinkExtension\Context\MinkContext
83. @mheap #phpworld Scenario: Searching for a page that does exist Given I am on "/wiki/Main_Page" When I fill in "search" with "Behavior Driven Development" And I press "searchButton" Then I should see "agile software development"
84. @mheap #phpworld Custom Steps
85. @mheap #phpworld Scenario: Searching for a page that does exist Given I am on "/wiki/Main_Page" When I fill in "search" with "Behavior Driven Development" And I press "searchButton" Then I should see "agile software development"
86. @mheap #phpworld Scenario: Searching for a page that does exist Given I am on "/wiki/Main_Page" When I search for "Behavior Driven Development" Then I should see "agile software development"
87. @mheap #phpworld Scenario: Searching for a page that does exist Given I am on "/wiki/Main_Page" When I search for "Behavior Driven Development" Then I should see "agile software development" 1 scenario (1 undefined) 3 steps (1 passed, 1 undefined, 1 skipped) 0m3.63s (10.62Mb) >> my_suite suite has undefined steps. Please choose the context to generate snippets: [0] None [1] FeatureContext [2] Behat\MinkExtension\Context\MinkContext
88. @mheap #phpworld [0] None [1] FeatureContext [2] Behat\MinkExtension\Context\MinkContext > 1 --- FeatureContext has missing steps. Define them with these snippets: /** * @When I search for :arg1 */ public function iSearchFor($arg1) { throw new PendingException(); }
89. @mheap #phpworld ./vendor/bin/behat --append-snippets
90. @mheap #phpworld [0] None [1] FeatureContext [2] Behat\MinkExtension\Context\MinkContext > 1 u features/bootstrap/FeatureContext.php - `I search for "Behavior Driven Development"` definition added
91. @mheap #phpworld Feature: Test Scenario: Searching for a page that does exist Given I am on "/wiki/Main_Page" When I search for "Behavior Driven Development" TODO: write pending definition Then I should see "agile software development" 1 scenario (1 pending) 3 steps (1 passed, 1 pending, 1 skipped) 0m2.23s (10.61Mb)
92. @mheap #phpworld /** @BeforeScenario */ public function gatherContexts( Behat\Behat\Hook\Scope\BeforeScenarioScope $scope ) { $environment = $scope ->getEnvironment(); $this ->minkContext = $environment ->getContext( 'Behat\MinkExtension\Context\MinkContext' ); }
93. @mheap #phpworld /** * Fills in form field with specified id|name|label|value * Example: When I fill in "username" with: "bwayne" * Example: And I fill in "bwayne" for "username" * * @When /^( ?:|I )fill in "(?P<field>( ?:[^"]| \\")*)" with "(?P<value>( ?:[^"]| \\")*)"$/ * @When /^( ?:|I )fill in "(?P<field>( ?:[^"]| \\")*)" with:$/ * @When /^( ?:|I )fill in "(?P<value>( ?:[^"]| \\")*)" for "(?P<field>( ?:[^"]| \\")*)"$/ */ public function fillField($field, $value) { $field = $this ->fixStepArgument($field); $value = $this ->fixStepArgument($value); $this ->getSession() ->getPage() ->fillField($field, $value); }
94. @mheap #phpworld /** * Presses button with specified id|name|title|alt|value * Example: When I press "Log In" * Example: And I press "Log In" * @When /^( ?:|I )press "(?P<button>( ?:[^"]| \\")*)"$/ */ public function pressButton($button) { $button = $this ->fixStepArgument($button); $this ->getSession() ->getPage() ->pressButton($button); }
95. @mheap #phpworld /** * @When I search for :term */ public function iSearchFor($term) { $this ->minkContext ->fillField('search', $term); $this ->minkContext ->pressButton('searchButton'); }
96. @mheap #phpworld $ ./vendor/bin/behat Feature: Test Scenario: Searching for a page that does exist Given I am on "/wiki/Main_Page" When I search for "Behavior Driven Development" Then I should see "agile software development" 1 scenario (1 passed) 3 steps (3 passed)
97. @mheap #phpworld Scenario: Successful login When I make a "GET" request to "/?username=michael&key=kangar00s" Then the response status code should be "200" And the "success" property equals "true" And the "user.name" property equals "michael”
98. @mheap #phpworld Scenario: Successful login When I log in as "Michael" with the password "kangar00s" Then the response status code should be "200" And the "success" property equals "true" And the "user.name" property equals "michael”
99. @mheap #phpworld Snapshot testing
100. @mheap #phpworld public function toJson(){ return json_encode(["id" => $this ->id]); } —— public function test_json_formatting_is_correct() { $order = new Order(1); $this ->assertMatchesSnapshot($order ->toJson()); }
101. @mheap #phpworld $ ./vendor/bin/phpunit There was 1 incomplete test: 1) ExampleTest ::test_json_formatting_is_correct Snapshot created for ExampleTest __test_json_formatting_is_correct
102. @mheap #phpworld $ ./vendor/bin/phpunit OK (1 test, 1 assertion)
103. @mheap #phpworld public function toJson(){ return json_encode(["id" => 'Q'.$this ->id]); }
104. @mheap #phpworld $ ./vendor/bin/phpunit 1) ExampleTest ::test_json_formatting_is_correct Failed asserting that two strings are equal. --- Expected +++ Actual @@ @@ -'{"id": 1}' +'{"id": "Q1"}' FAILURES! Tests: 1, Assertions: 1, Failures: 1.
105. @mheap #phpworld https: //github.com/spatie/phpunit-snapshot-assertions
106. @mheap #phpworld Conclusion
107. @mheap #phpworld We don’t have time for this
108. @mheap #phpworld Where should we start?
109. @mheap #phpworld Start refactoring
110. @mheap #phpworld Characterisation tests aren’t perfect
111. @mheap #phpworld The 4 stages of characterisation tests
112. @mheap #phpworld Examine your feature
113. @mheap #phpworld Characterize your feature
114. @mheap #phpworld Refactor your feature
115. @mheap #phpworld Delete your tests
116. @mheap #phpworld for characterization tests
117. @mheap #phpworld I’m Michael
118. @mheap #phpworld @mheap
119. @mheap #phpworld https://joind.in/21790