Behat for characterization tests

Once an API ships it doesn’t matter how it should behave – how it actually behaves is the important part. Users depend on the existing behaviour and we need a way to ensure that it doesn’t change Behat is a tool that was built to help design software, but it’s actually a great tool for capturing existing behaviour too. We’ve used these tools to gain confidence to refactor 5+ year old apps by capturing the existing behaviour before making changes. I want to share the secrets we learned with you.

1. @mheap #dpc18 for characterization tests
2. @mheap #dpc18 Imagine this…
3. @mheap #dpc18 $$$
4. @mheap #dpc18 $$$ $$$ $$$
5. @mheap #dpc18 $$$ Billing Rating Auth Report Masking Encoding $$$ $$$
6. @mheap #dpc18 $$$ Billing Rating Auth Report Masking Encoding $$$ $$$
7. @mheap #dpc18 $$$ $$$ $$$ Billing Rating Auth Report Masking Encoding public function provides() { return [Client ::class]; }
8. @mheap #dpc18
9. @mheap #dpc18 $ ls -l test total 0
10. @mheap #dpc18 YOU ARE IN A LEGACY CODEBASE > RUN TESTS YOU HAVE NO TESTS > READ SPEC YOU HAVE NO SPEC > WRITE FIX YOU ARE EATEN BY AN ELDER CODE HACK.
11. @mheap #dpc18 Imagine
12. @mheap #dpc18 Hello, I’m Michael
13. @mheap #dpc18 @mheap
14. @mheap #dpc18
16. @mheap #dpc18 Today
17. @mheap #dpc18 What is legacy code?
18. @mheap #dpc18 “code without tests” Legacy Code is
19. @mheap #dpc18 Legacy Code is “valuable code that we feel afraid to change” JB Rainsberger - Surviving Legacy Code with Golden Master and Sampling
20. @mheap #dpc18 What if I make a change and break something?
21. @mheap #dpc18 What if I make a change and change something?
22. @mheap #dpc18 Intended or Unintended
23. @mheap #dpc18 Intended or Unintended
24. @mheap #dpc18 Intended or Unintended
25. @mheap #dpc18 But there aren’t any tests!
26. @mheap #dpc18 class SalesUtil { const BQ = 1000.0; const BCR = 0.20; const OQM1 = 1.5; const OQM2 = self ::OQM1 * 2; static public function calculate($tSales) { if ($tSales <= static ::BQ) { return $tSales * static ::BCR; } else if ($tSales <= static ::BQ*2) { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1; } else { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1 + ($tSales - static ::BQ * 2) * static ::BCR * static ::OQM2; } } }
27. @mheap #dpc18 class SalesUtil { const BQ = 1000.0; const BCR = 0.20; const OQM1 = 1.5; const OQM2 = self ::OQM1 * 2; static public function calculate($tSales) { if ($tSales <= static ::BQ) { return $tSales * static ::BCR; } else if ($tSales <= static ::BQ*2) { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1; } else { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1 + ($tSales - static ::BQ * 2) * static ::BCR * static ::OQM2; } } }
28. @mheap #dpc18 class SalesUtil { const BQ = 1000.0; const BCR = 0.20; const OQM1 = 1.5; const OQM2 = self ::OQM1 * 2; static public function calculate($tSales) { if ($tSales <= static ::BQ) { return $tSales * static ::BCR; } else if ($tSales <= static ::BQ*2) { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1; } else { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1 + ($tSales - static ::BQ * 2) * static ::BCR * static ::OQM2; } } }
29. @mheap #dpc18 class SalesUtil { const BQ = 1000.0; const BCR = 0.20; const OQM1 = 1.5; const OQM2 = self ::OQM1 * 2; static public function calculate($tSales) { if ($tSales <= static ::BQ) { return $tSales * static ::BCR; } else if ($tSales <= static ::BQ*2) { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1; } else { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1 + ($tSales - static ::BQ * 2) * static ::BCR * static ::OQM2; } } }
30. @mheap #dpc18 use PHPUnit\Framework\TestCase; class SalesUtilTest extends TestCase { public function testX() { $this ->assertEquals(null, SalesUtil ::calculate(1000)); } }
31. @mheap #dpc18 use PHPUnit\Framework\TestCase; class SalesUtilTest extends TestCase { public function testX() { $this ->assertEquals(null, SalesUtil ::calculate(999)); } }
32. @mheap #dpc18 use PHPUnit\Framework\TestCase; class SalesUtilTest extends TestCase { public function testX() { $this ->assertEquals(null, SalesUtil ::calculate(1000)); } }
33. @mheap #dpc18 There was 1 failure: 1) SalesUtilTest ::testX Failed asserting that 200 matches expected null. /Users/michael/development/oss/characterisation-tests-examples/sales-util/test/SalesUtilTest.php:8 FAILURES! Tests: 1, Assertions: 1, Failures: 1.
34. @mheap #dpc18 use PHPUnit\Framework\TestCase; class SalesUtilTest extends TestCase { public function testSalesLTE1000Pay20PercentCommission() { $this ->assertEquals(200, SalesUtil ::calculate(1000)); } }
35. @mheap #dpc18 class SalesUtil { const BQ = 1000.0; const BCR = 0.20; const OQM1 = 1.5; const OQM2 = self ::OQM1 * 2; static public function calculate($tSales) { if ($tSales <= static ::BQ) { return $tSales * static ::BCR; } else if ($tSales <= static ::BQ*2) { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1; } else { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1 + ($tSales - static ::BQ * 2) * static ::BCR * static ::OQM2; } } }
36. @mheap #dpc18 There was 1 failure: 1) SalesUtilTest ::testX Failed asserting that 500.0 matches expected null. /Users/michael/development/oss/characterisation-tests-examples/sales-util/test/SalesUtilTest.php:13 FAILURES! Tests: 2, Assertions: 2, Failures: 1.
37. @mheap #dpc18 public function test2000Gets500Commission() { $this ->assertEquals(500, SalesUtil ::calculate(2000)); } public function testX() { $this ->assertEquals(null, SalesUtil ::calculate(1600)); }
38. @mheap #dpc18 There was 1 failure: 1) SalesUtilTest ::testX Failed asserting that 380.0 matches expected null. /Users/michael/development/oss/characterisation-tests-examples/sales-util/test/SalesUtilTest.php:19 FAILURES! Tests: 3, Assertions: 3, Failures: 1.
39. @mheap #dpc18 class SalesUtil { const BQ = 1000.0; const BCR = 0.20; const OQM1 = 1.5; const OQM2 = self ::OQM1 * 2; static public function calculate($tSales) { if ($tSales <= static ::BQ) { return $tSales * static ::BCR; } else if ($tSales <= static ::BQ*2) { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1; } else { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1 + ($tSales - static ::BQ * 2) * static ::BCR * static ::OQM2; } } }
40. @mheap #dpc18 Input Expected 0 0 10 2 500 100 1000 200 1001 200.3 1500 350 1982 494.6 1999.9 499.97 2000 500 2001 500.9 5000 3200 7890 5801 10000 7700
41. @mheap #dpc18 public function testX() { $this ->assertEquals(null, SalesUtil ::calculate(-123)); }
42. @mheap #dpc18 public function testNegativeSalesCommission() { $this ->assertEquals(-24.6, SalesUtil ::calculate(-123)); }
43. @mheap #dpc18 Do not change the code
44. @mheap #dpc18 Do not change the code
45. @mheap #dpc18 Do NOT change the code
46. @mheap https://en.wikipedia.org/wiki/Equivalence_partitioning What’s the minimum number of tests?
47. @mheap #dpc18 Using Behat
48. @mheap #dpc18 Choosing your pinch points
49. @mheap #dpc18 Choose HTTP
50. @mheap #dpc18 Choose Behat
51. @mheap #dpc18 Given I have 1 "Red Bucket" in my basket And I have 2 "Large Spades" in my basket When I add 1 "Red Bucket" in my basket Then I should see 2 "Red Buckets" in my basket And I should see 2 "Large Spades" in my basket
52. @mheap #dpc18 Given I have 1 "Red Bucket" in my basket And I have 2 "Large Spades" in my basket When I add 1 "Red Bucket" in my basket Then I should see 2 "Red Buckets" in my basket And I should see 2 "Large Spades" in my basket
53. @mheap #dpc18 Given I have 1 "Red Bucket" in my basket And I have 2 "Large Spades" in my basket When I add 1 "Red Bucket" in my basket Then I should see 2 "Red Buckets" in my basket And I should see 2 "Large Spades" in my basket
54. @mheap #dpc18 Given I have 1 "Red Bucket" in my basket And I have 2 "Large Spades" in my basket When I add 1 "Red Bucket" in my basket Then I should see 2 "Red Buckets" in my basket And I should see 2 "Large Spades" in my basket
55. @mheap #dpc18 Given I have 1 "Red Bucket" in my basket And I have 2 "Large Spades" in my basket When I add 1 "Red Bucket" in my basket Then I should see 2 "Red Buckets" in my basket And I should see 2 "Large Spades" in my basket
56. @mheap #dpc18 Be pragmatic
57. @mheap #dpc18 composer require behat/behat --dev vendor/bin/behat --init
58. @mheap #dpc18 Testing a HTTP API
59. @mheap #dpc18 $username = $_GET['username'] ?? error('username is required'); $key = $_GET['key'] ?? error('key is required'); $validUsers = [ "michael" => "kangar00s", "oscar" => "phparch17" ]; $validKey = $validUsers[strtolower($username)] ?? error('could not find user', 404); if ($key !== $validKey){ error('invalid apikey'); } output([ 'success' => 'true', 'user' => ['name' => $username] ]);
60. @mheap #dpc18 $username = $_GET['username'] ?? error('username is required'); $key = $_GET['key'] ?? error('key is required'); $validUsers = [ "michael" => "kangar00s", "oscar" => "phparch17" ]; $validKey = $validUsers[strtolower($username)] ?? error('could not find user', 404); if ($key !== $validKey){ error('invalid apikey'); } output([ 'success' => 'true', 'user' => ['name' => $username] ]);
61. @mheap #dpc18 $username = $_GET['username'] ?? error('username is required'); $key = $_GET['key'] ?? error('key is required'); $validUsers = [ "michael" => "kangar00s", "oscar" => "phparch17" ]; $validKey = $validUsers[strtolower($username)] ?? error('could not find user', 404); if ($key !== $validKey){ error('invalid apikey'); } output([ 'success' => 'true', 'user' => ['name' => $username] ]);
62. @mheap #dpc18 error('username is required'); error('key is required'); error('could not find user', 404); error('invalid apikey'); output([ 'success' => 'true', 'user' => ['name' => $username] ]); 1. Missing username 2. Missing key 3. Invalid username 4. Invalid key 5. Successful authentication
63. @mheap #dpc18 Scenario: No Username When I make a "GET" request to "/" Then the response status code should be "400" And the "error" property equals "username is required"
64. @mheap #dpc18 Scenario: No API Key When I make a "GET" request to "/?username=foo" Then the response status code should be "400" And the "error" property equals "key is required”
65. @mheap #dpc18 Scenario: Invalid Username When I make a "GET" request to "/?username=bananas&key=foo" Then the response status code should be "404" And the "error" property equals "could not find user” Scenario: Invalid API Key When I make a "GET" request to "/?username=michael&key=foo" Then the response status code should be "400" And the "error" property equals "invalid apikey"
66. @mheap #dpc18 Scenario: Successful login When I make a "GET" request to "/?username=michael&key=kangar00s" Then the response status code should be "200" And the "success" property equals "true" And the "user.name" property equals "michael”
67. @mheap #dpc18 Scenario Outline: Successful login When I make a "GET" request to "/?username=<name>&key=<key>" Then the response status code should be "200" And the "success" property equals "true" And the "user.name" property equals "<name>" Examples: | name | key | | michael | kangar00s | | oscar | phparch17 |
68. @mheap #dpc18 More complex tests
69. @mheap #dpc18 Given that header property "X-CustomAuthKey" is "Secret123" And that header property "Accept" is "application/json+vnd.v2" And that the request body is valid JSON ''' { "alpha":"beta", "count":3, "collection":["a","b","c"] } ''' When I make a "POST" request to "/account/balance" Then the response status code should be "200" And the "X-Remaining" header property equals "18" And the "balance.usd" property equals"1832.54" And the "balance.gbp" property equals "13.99" And the "balance.eur" property equals "19422.18"
70. @mheap #dpc18 When I make a "POST" request to "/account/balance" Then the response status code should be “200" And the "X-Remaining" header property equals "18" And the response body contains the JSON data ''' { "in_credit": true, "balance": { "usd": 1832.54, "gbp": 13.99, "eur": 19422.18 } } '''
71. @mheap #dpc18 When I make a "POST" request to "/account/balance" Then the response status code should be "200" And the "X-Remaining" header property equals "18" And the value of the "balance.usd" property matches the pattern “/^\d{1,3}\. \d{2}$/" And the value of the “last_updated” property matches the pattern "/^[0-9]{4} [\-][0-9]{2}[\-][0-9]{2} [0-9]{2}[:][0-9]{2}[:][0-9]{2}$/"
72. @mheap #dpc18 composer require datasift/testrest-extension
73. @mheap #dpc18 # behat.yml default: extensions: DataSift\BehatExtension: base_url: "http: //localhost:8080/" suites: default: contexts: - 'DataSift\BehatExtension\Context\RestContext'
74. @mheap #dpc18 HTTP Mocks
75. @mheap #dpc18 function get_balance($username) { return $client ->get( “http: //localhost:88/billing/check_balance/".$username ) ->getBody(); } $balance = get_balance($username); if ($username != 'michael') { error('wrong username'); } if (!$balance['has_credit']) { error('no credit remaining'); } success(['data' => 'valid account'])
76. @mheap #dpc18 Mountebank
77. @mheap #dpc18 npm install -g mountebank
78. @mheap #dpc18 Given Mountebank is running And a mock exists at "/billing/check_balance" it should return "200" with the body: ''' { "has_credit": false, "credit_limit": 0 } ''' And the mocks are created When I make a "GET" request to "/auth?username=michael" Then the response status code should be "403" And the response is JSON And the response body JSON equals ''' { "error": "no credit remaining" } '''
79. @mheap #dpc18 Given Mountebank is running And a mock exists at "/billing/check_balance" it should return "200" with the body: ''' { "has_credit": true, "credit_limit": 10000 } ''' And the mocks are created When I make a "GET" request to "/auth?username=michael" Then the response status code should be "200" And the response is JSON And the response body JSON equals ''' { "data": "valid account" } '''
80. @mheap #dpc18 Automating a browser
81. @mheap #dpc18 $ composer require --dev behat/mink-extension $ composer require --dev behat/mink-selenium2-driver $ java -Dwebdriver.chrome.driver=chromedriver -jar selenium- server-standalone-3.7.1.jar
82. @mheap #dpc18 # behat.yml default: extensions: Behat\MinkExtension: base_url: 'https: //wikipedia.org' sessions: default: selenium2: browser: "chrome" suites: my_suite: contexts: - \Behat\MinkExtension\Context\MinkContext
83. @mheap #dpc18 Scenario: Searching for a page that exists Given I am on "/wiki/Main_Page" When I fill in "search" with "Behavior Driven Development" And I press "searchButton" Then I should see "agile software development"
84. @mheap #dpc18 Custom Steps
85. @mheap #dpc18 Scenario: Searching for a page that exists Given I am on "/wiki/Main_Page" When I fill in "search" with "Behavior Driven Development" And I press "searchButton" Then I should see "agile software development"
86. @mheap #dpc18 Scenario: Searching for a page that exists Given I am on "/wiki/Main_Page" When I search for "Behavior Driven Development" Then I should see "agile software development"
87. @mheap #dpc18 Scenario: Searching for a page that exists Given I am on "/wiki/Main_Page" When I search for "Behavior Driven Development" Then I should see "agile software development" 1 scenario (1 undefined) 3 steps (1 passed, 1 undefined, 1 skipped) 0m3.63s (10.62Mb) >> my_suite suite has undefined steps. Please choose the context to generate snippets: [0] None [1] FeatureContext [2] Behat\MinkExtension\Context\MinkContext
88. @mheap #dpc18 [0] None [1] FeatureContext [2] Behat\MinkExtension\Context\MinkContext > 1 --- FeatureContext has missing steps. Define them with these snippets: /** * @When I search for :arg1 */ public function iSearchFor($arg1) { throw new PendingException(); }
89. @mheap #dpc18 ./vendor/bin/behat --append-snippets
90. @mheap #dpc18 [0] None [1] FeatureContext [2] Behat\MinkExtension\Context\MinkContext > 1 u features/bootstrap/FeatureContext.php - `I search for "Behavior Driven Development"` definition added
91. @mheap #dpc18 /** * @When I search for :arg1 */ public function iSearchFor($arg1) { throw new PendingException(); }
92. @mheap #dpc18 /** * @When I search for :term */ public function iSearchFor($term) { throw new PendingException(); }
93. @mheap #dpc18 Feature: Test Scenario: Searching for a page that exists Given I am on "/wiki/Main_Page" When I search for "Behavior Driven Development" TODO: write pending definition Then I should see "agile software development" 1 scenario (1 pending) 3 steps (1 passed, 1 pending, 1 skipped) 0m2.23s (10.61Mb)
94. @mheap #dpc18 /** @BeforeScenario */ public function gatherContexts( Behat\Behat\Hook\Scope\BeforeScenarioScope $scope ) { $environment = $scope ->getEnvironment(); $this ->minkContext = $environment ->getContext( 'Behat\MinkExtension\Context\MinkContext' ); }
95. @mheap #dpc18 /** * Fills in form field with specified id|name|label|value * Example: When I fill in "username" with: "bwayne" * Example: And I fill in "bwayne" for "username" * * @When /^( ?:|I )fill in "(?P<field>( ?:[^"]|\ ")*)" with "(?P<value>( ?:[^"]|\ ")*)"$/ * @When /^( ?:|I )fill in "(?P<field>( ?:[^"]|\ ")*)" with:$/ * @When /^( ?:|I )fill in "(?P<value>( ?:[^"]|\ ")*)" for "(?P<field>( ?:[^"]|\ ")*)"$/ */ public function fillField($field, $value) { $field = $this ->fixStepArgument($field); $value = $this ->fixStepArgument($value); $this ->getSession() ->getPage() ->fillField($field, $value); }
96. @mheap #dpc18 /** * Fills in form field with specified id|name|label|value * Example: When I fill in "username" with: "bwayne" * Example: And I fill in "bwayne" for "username" * * @When /^( ?:|I )fill in "(?P<field>( ?:[^"]|\ ")*)" with "(?P<value>( ?:[^"]|\ ")*)"$/ * @When /^( ?:|I )fill in "(?P<field>( ?:[^"]|\ ")*)" with:$/ * @When /^( ?:|I )fill in "(?P<value>( ?:[^"]|\ ")*)" for "(?P<field>( ?:[^"]|\ ")*)"$/ */ public function fillField($field, $value) { $field = $this ->fixStepArgument($field); $value = $this ->fixStepArgument($value); $this ->getSession() ->getPage() ->fillField($field, $value); }
97. @mheap #dpc18 /** * @When I search for :term */ public function iSearchFor($term) { $this ->minkContext ->fillField('search', $term); }
98. @mheap #dpc18 /** * Presses button with specified id|name|title|alt|value * Example: When I press "Log In" * Example: And I press "Log In" * @When /^( ?:|I )press "(?P<button>( ?:[^"]|\ ")*)"$/ */ public function pressButton($button) { $button = $this ->fixStepArgument($button); $this ->getSession() ->getPage() ->pressButton($button); }
99. @mheap #dpc18 /** * Presses button with specified id|name|title|alt|value * Example: When I press "Log In" * Example: And I press "Log In" * @When /^( ?:|I )press "(?P<button>( ?:[^"]|\ ")*)"$/ */ public function pressButton($button) { $button = $this ->fixStepArgument($button); $this ->getSession() ->getPage() ->pressButton($button); }
100. @mheap #dpc18 /** * @When I search for :term */ public function iSearchFor($term) { $this ->minkContext ->fillField('search', $term); $this ->minkContext ->pressButton('searchButton'); }
101. @mheap #dpc18 $ ./vendor/bin/behat Feature: Test Scenario: Searching for a page that exists Given I am on "/wiki/Main_Page" When I search for "Behavior Driven Development" Then I should see "agile software development" 1 scenario (1 passed) 3 steps (3 passed)
102. @mheap #dpc18 Scenario: Successful login When I make a "GET" request to "/?username=michael&key=kangar00s" Then the response status code should be "200" And the "success" property equals "true" And the "user.name" property equals "michael”
103. @mheap #dpc18 Scenario: Successful login When I log in as "Michael" with the password "kangar00s" Then the response status code should be "200" And the "success" property equals "true" And the "user.name" property equals "michael”
104. @mheap #dpc18 Snapshot testing
105. @mheap #dpc18 public function toJson(){ return json_encode(["id" => $this ->id]); } —— public function test_json_formatting_is_correct() { $order = new Order(1); $this ->assertMatchesSnapshot($order ->toJson()); }
106. @mheap #dpc18 $ ./vendor/bin/phpunit There was 1 incomplete test: 1) ExampleTest ::test_json_formatting_is_correct Snapshot created for ExampleTest __test_json_formatting_is_correct
107. @mheap #dpc18 $ ./vendor/bin/phpunit OK (1 test, 1 assertion)
108. @mheap #dpc18 public function toJson(){ return json_encode(["id" => 'Q'.$this ->id]); }
109. @mheap #dpc18 $ ./vendor/bin/phpunit 1) ExampleTest ::test_json_formatting_is_correct Failed asserting that two strings are equal. --- Expected +++ Actual @@ @@ -'{"id": 1}' +'{"id": "Q1"}' FAILURES! Tests: 1, Assertions: 1, Failures: 1.
110. @mheap #dpc18 https: //github.com/spatie/phpunit-snapshot-assertions
111. @mheap #dpc18 Conclusion
112. @mheap #dpc18 Behat
113. @mheap #dpc18 Mountebank
114. @mheap #dpc18 Mink
115. @mheap #dpc18 Snapshot Testing
116. @mheap #dpc18 We don’t have time for this
117. @mheap #dpc18 Where should we start?
118. @mheap #dpc18 Start refactoring
119. @mheap #dpc18 Characterisation tests aren’t perfect
120. @mheap #dpc18 The 4 stages of characterisation tests
121. @mheap #dpc18 Examine your feature
122. @mheap #dpc18 Characterize your feature
123. @mheap #dpc18 Refactor your feature
124. @mheap #dpc18 Delete your tests
125. @mheap #dpc18 for characterization tests
126. @mheap #dpc18 I’m Michael
127. @mheap #dpc18 @mheap
128. @mheap #dpc18 https://joind.in/talk/b7df0
129. @mheap #dpc18 Michael @mheap https://joind.in/talk/b7df0 Thankyou!