Behat for characterization tests

Behat for characterization tests

Once an API ships it doesn’t matter how it should behave – how it actually behaves is the important part. Users depend on the existing behaviour and we need a way to ensure that it doesn’t change Behat is a tool that was built to help design software, but it’s actually a great tool for capturing existing behaviour too. We’ve used these tools to gain confidence to refactor 5+ year old apps by capturing the existing behaviour before making changes. I want to share the secrets we learned with you.

Bbf9decfbfc2ab5b450ec503749ded28?s=128

Michael Heap

May 20, 2018
Tweet

Transcript

  1. @mheap #phpkonf for characterization tests

  2. @mheap #phpkonf Imagine this…

  3. @mheap #phpkonf $$$

  4. @mheap #phpkonf $$$ $$$ $$$

  5. @mheap #phpkonf $$$ Billing Rating Auth Report Masking Encoding $$$

    $$$
  6. @mheap #phpkonf $$$ Billing Rating Auth Report Masking Encoding $$$

    $$$
  7. @mheap #phpkonf $$$ $$$ $$$ Billing Rating Auth Report Masking

    Encoding public function provides() { return [Client ::class]; }
  8. @mheap #phpkonf

  9. @mheap #phpkonf $ ls -l test total 0

  10. @mheap https://twitter.com/brianwisti/status/503987766032494592 YOU ARE IN A LEGACY CODEBASE > RUN

    TESTS YOU HAVE NO TESTS > READ SPEC YOU HAVE NO SPEC > WRITE FIX YOU ARE EATEN BY AN ELDER CODE HACK.
  11. @mheap #phpkonf Imagine

  12. @mheap #phpkonf Hello, I’m Michael

  13. @mheap #phpkonf @mheap

  14. @mheap #phpkonf

  15. None
  16. @mheap #phpkonf Today

  17. @mheap #phpkonf What is legacy code?

  18. @mheap Michael Feathers - Working Effectively with Legacy Code “code

    without tests” Legacy Code is
  19. @mheap Legacy Code is “valuable code that we feel afraid

    to change” JB Rainsberger - Surviving Legacy Code with Golden Master and Sampling
  20. @mheap #phpkonf What if I make a change and break

    something?
  21. @mheap #phpkonf What if I make a change and change

    something?
  22. @mheap #phpkonf Intended or Unintended

  23. @mheap #phpkonf Intended or Unintended

  24. @mheap #phpkonf Intended or Unintended

  25. @mheap #phpkonf But there aren’t any tests!

  26. @mheap #phpkonf class SalesUtil { const BQ = 1000.0; const

    BCR = 0.20; const OQM1 = 1.5; const OQM2 = self ::OQM1 * 2; static public function calculate($tSales) { if ($tSales <= static ::BQ) { return $tSales * static ::BCR; } else if ($tSales <= static ::BQ*2) { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1; } else { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1 + ($tSales - static ::BQ * 2) * static ::BCR * static ::OQM2; } } }
  27. @mheap #phpkonf class SalesUtil { const BQ = 1000.0; const

    BCR = 0.20; const OQM1 = 1.5; const OQM2 = self ::OQM1 * 2; static public function calculate($tSales) { if ($tSales <= static ::BQ) { return $tSales * static ::BCR; } else if ($tSales <= static ::BQ*2) { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1; } else { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1 + ($tSales - static ::BQ * 2) * static ::BCR * static ::OQM2; } } }
  28. @mheap #phpkonf class SalesUtil { const BQ = 1000.0; const

    BCR = 0.20; const OQM1 = 1.5; const OQM2 = self ::OQM1 * 2; static public function calculate($tSales) { if ($tSales <= static ::BQ) { return $tSales * static ::BCR; } else if ($tSales <= static ::BQ*2) { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1; } else { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1 + ($tSales - static ::BQ * 2) * static ::BCR * static ::OQM2; } } }
  29. @mheap #phpkonf class SalesUtil { const BQ = 1000.0; const

    BCR = 0.20; const OQM1 = 1.5; const OQM2 = self ::OQM1 * 2; static public function calculate($tSales) { if ($tSales <= static ::BQ) { return $tSales * static ::BCR; } else if ($tSales <= static ::BQ*2) { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1; } else { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1 + ($tSales - static ::BQ * 2) * static ::BCR * static ::OQM2; } } }
  30. @mheap #phpkonf use PHPUnit\Framework\TestCase; class SalesUtilTest extends TestCase { public

    function testX() { $this ->assertEquals(null, SalesUtil ::calculate(1000)); } }
  31. @mheap #phpkonf use PHPUnit\Framework\TestCase; class SalesUtilTest extends TestCase { public

    function testX() { $this ->assertEquals(null, SalesUtil ::calculate(999)); } }
  32. @mheap #phpkonf use PHPUnit\Framework\TestCase; class SalesUtilTest extends TestCase { public

    function testX() { $this ->assertEquals(null, SalesUtil ::calculate(1000)); } }
  33. @mheap #phpkonf There was 1 failure: 1) SalesUtilTest ::testX Failed

    asserting that 200 matches expected null. /Users/michael/development/oss/characterisation-tests-examples/sales-util/test/SalesUtilTest.php:8 FAILURES! Tests: 1, Assertions: 1, Failures: 1.
  34. @mheap #phpkonf use PHPUnit\Framework\TestCase; class SalesUtilTest extends TestCase { public

    function testSalesLTE1000Pay20PercentCommission() { $this ->assertEquals(200, SalesUtil ::calculate(1000)); } }
  35. @mheap #phpkonf class SalesUtil { const BQ = 1000.0; const

    BCR = 0.20; const OQM1 = 1.5; const OQM2 = self ::OQM1 * 2; static public function calculate($tSales) { if ($tSales <= static ::BQ) { return $tSales * static ::BCR; } else if ($tSales <= static ::BQ*2) { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1; } else { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1 + ($tSales - static ::BQ * 2) * static ::BCR * static ::OQM2; } } }
  36. @mheap #phpkonf There was 1 failure: 1) SalesUtilTest ::testX Failed

    asserting that 500.0 matches expected null. /Users/michael/development/oss/characterisation-tests-examples/sales-util/test/SalesUtilTest.php:13 FAILURES! Tests: 2, Assertions: 2, Failures: 1.
  37. @mheap #phpkonf public function test2000Gets500Commission() { $this ->assertEquals(500, SalesUtil ::calculate(2000));

    } public function testX() { $this ->assertEquals(null, SalesUtil ::calculate(1600)); }
  38. @mheap #phpkonf There was 1 failure: 1) SalesUtilTest ::testX Failed

    asserting that 380.0 matches expected null. /Users/michael/development/oss/characterisation-tests-examples/sales-util/test/SalesUtilTest.php:19 FAILURES! Tests: 3, Assertions: 3, Failures: 1.
  39. @mheap #phpkonf class SalesUtil { const BQ = 1000.0; const

    BCR = 0.20; const OQM1 = 1.5; const OQM2 = self ::OQM1 * 2; static public function calculate($tSales) { if ($tSales <= static ::BQ) { return $tSales * static ::BCR; } else if ($tSales <= static ::BQ*2) { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1; } else { return static ::BQ * static ::BCR + ($tSales - static ::BQ) * static ::BCR * static ::OQM1 + ($tSales - static ::BQ * 2) * static ::BCR * static ::OQM2; } } }
  40. @mheap #phpkonf Input Expected 0 0 10 2 500 100

    1000 200 1001 200.3 1500 350 1982 494.6 1999.9 499.97 2000 500 2001 500.9 5000 3200 7890 5801 10000 7700
  41. @mheap #phpkonf public function testX() { $this ->assertEquals(null, SalesUtil ::calculate(-123));

    }
  42. @mheap #phpkonf public function testNegativeSalesCommission() { $this ->assertEquals(-24.6, SalesUtil ::calculate(-123));

    }
  43. @mheap #phpkonf Do not change the code

  44. @mheap #phpkonf Do not change the code

  45. @mheap #phpkonf Do NOT change the code

  46. @mheap https://en.wikipedia.org/wiki/Equivalence_partitioning What’s the minimum number of tests?

  47. @mheap #phpkonf We’re half way! (and we’ve barely mentioned Behat)

  48. @mheap #phpkonf Choosing your pinch points

  49. @mheap #phpkonf Choose HTTP

  50. @mheap #phpkonf Choose Behat

  51. @mheap #phpkonf Given I have 1 "Red Bucket" in my

    basket And I have 2 "Large Spades" in my basket When I add 1 "Red Bucket" in my basket Then I should see 2 "Red Buckets" in my basket And I should see 2 "Large Spades" in my basket
  52. @mheap #phpkonf Given I have 1 "Red Bucket" in my

    basket And I have 2 "Large Spades" in my basket When I add 1 "Red Bucket" in my basket Then I should see 2 "Red Buckets" in my basket And I should see 2 "Large Spades" in my basket
  53. @mheap #phpkonf Given I have 1 "Red Bucket" in my

    basket And I have 2 "Large Spades" in my basket When I add 1 "Red Bucket" in my basket Then I should see 2 "Red Buckets" in my basket And I should see 2 "Large Spades" in my basket
  54. @mheap #phpkonf Given I have 1 "Red Bucket" in my

    basket And I have 2 "Large Spades" in my basket When I add 1 "Red Bucket" in my basket Then I should see 2 "Red Buckets" in my basket And I should see 2 "Large Spades" in my basket
  55. @mheap #phpkonf Given I have 1 "Red Bucket" in my

    basket And I have 2 "Large Spades" in my basket When I add 1 "Red Bucket" in my basket Then I should see 2 "Red Buckets" in my basket And I should see 2 "Large Spades" in my basket
  56. @mheap #phpkonf Be pragmatic

  57. @mheap #phpkonf composer require behat/behat --dev vendor/bin/behat --init

  58. @mheap #phpkonf Testing a HTTP API

  59. @mheap #phpkonf $username = $_GET['username'] ?? error('username is required'); $key

    = $_GET['key'] ?? error('key is required'); $validUsers = [ "michael" => "kangar00s", "oscar" => "phparch17" ]; $validKey = $validUsers[strtolower($username)] ?? error('could not find user', 404); if ($key !== $validKey){ error('invalid apikey'); } output([ 'success' => 'true', 'user' => ['name' => $username] ]);
  60. @mheap #phpkonf $username = $_GET['username'] ?? error('username is required'); $key

    = $_GET['key'] ?? error('key is required'); $validUsers = [ "michael" => "kangar00s", "oscar" => "phparch17" ]; $validKey = $validUsers[strtolower($username)] ?? error('could not find user', 404); if ($key !== $validKey){ error('invalid apikey'); } output([ 'success' => 'true', 'user' => ['name' => $username] ]);
  61. @mheap #phpkonf $username = $_GET['username'] ?? error('username is required'); $key

    = $_GET['key'] ?? error('key is required'); $validUsers = [ "michael" => "kangar00s", "oscar" => "phparch17" ]; $validKey = $validUsers[strtolower($username)] ?? error('could not find user', 404); if ($key !== $validKey){ error('invalid apikey'); } output([ 'success' => 'true', 'user' => ['name' => $username] ]);
  62. @mheap #phpkonf error('username is required'); error('key is required'); error('could not

    find user', 404); error('invalid apikey'); output([ 'success' => 'true', 'user' => ['name' => $username] ]); 1. Missing username 2. Missing key 3. Invalid username 4. Invalid key 5. Successful authentication
  63. @mheap #phpkonf Scenario: No Username When I make a "GET"

    request to "/" Then the response status code should be "400" And the "error" property equals "username is required"
  64. @mheap #phpkonf Scenario: No API Key When I make a

    "GET" request to "/?username=foo" Then the response status code should be "400" And the "error" property equals "key is required”
  65. @mheap #phpkonf Scenario: Invalid Username When I make a "GET"

    request to "/?username=bananas&key=foo" Then the response status code should be "404" And the "error" property equals "could not find user” Scenario: Invalid API Key When I make a "GET" request to "/?username=michael&key=foo" Then the response status code should be "400" And the "error" property equals "invalid apikey"
  66. @mheap #phpkonf Scenario: Successful login When I make a "GET"

    request to "/?username=michael&key=kangar00s" Then the response status code should be "200" And the "success" property equals "true" And the "user.name" property equals "michael”
  67. @mheap #phpkonf Scenario Outline: Successful login When I make a

    "GET" request to "/?username=<name>&key=<key>" Then the response status code should be "200" And the "success" property equals "true" And the "user.name" property equals "<name>" Examples: | name | key | | michael | kangar00s | | oscar | phparch17 |
  68. @mheap #phpkonf More complex tests

  69. @mheap #phpkonf Given that header property "X-CustomAuthKey" is "Secret123" And

    that header property "Accept" is "application/json+vnd.v2" And that the request body is valid JSON ''' { "alpha":"beta", "count":3, "collection":["a","b","c"] } ''' When I make a "POST" request to "/account/balance" Then the response status code should be "200" And the "X-Remaining" header property equals "18" And the "balance.usd" property equals"1832.54" And the "balance.gbp" property equals "13.99" And the "balance.eur" property equals "19422.18"
  70. @mheap #phpkonf When I make a "POST" request to "/account/balance"

    Then the response status code should be “200" And the "X-Remaining" header property equals "18" And the response body contains the JSON data ''' { "in_credit": true, "balance": { "usd": 1832.54, "gbp": 13.99, "eur": 19422.18 } } '''
  71. @mheap #phpkonf When I make a "POST" request to "/account/balance"

    Then the response status code should be "200" And the "X-Remaining" header property equals "18" And the value of the "balance.usd" property matches the pattern “/^\d{1,3}\. \d{2}$/" And the value of the “last_updated” property matches the pattern "/^[0-9]{4} [\-][0-9]{2}[\-][0-9]{2} [0-9]{2}[:][0-9]{2}[:][0-9]{2}$/"
  72. @mheap #phpkonf composer require datasift/testrest-extension

  73. @mheap #phpkonf # behat.yml default: extensions: DataSift\BehatExtension: base_url: "http: //localhost:8080/"

    suites: default: contexts: - 'DataSift\BehatExtension\Context\RestContext'
  74. @mheap #phpkonf HTTP Mocks

  75. @mheap #phpkonf function get_balance($username) { return $client ->get( “http: //localhost:88/billing/check_balance/".$username

    ) ->getBody(); } $balance = get_balance($username); if ($username != 'michael') { error('wrong username'); } if (!$balance['has_credit']) { error('no credit remaining'); } success(['data' => 'valid account'])
  76. @mheap #phpkonf Mountebank

  77. @mheap #phpkonf npm install -g mountebank

  78. @mheap #phpkonf Given Mountebank is running And a mock exists

    at "/billing/check_balance" it should return "200" with the body: ''' { "has_credit": false, "credit_limit": 0 } ''' And the mocks are created When I make a "GET" request to "/auth?username=michael" Then the response status code should be "403" And the response is JSON And the response body JSON equals ''' { "error": "no credit remaining" } '''
  79. @mheap #phpkonf Given Mountebank is running And a mock exists

    at "/billing/check_balance" it should return "200" with the body: ''' { "has_credit": true, "credit_limit": 10000 } ''' And the mocks are created When I make a "GET" request to "/auth?username=michael" Then the response status code should be "200" And the response is JSON And the response body JSON equals ''' { "data": "valid account" } '''
  80. @mheap #phpkonf Automating a browser

  81. @mheap #phpkonf $ composer require --dev behat/mink-extension $ composer require

    --dev behat/mink-selenium2-driver $ java -Dwebdriver.chrome.driver=chromedriver -jar selenium- server-standalone-3.7.1.jar
  82. @mheap #phpkonf # behat.yml default: extensions: Behat\MinkExtension: base_url: 'https: //wikipedia.org'

    sessions: default: selenium2: browser: "chrome" suites: my_suite: contexts: - \Behat\MinkExtension\Context\MinkContext
  83. @mheap #phpkonf Scenario: Searching for a page that does exist

    Given I am on "/wiki/Main_Page" When I fill in "search" with "Behavior Driven Development" And I press "searchButton" Then I should see "agile software development"
  84. @mheap #phpkonf Custom Steps

  85. @mheap #phpkonf Scenario: Searching for a page that does exist

    Given I am on "/wiki/Main_Page" When I fill in "search" with "Behavior Driven Development" And I press "searchButton" Then I should see "agile software development"
  86. @mheap #phpkonf Scenario: Searching for a page that does exist

    Given I am on "/wiki/Main_Page" When I search for "Behavior Driven Development" Then I should see "agile software development"
  87. @mheap #phpkonf Scenario: Searching for a page that does exist

    Given I am on "/wiki/Main_Page" When I search for "Behavior Driven Development" Then I should see "agile software development" 1 scenario (1 undefined) 3 steps (1 passed, 1 undefined, 1 skipped) 0m3.63s (10.62Mb) >> my_suite suite has undefined steps. Please choose the context to generate snippets: [0] None [1] FeatureContext [2] Behat\MinkExtension\Context\MinkContext
  88. @mheap #phpkonf [0] None [1] FeatureContext [2] Behat\MinkExtension\Context\MinkContext > 1

    --- FeatureContext has missing steps. Define them with these snippets: /** * @When I search for :arg1 */ public function iSearchFor($arg1) { throw new PendingException(); }
  89. @mheap #phpkonf ./vendor/bin/behat --append-snippets

  90. @mheap #phpkonf [0] None [1] FeatureContext [2] Behat\MinkExtension\Context\MinkContext > 1

    u features/bootstrap/FeatureContext.php - `I search for "Behavior Driven Development"` definition added
  91. @mheap #phpkonf Feature: Test Scenario: Searching for a page that

    does exist Given I am on "/wiki/Main_Page" When I search for "Behavior Driven Development" TODO: write pending definition Then I should see "agile software development" 1 scenario (1 pending) 3 steps (1 passed, 1 pending, 1 skipped) 0m2.23s (10.61Mb)
  92. @mheap #phpkonf /** @BeforeScenario */ public function gatherContexts( Behat\Behat\Hook\Scope\BeforeScenarioScope $scope

    ) { $environment = $scope ->getEnvironment(); $this ->minkContext = $environment ->getContext( 'Behat\MinkExtension\Context\MinkContext' ); }
  93. @mheap #phpkonf /** * Fills in form field with specified

    id|name|label|value * Example: When I fill in "username" with: "bwayne" * Example: And I fill in "bwayne" for "username" * * @When /^( ?:|I )fill in "(?P<field>( ?:[^"]|\ ")*)" with "(?P<value>( ?:[^"]|\ ")*)"$/ * @When /^( ?:|I )fill in "(?P<field>( ?:[^"]|\ ")*)" with:$/ * @When /^( ?:|I )fill in "(?P<value>( ?:[^"]|\ ")*)" for "(?P<field>( ?:[^"]|\ ")*)"$/ */ public function fillField($field, $value) { $field = $this ->fixStepArgument($field); $value = $this ->fixStepArgument($value); $this ->getSession() ->getPage() ->fillField($field, $value); }
  94. @mheap #phpkonf /** * Presses button with specified id|name|title|alt|value *

    Example: When I press "Log In" * Example: And I press "Log In" * @When /^( ?:|I )press "(?P<button>( ?:[^"]|\ ")*)"$/ */ public function pressButton($button) { $button = $this ->fixStepArgument($button); $this ->getSession() ->getPage() ->pressButton($button); }
  95. @mheap #phpkonf /** * @When I search for :term */

    public function iSearchFor($term) { $this ->minkContext ->fillField('search', $term); $this ->minkContext ->pressButton('searchButton'); }
  96. @mheap #phpkonf $ ./vendor/bin/behat Feature: Test Scenario: Searching for a

    page that does exist Given I am on "/wiki/Main_Page" When I search for "Behavior Driven Development" Then I should see "agile software development" 1 scenario (1 passed) 3 steps (3 passed)
  97. @mheap #phpkonf Scenario: Successful login When I make a "GET"

    request to "/?username=michael&key=kangar00s" Then the response status code should be "200" And the "success" property equals "true" And the "user.name" property equals "michael”
  98. @mheap #phpkonf Scenario: Successful login When I log in as

    "Michael" with the password "kangar00s" Then the response status code should be "200" And the "success" property equals "true" And the "user.name" property equals "michael”
  99. @mheap #phpkonf Snapshot testing

  100. @mheap #phpkonf public function toJson(){ return json_encode(["id" => $this ->id]);

    } —— public function test_json_formatting_is_correct() { $order = new Order(1); $this ->assertMatchesSnapshot($order ->toJson()); }
  101. @mheap #phpkonf $ ./vendor/bin/phpunit There was 1 incomplete test: 1)

    ExampleTest ::test_json_formatting_is_correct Snapshot created for ExampleTest __test_json_formatting_is_correct
  102. @mheap #phpkonf $ ./vendor/bin/phpunit OK (1 test, 1 assertion)

  103. @mheap #phpkonf public function toJson(){ return json_encode(["id" => 'Q'.$this ->id]);

    }
  104. @mheap #phpkonf $ ./vendor/bin/phpunit 1) ExampleTest ::test_json_formatting_is_correct Failed asserting that

    two strings are equal. --- Expected +++ Actual @@ @@ -'{"id": 1}' +'{"id": "Q1"}' FAILURES! Tests: 1, Assertions: 1, Failures: 1.
  105. @mheap #phpkonf https: //github.com/spatie/phpunit-snapshot-assertions

  106. @mheap #phpkonf Conclusion

  107. @mheap #phpkonf We don’t have time for this

  108. @mheap #phpkonf Where should we start?

  109. @mheap #phpkonf Start refactoring

  110. @mheap #phpkonf Characterisation tests aren’t perfect

  111. @mheap #phpkonf The 4 stages of characterisation tests

  112. @mheap #phpkonf Examine your feature

  113. @mheap #phpkonf Characterize your feature

  114. @mheap #phpkonf Refactor your feature

  115. @mheap #phpkonf Delete your tests

  116. @mheap #phpkonf for characterization tests

  117. @mheap #phpkonf I’m Michael

  118. @mheap #phpkonf @mheap

  119. @mheap #phpkonf https://joind.in/talk/dc4a7

  120. @mheap #phpkonf Michael @mheap https://joind.in/talk/dc4a7 Thankyou!