Property based testing

How to improve your tests using the technique of property based testing.

Transcript

1. FIND MORE BUGS WITH PROPERTY BASED TESTING SUZANNE HAMILTON
2. MANUAL TESTING
3. MANUAL TESTING TEST SCRIPTS
4. MANUAL TESTING VS EXPLORATORY TESTING TEST SCRIPTS
5. AUTOMATED TESTING
6. AUTOMATED TESTING EXAMPLE BASED TESTS
7. AUTOMATED TESTING VS ? EXAMPLE BASED TESTS
8. AUTOMATED TESTING VS PROPERTY BASED TESTS EXAMPLE BASED TESTS
9. EXAMPLE: THE GILDED ROSE KATA
11. Sandi Metz All the Little Things https://youtu.be/8bZh5LMaSmE
12. class GildedRose(object): def __init__(self, name, quality, days_remaining): self.name = name self.quality = quality self.days_remaining = days_remaining def tick(self): # ...
13. class GildedRose(object): def __init__(self, name, quality, days_remaining): self.name = name self.quality = quality self.days_remaining = days_remaining def tick(self): # ...
14. class GildedRose(object): def __init__(self, name, quality, days_remaining): self.name = name self.quality = quality self.days_remaining = days_remaining def tick(self): # ...
15. def tick(self): if self.name != "Aged Brie" and self.name != "Backstage passes to a TAFKAL80ETC concert": if self.quality > 0: if self.name != "Sulfuras, Hand of Ragnaros": self.quality = self.quality - 1 else: if self.quality < 50: self.quality = self.quality + 1 if self.name == "Backstage passes to a TAFKAL80ETC concert": if self.days_remaining < 11: if self.quality < 50: self.quality = self.quality + 1 if self.days_remaining < 6: if self.quality < 50: self.quality = self.quality + 1 if self.name != "Sulfuras, Hand of Ragnaros": self.days_remaining = self.days_remaining - 1 if self.days_remaining < 0: if self.name != "Aged Brie": if self.name != "Backstage passes to a TAFKAL80ETC concert": if self.quality > 0: if self.name != "Sulfuras, Hand of Ragnaros": self.quality = self.quality - 1 else: self.quality = self.quality - self.quality else: if self.quality < 50: self.quality = self.quality + 1
16. GILDED ROSE REQUIREMENTS ‣ On each tick, the system lowers the days remaining
17. def test_lowers_days_remaining():
18. def test_lowers_days_remaining(): item = GildedRose(
19. def test_lowers_days_remaining(): item = GildedRose("some item"
20. def test_lowers_days_remaining(): item = GildedRose("some item", quality =
21. def test_lowers_days_remaining(): item = GildedRose("some item", quality = 3
22. def test_lowers_days_remaining(): item = GildedRose("some item", quality = 3, days_remaining =
23. def test_lowers_days_remaining(): item = GildedRose("some item", quality = 3, days_remaining = 8)
24. def test_lowers_days_remaining(): item = GildedRose("some item", quality = 3, days_remaining = 8) item.tick()
25. def test_lowers_days_remaining(): item = GildedRose("some item", quality = 3, days_remaining = 8) item.tick() assert item.days_remaining == 7
26. def test_lowers_days_remaining(): item = GildedRose("some item", quality = 3, days_remaining = 8) item.tick() assert item.days_remaining == 7
27. def test_lowers_days_remaining(): item = GildedRose("some item", quality = 3, days_remaining = 8) item.tick() assert item.days_remaining == 7
28. def test_lowers_days_remaining(): item = GildedRose("some item", quality = 3, days_remaining = 8) item.tick() assert item.days_remaining == 7
29. def test_lowers_days_remaining(): item = GildedRose("some item", quality = 3, days_remaining = 8) item.tick() assert item.days_remaining == 7
30. def test_lowers_days_remaining(name, quality, days_remaining): item = GildedRose(name, quality, days_remaining) item.tick() assert item.days_remaining == 7
31. def test_lowers_days_remaining(name, quality, days_remaining): item = GildedRose(name, quality, days_remaining) item.tick() assert item.days_remaining == 7
32. def test_lowers_days_remaining(name, quality, days_remaining): item = GildedRose(name, quality, days_remaining) item.tick() assert item.days_remaining == days_remaining - 1
33. def test_lowers_days_remaining(name, quality, days_remaining): item = GildedRose(name, quality, days_remaining) item.tick() assert item.days_remaining < days_remaining
34. def test_lowers_days_remaining(name, quality, days_remaining): item = GildedRose(name, quality, days_remaining) item.tick() assert item.days_remaining < days_remaining
35. from hypothesis import given from hypothesis.strategies import integers, text @given(text(), integers(), integers()) def test_lowers_days_remaining(name, quality, days_remaining): item = GildedRose(name, quality, days_remaining) item.tick() assert item.days_remaining < days_remaining
36. from hypothesis import given from hypothesis.strategies import integers, text @given(text(), integers(), integers()) def test_lowers_days_remaining(name, quality, days_remaining): item = GildedRose(name, quality, days_remaining) item.tick() assert item.days_remaining < days_remaining
37. from hypothesis import given from hypothesis.strategies import integers, text @given(text(), integers(), integers()) def test_lowers_days_remaining(name, quality, days_remaining): item = GildedRose(name, quality, days_remaining) item.tick() assert item.days_remaining < days_remaining
38. from hypothesis import given from hypothesis.strategies import integers, text @given(text(), integers(), integers()) def test_lowers_days_remaining(name, quality, days_remaining): item = GildedRose(name, quality, days_remaining) item.tick() assert item.days_remaining < days_remaining
39. from hypothesis import given from hypothesis.strategies import integers, text @given(text(), integers(), integers()) def test_lowers_days_remaining(name, quality, days_remaining): item = GildedRose(name, quality, days_remaining) item.tick() assert item.days_remaining < days_remaining
40. GILDED ROSE REQUIREMENTS ‣ Quality is never more than 50
 

41. GILDED ROSE REQUIREMENTS ‣ Quality is never more than 50 ‣ "Aged Brie" increases in quality the older it gets
42. def test_aged_brie_quality_limit_is_50():
43. def test_aged_brie_quality_limit_is_50(): item = GildedRose("Aged Brie"
44. def test_aged_brie_quality_limit_is_50(): item = GildedRose("Aged Brie", quality = 50
45. def test_aged_brie_quality_limit_is_50(): item = GildedRose("Aged Brie", quality = 50, days_remaining = 8)
46. def test_aged_brie_quality_limit_is_50(): item = GildedRose("Aged Brie", quality = 50, days_remaining = 8) item.tick()
47. def test_aged_brie_quality_limit_is_50(): item = GildedRose("Aged Brie", quality = 50, days_remaining = 8) item.tick() assert item.quality == 50
48. def test_aged_brie_quality_limit_is_50(): item = GildedRose("Aged Brie", quality = 50, days_remaining = 8) item.tick() assert item.quality == 50
49. def test_quality_limit_is_50(): item = GildedRose("Aged Brie", quality = 50, days_remaining = 8) item.tick() assert item.quality == 50
50. @given(text(), integers(), integers()) def test_quality_limit_is_50(name, quality, days_remaining): item = GildedRose(name, quality, days_remaining) item.tick() assert item.quality == 50
51. @given(text(), integers(), integers()) def test_quality_limit_is_50(name, quality, days_remaining): item = GildedRose(name, quality, days_remaining) item.tick() assert item.quality == 50
52. @given(text(), integers(), integers()) def test_quality_limit_is_50(name, quality, days_remaining): item = GildedRose(name, quality, days_remaining) item.tick() assert item.quality <= 50
53. @given(text(), integers(), integers()) def test_quality_limit_is_50(name, quality, days_remaining): item = GildedRose(name, quality, days_remaining) item.tick() assert item.quality <= 50
54. @given(item_name(), integers(), integers()) def test_quality_limit_is_50(name, quality, days_remaining): item = GildedRose(name, quality, days_remaining) item.tick() assert item.quality <= 50
55. def item_name():
56. def item_name(): return one_of(
57. def item_name(): return one_of( text()
58. def item_name(): return one_of( text(), sampled_from([ "Sulfuras, Hand of Ragnaros", "Backstage passes to a TAFKAL80ETC concert", "Aged Brie" ]) )
59. def item_name(): return one_of( text(), sampled_from([ "Sulfuras, Hand of Ragnaros", "Backstage passes to a TAFKAL80ETC concert", "Aged Brie" ]) )
60. @given(item_name(), integers(), integers()) def test_quality_limit_is_50(name, quality, days_remaining): item = GildedRose(name, quality, days_remaining) item.tick() assert item.quality <= 50
61. @given(item_name(), integers(), integers()) def test_quality_limit_is_50(name, quality, days_remaining): item = GildedRose(name, quality, days_remaining) item.tick() assert item.quality <= 50
62. ===================================== FAILURES ====================================== _____________________________ test_quality_limit_is_50 ______________________________ @given(text(), integers(), integers()) > def test_quality_limit_is_50(name, quality, days_remaining): tests/test_gilded_rose.py:34: ... Stack trace name = '', quality = 53, days_remaining = 0 @given(text(), integers(), integers()) def test_quality_limit_is_50(name, quality, days_remaining): item = GildedRose(name, quality, days_remaining) item.tick() > assert item.quality <= 50 E AssertionError: assert 51 <= 50 E + where 51 = GildedRose(name: '', quality: 51, days_remaining: -1).quality tests/test_gilded_rose.py:38: AssertionError ------------------------------------ Hypothesis ------------------------------------- Falsifying example: test_quality_limit_is_50(name='', quality=53, days_remaining=0) ======================== 1 failed, 1 passed in 0.21 seconds =========================
63. ===================================== FAILURES ====================================== _____________________________ test_quality_limit_is_50 ______________________________ @given(text(), integers(), integers()) > def test_quality_limit_is_50(name, quality, days_remaining): tests/test_gilded_rose.py:34: ... Stack trace name = '', quality = 53, days_remaining = 0 @given(text(), integers(), integers()) def test_quality_limit_is_50(name, quality, days_remaining): item = GildedRose(name, quality, days_remaining) item.tick() > assert item.quality <= 50 E AssertionError: assert 51 <= 50 E + where 51 = GildedRose(name: '', quality: 51, days_remaining: -1).quality tests/test_gilded_rose.py:38: AssertionError ------------------------------------ Hypothesis ------------------------------------- Falsifying example: test_quality_limit_is_50(name='', quality=53, days_remaining=0) ======================== 1 failed, 1 passed in 0.21 seconds =========================
64. ===================================== FAILURES ====================================== _____________________________ test_quality_limit_is_50 ______________________________ @given(text(), integers(), integers()) > def test_quality_limit_is_50(name, quality, days_remaining): tests/test_gilded_rose.py:34: ... Stack trace name = '', quality = 53, days_remaining = 0 @given(text(), integers(), integers()) def test_quality_limit_is_50(name, quality, days_remaining): item = GildedRose(name, quality, days_remaining) item.tick() > assert item.quality <= 50 E AssertionError: assert 51 <= 50 E + where 51 = GildedRose(name: '', quality: 51, days_remaining: -1).quality tests/test_gilded_rose.py:38: AssertionError ------------------------------------ Hypothesis ------------------------------------- Falsifying example: test_quality_limit_is_50(name='', quality=53, days_remaining=0) ======================== 1 failed, 1 passed in 0.21 seconds =========================
65. ===================================== FAILURES ====================================== _____________________________ test_quality_limit_is_50 ______________________________ @given(text(), integers(), integers()) > def test_quality_limit_is_50(name, quality, days_remaining): tests/test_gilded_rose.py:34: ... Stack trace name = '', quality = 53, days_remaining = 0 @given(text(), integers(), integers()) def test_quality_limit_is_50(name, quality, days_remaining): item = GildedRose(name, quality, days_remaining) item.tick() > assert item.quality <= 50 E AssertionError: assert 51 <= 50 E + where 51 = GildedRose(name: '', quality: 51, days_remaining: -1).quality tests/test_gilded_rose.py:38: AssertionError ------------------------------------ Hypothesis ------------------------------------- Falsifying example: test_quality_limit_is_50(name='', quality=53, days_remaining=0) ======================== 1 failed, 1 passed in 0.21 seconds =========================
66. Trying example: test_quality_limit_is_50(name='\U000c762b\U000ac9f5\U000f15be\x00\U000b83a4.\U0009c1f0', quality=68, days_remaining=-4089)
67. Trying example: test_quality_limit_is_50(name='\U000c762b\U000ac9f5\U000f15be\x00\U000b83a4.\U0009c1f0', quality=68, days_remaining=-4089) ... stack trace ... AssertionError: assert 66 <= 50
68. Trying example: test_quality_limit_is_50(name='\U000c762b\U000ac9f5\U000f15be\x00\U000b83a4.\U0009c1f0', quality=68, days_remaining=-4089) ... stack trace ... AssertionError: assert 66 <= 50 ... lots of attempts at simplification ...
69. Trying example: test_quality_limit_is_50(name='\U000c762b\U000ac9f5\U000f15be\x00\U000b83a4.\U0009c1f0', quality=68, days_remaining=-4089) ... stack trace ... AssertionError: assert 66 <= 50 ... lots of attempts at simplification ... Falsifying example: test_quality_limit_is_50(name='', quality=53, days_remaining=0)
70. COMMON TACTICS
71. ROUND TRIP ▸ Serialize/deserialize ▸ Encrypt/decrypt assert decode(encode(data)) == data
72. IDEMPOTENCE ▸ Clean up data ▸ PUT endpoint assert clean(clean(data)) == clean(data)
73. TEST ORACLE ▸ Refactoring ▸ Performance improvement assert new_function(input) == old_function(input)
74. TRY IT OUT
75. Python Hypothesis JavaScript JSVerify Java QuickTheories Scala ScalaCheck C# and F# FsCheck PHP Eris Clojure test.check (Ruby) Hypothesis for Ruby (pre-alpha) SOME LIBRARIES
76. READ MORE ▸ Other libraries:
 https://hypothesis.works/articles/ quickcheck-in-every-language/ ▸ Other property tactics:
 https://fsharpforfunandprofit.com/posts/ property-based-testing-2/ @suzyhamilton