Functions for nothing, and your tests for free

An intro to property-based testing, and using F# for testing.

Presented at the ALT.NET Melbourne meetup.

Transcript

1. Functions for nothing, and your tests for free George Pollard (@porges)
2. Functions for nothing, and your tests for free { F#
3. Functions for nothing, and your tests for free } { F# FsCheck & property-based testing
5. ‘’
10. : • What are property-based tests? • What do Hedgehog/FsCheck do? • Applying property-based testing • Patterns • What not to do • F# for testing • Demo: model-based testing
11. • What are property-based tests? • What do Hedgehog/FsCheck do? • Applying property-based testing • Patterns • What not to do • F# for testing • Demo: model-based testing
12. What is the point of property-based testing?
13. What problem does it solve?
14. [Fact] public void TestUppercase() { var input = "hello"; Assert.Equal("HELLO", input.ToUpper()); } [<Fact>] let testUppercase() = let input = "hello" Assert.Equal("HELLO", input.ToUpper())
15. [Theory] [InlineData("hello", "HELLO")] public void TestUppercase(string input, string output) { Assert.Equal(output, input.ToUpper()); } [<Theory>] [<InlineData("hello", "HELLO")>] let testUppercase (input : string) (output : string) = Assert.Equal(output, input.ToUpper())
16. [Theory] [InlineData("hello", "HELLO")] public void TestUppercase(string input, string output) { Assert.Equal(output, input.ToUpper()); } [<Theory>] [<InlineData("hello", "HELLO")>] let testUppercase (input : string) (output : string) = Assert.Equal(output, input.ToUpper())
17. [Theory] [InlineData("hello", "HELLO")] public void TestUppercase(string input, string output) { Assert.Equal(output, input.ToUpper()); } [<Theory>] [<InlineData("hello", "HELLO")>] let testUppercase (input : string) (output : string) = Assert.Equal(output, input.ToUpper())
19. [Theory] [InlineData("beer", "BEER")] [InlineData("0", "0")] [InlineData("lizard", "LIZARD")] [InlineData("sfdeljknesv", "SFDELJKNESV")] public void TestUppercase(string input, string output) { Assert.Equal(output, input.ToUpper()); }
20. public void TestUppercase(string input) { var output = input.ToUpper(); Assert.DoesNotContain(output, char.IsLower); } let testUppercase (input : string) = let output = input.ToUpper() Assert.DoesNotContain(output, Char.IsLower)
21. public void NoLowercaseInToUpperResult(string input) { var output = input.ToUpper(); Assert.DoesNotContain(output, char.IsLower); } let noLowercaseInToUpperResult (input : string) = let output = input.ToUpper() Assert.DoesNotContain(output, Char.IsLower)
22. • What are property-based tests? • What do Hedgehog/FsCheck do? • Applying property-based testing • Patterns • What not to do • F# for testing • Demo: model-based testing
23. What are FsCheck/Hedgehog?: Libraries for randomized execution of property-based tests
24. What do they provide?: A way to generate (and shrink) random test data
25. What do they provide?: A way to generate (and shrink) random test data Integration with test runner
26. public void NoLowercaseInToUpperResult(string input) { var output = input.ToUpper(); Assert.DoesNotContain(output, char.IsLower); } let noLowercaseInToUpperResult (input : string) = let output = input.ToUpper() Assert.DoesNotContain(output, Char.IsLower)
27. [Property] public void ToUpperLeavesNoLowercase(NonNull<string> input) { var output = input.Get.ToUpper(); Assert.DoesNotContain(output, char.IsLower); } [<Property>] let noLowercaseInToUpperResult (input : NonNull<string>) = let output = input.Get.ToUpper() Assert.DoesNotContain(output, Char.IsLower)
28. [Property] public void ToUpperLeavesNoLowercase(NonNull<string> input) { var output = input.Get.ToUpper(); Assert.DoesNotContain(output, char.IsLower); } [<Property>] let noLowercaseInToUpperResult (input : NonNull<string>) = let output = input.Get.ToUpper() Assert.DoesNotContain(output, Char.IsLower)
29. [Property] public void ToUpperLeavesNoLowercase(NonNull<string> input) { var output = input.Get.ToUpper(); Assert.DoesNotContain(output, char.IsLower); } [<Property>] let noLowercaseInToUpperResult (input : NonNull<string>) = let output = input.Get.ToUpper() Assert.DoesNotContain(output, Char.IsLower)
30. Writing your own generator: C#: static Gen<string> FoodGenerator = Gen.Elements("fish", "carrot", "shoes"); static Gen<Dog> DogGenerator = from name in Arb.Generate<string>() from colours in Gen.ArrayOf(Arb.Generate<Color>()) from food in FoodGenerator select new Dog(name, colours, food);
31. Writing your own generator: F#: let foodGenerator = Gen.elements [ "fish"; "carrot"; "shoes" ] let dogGenerator = gen { let! name = Arb.generate<string> let! colours = Gen.arrayOf Arb.generate<Color> let! food = foodGenerator return Dog(name, colours, food) }
34. "spades": …, "/specialkey": …, "diamonds": …, "/magic": …,
35. "spades": …, "diamonds": …, "/specialkey": …, "/magic": …,
36. "繅": …, "㉒": …, "/specialkey": …, "/magic": … "⚾/fsfasd": …,
37. if (key.StartsWith("/")) { … }
38. "ͿԤԥԨԩԪԫԬԭԮԯ֍֎֏ࢻ ࢺࢹࢸࢷࢶࢯࢮࢭࢬࢫࢪࢩࢨࢧࢦࢥࢤࢣࢢࢡࢠ ࢽࢼ෦෧෨෩෪෫෬෭෮෯Ⴧ᠚᠛᠜᠝᠞₶₷₸₻₼₽₾₿↊↋⚽⚾⚿✀ ➕➖➗➰➿⟋⟍⭍⭎⭏⭚⭛⭜⭝⭞⭟⭠⭡⭢⭣⭤⭥⭦⭧⭨⭩⭪⭫⭬⭭⭮⭯ ⭰⭱⭲⭳⭶⭷⭸⭹⭺⭻⭼⭽⭾⭿⮀⮁⮂⮃⮄⮅⮆⮇⮈⮉⮊⮋⮌⮍ ⮎⮏⮐⮑⮒⮓⮔⮕⮘⮝⮞⮟⮠⮡⮢⮣⮤⮥⮦⮧⮨⮩⮪⮫⮬⮭⮮⮯ ⮰⮱⮲⮳⮴⮵⮶⮷⮸⮹⮙⮚⮛⮜⮽⮾⮿⯀⯁⯂⯃⯄⯅⯆⯇⯈⯊⯋⯌ ⯍⯎⯏⯐⸴⸵⸶⸷⸸⸹⸺⸻⸼⸽⸾⸿⹀⹁⹂ꞏꞒꞓꞔꞕꞖꞗꞘꞙꞚꞛꞜꞝꞞꞟꞪꬰꬱꬲꬳꬴꬵꬶꬷꬸꬹꬺ ꬻꬼꬽꬾꬿꭀꭁꭂꭃꭄꭅꭆꭇꭈꭉꭊꭋꭌꭍꭎꭏꭐꭑꭒꭓꭔꭕꭗꭘꭙꭚ꭛ꭜꭝꭞꭟꭠꭡꭢꭣꭤꭥ/"
39. "ͿԤԥԨԩԪԫԬԭԮԯ֍֎֏ࢻ ࢺࢹࢸࢷࢶࢯࢮࢭࢬࢫࢪࢩࢨࢧࢦࢥࢤࢣࢢࢡࢠ ࢽࢼ෦෧෨෩෪෫෬෭෮෯Ⴧ᠚᠛᠜᠝᠞₶₷₸₻₼₽₾₿↊↋⚽⚾⚿✀ ➕➖➗➰➿⟋⟍⭍⭎⭏⭚⭛⭜⭝⭞⭟⭠⭡⭢⭣⭤⭥⭦⭧⭨⭩⭪⭫⭬⭭⭮⭯ ⭰⭱⭲⭳⭶⭷⭸⭹⭺⭻⭼⭽⭾⭿⮀⮁⮂⮃⮄⮅⮆⮇⮈⮉⮊⮋⮌⮍ ⮎⮏⮐⮑⮒⮓⮔⮕⮘⮝⮞⮟⮠⮡⮢⮣⮤⮥⮦⮧⮨⮩⮪⮫⮬⮭⮮⮯ ⮰⮱⮲⮳⮴⮵⮶⮷⮸⮹⮙⮚⮛⮜⮽⮾⮿⯀⯁⯂⯃⯄⯅⯆⯇⯈⯊⯋⯌ ⯍⯎⯏⯐⸴⸵⸶⸷⸸⸹⸺⸻⸼⸽⸾⸿⹀⹁⹂ꞏꞒꞓꞔꞕꞖꞗꞘꞙꞚꞛꞜꞝꞞꞟꞪꬰꬱꬲꬳꬴꬵꬶꬷꬸꬹꬺ ꬻꬼꬽꬾꬿꭀꭁꭂꭃꭄꭅꭆꭇꭈꭉꭊꭋꭌꭍꭎꭏꭐꭑꭒꭓꭔꭕꭗꭘꭙꭚ꭛ꭜꭝꭞꭟꭠꭡꭢꭣꭤꭥ/"
40. if (key.StartsWith("/", StringComparison.Ordinal)) { … }
42. ^[a-z0-9]{3,24}$
43. ^[a-z0-9]{3,24}$ 1. Start of string,
44. ^[a-z0-9]{3,24}$ 1. Start of string, 2. alphanumeric characters…
45. ^[a-z0-9]{3,24}$ 1. Start of string, 2. alphanumeric characters 3. … 3-24 times,
46. ^[a-z0-9]{3,24}$ 1. Start of string, 2. alphanumeric characters 3. … 3-24 times, 4. end of string.
47. ^[a-z0-9]{3,24}$ 1. Start of string, 2. alphanumeric characters 3. … 3-24 times, 4. end of string?
48. ^[a-z0-9]{3,24}$ 1. Start of string, 2. alphanumeric characters 3. … 3-24 times, 4. end of string… possibly preceded by a newline.
49. ^[a-z0-9]{3,24}\z 1. Start of string, 2. alphanumeric characters 3. … 3-24 times, 4. end of string.
50. Lessons • Imagination & ability
51. • What are property-based tests? • What do Hedgehog/FsCheck do? • Applying property-based testing • Patterns • What not to do • F# for testing • Demo: model-based testing
52. Applying property-based testing: How can I use it with my existing code? What properties do I test for?
54. Do: ‘Upgrade’ parameterized tests: Replace ‘canned examples’ Or: you can keep the existing one and add a randomized version
55. Do: check idempotence: ‘once is enough’ | f(f(x)) == f(x) Examples: Uppercase, Absolute, etc
56. Do: check idempotence: ‘once is enough’ | x.f(…); | x.f(…); y.f(…); | Assert(x == y); e.g. Create/Delete
57. Do: check roundtripping: ‘there and back’ | from(to(x)) == x Examples: Serialization
58. Do: check roundtripping: ‘there and back’ | store(x); | y = get(x); | Assert(x == y); e.g. PUT/GET
59. Do: check roundtripping: ‘involution’/self-inverse | f(f(x)) == x Examples: Reverse, Negate, etc
60. Do: check invariants: ‘things that don’t change’ | inv(f(x)) == inv(x) Examples: Reverse, Select/Map, etc (length)
61. Do: check commutativity: ‘order doesn’t matter’ | f(x, y) == f(y, x) Examples: Addition, Multiplication, etc Min, Max, etc
62. Do: check commutativity: ‘order doesn’t matter’ f.x(); f.y(); f.z(); == f.z(); f.y(); f.x();
63. Do: check associativity: ‘any grouping’ f(x, f(y, z)) == f(f(x, y), z)
64. Do: check associativity: ‘any grouping’ e.g. Reduce x z w … = (((x y) z) w … (x y) (z w)
65. Do: use oracles: Trusted implementation that gives the correct result f(x) == trusted_f(x)
66. Do: use oracles: Use them to verify: • Reimplementations/rewrites • Optimized implementations
67. Do: Use model-based testing: Create a simpler version that doesn’t implement the full specification Use it to check only those parts
68. Do: Use model-based testing: 1. Describe actions as values 2. Randomly generate actions 3. Apply to all implementations 4. Verify real against model
70. Do not: Re-run failures: Throws away the specific failure
71. Do not: Re-run failures: Throws away the specific failure Instead: Investigate failure, save as new example-based test
72. Do not: Re-implement real code: Commonly arises when testing validation
73. Do not: Re-implement real code: Commonly arises when testing validation Instead: Test negative cases
74. Do not: Perform ad-hoc filtering: For conditional tests, don’t filter input values yourself
75. Do not: Perform ad-hoc filtering: For conditional tests, don’t filter input values yourself Instead: Use the framework support
76. Do not: Perform ad-hoc filtering:
77. • What are property-based tests? • What do Hedgehog/FsCheck do? • Applying property-based testing • Patterns • What not to do • F# for testing • Demo: model-based testing
78. (CC-BY) https://www.flickr.com/photos/streetmatt/15761889077
79. type PlayingCard = { rank : Rank; suit : Suit }
80. type PlayingCard = { rank : Rank; suit : Suit }
81. type PlayingCard = { rank : Rank; suit : Suit }
82. class PlayingCard { public PlayingCard(Rank rank, Suit suit) { Rank = rank; Suit = suit; } public Rank Rank { get; } public Suit Suit { get; } }
83. class PlayingCard : IEquatable<PlayingCard> { public PlayingCard(Rank rank, Suit suit) { Rank = rank; Suit = suit; } public Rank Rank { get; } public Suit Suit { get; } public bool Equals(PlayingCard other) => other != null && Rank == other.Rank && Suit == other.Suit; public override bool Equals(object obj) => Equals(obj as PlayingCard); public override int GetHashCode() => (Rank.GetHashCode() * 137) ^ Suit.GetHashCode(); }
84. class PlayingCard : IEquatable<PlayingCard> , IComparable<PlayingCard> { public PlayingCard(Rank rank, Suit suit) { Rank = rank; Suit = suit; } public Rank Rank { get; } public Suit Suit { get; } public int CompareTo(PlayingCard other) { var rankCompare = Rank.CompareTo(other.Rank); if (rankCompare != 0) { return rankCompare; } return Suit.CompareTo(other.Suit); } public bool Equals(PlayingCard other) => other != null && Rank == other.Rank && Suit == other.Suit; public override bool Equals(object obj) => Equals(obj as PlayingCard); public override int GetHashCode() => (Rank.GetHashCode() * 137) ^ Suit.GetHashCode(); }
85. class PlayingCard : IEquatable<PlayingCard> , IComparable<PlayingCard> { public PlayingCard(Rank rank, Suit suit) { Rank = rank; Suit = suit; } public Rank Rank { get; } public Suit Suit { get; } public int CompareTo(PlayingCard other) { var rankCompare = Rank.CompareTo(other.Rank); if (rankCompare != 0) { return rankCompare; } return Suit.CompareTo(other.Suit); } public bool Equals(PlayingCard other) => other != null && Rank == other.Rank && Suit == other.Suit; public override bool Equals(object obj) => Equals(obj as PlayingCard); public override int GetHashCode() => (Rank.GetHashCode() * 137) ^ Suit.GetHashCode(); public override string ToString() => $"Rank: {Rank} Suit: {Suit}"; }
86. class PlayingCard : IEquatable<PlayingCard> , IComparable<PlayingCard> { public PlayingCard(Rank rank, Suit suit) { Rank = rank; Suit = suit; } public Rank Rank { get; } public Suit Suit { get; } public int CompareTo(PlayingCard other) { var rankCompare = Rank.CompareTo(other.Rank); if (rankCompare != 0) { return rankCompare; } return Suit.CompareTo(other.Suit); } public bool Equals(PlayingCard other) => other != null && Rank == other.Rank && Suit == other.Suit; public override bool Equals(object obj) => Equals(obj as PlayingCard); public override int GetHashCode() => (Rank.GetHashCode() * 137) ^ Suit.GetHashCode(); public override string ToString() => $"Rank: {Rank} Suit: {Suit}"; public static bool operator ==(PlayingCard left, PlayingCard right) => Equals(left, right); public static bool operator !=(PlayingCard left, PlayingCard right) => !Equals(left, right); public static bool operator <(PlayingCard left, PlayingCard right) => left.CompareTo(right) < 0; public static bool operator >(PlayingCard left, PlayingCard right) => left.CompareTo(right) > 0; public static bool operator >=(PlayingCard left, PlayingCard right) => left.CompareTo(right) >= 0; public static bool operator <=(PlayingCard left, PlayingCard right) => left.CompareTo(right) <= 0; }
87. type PlayingCard = { rank : Rank; suit : Suit }
89. type Suit = Spades | Clubs | Hearts | Diamonds
90. type Result<'t> = | Failure of string | Success of 't
91. class Result<T> { class Success : Result<T> { private T _value; // ... } class Failure : Result<T> { private string _value; // ... } } type Result<'t> = | Failure of string | Success of 't
93. type Result<'t> = | Failure of string | Success of ‘t let getPerson (name : string) : Result<Person> = …
94. type Result<'t> = | Failure of string | Success of ‘t let getPerson (name : string) : Result<Person> = … let myFunc () = match getPerson "George" with | Failure msg -> … | Success p -> …
95. type Result<'t> = | Failure of string | Success of ‘t let getPerson (name : string) : Result<Person> = … let myFunc () = match getPerson "George" with | Failure "Currently presenting" -> … | Failure msg -> … | Success p -> …
97. [Fact] public void AccountBalanceIncreasesWhenDepositIsMade() { var bankAccount = new BankAccount(); bankAccount.Deposit(100); Assert.Equal(100, bankAccount.Balance); }
98. [Fact] public void AccountBalanceIncreasesWhenDepositIsMade() { var bankAccount = new BankAccount(); bankAccount.Deposit(100); Assert.Equal(100, bankAccount.Balance); }
99. [Fact] public void Account_Balance_Increases_When_Deposit_Is_Made() { var bankAccount = new BankAccount(); bankAccount.Deposit(100); Assert.Equal(100, bankAccount.Balance); }
101. [<Fact>] let ``Account balance increases when deposit is made`` () = let account = deposit emptyAccount 100 Assert.Equal(100, account.balance)
102. [<Fact>] let ``Great test `` () = let account = deposit emptyAccount 100 Assert.Equal(100, account.balance)
104. Further reading QuickCheck (paper): https://www.cs.tufts.edu/~nr/cs257/archive/john-hughes/quick.pdf Paket (about oracles): http://www.navision-blog.de/blog/2016/03/21/property-based-testing-in- the-real-world/ F#: ‘F# for fun and profit’ - property-based testing - domain modeling made functional