Controllable Randomness in Unit Tests

Talk by Alexander Voronov

Originally posted here: https://speakerdeck.com/alexandervoronov/controllable-randomness-in-unit-tests

Речь пойдет о:
- В чем польза случайных данных в юнит тестах.
- Зачем и как контролировать случайность.
- Какие инструменты нам помогут достичь этого.
- Как это внедрить в свой проект.

Source code:
https://gist.github.com/a-voronov/12fcc2139fa2d14e31b256b57ef83f27

This talk was made for CocoaFriday #2 ( https://cocoaheads.org.ua/cocoafriday/2 ) which took place Apr 5, 2019.

Video: https://youtu.be/JaRMS8J87WE

Transcript

1. CONTROLLABLE RANDOMNESS IN UNIT TESTS @aleks_voronov • CocoaFriday#2
2. TESTING IS GOOD @aleks_voronov • CocoaFriday#2
3. TESTS AND DATA @aleks_voronov • CocoaFriday#2
4. { "id": 6253282, "id_str": "6253282", "name": "Twitter API", "screen_name": "twitterapi", "location": "San Francisco, CA", "url": "https://dev.twitter.com", "description": "The Real Twitter API.", "derived": { "locations": [ {} ] }, "protected": true, "verified": false, "followers_count": 21, "friends_count": 32, "listed_count": 9274, "favourites_count": 13, "statuses_count": 42, "created_at": "Mon Nov 29 21:18:15 +0000 2010", "geo_enabled": true, "lang": "zh-cn", "contributors_enabled": false, "profile_background_color": "e8f2f7", "profile_background_image_url": "http://a2.twimg.com/profile_background_images/229557229/twitterapi-bg.png", "profile_background_image_url_https": "https://si0.twimg.com/profile_background_images/229557229/twitterapi-bg.png", "profile_background_tile": false, "profile_banner_url": "https://si0.twimg.com/profile_banners/819797/1348102824", "profile_image_url": "http://abs.twimg.com/sticky/default_profile_images/default_profile_normal.png", "profile_image_url_https": "https://abs.twimg.com/sticky/default_profile_images/default_profile_normal.png", "profile_link_color": "0094C2", "profile_sidebar_border_color": "0094C2", "profile_sidebar_fill_color": "a9d9f1", "profile_text_color": "437792", "profile_use_background_image": true, "default_profile": false, "default_profile_image": false, "withheld_in_countries": [ "GR", "HK", "MY" ], "withheld_scope": "user" } @aleks_voronov • CocoaFriday#2
5. MANUAL DATA INATTENTIONAL BLINDNESS @aleks_voronov • CocoaFriday#2
6. PREDEFINED (DERIVED) DATA YAML • JSON • PLIST • SQLITE @aleks_voronov • CocoaFriday#2
7. DATA GENERATED ON THE FLY ! ▸ Specification OpenAPI • Swift Type System ▸ Behavior Properties SwiftCheck @aleks_voronov • CocoaFriday#2
8. MIXED APPROACH Generated by specification • using predefined data sets • and manual clarifications @aleks_voronov • CocoaFriday#2
9. IMPLEMENTATION @aleks_voronov • CocoaFriday#2
10. GENERATING RANDOM NUMBERS ! @aleks_voronov • CocoaFriday#2
11. MODULO BIAS @aleks_voronov • CocoaFriday#2
12. func random(from: Int, to: Int) -> Int { return Int(arc4random_uniform(UInt32(to) - UInt32(from) + 1) + UInt32(from)) } @aleks_voronov • CocoaFriday#2
13. @aleks_voronov • CocoaFriday#2
14. 4.2 protocol RandomNumberGenerator { mutating func next() -> UInt64 ... } github.com/apple/swift-evolution/blob/master/proposals/0202-random-unification.md @aleks_voronov • CocoaFriday#2
15. 4.2 struct SystemRandomNumberGenerator: RandomNumberGenerator { ... } github.com/apple/swift-evolution/blob/master/proposals/0202-random-unification.md @aleks_voronov • CocoaFriday#2
16. 4.2 Bool • Int • Double • Array • Set • Dictionary random() -> T random<G: RandomNumberGenerator>(inout G) -> T github.com/apple/swift-evolution/blob/master/proposals/0202-random-unification.md @aleks_voronov • CocoaFriday#2
17. GENERIC 'RANDOM' INTERFACE protocol Random { static func random<G: RandomNumberGenerator>( using generator: inout G ) -> Self } @aleks_voronov • CocoaFriday#2
18. CONFORMING BASICS extension Character: Random { ... } extension UnicodeScalar: Random { ... } extension FloatingPoint where Self: Random { ... } extension FixedWidthInteger where Self: Random { ... } @aleks_voronov • CocoaFriday#2
19. extension Bool: Random { } extension Float: Random { } extension Double: Random { } extension Int: Random { } extension Int64: Random { } extension Int32: Random { } extension Int16: Random { } extension Int8: Random { } extension UInt: Random { } extension UInt64: Random { } extension UInt32: Random { } extension UInt16: Random { } extension UInt8: Random { } @aleks_voronov • CocoaFriday#2
20. RANDOM COLLECTIONS extension Array: Random where Element: Random { static func random<G: RandomNumberGenerator>( using generator: inout G ) -> [Element] { ... } } @aleks_voronov • CocoaFriday#2
21. RANDOM ENUMS protocol RandomAll: Random { static func allRandom<G: RandomNumberGenerator>( using generator: inout G ) -> [Self] } @aleks_voronov • CocoaFriday#2
22. OPTIONAL EXAMPLE extension Optional: RandomAll where Wrapped: Random { public static func allRandom<G: RandomNumberGenerator>( using generator: inout G ) -> [Wrapped?] { return [.none, .some(.random(using: &generator))] } } extension Optional: Random where Wrapped: Random {} @aleks_voronov • CocoaFriday#2
23. RANDOM TUPLES func randomTuple< A: Random, B: Random, G: RandomNumberGenerator >(using generator: inout G) -> (A, B) { return ( .random(using: &generator), .random(using: &generator) ) } @aleks_voronov • CocoaFriday#2
24. AND SO ON ... @aleks_voronov • CocoaFriday#2
25. LET'S USE IT struct User { let id: String let name: String let age: Int let friends: (followers: Int, followees: Int) } @aleks_voronov • CocoaFriday#2
26. extension User: Random { static func random<G: RandomNumberGenerator>( using generator: inout G ) -> User { return User( id: .random(using: &generator), name: .random(using: &generator), age: .random(using: &generator), friends: randomTuple(using: &generator) ) } } @aleks_voronov • CocoaFriday#2
27. USAGE User.random(using: &generator) @aleks_voronov • CocoaFriday#2
28. TESTING ... @aleks_voronov • CocoaFriday#2
29. FAILED USER.NAME CONTAINS INVALID SYMBOLS @aleks_voronov • CocoaFriday#2
30. REPLAYING 'RANDOM' TESTS @aleks_voronov • CocoaFriday#2
31. PSEUDO-RANDOM NUMBER GENERATOR SEEDS @aleks_voronov • CocoaFriday#2
32. @aleks_voronov • CocoaFriday#2
33. struct PRNG: RandomNumberGenerator { typealias State = UInt64 private(set) var state: State init() { var generator = SystemRandomNumberGenerator() state = .random(using: &generator) } init(seed: State) { state = seed } mutating func next() -> UInt64 { state = magic(state) return state } } @aleks_voronov • CocoaFriday#2
34. ! USAGE ▸ XCTest.setUp ? ▸ Quick.beforeAll ? ▸ NSPrincipalClass ! @aleks_voronov • CocoaFriday#2
35. If an NSPrincipalClass key is declared in the test bundle's Info.plist file, XCTest automatically creates a single instance of that class when the test bundle is loaded ... – developer.apple.com/documentation/... @aleks_voronov • CocoaFriday#2
36. public var R = PRNG() class RandomSeed: NSObject { override init() { super.init() let seed = "\(R.state)" print(seed) } } @aleks_voronov • CocoaFriday#2
37. LOUD AND CLEAR @aleks_voronov • CocoaFriday#2
38. 10 developer.apple.com/videos/play/wwdc2018/403/ @aleks_voronov • CocoaFriday#2
39. GET EVERY FAILED TEST SEED ✅ ✅ ❌ ✅ ❌ –––––––8–––––12–––––––––––23–42 @aleks_voronov • CocoaFriday#2
40. XCTESTOBSERVATIONCENTER @aleks_voronov • CocoaFriday#2
41. If an NSPrincipalClass key is declared in the test bundle's Info.plist file, XCTest automatically creates a single instance of that class when the test bundle is loaded. You can use this instance as a place to register observers or do other pretesting global setup before testing for that bundle begins. – developer.apple.com/documentation/xctest/xctestobservationcenter @aleks_voronov • CocoaFriday#2
42. XCTESTOBSERVATION ▸ START: Save Seed 1 ▸ FAIL: Log Seed * ▸ SUCCESS: Do nothing * developer.apple.com/documentation/xctest/xctestobservation @aleks_voronov • CocoaFriday#2
43. @aleks_voronov • CocoaFriday#2
44. SOURCERY github.com/krzysztofzablocki/Sourcery @aleks_voronov • CocoaFriday#2
45. import Foundation {# So far getting imports hardcoded from config, correct approach might be found here: https://github.com/krzysztofzablocki/Sourcery/issues/670 #} {% for import in argument.imports %} import {{ import }} {% endfor %} {% if argument.testable %}{% for testable in argument.testable %} @testable import {{ testable }} {% endfor %}{% endif %} {% macro randomValue type %}{% if type.kind == "protocol" %}{{ type.inheritedTypes.0.name }}{% endif %}{% endmacro %} // MARK: - Structs {# Random Struct #} {% macro rng %}&{{ argument.rng }}{% endmacro %} {% macro customRandomValueType variable %}{% if variable.annotations.random %}{{ variable.annotations.random }}{% endif %}{% endmacro %} {% macro randomValueUsingGenerator variable %}{% call customRandomValueType variable %}{% if variable.isTuple %}randomTuple(using: &generator){% else %}.random(using: &generator){% endif %}{% endmacro %} {% macro randomValueUsingRNG variable %}{% call customRandomValueType variable %}{% if variable.isTuple %}randomTuple(using: {% call rng %}){% else %}.random(using: {% call rng %}){% endif %}{% endmacro %} {% for type in types.structs where type|annotated:"Random" %} extension {{ type.name }}: Random { public static func random<G: RandomNumberGenerator>(using generator: inout G) -> {{ type.name }} { return {{ type.name }}( {% for variable in type.variables where not variable.isComputed %} {{ variable.name }}: {% call randomValueUsingGenerator variable %}{% if not forloop.last %},{% endif %} {% endfor %} ) } {% if argument.rng %} {# user-friendly version, so that you don't have to bother with closures and incoming generators. But it's tightly coupled to `rng` argument value from sourcery config #} public static func random( {% for variable in type.variables where not variable.isComputed %} {{ variable.name }}: {{ variable.typeName }} = {% call randomValueUsingRNG variable %}{% if not forloop.last %},{% endif %} {% endfor %} ) -> {{ type.name }} { return {{ type.name }}( {% for variable in type.variables where not variable.isComputed %} {{ variable.name }}: {{ variable.name }}{% if not forloop.last %},{% endif %} {% endfor %} ) } {% else %} {# this one is generated additionally, so that you can have everything random except for some selected fields #} public static func random<G: RandomNumberGenerator>( _ generator: inout G, {% for variable in type.variables where not variable.isComputed %} {{ variable.name }}: (inout G) -> {{ variable.typeName }} = { generator in {% call randomValueUsingGenerator variable %} }{% if not forloop.last %},{% endif %} {% endfor %} ) -> {{ type.name }} { return {{ type.name }}( {% for variable in type.variables where not variable.isComputed %} {{ variable.name }}: {{ variable.name }}(&generator){% if not forloop.last %},{% endif %} {% endfor %} ) } {% endif %} } {% endfor %} // MARK: - Enums {# Random Enum #} {% macro randomEnumCaseUsingGenerator case %}.{{ case.name }}{% if case.hasAssociatedValue %}({% for value in case.associatedValues %}{% if value.localName %}{{ value.localName }}: {% endif %}{% call randomValueUsingGenerator value %}{% if not forloop.last %}, {% endif %}{% endfor %}){% endif %}{% endmacro %} {% macro randomIfCaseUsingGenerator case %}{% if case.hasAssociatedValue %}{{ case.name }}(&generator){% else %}.{{ case.name }}{% endif %}{% endmacro %} {% macro randomEnumCaseUsingRNG case %}.{{ case.name }}{% if case.hasAssociatedValue %}({% for value in case.associatedValues %}{% if value.localName %}{{ value.localName }}: {% endif %}{% call randomValueUsingRNG value %}{% if not forloop.last %}, {% endif %}{% endfor %}){% endif %}{% endmacro %} {% macro randomIfCaseUsingRNG case %}{% if case.hasAssociatedValue %}{{ case.name }}{% else %}.{{ case.name }}{% endif %}{% endmacro %} {% for enum in types.enums where enum|annotated:"Random" %} {% if enum.cases.count == 0 %} #warning("`{{ enum.name }}` is an uninhabitant type and no value of it can be created, thus it can't have random value as well.") // extension {{ enum.name }}: Random { } {% elif not enum.hasAssociatedValues %} extension {{ enum.name }}: Random { {# I'd expect to check `enum.based` property, but it's always empty ¯\_(ϑ)_/¯ #} {% if enum.rawTypeName.name != "CaseIterable" and enum.inheritedTypes|join:" "|!contains:"CaseIterable" %} #warning("Please, conform to `CaseIterable` protocol, so that compiler takes care of this") public static let allCases: [{{ enum.name }}] = [{% for case in enum.cases %}.{{ case.name }}{% if not forloop.last %}, {% endif %}{% endfor %}] {% endif %} public static func random<G: RandomNumberGenerator>(using generator: inout G) -> {{ enum.name }} { return allCases.randomElement(using: &generator)! } } {% else %} extension {{ enum.name }}: RandomAll { public static func allRandom<G: RandomNumberGenerator>(using generator: inout G) -> [{{ enum.name }}] { {% if enum.cases.count == 1 %} return [{% call randomEnumCaseUsingGenerator enum.cases.0 %}] {% else %} return [{% for case in enum.cases %} {% call randomEnumCaseUsingGenerator case %}{% if not forloop.last %},{% endif %}{% endfor %} ] {% endif %} } {# alternative version, so that you don't have to bother with closures and incoming generators but it's tightly coupled to `rng` argument value from sourcery config and it executes and calculates randoms for all cases even though only one will be needed (thus a bit more expensive) #} {% if argument.rng %} public static func random( {% for case in enum.cases where case.associatedValues %} {{ case.name }}: {{ enum.name }} = {% call randomEnumCaseUsingRNG case %}{% if not forloop.last %},{% endif %} {% endfor %} ) -> {{ enum.name }} { {% if enum.cases.count == 1 %} return {% call randomIfCaseUsingRNG enum.cases.0 %} {% else %} return [ {% for case in enum.cases %} {% call randomIfCaseUsingRNG case %}{% if not forloop.last %},{% endif %} {% endfor %} ].randomElement(using: {% call rng %})! {% endif %} } {% else %} {# same for enum - you can override `random` for some cases and leave others generated by default #} public static func random<G: RandomNumberGenerator>( _ generator: inout G, {% for case in enum.cases where case.associatedValues %} {{ case.name }}: (inout G) -> {{ enum.name }} = { generator in {% call randomEnumCaseUsingGenerator case %} }{% if not forloop.last %},{% endif %} {% endfor %} ) -> {{ enum.name }} { {% if enum.cases.count == 1 %} return {% call randomIfCaseUsingGenerator enum.cases.0 %} {% else %} return [ {% for case in enum.cases %} {% call randomIfCaseUsingGenerator case %}{% if not forloop.last %},{% endif %} {% endfor %} ].randomElement(using: &generator)! {% endif %} } {% endif %} } {% endif %} {% endfor %} @aleks_voronov • CocoaFriday#2
46. ARE WE THERE YET? @aleks_voronov • CocoaFriday#2
47. FASTLANE + XCPRETTY OUTPUT @aleks_voronov • CocoaFriday#2
48. INTERESTING ▸ pointfree.co : ep30 - ep32 • ep47 - ep50 ▸ github.com/pointfreeco/swift-gen ▸ github.com/typelift/SwiftCheck ▸ ieeexplore.ieee.org/document/6963470 : Oracles @aleks_voronov • CocoaFriday#2
49. THANKS! GIST.GITHUB.COM/A-VORONOV/12FCC2139FA2D14E31B256B57EF83F27 @ALEKS_VORONOV @aleks_voronov • CocoaFriday#2