Testing without Xcode - CMD+U 2016

Transcript

1. Testing without
   Xcode
   Kyle Fuller
   

   View Slide

2. View Slide

3. View Slide

4. View Slide

5. View Slide

6. View Slide

7. View Slide

8. Open Source
   Libraries
   

   View Slide

9. Existing code on
   other Platforms
   

   View Slide

10. Editors
    

    View Slide

11. Who am I
    

    View Slide

12. View Slide

13. Agenda
    • How other communities do testing
    • How testing is done in Swift today
    • How we can do testing in Swift in the future
    

    View Slide

14. How do other communities
    solve testing?
    • Ruby !
    • C / C++ ䷢
    

    View Slide

15. Testing in Ruby
    describe '.authenticate' do
    context 'when logged in' do
    it { is_expected.to respond_with 200 }
    end
    end
    

    View Slide

16. Testing in Ruby
    $ rspec test.rb
    Finished in 0.0016 seconds (files took 0.2153 seconds to load)
    5 examples, 0 failures
    

    View Slide

17. Testing in C / C++
    int main(int argc, const char *argv[]) {
    user_t user = user_alloc("kyle");
    // Test User's name
    assert(user.name == "kyle");
    // Test User's priviledges
    assert(user.is_admin, 0);
    return 0;
    }
    

    View Slide

18. Testing in C / C++
    $ clang test.c -o tests
    $ ./tests
    Tests passed
    

    View Slide

19. Testing in C / C++
    #define CATCH_CONFIG_MAIN
    #include "catch.hpp"
    TEST_CASE( "Factorials are computed", "[factorial]" ) {
    REQUIRE( Factorial(1) == 1 );
    REQUIRE( Factorial(2) == 2 );
    REQUIRE( Factorial(3) == 6 );
    REQUIRE( Factorial(10) == 3628800 );
    }
    

    View Slide

20. Test Framework ~= Test Runner
    

    View Slide

21. Testing in Swift
    • assert
    • XCTest
    • Quick (uses XCTest)
    • Spectre
    • Ploughman
    • Custom testing
    • CLI tool
    

    View Slide

22. Testing with
    assert
    

    View Slide

23. assert()
    

    View Slide

24. assert(person.name == "Kyle")
    

    View Slide

25. assert(person.name == "Kyle",
    "Persons name should be Kyle")
    

    View Slide

26. struct Person: CustomStringConvertible {
    let name: String
    var description: String {
    return name
    }
    }
    let kyle = Person(name: "Kyle")
    assert(kyle.description == "Kyle",
    "conforms to CustomStringConvertible")
    

    View Slide

27. $ swift test.swift
    

    View Slide

28. Testing in a
    playground
    

    View Slide

29. Demo
    

    View Slide

30. assert() testing
    Upsides
    • Extremely simple
    • Built into stdlib
    Downsides
    • Hard to manage
    • Crashes your process, not all tests are ran
    

    View Slide

31. XCTest
    

    View Slide

32. CompileSwift normal x86_64
    /Users/kyle/Projects/QueryKit/QueryKit/Tests/SortDescriptorTests.swift cd
    /Users/kyle/Projects/QueryKit/QueryKit
    /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swift
    -frontend -c /Users/kyle/Projects/QueryKit/QueryKit/Tests/AttributeTests.swift
    /Users/kyle/Projects/QueryKit/QueryKit/Tests/QueryKitTests.swift -primary-file
    /Users/kyle/Projects/QueryKit/QueryKit/Tests/SortDescriptorTests.swift
    ...
    Test Suite 'All tests' started at 2016-07-08 01:59:05.062
    Test Suite 'QueryKitTests.xctest' started at 2016-07-08 01:59:05.063
    Test Suite 'AttributeTests' started at 2016-07-08 01:59:05.063
    Test Case '-[QueryKitTests.AttributeTests testAscendingSortDescriptor]' started.
    Test Case '-[QueryKitTests.AttributeTests testAscendingSortDescriptor]' passed (0.002 seconds).
    Test Case '-[QueryKitTests.AttributeTests testAttributeExpression]' started.
    Test Case '-[QueryKitTests.AttributeTests testAttributeExpression]' passed (0.001 seconds).
    Test Suite 'SortDescriptorTests' passed at 2016-07-08 01:59:05.362.
    Executed 2 tests, with 0 failures (0 unexpected) in 0.001 (0.003) seconds
    Test Suite 'QueryKitTests.xctest' passed at 2016-07-08 01:59:05.363.
    Executed 75 tests, with 0 failures (0 unexpected) in 0.240 (0.300) seconds
    Test Suite 'All tests' passed at 2016-07-08 01:59:05.364.
    Executed 75 tests, with 0 failures (0 unexpected) in 0.240 (0.302) seconds
    ** TEST SUCCEEDED **
    

    View Slide

33. xctool & xcpretty
    

    View Slide

34. https://github.com/supermarin/xcpretty
    

    View Slide

35. View Slide

36. Test Anywhere Protocol (TAP)
    ...
    ok 71 - testTypeSafeOrderBySortDescriptor
    ok 72 - testTypeSafeOrderBySortDescriptors
    ok 73 - testTypeSafeRelatedFilterPredicate
    ok 74 - testAscendingSortDescriptor
    ok 75 - testDescendingSortDescriptor
    1..75
    

    View Slide

37. View Slide

38. How does XCTest work?
    $ xcrun -sdk macosx xctest
    Usage: xctest
    

    View Slide

39. import XCTest
    class PersonTests : XCTestCase {
    func testCustomStringConvertible() {
    let kyle = Person(name: "Kyle")
    XCTAssertEqual(kyle.description, "Kyle")
    }
    }
    

    View Slide

40. How does XCTest work without
    Objective-C?
    import XCTest
    extension PersonTests {
    static var allTests : [(String, PersonTests -> () throws -> Void)] {
    return [
    ("testCustomStringConvertible", testCustomStringConvertible),
    ]
    }
    }
    XCTMain([
    testCase(PersonTests.allTests),
    ])
    

    View Slide

41. View Slide

42. Spectre
    http://spectre.fuller.li/
    

    View Slide

43. describe("Person") {
    let kyle = Person(name: "Kyle")
    $0.it("has a description") {
    try expect(kyle.description) == "Kyle"
    }
    }
    

    View Slide

44. describe("Person") {
    let kyle = Person(name: "Kyle")
    $0.it("has a description") {
    try expect(kyle.description) == "Kyle"
    }
    }
    

    View Slide

45. try expect(person.description) == "Kyle"
    

    View Slide

46. func == (lhs: Expectation, rhs: T?) throws
    

    View Slide

47. Reporters
    

    View Slide

48. View Slide

49. View Slide

50. Custom Reporters
    

    View Slide

51. Testing in
    Playground
    

    View Slide

52. View Slide

53. Demo
    

    View Slide

54. Spectre Runner
    

    View Slide

55. $ ./tests --help
    FLAGS:
    --tap - Test Anywhere Protocol Reporter
    --t - Dot Reporter
    

    View Slide

56. https://github.com/Quick/Quick/
    

    View Slide

57. import Quick
    import Nimble
    class TableOfContentsSpec: QuickSpec {
    override func spec() {
    describe("the 'Documentation' directory") {
    it("has everything you need to get started") {
    let sections = Directory("Documentation").sections
    expect(sections).to(contain("Installing Quick"))
    }
    }
    }
    }
    

    View Slide

58. Cucumber
    

    View Slide

59. Feature: An array
    Scenario: Appending to an array
    Given I have an empty array
    When I add 1 to the array
    Then I should have 1 item in the array
    Scenario: Filtering an array
    Given I have an array with the numbers 1 through 5
    When I filter the array for even numbers
    Then I should have 2 items in the array
    

    View Slide

60. Ploughman
    https://github.com/kylef/Ploughman/
    

    View Slide

61. Given I have an empty array
    var array: [Int]? = nil
    given("^I have an empty array$") { _ in
    array = []
    }
    

    View Slide

62. When I add X to the array
    when("^I add (\\d) to the array$") { match in
    let number = Int(match.groups[1])!
    array.append(number)
    }
    

    View Slide

63. Then I should have X items in the
    array
    then("^I should have (\\d) items? in the array$") { match in
    let count = Int(match.groups[1])!
    try expect(array.count) == count
    }
    

    View Slide

64. $ ./runner scenarios/array.gherkin
    

    View Slide

65. Real World Example
    

    View Slide

66. Feature: Basic Nest compliance
    Scenario: Success response status code
    When I make a GET request to /hello
    Then I should have a 200 response
    Scenario: Response header
    When I make a GET request to /hello
    Then I should see the header 'Content-Type' with the value 'text/plain'
    Scenario: Response body
    When I make a GET request to /hello
    Then The contents of the body should be 'Hello World'
    Scenario: Response body (POST)
    When I make a POST request with body 'Swift' to /hello
    Then The contents of the body should be 'Hello Swift'
    

    View Slide

67. $ NestTestSuite --host http://localhost:8080 Features/*.feature
    -> Basic Nest compliance
    -> Performing a simple GET request
    1 scenarios passed, 0 scenarios failed.
    

    View Slide

68. Build Tools
    

    View Slide

69. Xcode
    

    View Slide

70. DIY
    $ swiftc Sources/*.swift -module-name Hello -emit-library -emit-module -o Hello.dylib
    $ swiftc Tests/*.swift -lHello -o run-tests
    $ ./run-tests
    

    View Slide

71. DIY - Make (Makefile)
    Hello.dylib:
    swiftc Sources/*.swift -module-name Hello -emit-library -emit-module -o Hello.dylib
    run-tests: Hello.dylib
    swiftc Tests/*.swift -lHello -o run-tests
    .PHONY: test
    test: run-tests
    ./run-tests
    

    View Slide

72. $ make test
    

    View Slide

73. Swift Package Manager
    

    View Slide

74. import PackageDescription
    let package = Package(
    name: "Curassow",
    dependencies: [
    .Package(url: "https://github.com/kylef/Commander.git",
    majorVersion: 0, minor: 4),
    ]
    )
    

    View Slide

75. $ swift build
    

    View Slide

76. import PackageDescription
    let package = Package(
    name: "Curassow",
    testDependencies: [
    .Package(url: "https://github.com/kylef/Spectre.git",
    majorVersion: 0, minor: 7),
    ]
    )
    

    View Slide

77. $ swift test
    

    View Slide

78. LinuxMain.swift
    import XCTest
    extension PersonTests {
    static var allTests : [(String, PersonTests -> () throws -> Void)] {
    return [
    ("testCustomStringConvertible", testCustomStringConvertible),
    ]
    }
    }
    XCTMain([
    testCase(PersonTests.allTests),
    ])
    

    View Slide

79. Custom Testing
    with swift test
    

    View Slide

80. !
    swim
    https://github.com/kylef/swim/
    

    View Slide

81. swim Goals
    • swim is compatible with multiple versions of
    Swift (2.x, 3.x, etc)
    • swim allows you to test your Swift project with
    third-party testing frameworks.
    • swim is compatible with common Swift Package
    Manager libraries.
    • swim is easy to install (does not depend on
    Swift)
    

    View Slide

82. $ pip install swim
    

    View Slide

83. Package.swift
    let package = Package(
    name: "Stencil",
    testDependencies: [
    .Package(url: "https://github.com/kylef/Spectre", majorVersion: 0),
    ]
    )
    Terminal
    $ swim build
    $ swim test
    

    View Slide

84. swiftenv
    https://swiftenv.fuller.li/
    

    View Slide

85. $ swiftenv install 2.2.1
    

    View Slide

86. $ swiftenv local DEVELOPMENT-SNAPSHOT-2016-02-08-a
    

    View Slide

87. $ cat .swift-version
    DEVELOPMENT-SNAPSHOT-2016-02-08-a
    

    View Slide

88. $ swift --version
    Apple Swift version 2.2.1
    $ cd Stencil
    $ swift --version
    Apple Swift version 3.0-dev
    

    View Slide

89. swiftenv
    https://swiftenv.fuller.li/
    

    View Slide

90. $ brew install kylef/formulae/swiftenv
    

    View Slide

91. Testing against
    multiple versions
    of Swift
    

    View Slide

92. $ env SWIFT_VERSION=2.2 swift test --clean
    $ env SWIFT_VERSION=2.3 swift test --clean
    $ env SWIFT_VERSION=3.0 swift test --clean
    

    View Slide

93. Swift on Travis CI
    https://swiftenv.fuller.li/en/latest/
    integrations/travis-ci.html
    

    View Slide

94. os:
    - linux
    - osx
    install:
    - eval "$(curl -sL https://github.com/kylef/.../swiftenv-install.sh)"
    

    View Slide

95. script: swift test
    

    View Slide

96. env:
    - SWIFT_VERSION=2.2
    - SWIFT_VERSION=2.2.1
    

    View Slide

97. • Early stages for testing OSS Swift
    • New tools are emerging
    • Swift Package Manager is evolving
    • Test against all platforms
    

    View Slide

98. kylefuller
    https://fuller.li/talks
    

    View Slide