iOS Testing mit XCTest & Co.- Mobile Testing Days 2015

Talk gehalten auf den Mobile Testing Days in Berlin am 28. Mai 2015.

Transcript

1. Thimo Bess | arconsis IT-Solutions GmbH iOS Testing mit XCTest & Co.
2. zu meiner Person • Thimo Bess • Software-Engineer bei arconsis IT-Solutions GmbH in Karlsruhe • iOS-Entwicklung • Webanwendungen
3. Inhalt • Tests/Warum? • XCTests • OCMock • OCHamcrest • OHHTTPStubs • Kiwi • Xcode Bots • Code Coverage
4. Tests? Warum? • zu aufwändig • hohe Kosten • wenig Nutzen • oder nicht?
5. Tests! - Darum. • Weniger Bugs • Bessere Dokumentation • Modulares Design • Geringere Kosten bei Erweiterungen • Ermöglichen Refactoring • Lassen uns besser schlafen
6. XCTests
7. XCTestCase Lifecycle @interface MyTestCase : XCTestCase @end @implementation MyTestCase + (void)setUp {} // beforeAll - (void)setUp {} // beforeEach - (void)test* {} + (void)tearDown {} // afterAll - (void)tearDown {} // afterEach @end
8. XCTestCase Lifecycle @interface MyTestCase : XCTestCase @end @implementation MyTestCase + (void)setUp {} // beforeAll - (void)setUp {} // beforeEach - (void)test* {} + (void)tearDown {} // afterAll - (void)tearDown {} // afterEach @end
9. XCTestCase Lifecycle @interface MyTestCase : XCTestCase @end @implementation MyTestCase + (void)setUp {} // beforeAll - (void)setUp {} // beforeEach - (void)test* {} + (void)tearDown {} // afterAll - (void)tearDown {} // afterEach @end
10. XCTestCase Lifecycle @interface MyTestCase : XCTestCase @end @implementation MyTestCase + (void)setUp {} // beforeAll - (void)setUp {} // beforeEach - (void)test* {} + (void)tearDown {} // afterAll - (void)tearDown {} // afterEach @end
11. XCTestCase Lifecycle @interface MyTestCase : XCTestCase @end @implementation MyTestCase + (void)setUp {} // beforeAll - (void)setUp {} // beforeEach - (void)test* {} + (void)tearDown {} // afterAll - (void)tearDown {} // afterEach @end
12. XCTestCase Lifecycle @interface MyTestCase : XCTestCase @end @implementation MyTestCase + (void)setUp {} // beforeAll - (void)setUp {} // beforeEach - (void)test {} + (void)tearDown {} // afterAll - (void)tearDown {} // afterEach @end
13. XCTestCase Example - (void) testLoginWithUserAndPassword_shouldReturnValidUser { // Given: Arrange all the things needed to exercise // the code // When: Act on the code you want to verify // Then: Assert that the code worked as expected }
14. XCTestCase Example - (void) testLoginWithUserAndPassword_shouldReturnValidUser { // Given: Arrange all the things needed to exercise the code UserManager *sut = [UserManager new]; NSString *username = @"Peter"; NSString *password = @"secret"; // When: Act on the code you want to verify User *user = [sut loginUser:username withPassword:password]; // Then: Assert that the code worked as expected XCTAssertNotNil(user, "User should not be nil"); XCTAssertEqualObjects(@"Peter", user.username, @"Expected username"); XCTAssertNotNil(user.sessionId, "SessionId should not be nil"); XCTAssertTrue(user.sessionId.length >= 5); }
15. XCTestCase Example - (void) testLoginWithUserAndPassword_shouldReturnValidUser { // Given: Arrange all the things needed to exercise the code UserManager *sut = [UserManager new]; NSString *username = @"Peter"; NSString *password = @"secret"; // When: Act on the code you want to verify User *user = [sut loginUser:username withPassword:password]; // Then: Assert that the code worked as expected XCTAssertNotNil(user, "User should not be nil"); XCTAssertEqualObjects(@"Peter", user.username, @"Expected username"); XCTAssertNotNil(user.sessionId, "SessionId should not be nil"); XCTAssertTrue(user.sessionId.length >= 5); }
16. XCTestCase Example - (void) testLoginWithUserAndPassword_shouldReturnValidUser { // Given: Arrange all the things needed to exercise the code UserManager *sut = [UserManager new]; NSString *username = @"Peter"; NSString *password = @"secret"; // When: Act on the code you want to verify User *user = [sut loginUser:username withPassword:password]; // Then: Assert that the code worked as expected XCTAssertNotNil(user, "User should not be nil"); XCTAssertEqualObjects(@"Peter", user.username, @"Expected username"); XCTAssertNotNil(user.sessionId, "SessionId should not be nil"); XCTAssertTrue(user.sessionId.length >= 5); }
17. XCTestCase Example - (void) testLoginWithUserAndPassword_shouldReturnValidUser { // Given: Arrange all the things needed to exercise the code UserManager *sut = [UserManager new]; NSString *username = @"Peter"; NSString *password = @"secret"; // When: Act on the code you want to verify User *user = [sut loginUser:username withPassword:password]; // Then: Assert that the code worked as expected XCTAssertNotNil(user, "User should not be nil"); XCTAssertEqualObjects(@"Peter", user.username, @"Expected username"); XCTAssertNotNil(user.sessionId, "SessionId should not be nil"); XCTAssertTrue(user.sessionId.length >= 5); }
18. XCTestCase Example - (void) testLoginWithUserAndPassword_shouldReturnValidUser { // Given: Arrange all the things needed to exercise the code UserManager *sut = [UserManager new]; NSString *username = @"Peter"; NSString *password = @"secret"; // When: Act on the code you want to verify User *user = [sut loginUser:username withPassword:password]; // Then: Assert that the code worked as expected XCTAssertNotNil(user, "User should not be nil"); XCTAssertEqualObjects(@"Peter", user.username, @"Expected username"); XCTAssertNotNil(user.sessionId, "SessionId should not be nil"); XCTAssertTrue(user.sessionId.length >= 5); }
19. XCTestCase Example - (void) testLoginWithUserAndPassword_shouldReturnValidUser { // Given: Arrange all the things needed to exercise the code UserManager *sut = [UserManager new]; NSString *username = @"Peter"; NSString *password = @"secret"; // When: Act on the code you want to verify User *user = [sut loginUser:username withPassword:password]; // Then: Assert that the code worked as expected XCTAssertNotNil(user, "User should not be nil"); XCTAssertEqualObjects(@"Peter", user.username, @"Expected username"); XCTAssertNotNil(user.sessionId, "SessionId should not be nil"); XCTAssertTrue(user.sessionId.length >= 5); }
20. XCTAssert (1) objects and values XCTFail(format..)
 XCTAssertNil(a1, format..)
 XCTAssertNotNil(a1, format..)
 XCTAssert(a1, format..)
 XCTAssertTrue(a1, format..)
 XCTAssertFalse(a1, format..)
 XCTAssertEqualObjects(a1, a2, format..)
 XCTAssertEquals(a1, a2, format..)
 XCTAssertEqualsWithAccuracy(a1, a2, accuracy, format..)

21. XCTAssert (2) Exceptions XCTAssertThrows(expression, format..)
 XCTAssertThrowsSpecific(expression, specificException, format..) XCTAssertThrowsSpecificNamed(expression, specificException, exceptionName, format..) 
 XCTAssertNoThrow(expression, format..) XCTAssertNoThrowSpecific(expression, specificException, format..) XCTAssertNoThrowSpecificNamed(expression, specificException, exceptionName, format..)
22. XCTestCase: Exceptions - (void)testLoginWithNullCredentials_ShouldThrowException { // Default SUT already configured in setUp XCTAssertThrowsSpecificNamed([self.sut loginUser:nil withPassword:nil], NSException, NSInvalidArgumentException, @"should throw NSInvalidArgumentException"); }
23. Vorteile XCTest • Guter Xcode Support • Ausführung einzelner Tests möglich
24. Nachteile XCTest • Fehlermeldungen oft wenig hilfreich (z.B. XCAssertTrue, XCAssertFalse) • Assertions selbst schlecht lesbar
25. Grenzen von XCTest • Matcher • Mocks/Stubs • Network Requests
26. OCHamcrest pod 'OCHamcrest'
27. OCHamcrest • iOS & Mac OS X • Matcher Library
28. Matcher (Auszug) • Object • equalTo • notNullValue • sameInstance • Collections • hasEntry, hasKey, hasValue
29. Beispiel - (void) testEquals { Biscuit* theBiscuit = [[Biscuit alloc] initWithName:@"Ginger"]; Biscuit* myBiscuit = [[Biscuit alloc] initWithName:@"Ginger"]; assertThat(theBiscuit, equalTo(myBiscuit)); assertThat(theBiscuit, is(equalTo(myBiscuit))); }
30. Beispiel - (void) testEquals { Biscuit* theBiscuit = [[Biscuit alloc] initWithName:@"Ginger"]; Biscuit* myBiscuit = [[Biscuit alloc] initWithName:@"Ginger"]; assertThat(theBiscuit, equalTo(myBiscuit)); assertThat(theBiscuit, is(equalTo(myBiscuit))); }
31. OCMock pod 'OCMock'
32. Mock • „to mock“ = „etwas vortäuschen“ • sinnvoll wenn, das „echte“ Objekt • noch nicht existiert • nicht deterministische Ergebnisse liefert • nicht oder schwer rückgängig zu machende Prozesse anstößt
33. Mock #import <OCMock/OCMock.h> - (void)testMockedUser { User *user = [self createMockUserWithName:@"Peter" andSessionId:@"secret"]; OCMVerifyAll(user); XCTAssertEqualObjects(@"Peter", user.username, @"Expected username"); XCTAssertEqualObjects(@"secret", user.sessionId, @"Expected sessionId"); }
34. Mock - (User *)createMockUserWithName:(NSString *)username andSessionId:(NSString *)sessionId { id userMock = OCMClassMock([User class]); OCMExpect([userMock username]).andReturn(username); OCMExpect([userMock sessionId]).andReturn(sessionId); return userMock; }
35. Mock - (UserAPI *)createMockUserAPIForMockUser:(User *)user { id apiMock = OCMClassMock([UserAPI class]); OCMExpect([apiMock authenticate:OCMOCK_ANY withPassword:OCMOCK_ANY).andReturn(user); return apiMock; }
36. Mock - (void)testLoginWithUserAndPassword_shouldReturnValidUser { // Given: Arrange all the things needed to exercise the code User *mockUser = [self createMockUserWithName:@"Peter" andSessionId:@"123456789"]; UserAPI *mockApi = [self createMockUserAPIForMockUser:mockUser]; self.sut = [[UserManager alloc] initWithApi:mockApi]; // When: Act on the code you want to verify User *user = [self.sut loginUser:@"Peter" withPassword:@"123456789"]; // Then: Assert that the code worked as expected OCMVerifyAll(user); XCTAssertEqualObjects(@"Peter", user.username, @"Expected username"); XCTAssertEqualObjects(@"123456789", user.sessionId, @"Expected sessionId"); }
37. Mock - (void)testLoginWithUserAndPassword_shouldReturnValidUser { // Given: Arrange all the things needed to exercise the code User *mockUser = [self createMockUserWithName:@"Peter" andSessionId:@"123456789"]; UserAPI *mockApi = [self createMockUserAPIForMockUser:mockUser]; self.sut = [[UserManager alloc] initWithApi:mockApi]; // When: Act on the code you want to verify User *user = [self.sut loginUser:@"Peter" withPassword:@"123456789"]; // Then: Assert that the code worked as expected OCMVerifyAll(user); XCTAssertEqualObjects(@"Peter", user.username, @"Expected username"); XCTAssertEqualObjects(@"123456789", user.sessionId, @"Expected sessionId"); }
38. Mock - (void)testLoginWithUserAndPassword_shouldReturnValidUser { // Given: Arrange all the things needed to exercise the code User *mockUser = [self createMockUserWithName:@"Peter" andSessionId:@"123456789"]; UserAPI *mockApi = [self createMockUserAPIForMockUser:mockUser]; self.sut = [[UserManager alloc] initWithApi:mockApi]; // When: Act on the code you want to verify User *user = [self.sut loginUser:@"Peter" withPassword:@"123456789"]; // Then: Assert that the code worked as expected OCMVerifyAll(user); XCTAssertEqualObjects(@"Peter", user.username, @"Expected username"); XCTAssertEqualObjects(@"123456789", user.sessionId, @"Expected sessionId"); }
39. Mock - (void)testLoginWithUserAndPassword_shouldReturnValidUser { // Given: Arrange all the things needed to exercise the code User *mockUser = [self createMockUserWithName:@"Peter" andSessionId:@"123456789"]; UserAPI *mockApi = [self createMockUserAPIForMockUser:mockUser]; self.sut = [[UserManager alloc] initWithApi:mockApi]; // When: Act on the code you want to verify User *user = [self.sut loginUser:@"Peter" withPassword:@"123456789"]; // Then: Assert that the code worked as expected OCMVerifyAll(mockUser); OCMVerifyAll(mockApi); XCTAssertEqualObjects(@"Peter", user.username, @"Expected username"); XCTAssertEqualObjects(@"123456789", user.sessionId, @"Expected sessionId"); }
40. Mock async callbacks • Mit Hilfe von NSInvocation • NSInvocation beinhaltet • Klasse • Funktion • Parameter …
41. Beispiel - (void)testAsyncMock { UserAPI *api = [self createAsyncMockUserAPIForMockUser:[User new]]; [api authenticate:@"Peter" withPassword:@"Paul" onSuccess:^(User *user) { XCTAssertNotNil(user); } onFailure:^(NSError *error) { XCTFail("Should not be called!"); }]; }
42. Mock async callbacks - (UserAPI *)createAsyncMockUserAPIForMockUser:(User *)user { id apiMock = OCMClassMock([UserAPI class]); OCMExpect([apiMock authenticate:OCMOCK_ANY withPassword:OCMOCK_ANY onSuccess:OCMOCK_ANY onFailure:OCMOCK_ANY]).andDo(^(NSInvocation *invocation) { void(^successBlock)(User *) = nil; // index 0 and 1 reserved for invocation object => index 4 [invocation getArgument:&successBlock atIndex:4]; successBlock(user); }); return apiMock; }
43. Testen mit CoreData
44. Testen mit CoreData • In-Memory Datenbank für die Tests benutzen • PersistentStoreType: NSInMemoryStoreType • wird nach jedem Test zurückgesetzt
45. Testen mit CoreData [_persistentStoreCoordinator addPersistentStoreWithType:NSInMemoryStoreType configuration:nil URL:storeURL options:options error:&error]
46. Testen mit CoreData + (void)setUp { // Create db dedicated to test class [MagicalRecord setupCoreDataStackWithInMemoryStore]; } +(void)tearDown { [MagicalRecord cleanUp]; } - (void)setUp { // Cleanup managed object context before each test [[NSManagedObjectContext MR_defaultContext] reset]; }
47. Testen von asynchronen Methoden
48. Beispiel - (void)testLoginWithWrongCredentials_ShouldReturnFailure { sut.api = [self createMockApiWithInvalidUser]; [self.sut loginUser:@"Paul" withPassword:@"secret" onSuccess:^(User *user) { XCTFail(@"Should not be called!"); } onFailure:^(NSError *e) { XCTAssertTrue(e.code == 401); // Unauthorized }]; }
49. Sehen Sie das Problem?
50. Problem • Testmethode nicht asynchron • unter Umständen beendet vor dem Callback der asynchronen Methode
51. Lösung • warten auf den Callback der asynchronen Methode
52. Beispiel - (void)testLoginWithWrongCredentials_ShouldReturnFailure { sut.api = [self createMockApiWithInvalidUser]; XCTestExpectation *expectation = [self expectationWithDescription:@"methodCalled"]; [self.sut loginUser:@"Paul" withPassword:@"secret" onSuccess:^(User *user) { XCTFail(@"Should not be called!"); } onFailure:^(NSError *e) { XCTAssertTrue(e.code == 401); // Unauthenticated [expectation fulfill]; }]; [self waitForExpectationsWithTimeout:5 handler:nil]; }
53. Beispiel - (void)testLoginWithWrongCredentials_ShouldReturnFailure { sut.api = [self createMockApiWithInvalidUser]; XCTestExpectation *expectation = [self expectationWithDescription:@"methodCalled"]; [self.sut loginUser:@"Paul" withPassword:@"secret" onSuccess:^(User *user) { XCTFail(@"Should not be called!"); } onFailure:^(NSError *e) { XCTAssertTrue(e.code == 401); // Unauthenticated [expectation fulfill]; }]; [self waitForExpectationsWithTimeout:5 handler:nil]; }
54. Beispiel - (void)testLoginWithWrongCredentials_ShouldReturnFailure { sut.api = [self createMockApiWithInvalidUser]; XCTestExpectation *expectation = [self expectationWithDescription:@"methodCalled"]; [self.sut loginUser:@"Paul" withPassword:@"secret" onSuccess:^(User *user) { XCTFail(@"Should not be called!"); } onFailure:^(NSError *e) { XCTAssertTrue(e.code == 401); // Unauthenticated [expectation fulfill]; }]; [self waitForExpectationsWithTimeout:5 handler:nil]; }
55. Beispiel - (void)testLoginWithWrongCredentials_ShouldReturnFailure { sut.api = [self createMockApiWithInvalidUser]; XCTestExpectation *expectation = [self expectationWithDescription:@"methodCalled"]; [self.sut loginUser:@"Paul" withPassword:@"secret" onSuccess:^(User *user) { XCTFail(@"Should not be called!"); } onFailure:^(NSError *e) { XCTAssertTrue(e.code == 401); // Unauthenticated [expectation fulfill]; }]; [self waitForExpectationsWithTimeout:5 handler:nil]; }
56. Testen von Notifications
57. Testen von Notifications - (void)testMockNotifications { id mock = [OCMockObject observerMock]; // OCMock adds a custom methods to NSNotificationCenter via a category [[NSNotificationCenter defaultCenter] addMockObserver:mock name:TDDAppExampleNotification object:nil]; [[mock expect] notificationWithName:TDDAppExampleNotification object:[OCMArg any]]; ServiceMgr *sm = [self createServiceMgr]; // made-up factory method [sm callFancyBackendAPIAndSendNotification:@"Penny? Penny? Penny?"]; [mock verify]; }
58. Testen von Notifications - (void)testMockNotifications { id mock = [OCMockObject observerMock]; // OCMock adds a custom methods to NSNotificationCenter via a category [[NSNotificationCenter defaultCenter] addMockObserver:mock name:TDDAppExampleNotification object:nil]; [[mock expect] notificationWithName:TDDAppExampleNotification object:[OCMArg any]]; ServiceMgr *sm = [self createServiceMgr]; // made-up factory method [sm callFancyBackendAPIAndSendNotification:@"Penny? Penny? Penny?"]; [mock verify]; }
59. Testen von Notifications - (void)testMockNotifications { id mock = [OCMockObject observerMock]; // OCMock adds a custom methods to NSNotificationCenter via a category [[NSNotificationCenter defaultCenter] addMockObserver:mock name:TDDAppExampleNotification object:nil]; [[mock expect] notificationWithName:TDDAppExampleNotification object:[OCMArg any]]; ServiceMgr *sm = [self createServiceMgr]; // made-up factory method [sm callFancyBackendAPIAndSendNotification:@"Penny? Penny? Penny?"]; [mock verify]; }
60. Testen von Notifications - (void)testMockNotifications { id mock = [OCMockObject observerMock]; // OCMock adds a custom methods to NSNotificationCenter via a category [[NSNotificationCenter defaultCenter] addMockObserver:mock name:TDDAppExampleNotification object:nil]; [[mock expect] notificationWithName:TDDAppExampleNotification object:[OCMArg any]]; ServiceMgr *sm = [self createServiceMgr]; // made-up factory method [sm callFancyBackendAPIAndSendNotification:@"Penny? Penny? Penny?"]; [mock verify]; }
61. OHHTTPStubs pod 'OHHTTPStubs'
62. Stub • „stub“ = Stummel, Stumpf • ersetzt Programmcode, • der noch nicht entwickelt ist oder • auf einem entfernten System läuft
63. OHHTTPStubs • Netzwerk stubs • Entkopplung unterhalb des Clients • Tests ohne Abhängigkeiten zu externen Systemen • Reproduzierbare Ergebnisse
64. Beispiel [OHHTTPStubs stubRequestsPassingTest:^BOOL(NSURLRequest *request) { return YES; } withStubResponse:^OHHTTPStubsResponse *(NSURLRequest *request) { NSString *path = OHPathForFileInBundle(@"stub_response.json", nil); NSDictionary *headers = @{@“Content-Type" : @“application/json”}; return [OHHTTPStubsResponse responseWithFileAtPath:path statusCode:200 headers:headers]; }];
65. Kiwi pod 'Kiwi'
66. Kiwi • BDD library = Behaviour Driven Development • Tests sollen lesbarer werden
67. Specs (1) https://github.com/allending/Kiwi/wiki/Specs #import "Kiwi.h" SPEC_BEGIN(XmlHelperTestsSpec) describe(@"XmlHelperTests", ^{ context(@"string parsing", ^{ it(@"is non nil", ^{ [[variable shouldNot] beNil]; }); pending(@"is non nil", ^{ }); // todo }); }); SPEC_END
68. Specs (2) setUp and tearDown beforeAll(^{ }); beforeEach(^{ }); afterAll(^{ }); afterEach(^{ });
69. Expectations https://github.com/allending/Kiwi/wiki/Expectations // Examples: id car = [Car car]; [[car shouldNot] beNil]; [[car should] beKindOfClass:[Car class]]; [[car shouldNot] conformToProtocol:@protocol(FlyingMachine)]; [[[car should] have:4] wheels]; [[theValue(car.speed) should] equal:theValue(42.0f)]; [[[car should] receive] changeToGear:3];
70. Expectations https://github.com/allending/Kiwi/wiki/Expectations // Scalar values: [[theValue(1 + 1) should] equal:theValue(2)]; [[theValue(YES) shouldNot] equal:theValue(NO)]; [[theValue(20) should] beBetween:theValue(1) and:theValue(30.0)]; [[theValue(animal.mass) should]beGreaterThan:theValue(42.0f)]
71. Vorteile Kiwi • Test ist sehr gut lesbar
72. Nachteile Kiwi • Ausführung einzelner Tests nicht möglich
73. Xcode Bots
74. Xcode Bots • Automatisierte Tests • Statische Code Analyse • Distribution
75. Xcode Bots • Pro • Integration in Xcode • Einfache Konfiguration • Tests auf verschiedenen Simulatoren und Devices • Detaillierte Übersichtsseite
76. Xcode Bots • Contra • Schwer erweiterbar • Begrenzte Anpassungsmöglichkeiten
77. Xcode Bots
78. Xcode Bots
79. Xcode Bots
80. Code Coverage
81. Code Coverage • Report • gcov • Präsentation • Xcoverage • CoverStory • lcov
82. Code Coverage • Build Settings • GCC_GENERATE_TEST_COVERAGE_FILES = Yes • GCC_INSTRUMENT_PROGRAM_FLOW_ARCS = Yes • Erzeugt gcda und gcno Dateien im Derived Data Ordner
83. Xcoverage
84. XcodeCoverage (lcov) • https://github.com/jonreid/XcodeCoverage • Generierung von HTML Reports • Einfache Integration in CI Prozess
85. XcodeCoverage (lcov) 1. Scriptsammlung in Projekt kopieren 2. Run Script Build Phase hinzufügen
 XcodeCoverage/exportenv.sh 3. getcov ausführen —> HTML export
86. XcodeCoverage (lcov)
87. Recap • Tests/Warum? • XCTests • OCMock • OCHamcrest • OHHTTPStubs • Kiwi • Xcode Bots • Code Coverage
88. Quellen • OCMock (http://ocmock.org) • OCHamcrest (https://github.com/hamcrest/ OCHamcrest) • OHHTTPStubs (https://github.com/AliSoftware/ OHHTTPStubs) • Kiwi (https://github.com/kiwi-bdd/Kiwi/wiki/Getting- Started-with-Kiwi-2.0) • Xcoverage (http://www.xcoverageapp.com) • LCOV (http://ltp.sourceforge.net/coverage/lcov.php)
89. Danke thimo.bess@arconsis.com @drag0ndust