Upgrade to Pro — share decks privately, control downloads, hide ads and more …

'Testing iOS app with external hardware' by Viacheslav Volodko

'Testing iOS app with external hardware' by Viacheslav Volodko

This talk was made for CocoaHeads Kyiv #13 which took place Dec 16 2017.

CocoaHeads Ukraine

December 16, 2017
Tweet

More Decks by CocoaHeads Ukraine

Other Decks in Programming

Transcript

  1. TESTING IOS APP WITH EXTERNAL HARDWARE THE PROBLEM Bluetooth LE

    WiFi Cable Connection Bluetooth Classic
  2. TESTING IOS APP WITH EXTERNAL HARDWARE THE PROBLEM Bluetooth LE

    WiFi Cable Connection Bluetooth Classic
  3. TESTING IOS APP WITH EXTERNAL HARDWARE THE PROBLEM Bluetooth LE

    WiFi Cable Connection Bluetooth Classic
  4. TESTING IOS APP WITH EXTERNAL HARDWARE THE PROBLEM Bluetooth LE

    WiFi Cable Connection Bluetooth Classic JSON/XML
  5. TESTING IOS APP WITH EXTERNAL HARDWARE THE PROBLEM Bluetooth LE

    WiFi Cable Connection Bluetooth Classic JSON/XML HTTP
  6. TESTING IOS APP WITH EXTERNAL HARDWARE THE PROBLEM Bluetooth LE

    WiFi Cable Connection Bluetooth Classic JSON/XML HTTP TCP/IP
  7. TESTING IOS APP WITH EXTERNAL HARDWARE THE PROBLEM JSON/XML HTTP

    TCP/IP Packetization Bluetooth LE WiFi Cable Connection Bluetooth Classic
  8. TESTING IOS APP WITH EXTERNAL HARDWARE THE PROBLEM Backend https

    v.1.0.0 v.1.1.0 v.2.0.0 v.2.1.0 RESPONSIBLE FOR VERSIONING
  9. TESTING IOS APP WITH EXTERNAL HARDWARE THE PROBLEM Bluetooth LE

    WiFi Cable Connection Bluetooth Classic
  10. TESTING IOS APP WITH EXTERNAL HARDWARE THE PROBLEM Bluetooth LE

    WiFi Cable Connection Bluetooth Classic v.1.0.0 v.1.1.0 v.2.0.0 v.2.1.0 Firmware
  11. TESTING IOS APP WITH EXTERNAL HARDWARE THE PROBLEM Bluetooth LE

    WiFi Cable Connection Bluetooth Classic v.1.0.0 v.1.1.0 v.2.0.0 v.2.1.0 Firmware RESPONSIBLE FOR VERSIONING
  12. TESTING IOS APP WITH EXTERNAL HARDWARE THE PROBLEM Bluetooth LE

    WiFi Cable Connection Bluetooth Classic RESPONSIBLE FOR VERSIONING
  13. TESTING IOS APP WITH EXTERNAL HARDWARE THE PROBLEM Bluetooth LE

    WiFi Cable Connection Bluetooth Classic
  14. TESTING IOS APP WITH EXTERNAL HARDWARE THE PROBLEM Bluetooth LE

    WiFi Cable Connection Bluetooth Classic
  15. TESTING IOS APP WITH EXTERNAL HARDWARE RESOLUTION: ARCHITECTURE Connectivity Layer

    Bluetooth LE WiFi Cable Connection Bluetooth Classic • System layer, e.g. Core Bluetooth
  16. TESTING IOS APP WITH EXTERNAL HARDWARE RESOLUTION: ARCHITECTURE Connectivity Layer

    Bluetooth LE WiFi Cable Connection Bluetooth Classic Transport Layer • StateLESS layer • Request/Response • Device specific components • System layer, e.g. Core Bluetooth
  17. TESTING IOS APP WITH EXTERNAL HARDWARE RESOLUTION: ARCHITECTURE Connectivity Layer

    Bluetooth LE WiFi Cable Connection Bluetooth Classic Transport Layer • StateLESS layer • Request/Response • Device specific components • System layer, e.g. Core Bluetooth Business Logic Layer • StateFULL layer • Device agnostic components
  18. TESTING IOS APP WITH EXTERNAL HARDWARE RESOLUTION: ARCHITECTURE Connectivity Layer

    Bluetooth LE WiFi Cable Connection Bluetooth Classic Connectivity Layer Abstraction Transport Layer • StateLESS layer • Request/Response • Device specific components • System layer, e.g. Core Bluetooth Business Logic Layer • StateFULL layer • Device agnostic components
  19. TESTING IOS APP WITH EXTERNAL HARDWARE UNIT TESTING Connectivity Layer

    Connectivity Layer Abstraction Transport Layer • StateLESS layer • Request/Response • Device specific components • System layer, e.g. Core Bluetooth Business Logic Layer • StateFULL layer • Device agnostic components
  20. TESTING IOS APP WITH EXTERNAL HARDWARE UNIT TESTING Connectivity Layer

    Abstraction Transport Layer • StateLESS layer • Request/Response • Device specific components Business Logic Layer • StateFULL layer • Device agnostic components Mock Connectivity Layer
  21. TESTING IOS APP WITH EXTERNAL HARDWARE UNIT TESTING Connectivity Layer

    Abstraction Transport Layer • StateLESS layer • Request/Response • Device specific components Mock Connectivity Layer Unit Tests Cases
  22. Unit Tests Cases Mock Connectivity Layer TESTING IOS APP WITH

    EXTERNAL HARDWARE UNIT TESTING Connectivity Layer Abstraction Transport Layer • StateLESS layer • Request/Response • Device specific components
  23. Unit Tests Cases Mock Connectivity Layer TESTING IOS APP WITH

    EXTERNAL HARDWARE UNIT TESTING Connectivity Layer Abstraction Transport Layer • StateLESS layer • Request/Response • Device specific components Business Logic Layer • StateFULL layer • Device agnostic components
  24. Unit Tests Cases TESTING IOS APP WITH EXTERNAL HARDWARE UNIT

    TESTING Business Logic Layer • StateFULL layer • Device agnostic components Mock Transport Layer
  25. TESTING IOS APP WITH EXTERNAL HARDWARE UNIT TESTING ▸ Easy

    to cover edge cases ▸ Increases code modularity PROS CONS
  26. TESTING IOS APP WITH EXTERNAL HARDWARE UNIT TESTING ▸ Easy

    to cover edge cases ▸ Increases code modularity ▸ Decreases debugging time PROS CONS
  27. TESTING IOS APP WITH EXTERNAL HARDWARE UNIT TESTING ▸ Easy

    to cover edge cases ▸ Increases code modularity ▸ Decreases debugging time ▸ Easily automated (CI) PROS CONS
  28. TESTING IOS APP WITH EXTERNAL HARDWARE UNIT TESTING ▸ Easy

    to cover edge cases ▸ Increases code modularity ▸ Decreases debugging time ▸ Easily automated (CI) PROS ▸ Time consuming CONS
  29. TESTING IOS APP WITH EXTERNAL HARDWARE UNIT TESTING ▸ Easy

    to cover edge cases ▸ Increases code modularity ▸ Decreases debugging time ▸ Easily automated (CI) PROS ▸ Time consuming ▸ Increases code complexity CONS
  30. TESTING IOS APP WITH EXTERNAL HARDWARE UNIT TESTING ▸ Easy

    to cover edge cases ▸ Increases code modularity ▸ Decreases debugging time ▸ Easily automated (CI) PROS ▸ Time consuming ▸ Increases code complexity ▸ No real device in chain CONS
  31. TESTING IOS APP WITH EXTERNAL HARDWARE INTEGRATION TESTING WITH REAL

    DEVICE Connectivity Layer Connectivity Layer Abstraction Transport Layer • StateLESS layer • Request/Response • Device specific components • System layer, e.g. Core Bluetooth Business Logic Layer • StateFULL layer • Device agnostic components
  32. TESTING IOS APP WITH EXTERNAL HARDWARE INTEGRATION TESTING WITH REAL

    DEVICE Connectivity Layer Connectivity Layer Abstraction Transport Layer • StateLESS layer • Request/Response • Device specific components • System layer, e.g. Core Bluetooth Business Logic Layer • StateFULL layer • Device agnostic components Integration Tests
  33. TESTING IOS APP WITH EXTERNAL HARDWARE INTEGRATION TESTING WITH REAL

    DEVICE Connectivity Layer Connectivity Layer Abstraction Transport Layer • StateLESS layer • Request/Response • Device specific components • System layer, e.g. Core Bluetooth Integration Tests
  34. TESTING IOS APP WITH EXTERNAL HARDWARE INTEGRATION TESTING WITH REAL

    DEVICE ▸ Guarantee to work with Real device PROS CONS
  35. TESTING IOS APP WITH EXTERNAL HARDWARE INTEGRATION TESTING WITH REAL

    DEVICE ▸ Guarantee to work with Real device ▸ Nothing to mock: 
 faster than Unit Tests PROS CONS
  36. TESTING IOS APP WITH EXTERNAL HARDWARE INTEGRATION TESTING WITH REAL

    DEVICE ▸ Guarantee to work with Real device ▸ Nothing to mock: 
 faster than Unit Tests PROS ▸ Hard to cover corner cases CONS
  37. TESTING IOS APP WITH EXTERNAL HARDWARE INTEGRATION TESTING WITH REAL

    DEVICE ▸ Guarantee to work with Real device ▸ Nothing to mock: 
 faster than Unit Tests PROS ▸ Hard to cover corner cases ▸ Not always stable CONS
  38. TESTING IOS APP WITH EXTERNAL HARDWARE INTEGRATION TESTING WITH REAL

    DEVICE ▸ Guarantee to work with Real device ▸ Nothing to mock: 
 faster than Unit Tests PROS ▸ Hard to cover corner cases ▸ Not always stable ▸ Hard to automate (CI) CONS
  39. TESTING IOS APP WITH EXTERNAL HARDWARE INTEGRATION TESTING WITH REAL

    DEVICE ▸ Guarantee to work with Real device ▸ Nothing to mock: 
 faster than Unit Tests PROS ▸ Hard to cover corner cases ▸ Not always stable ▸ Hard to automate (CI) ▸ Run takes more time CONS
  40. TESTING IOS APP WITH EXTERNAL HARDWARE INTEGRATION TESTING WITH DEVICE

    SIMULATOR Connectivity Layer Connectivity Layer Abstraction Transport Layer • StateLESS layer • Request/Response • Device specific components • System layer, e.g. Core Bluetooth Integration Tests
  41. TESTING IOS APP WITH EXTERNAL HARDWARE INTEGRATION TESTING WITH DEVICE

    SIMULATOR Connectivity Layer Connectivity Layer Abstraction Transport Layer • StateLESS layer • Request/Response • Device specific components • System layer, e.g. Core Bluetooth Integration Tests Simulator
  42. Raspberry Pi + Python app TESTING IOS APP WITH EXTERNAL

    HARDWARE INTEGRATION TESTING WITH DEVICE SIMULATOR Connectivity Layer Connectivity Layer Abstraction Transport Layer • StateLESS layer • Request/Response • Device specific components • System layer, e.g. Core Bluetooth Integration Tests Simulator OSX app - BLE Peripheral Raspberry Pi + NodeJS app Arduino
  43. TESTING IOS APP WITH EXTERNAL HARDWARE INTEGRATION TESTING WITH DEVICE

    SIMULATOR ▸ Limited guarantee to work with Real device PROS CONS
  44. TESTING IOS APP WITH EXTERNAL HARDWARE INTEGRATION TESTING WITH DEVICE

    SIMULATOR ▸ Limited guarantee to work with Real device ▸ No device prototype needed PROS CONS
  45. TESTING IOS APP WITH EXTERNAL HARDWARE INTEGRATION TESTING WITH DEVICE

    SIMULATOR ▸ Limited guarantee to work with Real device ▸ No device prototype needed ▸ Easy to cover edge cases PROS CONS
  46. TESTING IOS APP WITH EXTERNAL HARDWARE INTEGRATION TESTING WITH DEVICE

    SIMULATOR ▸ Limited guarantee to work with Real device ▸ No device prototype needed ▸ Easy to cover edge cases PROS ▸ Need to follow specs very carefully CONS
  47. TESTING IOS APP WITH EXTERNAL HARDWARE INTEGRATION TESTING WITH DEVICE

    SIMULATOR ▸ Limited guarantee to work with Real device ▸ No device prototype needed ▸ Easy to cover edge cases PROS ▸ Need to follow specs very carefully ▸ An additional codebase to support CONS
  48. TESTING IOS APP WITH EXTERNAL HARDWARE INTEGRATION TESTING WITH DEVICE

    SIMULATOR ▸ Limited guarantee to work with Real device ▸ No device prototype needed ▸ Easy to cover edge cases PROS ▸ Need to follow specs very carefully ▸ An additional codebase to support ▸ Codebase grows with new Firmware/ Devices CONS
  49. TESTING IOS APP WITH EXTERNAL HARDWARE INTEGRATION TESTING WITHIN TEST

    HARNESS Connectivity Layer Connectivity Layer Abstraction Transport Layer • StateLESS layer • Request/Response • Device specific components • System layer, e.g. Core Bluetooth Integration Tests
  50. TESTING IOS APP WITH EXTERNAL HARDWARE INTEGRATION TESTING WITHIN TEST

    HARNESS Connectivity Layer Connectivity Layer Abstraction Transport Layer • StateLESS layer • Request/Response • Device specific components • System layer, e.g. Core Bluetooth Integration Tests
  51. TESTING IOS APP WITH EXTERNAL HARDWARE INTEGRATION TESTING WITHIN TEST

    HARNESS Connectivity Layer Connectivity Layer Abstraction Transport Layer • StateLESS layer • Request/Response • Device specific components • System layer, e.g. Core Bluetooth Integration Tests XCTest Bundle
  52. TESTING IOS APP WITH EXTERNAL HARDWARE INTEGRATION TESTING WITHIN TEST

    HARNESS Connectivity Layer Connectivity Layer Abstraction Transport Layer • StateLESS layer • Request/Response • Device specific components • System layer, e.g. Core Bluetooth Integration Tests Test Harness iOS App
  53. TESTING IOS APP WITH EXTERNAL HARDWARE INTEGRATION TESTING WITHIN TEST

    HARNESS ▸ Can share build with QA-, FW- Engineers PROS CONS
  54. TESTING IOS APP WITH EXTERNAL HARDWARE INTEGRATION TESTING WITHIN TEST

    HARNESS ▸ Can share build with QA-, FW- Engineers ▸ Easier to test new firmware/hardware PROS CONS
  55. TESTING IOS APP WITH EXTERNAL HARDWARE INTEGRATION TESTING WITHIN TEST

    HARNESS ▸ Can share build with QA-, FW- Engineers ▸ Easier to test new firmware/hardware ▸ Android: easy to test on exotic device PROS CONS
  56. TESTING IOS APP WITH EXTERNAL HARDWARE INTEGRATION TESTING WITHIN TEST

    HARNESS ▸ Can share build with QA-, FW- Engineers ▸ Easier to test new firmware/hardware ▸ Android: easy to test on exotic device PROS ▸ Need to reimplement some XCTest tools CONS
  57. TESTING IOS APP WITH EXTERNAL HARDWARE INTEGRATION TESTING WITHIN TEST

    HARNESS ▸ Can share build with QA-, FW- Engineers ▸ Easier to test new firmware/hardware ▸ Android: easy to test on exotic device PROS ▸ Need to reimplement some XCTest tools ▸ Need to keep transport layer in Framework CONS
  58. TESTING IOS APP WITH EXTERNAL HARDWARE INTEGRATION TESTING WITHIN TEST

    HARNESS ▸ Can share build with QA-, FW- Engineers ▸ Easier to test new firmware/hardware ▸ Android: easy to test on exotic device PROS ▸ Need to reimplement some XCTest tools ▸ Need to keep transport layer in Framework ▸ Need to support another codebase CONS
  59. TESTING IOS APP WITH EXTERNAL HARDWARE BEST PRACTICES ▸ Cover

    happy path + corner cases via Unit Tests
 CI - must-have
  60. TESTING IOS APP WITH EXTERNAL HARDWARE BEST PRACTICES ▸ Cover

    happy path + corner cases via Unit Tests
 CI - must-have ▸ Cover happy path ONLY via Integration Tests
 CI-optional
  61. TESTING IOS APP WITH EXTERNAL HARDWARE BEST PRACTICES ▸ Cover

    happy path + corner cases via Unit Tests
 CI - must-have ▸ Cover happy path ONLY via Integration Tests
 CI-optional ▸ No device - use Simulator
  62. TESTING IOS APP WITH EXTERNAL HARDWARE BEST PRACTICES ▸ Unstable

    connection - 
 MUST still cover with Integration Tests
 

  63. TESTING IOS APP WITH EXTERNAL HARDWARE BEST PRACTICES ▸ Unstable

    connection - 
 MUST still cover with Integration Tests
 
 ▸ When too many device/firmware versions - 
 use Integration Tests + Test Harness
  64. TESTING IOS APP WITH EXTERNAL HARDWARE SHOWCASE 1: COFFEE ROASTING

    MACHINE Coffee roasting machine Bluetooth LE WiFi
  65. TESTING IOS APP WITH EXTERNAL HARDWARE SHOWCASE 1: COFFEE ROASTING

    MACHINE Coffee roasting machine Bluetooth LE WiFi
  66. TESTING IOS APP WITH EXTERNAL HARDWARE SHOWCASE 1: COFFEE ROASTING

    MACHINE Coffee roasting machine Hardware Unit Hardware Unit Hardware Unit Hardware Unit
  67. TESTING IOS APP WITH EXTERNAL HARDWARE SHOWCASE 1: COFFEE ROASTING

    MACHINE Coffee roasting machine Hardware Unit Hardware Unit Hardware Unit Hardware Unit USB
  68. TESTING IOS APP WITH EXTERNAL HARDWARE SHOWCASE 1: COFFEE ROASTING

    MACHINE Coffee roasting machine Hardware Unit Hardware Unit Hardware Unit Hardware Unit USB
  69. TESTING IOS APP WITH EXTERNAL HARDWARE SHOWCASE 1: COFFEE ROASTING

    MACHINE Coffee roasting machine Bluetooth LE WiFi Hardware Hardware Hardware Hardware USB
  70. TESTING IOS APP WITH EXTERNAL HARDWARE SHOWCASE 1: COFFEE ROASTING

    MACHINE Coffee roasting machine Bluetooth LE WiFi USB
  71. TESTING IOS APP WITH EXTERNAL HARDWARE SHOWCASE 1: COFFEE ROASTING

    MACHINE Coffee roasting machine Bluetooth LE WiFi USB Simulator
  72. TESTING IOS APP WITH EXTERNAL HARDWARE SHOWCASE 1: COFFEE ROASTING

    MACHINE Coffee roasting machine Bluetooth LE WiFi USB Simulator Arduino Uno
  73. Simulator TESTING IOS APP WITH EXTERNAL HARDWARE SHOWCASE 1: COFFEE

    ROASTING MACHINE Coffee roasting machine Bluetooth LE WiFi USB Arduino Uno
  74. Simulator TESTING IOS APP WITH EXTERNAL HARDWARE SHOWCASE 1: COFFEE

    ROASTING MACHINE Coffee roasting machine Bluetooth LE WiFi USB Arduino Uno Integration Tests
  75. TESTING IOS APP WITH EXTERNAL HARDWARE SHOWCASE 1: COFFEE ROASTING

    MACHINE Coffee roasting machine Bluetooth LE WiFi USB Integration Tests
  76. TESTING IOS APP WITH EXTERNAL HARDWARE SHOWCASE 1: COFFEE ROASTING

    MACHINE Coffee roasting machine Bluetooth LE WiFi USB Integration Tests 1 day
  77. TESTING IOS APP WITH EXTERNAL HARDWARE SHOWCASE 2: MEDICAL WEARABLE

    Data synchronization algorithm Mock Transport Layer Unit Tests Cases
  78. TESTING IOS APP WITH EXTERNAL HARDWARE SHOWCASE 2: MEDICAL WEARABLE

    Data synchronization algorithm Transport Layer Integration Tests Cases Test Harness iOS App
  79. TESTING IOS APP WITH EXTERNAL HARDWARE SHOWCASE 2: MEDICAL WEARABLE

    ▸ 3 weeks of work ▸ 2 developers: 1 iOS, 1 Android
  80. TESTING IOS APP WITH EXTERNAL HARDWARE SHOWCASE 2: MEDICAL WEARABLE

    ▸ 3 weeks of work ▸ 2 developers: 1 iOS, 1 Android ▸ Reliable data synchronization
  81. TESTING IOS APP WITH EXTERNAL HARDWARE INSTEAD OF CONCLUSIONS ▸

    Don’t sell your customer auto tests - 
 Include those to estimates
  82. TESTING IOS APP WITH EXTERNAL HARDWARE INSTEAD OF CONCLUSIONS ▸

    Don’t sell your customer auto tests - 
 Include those to estimates ▸ 100% code coverage is never needed - 
 Find level needed by your app
  83. TESTING IOS APP WITH EXTERNAL HARDWARE INSTEAD OF CONCLUSIONS ▸

    Don’t sell your customer auto tests - 
 Include those to estimates ▸ 100% code coverage is never needed - 
 Find level needed by your app ▸ Try both sides of the force: 
 Both under-testing and over-testing