DevOpsPorto Meetup15: Acceptance Testing for Continuous Delivery by Dave Farley

Talk delivered by Dave Farley

Transcript

1. Dave Farley
   http://www.davefarley.net
   @davefarley77
   http://www.continuous-delivery.co.uk
   Acceptance Testing for
   Continuous Delivery
   

   View Slide

2. The Role of Acceptance Testing
   Local Dev. Env.
   Source
   Repository
   

   View Slide

3. The Role of Acceptance Testing
   Artifact
   Repository
   Local Dev. Env.
   Deployment Pipeline
   Commit
   Production Env.
   Deployment
   App.
   Commit
   Acceptance
   Manual
   Perf1
   Perf2
   Staged
   Production
   Source
   Repository
   Acceptance
   Component
   Performance
   System
   Performance
   Staging Env.
   Deployment
   App.
   Manual Test Env.
   Deployment
   App.
   

   View Slide

4. The Role of Acceptance Testing
   Artifact
   Repository
   Local Dev. Env.
   Deployment Pipeline
   Commit
   Production Env.
   Deployment
   App.
   Commit
   Acceptance
   Manual
   Perf1
   Perf2
   Staged
   Production
   Source
   Repository
   Acceptance
   Component
   Performance
   System
   Performance
   Staging Env.
   Deployment
   App.
   Manual Test Env.
   Deployment
   App.
   Staging Env.
   Deployment
   App.
   Manual Test Env.
   Deployment
   App.
   Component
   Performance
   System
   Performance
   Acceptance
   

   View Slide

5. The Role of Acceptance Testing
   Artifact
   Repository
   Local Dev. Env.
   Deployment Pipeline
   Commit
   Production Env.
   Deployment
   App.
   Commit
   Acceptance
   Manual
   Perf1
   Perf2
   Staged
   Production
   Source
   Repository
   Acceptance
   Component
   Performance
   System
   Performance
   Staging Env.
   Deployment
   App.
   Manual Test Env.
   Deployment
   App.
   Staging Env.
   Deployment
   App.
   Manual Test Env.
   Deployment
   App.
   Component
   Performance
   System
   Performance
   Acceptance
   

   View Slide

6. What is Acceptance Testing?
   Asserts that the code does what the users want.
   

   View Slide

7. What is Acceptance Testing?
   An automated “definition of done”
   

   View Slide

8. What is Acceptance Testing?
   Asserts that the code works in a “production-like” test environment.
   

   View Slide

9. What is Acceptance Testing?
   A test of the deployment and configuration of a whole system.
   

   View Slide

10. What is Acceptance Testing?
    Provides timely feedback on stories - closes a feedback loop.
    

    View Slide

11. What is Acceptance Testing?
    Acceptance Testing, ATDD, BDD, Specification by
    Example, Executable Specifications.
    

    View Slide

12. What is Acceptance Testing?
    A Good Acceptance Test is:
    An Executable Specification of the
    Behaviour of the System
    

    View Slide

13. What is Acceptance Testing?
    Unit Test Code
    Idea
    Executable
    spec.
    Build Release
    

    View Slide

14. What is Acceptance Testing?
    Unit Test Code
    Idea
    Executable
    spec.
    Build Release
    

    View Slide

15. What is Acceptance Testing?
    Unit Test Code
    Idea
    Executable
    spec.
    Build Release
    

    View Slide

16. So What’s So Hard?
    • Tests break when the SUT changes (Particularly UI)
    • Tests are complex to develop
    • This is a problem of design, the tests are too tightly-coupled to the
    SUT!
    • The history is littered with poor implementations:
    • UI Record-and-playback Systems
    • Record-and-playback of production data
    • Dumps of production data to test systems
    • Nasty automated testing products.
    

    View Slide

17. So What’s So Hard?
    • Tests break when the SUT changes (Particularly UI)
    • Tests are complex to develop
    • This is a problem of design, the tests are too tightly-coupled to the
    SUT!
    • The history is littered with poor implementations:
    • UI Record-and-playback Systems
    • Record-and-playback of production data
    • Dumps of production data to test systems
    • Nasty automated testing products.
    Anti-Pattern!
    Anti-Pattern!
    Anti-Pattern!
    Anti-Pattern!
    

    View Slide

18. Who Owns the Tests?
    • Anyone can write a test
    • Developers are the people that will break tests
    • Therefore Developers own the responsibility to keep them working
    • Separate Testing/QA team owning automated tests
    

    View Slide

19. Who Owns the Tests?
    • Anyone can write a test
    • Developers are the people that will break tests
    • Therefore Developers own the responsibility to keep them working
    • Separate Testing/QA team owning automated tests
    Anti-Pattern!
    

    View Slide

20. Who Owns the Tests?
    Developers Own
    Acceptance Tests!
    

    View Slide

21. Properties of Good Acceptance Tests
    • “What” not “How”
    • Isolated from other tests
    • Repeatable
    • Uses the language of the problem domain
    • Tests ANY change
    • Efficient
    

    View Slide

22. Properties of Good Acceptance Tests
    • “What” not “How”
    • Isolated from other tests
    • Repeatable
    • Uses the language of the problem domain
    • Tests ANY change
    • Efficient
    

    View Slide

23. Public API FIX API
    Trade
    Reporting
    Gateway
    …
    “What” not “How”
    API Traders
    Clearing
    Destination
    Other
    external
    end-points
    Market
    Makers
    UI
    Traders
    

    View Slide

24. Public API FIX API
    Trade
    Reporting
    Gateway
    …
    “What” not “How”
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    

    View Slide

25. Public API FIX API
    Trade
    Reporting
    Gateway
    …
    FIX API
    “What” not “How”
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    

    View Slide

26. Public API FIX API
    Trade
    Reporting
    Gateway
    …
    “What” not “How”
    API Traders
    Clearing
    Destination
    Other
    external
    end-points
    Market
    Makers
    UI
    Traders
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    

    View Slide

27. Public API FIX API
    Trade
    Reporting
    Gateway
    …
    “What” not “How”
    FIX API
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    API
    External
    Stubs
    FIX-API
    UI FIX-API
    FIX-API
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    

    View Slide

28. Public API FIX API
    Trade
    Reporting
    Gateway
    …
    “What” not “How”
    FIX API
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    API
    External
    Stubs
    FIX-API
    UI FIX-API
    

    View Slide

29. Public API FIX API
    Trade
    Reporting
    Gateway
    …
    “What” not “How”
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    API
    External
    Stubs
    FIX-API
    UI FIX-API
    

    View Slide

30. Public API FIX API
    Trade
    Reporting
    Gateway
    …
    “What” not “How”
    API
    External
    Stubs
    FIX-API
    UI FIX-API
    Test infrastructure common to all acceptance tests
    

    View Slide

31. “What” not “How” - Separate Deployment from Testing
    • Every Test should control its start conditions, and so should start
    and init the app.
    • Acceptance Test deployment should be a rehearsal for
    Production Release
    • This separation of concerns provides an opportunity for
    optimisation
    • Parallel tests in a shared environment
    • Lower test start-up overhead
    

    View Slide

32. “What” not “How” - Separate Deployment from Testing
    • Every Test should control its start conditions, and so should start
    and init the app.
    • Acceptance Test deployment should be a rehearsal for
    Production Release
    • This separation of concerns provides an opportunity for
    optimisation
    • Parallel tests in a shared environment
    • Lower test start-up overhead
    Anti-Pattern!
    

    View Slide

33. Properties of Good Acceptance Tests
    • “What” not “How”
    • Isolated from other tests
    • Repeatable
    • Uses the language of the problem domain
    • Tests ANY change
    • Efficient
    

    View Slide

34. Properties of Good Acceptance Tests
    • “What” not “How”
    • Isolated from other tests
    • Repeatable
    • Uses the language of the problem domain
    • Tests ANY change
    • Efficient
    

    View Slide

35. Test Isolation
    • Any form of testing is about evaluating something in controlled
    circumstances
    • Isolation works on multiple levels
    • Isolating the System under test
    • Isolating test cases from each other
    • Isolating test cases from themselves (temporal isolation)
    • Isolation is a vital part of your Test Strategy
    

    View Slide

36. Test Isolation - Isolating the System Under Test
    

    View Slide

37. Test Isolation - Isolating the System Under Test
    External System
    ‘A’
    External System
    ‘C’
    System Under Test
    ‘B’
    

    View Slide

38. Test Isolation - Isolating the System Under Test
    External System
    ‘A’
    External System
    ‘C’
    System Under Test
    ‘B’
    

    View Slide

39. Test Isolation - Isolating the System Under Test
    External System
    ‘A’
    External System
    ‘C’
    System Under Test
    ‘B’
    

    View Slide

40. Test Isolation - Isolating the System Under Test
    External System
    ‘A’
    External System
    ‘C’
    System Under Test
    ‘B’
    ?
    

    View Slide

41. Test Isolation - Isolating the System Under Test
    External System
    ‘A’
    External System
    ‘C’
    System Under Test
    ‘B’
    Anti-Pattern!
    

    View Slide

42. Test Isolation - Isolating the System Under Test
    System Under Test
    ‘B’
    Test Cases
    Verifiable
    Output
    

    View Slide

43. Test Isolation - Validating The Interfaces
    

    View Slide

44. Test Isolation - Validating The Interfaces
    External System
    ‘A’
    External System
    ‘C’
    System Under Test
    ‘B’
    

    View Slide

45. Test Isolation - Validating The Interfaces
    External System
    ‘A’
    External System
    ‘C’
    System Under Test
    ‘B’
    

    View Slide

46. Test Isolation - Validating The Interfaces
    External System
    ‘A’
    External System
    ‘C’
    Test Cases
    Verifiable
    Output
    System Under Test
    ‘B’
    Test Cases
    Verifiable
    Output
    Test Cases
    Verifiable
    Output
    

    View Slide

47. Test Isolation - Validating The Interfaces
    External System
    ‘A’
    External System
    ‘C’
    Test Cases
    Verifiable
    Output
    System Under Test
    ‘B’
    Test Cases
    Verifiable
    Output
    Test Cases
    Verifiable
    Output
    

    View Slide

48. Test Isolation - Isolating Test Cases
    • Assuming multi-user systems…
    • Tests should be efficient - We want to run LOTS!
    • What we really want is to deploy once, and run LOTS of tests
    • So we must avoid ANY dependencies between tests…
    • Use natural functional isolation e.g.
    • If testing Amazon, create a new account and a new book/product for every test-case
    • If testing eBay create a new account and a new auction for every test-case
    • If testing GitHub, create a new account and a new repository for every test-case
    • …
    

    View Slide

49. • We want repeatable results
    • If I run my test-case twice it should work both times
    Test Isolation - Temporal Isolation
    

    View Slide

50. • We want repeatable results
    • If I run my test-case twice it should work both times
    Test Isolation - Temporal Isolation
    def test_should_place_an_order(self):
    self.store.createBook(“Continuous Delivery”);
    order = self.store.placeOrder(book=“Continuous Delivery")
    self.store.assertOrderPlaced(order)
    

    View Slide

51. • We want repeatable results
    • If I run my test-case twice it should work both times
    Test Isolation - Temporal Isolation
    def test_should_place_an_order(self):
    self.store.createBook(“Continuous Delivery”);
    order = self.store.placeOrder(book=“Continuous Delivery")
    self.store.assertOrderPlaced(order)
    

    View Slide

52. • We want repeatable results
    • If I run my test-case twice it should work both times
    Test Isolation - Temporal Isolation
    def test_should_place_an_order(self):
    self.store.createBook(“Continuous Delivery”);
    order = self.store.placeOrder(book=“Continuous Delivery")
    self.store.assertOrderPlaced(order)
    

    View Slide

53. • We want repeatable results
    • If I run my test-case twice it should work both times
    Test Isolation - Temporal Isolation
    def test_should_place_an_order(self):
    self.store.createBook(“Continuous Delivery”);
    order = self.store.placeOrder(book=“Continuous Delivery")
    self.store.assertOrderPlaced(order)
    Continuous Delivery
    

    View Slide

54. • We want repeatable results
    • If I run my test-case twice it should work both times
    Test Isolation - Temporal Isolation
    def test_should_place_an_order(self):
    self.store.createBook(“Continuous Delivery”);
    order = self.store.placeOrder(book=“Continuous Delivery")
    self.store.assertOrderPlaced(order)
    Continuous Delivery
    

    View Slide

55. • We want repeatable results
    • If I run my test-case twice it should work both times
    Test Isolation - Temporal Isolation
    def test_should_place_an_order(self):
    self.store.createBook(“Continuous Delivery”);
    order = self.store.placeOrder(book=“Continuous Delivery")
    self.store.assertOrderPlaced(order)
    Continuous Delivery1234
    

    View Slide

56. • We want repeatable results
    • If I run my test-case twice it should work both times
    Test Isolation - Temporal Isolation
    def test_should_place_an_order(self):
    self.store.createBook(“Continuous Delivery”);
    order = self.store.placeOrder(book=“Continuous Delivery")
    self.store.assertOrderPlaced(order)
    Continuous Delivery1234
    Continuous Delivery6789
    

    View Slide

57. • We want repeatable results
    • If I run my test-case twice it should work both times
    Test Isolation - Temporal Isolation
    def test_should_place_an_order(self):
    self.store.createBook(“Continuous Delivery”);
    order = self.store.placeOrder(book=“Continuous Delivery")
    self.store.assertOrderPlaced(order)
    Continuous Delivery1234
    Continuous Delivery6789
    • Alias your functional isolation entities
    • In your test case create account ‘Dave’ in reality, in the test infrastructure, ask the
    application to create account ‘Dave2938472398472’ and alias it to ‘Dave’ in your test
    infrastructure.
    

    View Slide

58. Properties of Good Acceptance Tests
    • “What” not “How”
    • Isolated from other tests
    • Repeatable
    • Uses the language of the problem domain
    • Tests ANY change
    • Efficient
    

    View Slide

59. Properties of Good Acceptance Tests
    • “What” not “How”
    • Isolated from other tests
    • Repeatable
    • Uses the language of the problem domain
    • Tests ANY change
    • Efficient
    

    View Slide

60. Repeatability - Test Doubles
    External System
    

    View Slide

61. Repeatability - Test Doubles
    External System
    Local Interface
    to External System
    

    View Slide

62. Repeatability - Test Doubles
    External System
    Local Interface
    to External System
    Communications
    to External System
    

    View Slide

63. Repeatability - Test Doubles
    External System
    Local Interface
    to External System
    Communications
    to External System
    TestStub
    Simulating External System
    Local Interface
    to External System
    

    View Slide

64. Repeatability - Test Doubles
    External System
    Local Interface
    to External System
    Communications
    to External System
    TestStub
    Simulating External System
    Local Interface
    to External System
    Production
    Test
    Environm
    ent
    kjhaskjhdkjhkjh askjhl lkjasl dkjas lkajl ajsd
    lkjalskjlakjsdlkajsld j
    lkajsdlkajsldkj
    lkjlakjsldkjlka laskj ljl akjl kajsldijupoqwiuepoq dlkjl iu
    lkajsodiuqpwouoi la
    ]laksjdiuqoiwuoijds
    oijasodiaosidjuoiasud
    kjhaskjhdkjhkjh askjhl lkjasl dkjas lkajl ajsd
    lkjalskjlakjsdlkajsld j
    lkajsdlkajsldkj
    lkjlakjsldkjlka laskj ljl akjl kajsldijupoqwiuepoq dlkjl iu
    lkajsodiuqpwouoi la
    ]laksjdiuqoiwuoijds
    oijasodiaosidjuoiasud
    Configuration
    

    View Slide

65. Test Doubles As Part of Test Infrastructure
    TestStub
    Simulating External System
    Local Interface
    to External System
    

    View Slide

66. Test Doubles As Part of Test Infrastructure
    TestStub
    Simulating External System
    Local Interface
    to External System
    

    View Slide

67. Test Doubles As Part of Test Infrastructure
    TestStub
    Simulating External System
    Local Interface
    to External System
    Public Interface
    

    View Slide

68. Test Doubles As Part of Test Infrastructure
    TestStub
    Simulating External System
    Local Interface
    to External System
    Public Interface
    

    View Slide

69. Test Doubles As Part of Test Infrastructure
    TestStub
    Simulating External System
    Local Interface
    to External System
    Test Infrastructure
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    Test Infrastructure
    Back-Channel
    Public Interface
    System
    Under Test
    

    View Slide

70. Properties of Good Acceptance Tests
    • “What” not “How”
    • Isolated from other tests
    • Repeatable
    • Uses the language of the problem domain
    • Tests ANY change
    • Efficient
    

    View Slide

71. Properties of Good Acceptance Tests
    • “What” not “How”
    • Isolated from other tests
    • Repeatable
    • Uses the language of the problem domain
    • Tests ANY change
    • Efficient
    

    View Slide

72. Language of the Problem Domain - DSL
    • A Simple ‘DSL’ Solves many of our problems
    • Ease of TestCase creation
    • Readability
    • Ease of Maintenance
    • Separation of “What” from “How”
    • Test Isolation
    • The Chance to abstract complex set-up and scenarios
    • …
    

    View Slide

73. Language of the Problem Domain - DSL
    @Test
    public void shouldSupportPlacingValidBuyAndSellLimitOrders()
    {
    trading.selectDealTicket("instrument");
    trading.dealTicket.placeOrder("type: limit", ”bid: 4@10”);
    trading.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to buy 4.00 contracts at 10.0");
    trading.dealTicket.dismissFeedbackMessage();
    trading.dealTicket.placeOrder("type: limit", ”ask: 4@9”);
    trading.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to sell 4.00 contracts at 9.0");
    }
    

    View Slide

74. Language of the Problem Domain - DSL
    @Test
    public void shouldSupportPlacingValidBuyAndSellLimitOrders()
    {
    trading.selectDealTicket("instrument");
    trading.dealTicket.placeOrder("type: limit", ”bid: 4@10”);
    trading.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to buy 4.00 contracts at 10.0");
    trading.dealTicket.dismissFeedbackMessage();
    trading.dealTicket.placeOrder("type: limit", ”ask: 4@9”);
    trading.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to sell 4.00 contracts at 9.0");
    }
    @Test
    public void shouldSuccessfullyPlaceAnImmediateOrCancelBuyMarketOrder()
    {
    fixAPIMarketMaker.placeMassOrder("instrument", "ask: 11@52", "ask: 10@51", "ask: 10@50", "bid: 10@49");
    fixAPI.placeOrder("instrument", "side: buy", "quantity: 4", "goodUntil: Immediate", "allowUnmatched: true");
    fixAPI.waitForExecutionReport("executionType: Fill", "orderStatus: Filled",
    "side: buy", "quantity: 4", "matched: 4", "remaining: 0",
    "executionPrice: 50", "executionQuantity: 4");
    }
    

    View Slide

75. Language of the Problem Domain - DSL
    @Test
    public void shouldSupportPlacingValidBuyAndSellLimitOrders()
    {
    trading.selectDealTicket("instrument");
    trading.dealTicket.placeOrder("type: limit", ”bid: 4@10”);
    trading.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to buy 4.00 contracts at 10.0");
    trading.dealTicket.dismissFeedbackMessage();
    trading.dealTicket.placeOrder("type: limit", ”ask: 4@9”);
    trading.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to sell 4.00 contracts at 9.0");
    }
    @Test
    public void shouldSuccessfullyPlaceAnImmediateOrCancelBuyMarketOrder()
    {
    fixAPIMarketMaker.placeMassOrder("instrument", "ask: 11@52", "ask: 10@51", "ask: 10@50", "bid: 10@49");
    fixAPI.placeOrder("instrument", "side: buy", "quantity: 4", "goodUntil: Immediate", "allowUnmatched: true");
    fixAPI.waitForExecutionReport("executionType: Fill", "orderStatus: Filled",
    "side: buy", "quantity: 4", "matched: 4", "remaining: 0",
    "executionPrice: 50", "executionQuantity: 4");
    }
    @Before
    public void beforeEveryTest()
    {
    adminAPI.createInstrument("name: instrument");
    registrationAPI.createUser("user");
    registrationAPI.createUser("marketMaker", "accountType: MARKET_MAKER");
    tradingUI.loginAsLive("user");
    }
    

    View Slide

76. Language of the Problem Domain - DSL
    public void placeOrder(final String... args)
    {
    final DslParams params =
    new DslParams(args,
    new OptionalParam("type").setDefault("Limit").setAllowedValues("limit", "market", "StopMarket"),
    new OptionalParam("side").setDefault("Buy").setAllowedValues("buy", "sell"),
    new OptionalParam("price"),
    new OptionalParam("triggerPrice"),
    new OptionalParam("quantity"),
    new OptionalParam("stopProfitOffset"),
    new OptionalParam("stopLossOffset"),
    new OptionalParam("confirmFeedback").setDefault("true"));
    getDealTicketPageDriver().placeOrder(params.value("type"),
    params.value("side"),
    params.value("price"),
    params.value("triggerPrice"),
    params.value("quantity"),
    params.value("stopProfitOffset"),
    params.value("stopLossOffset"));
    if (params.valueAsBoolean("confirmFeedback"))
    {
    getDealTicketPageDriver().clickOrderFeedbackConfirmationButton();
    }
    LOGGER.debug("placeOrder(" + Arrays.deepToString(args) + ")");
    }
    

    View Slide

77. Language of the Problem Domain - DSL
    @Test
    public void shouldSupportPlacingValidBuyAndSellLimitOrders()
    {
    tradingUI.showDealTicket("instrument");
    tradingUI.dealTicket.placeOrder("type: limit", ”bid: 4@10”);
    tradingUI.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to buy 4.00 contracts at 10.0");
    tradingUI.dealTicket.dismissFeedbackMessage();
    tradingUI.dealTicket.placeOrder("type: limit", ”ask: 4@9”);
    tradingUI.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to sell 4.00 contracts at 9.0");
    }
    @Test
    public void shouldSuccessfullyPlaceAnImmediateOrCancelBuyMarketOrder()
    {
    fixAPIMarketMaker.placeMassOrder("instrument", "ask: 11@52", "ask: 10@51", "ask: 10@50", "bid: 10@49");
    fixAPI.placeOrder("instrument", "side: buy", "quantity: 4", "goodUntil: Immediate", "allowUnmatched: true");
    fixAPI.waitForExecutionReport("executionType: Fill", "orderStatus: Filled",
    "side: buy", "quantity: 4", "matched: 4", "remaining: 0",
    "executionPrice: 50", "executionQuantity: 4");
    }
    

    View Slide

78. Language of the Problem Domain - DSL
    @Test
    public void shouldSupportPlacingValidBuyAndSellLimitOrders()
    {
    tradingUI.showDealTicket("instrument");
    tradingUI.dealTicket.placeOrder("type: limit", ”bid: 4@10”);
    tradingUI.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to buy 4.00 contracts at 10.0");
    tradingUI.dealTicket.dismissFeedbackMessage();
    tradingUI.dealTicket.placeOrder("type: limit", ”ask: 4@9”);
    tradingUI.dealTicket.checkFeedbackMessage("You have successfully sent a limit order to sell 4.00 contracts at 9.0");
    }
    @Test
    public void shouldSuccessfullyPlaceAnImmediateOrCancelBuyMarketOrder()
    {
    fixAPIMarketMaker.placeMassOrder("instrument", "ask: 11@52", "ask: 10@51", "ask: 10@50", "bid: 10@49");
    fixAPI.placeOrder("instrument", "side: buy", "quantity: 4", "goodUntil: Immediate", "allowUnmatched: true");
    fixAPI.waitForExecutionReport("executionType: Fill", "orderStatus: Filled",
    "side: buy", "quantity: 4", "matched: 4", "remaining: 0",
    "executionPrice: 50", "executionQuantity: 4");
    }
    

    View Slide

79. Language of the Problem Domain - DSL
    @Channel(fixApi, dealTicket, publicApi)
    @Test
    public void shouldSuccessfullyPlaceAnImmediateOrCancelBuyMarketOrder()
    {
    trading.placeOrder("instrument", "side: buy", “price: 123.45”, "quantity: 4", "goodUntil: Immediate”);
    trading.waitForExecutionReport("executionType: Fill", "orderStatus: Filled",
    "side: buy", "quantity: 4", "matched: 4", "remaining: 0",
    "executionPrice: 123.45", "executionQuantity: 4");
    }
    

    View Slide

80. Language of the Problem Domain - DSL
    @Channel(fixApi, dealTicket, publicApi)
    @Test
    public void shouldSuccessfullyPlaceAnImmediateOrCancelBuyMarketOrder()
    {
    trading.placeOrder("instrument", "side: buy", “price: 123.45”, "quantity: 4", "goodUntil: Immediate”);
    trading.waitForExecutionReport("executionType: Fill", "orderStatus: Filled",
    "side: buy", "quantity: 4", "matched: 4", "remaining: 0",
    "executionPrice: 123.45", "executionQuantity: 4");
    }
    

    View Slide

81. Properties of Good Acceptance Tests
    • “What” not “How”
    • Isolated from other tests
    • Repeatable
    • Uses the language of the problem domain
    • Tests ANY change
    • Efficient
    

    View Slide

82. Properties of Good Acceptance Tests
    • “What” not “How”
    • Isolated from other tests
    • Repeatable
    • Uses the language of the problem domain
    • Tests ANY change
    • Efficient
    

    View Slide

83. Testing with Time
    • Test Cases should be deterministic
    • Time is a problem for determinism - There are two options:
    • Ignore time
    • Control time
    

    View Slide

84. Testing With Time - Ignore Time
    Mechanism
    Filter out time-based values in your test infrastructure so that they
    are ignored
    Pros:
    • Simple!
    Cons:
    • Can miss errors
    • Prevents any hope of testing complex time-based scenarios
    

    View Slide

85. Mechanism
    Treat Time as an external dependency, like any external system -
    and Fake it!
    Pros:
    • Very Flexible!
    • Can simulate any time-based scenario, with time under the control of the test
    case.
    Cons:
    • Slightly more complex infrastructure
    Testing With Time - Controlling Time
    

    View Slide

86. Testing With Time - Controlling Time
    @Test
    public void shouldBeOverdueAfterOneMonth()
    {
    book = library.borrowBook(“Continuous Delivery”);
    assertFalse(book.isOverdue());
    time.travel(“+1 week”);
    assertFalse(book.isOverdue());
    time.travel(“+4 weeks”);
    assertTrue(book.isOverdue());
    }
    

    View Slide

87. Testing With Time - Controlling Time
    @Test
    public void shouldBeOverdueAfterOneMonth()
    {
    book = library.borrowBook(“Continuous Delivery”);
    assertFalse(book.isOverdue());
    time.travel(“+1 week”);
    assertFalse(book.isOverdue());
    time.travel(“+4 weeks”);
    assertTrue(book.isOverdue());
    }
    

    View Slide

88. Testing With Time - Controlling Time
    

    View Slide

89. Testing With Time - Controlling Time
    Test Infrastructure
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    System
    Under Test
    public void someTimeDependentMethod()
    {
    time = System.getTime();
    }
    System
    Under Test
    

    View Slide

90. Testing With Time - Controlling Time
    Test Infrastructure
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    System
    Under Test
    include Clock;
    public void someTimeDependentMethod()
    {
    time = Clock.getTime();
    }
    System
    Under Test
    

    View Slide

91. Testing With Time - Controlling Time
    Test Infrastructure
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    System
    Under Test
    include Clock;
    public void someTimeDependentMethod()
    {
    time = Clock.getTime();
    }
    public class Clock {
    public static clock = new SystemClock();
    public static void setTime(long newTime) {
    clock.setTime(newTime);
    }
    public static long getTime() {
    return clock.getTime();
    }
    System
    Under Test
    

    View Slide

92. Testing With Time - Controlling Time
    Test Infrastructure
    Test
    Case
    Test
    Case
    Test
    Case
    Test
    Case
    System
    Under Test
    include Clock;
    public void someTimeDependentMethod()
    {
    time = Clock.getTime();
    }
    public void onInit() {
    // Remote Call - back-channel
    systemUnderTest.setClock(new TestClock());
    }
    public void time-travel(String time) {
    long newTime = parseTime(time);
    // Remote Call - back-channel
    systemUnderTest.setTime(newTime);
    }
    Test Infrastructure
    Back-Channel
    public class Clock {
    public static clock = new SystemClock();
    public static void setTime(long newTime) {
    clock.setTime(newTime);
    }
    public static long getTime() {
    return clock.getTime();
    }
    System
    Under Test
    

    View Slide

93. Test Environment Types
    • Some Tests need special treatment.
    • Tag Tests with properties and allocate them dynamically:
    

    View Slide

94. Test Environment Types
    • Some Tests need special treatment.
    • Tag Tests with properties and allocate them dynamically:
    @TimeTravel
    @Test
    public void shouldDoSomethingThatNeedsFakeTime()
    …
    @Destructive
    @Test
    public void shouldDoSomethingThatKillsPartOfTheSystem()
    …
    @FPGA(version=1.3)
    @Test
    public void shouldDoSomethingThatRequiresSpecificHardware()
    …
    

    View Slide

95. Test Environment Types
    • Some Tests need special treatment.
    • Tag Tests with properties and allocate them dynamically:
    @TimeTravel
    @Test
    public void shouldDoSomethingThatNeedsFakeTime()
    …
    @Destructive
    @Test
    public void shouldDoSomethingThatKillsPartOfTheSystem()
    …
    @FPGA(version=1.3)
    @Test
    public void shouldDoSomethingThatRequiresSpecificHardware()
    …
    

    View Slide

96. Test Environment Types
    

    View Slide

97. Test Environment Types
    

    View Slide

98. Properties of Good Acceptance Tests
    • “What” not “How”
    • Isolated from other tests
    • Repeatable
    • Uses the language of the problem domain
    • Tests ANY change
    • Efficient
    

    View Slide

99. Properties of Good Acceptance Tests
    • “What” not “How”
    • Isolated from other tests
    • Repeatable
    • Uses the language of the problem domain
    • Tests ANY change
    • Efficient
    

    View Slide

100. Production-like Test Environments
     

     View Slide

101. Production-like Test Environments
     

     View Slide

102. Production-like Test Environments
     

     View Slide

103. Production-like Test Environments
     

     View Slide

104. Production-like Test Environments
     

     View Slide

105. Production-like Test Environments
     

     View Slide

106. Production-like Test Environments
     

     View Slide

107. Make Test Cases Internally Synchronous
     

     View Slide

108. Make Test Cases Internally Synchronous
     • Look for a “Concluding Event” listen for that in your DSL to report
     an async call as complete
     

     View Slide

109. Make Test Cases Internally Synchronous
     Example DSL level Implementation…
     public String placeOrder(String params…)
     {
     orderSent = sendAsyncPlaceOrderMessage(parseOrderParams(params));
     return waitForOrderConfirmedOrFailOnTimeOut(orderSent);
     }
     • Look for a “Concluding Event” listen for that in your DSL to report
     an async call as complete
     

     View Slide

110. Make Test Cases Internally Synchronous
     Example DSL level Implementation…
     public String placeOrder(String params…)
     {
     orderSent = sendAsyncPlaceOrderMessage(parseOrderParams(params));
     return waitForOrderConfirmedOrFailOnTimeOut(orderSent);
     }
     • Look for a “Concluding Event” listen for that in your DSL to report
     an async call as complete
     

     View Slide

111. Make Test Cases Internally Synchronous
     • Look for a “Concluding Event” listen for that in your DSL to report
     an async call as complete
     • If you really have to, implement a 

     “poll-and-timeout” mechanism in your test-infrastructure
     • Never, Never, Never, put a “wait(xx)” and expect your tests to be
     (a) Reliable or (b) Efficient!
     • Look for a “Concluding Event” listen for that in your DSL to report
     an async call as complete
     

     View Slide

112. Make Test Cases Internally Synchronous
     • Look for a “Concluding Event” listen for that in your DSL to report
     an async call as complete
     • If you really have to, implement a 

     “poll-and-timeout” mechanism in your test-infrastructure
     • Never, Never, Never, put a “wait(xx)” and expect your tests to be
     (a) Reliable or (b) Efficient!
     • Look for a “Concluding Event” listen for that in your DSL to report
     an async call as complete
     Anti-Pattern!
     

     View Slide

113. Scaling-Up
     Artifact
     Repository
     Deployment Pipeline
     Acceptance
     Commit
     Component
     Performance
     System
     Performance
     Staging Env.
     Deployment
     App.
     Production Env.
     Deployment
     App.
     Source
     Repository
     Manual Test Env.
     Deployment
     App.
     

     View Slide

114. Scaling-Up
     Artifact
     Repository
     Deployment Pipeline
     Acceptance
     Commit
     Component
     Performance
     System
     Performance
     Staging Env.
     Deployment
     App.
     Production Env.
     Deployment
     App.
     Source
     Repository
     Manual Test Env.
     Deployment
     App.
     Deployment Pipeline
     Commit
     Manual Test Env.
     Deployment
     App.
     Artifact
     Repository
     Acceptance Acceptance Test
     Environment
     

     View Slide

115. Scaling-Up
     Artifact
     Repository
     Deployment Pipeline
     Acceptance
     Commit
     Component
     Performance
     System
     Performance
     Staging Env.
     Deployment
     App.
     Production Env.
     Deployment
     App.
     Source
     Repository
     Manual Test Env.
     Deployment
     App.
     Deployment Pipeline
     Commit
     Manual Test Env.
     Deployment
     App.
     Artifact
     Repository
     Acceptance Acceptance Test
     Environment
     A
     A
     

     View Slide

116. Scaling-Up
     Artifact
     Repository
     Deployment Pipeline
     Acceptance
     Commit
     Component
     Performance
     System
     Performance
     Staging Env.
     Deployment
     App.
     Production Env.
     Deployment
     App.
     Source
     Repository
     Manual Test Env.
     Deployment
     App.
     Deployment Pipeline
     Commit
     Manual Test Env.
     Deployment
     App.
     Artifact
     Repository
     Acceptance Acceptance Test
     Environment
     A
     A
     

     View Slide

117. Scaling-Up
     Artifact
     Repository
     Deployment Pipeline
     Acceptance
     Commit
     Component
     Performance
     System
     Performance
     Staging Env.
     Deployment
     App.
     Production Env.
     Deployment
     App.
     Source
     Repository
     Manual Test Env.
     Deployment
     App.
     Deployment Pipeline
     Commit
     Manual Test Env.
     Deployment
     App.
     Artifact
     Repository
     Acceptance Acceptance Test
     Environment
     Test Host
     Test Host
     Test Host
     Test Host
     Test Host
     A
     A
     

     View Slide

118. Scaling-Up
     Artifact
     Repository
     Deployment Pipeline
     Acceptance
     Commit
     Component
     Performance
     System
     Performance
     Staging Env.
     Deployment
     App.
     Production Env.
     Deployment
     App.
     Source
     Repository
     Manual Test Env.
     Deployment
     App.
     Deployment Pipeline
     Commit
     Manual Test Env.
     Deployment
     App.
     Artifact
     Repository
     Acceptance
     Acceptance
     Acceptance Test
     Environment
     Test Host
     Test Host
     Test Host
     Test Host
     Test Host
     A
     A
     

     View Slide

119. Anti-Patterns in Acceptance Testing
     

     View Slide

120. Anti-Patterns in Acceptance Testing
     • Don’t use UI Record-and-playback Systems
     

     View Slide

121. Anti-Patterns in Acceptance Testing
     • Don’t use UI Record-and-playback Systems
     • Don’t Record-and-playback production data. This has a role, but it is NOT Acceptance
     Testing
     

     View Slide

122. Anti-Patterns in Acceptance Testing
     • Don’t use UI Record-and-playback Systems
     • Don’t Record-and-playback production data. This has a role, but it is NOT Acceptance
     Testing
     • Don’t dump production data to your test systems, instead define the absolute minimum data
     that you need
     

     View Slide

123. Anti-Patterns in Acceptance Testing
     • Don’t use UI Record-and-playback Systems
     • Don’t Record-and-playback production data. This has a role, but it is NOT Acceptance
     Testing
     • Don’t dump production data to your test systems, instead define the absolute minimum data
     that you need
     • Don’t assume Nasty Automated Testing Products(tm) will do what you need. Be very sceptical
     about them. Start with YOUR strategy and evaluate tools against that.
     

     View Slide

124. Anti-Patterns in Acceptance Testing
     • Don’t use UI Record-and-playback Systems
     • Don’t Record-and-playback production data. This has a role, but it is NOT Acceptance
     Testing
     • Don’t dump production data to your test systems, instead define the absolute minimum data
     that you need
     • Don’t assume Nasty Automated Testing Products(tm) will do what you need. Be very sceptical
     about them. Start with YOUR strategy and evaluate tools against that.
     • Don’t have a separate Testing/QA team! Quality is down to everyone - Developers own
     Acceptance Tests!!!
     

     View Slide

125. Anti-Patterns in Acceptance Testing
     • Don’t use UI Record-and-playback Systems
     • Don’t Record-and-playback production data. This has a role, but it is NOT Acceptance
     Testing
     • Don’t dump production data to your test systems, instead define the absolute minimum data
     that you need
     • Don’t assume Nasty Automated Testing Products(tm) will do what you need. Be very sceptical
     about them. Start with YOUR strategy and evaluate tools against that.
     • Don’t have a separate Testing/QA team! Quality is down to everyone - Developers own
     Acceptance Tests!!!
     • Don’t let every Test start and init the app. Optimise for Cycle-Time, be efficient in your use of
     test environments.
     

     View Slide

126. Anti-Patterns in Acceptance Testing
     • Don’t use UI Record-and-playback Systems
     • Don’t Record-and-playback production data. This has a role, but it is NOT Acceptance
     Testing
     • Don’t dump production data to your test systems, instead define the absolute minimum data
     that you need
     • Don’t assume Nasty Automated Testing Products(tm) will do what you need. Be very sceptical
     about them. Start with YOUR strategy and evaluate tools against that.
     • Don’t have a separate Testing/QA team! Quality is down to everyone - Developers own
     Acceptance Tests!!!
     • Don’t let every Test start and init the app. Optimise for Cycle-Time, be efficient in your use of
     test environments.
     • Don’t include Systems outside of your control in your Acceptance Test Scope
     

     View Slide

127. Anti-Patterns in Acceptance Testing
     • Don’t use UI Record-and-playback Systems
     • Don’t Record-and-playback production data. This has a role, but it is NOT Acceptance
     Testing
     • Don’t dump production data to your test systems, instead define the absolute minimum data
     that you need
     • Don’t assume Nasty Automated Testing Products(tm) will do what you need. Be very sceptical
     about them. Start with YOUR strategy and evaluate tools against that.
     • Don’t have a separate Testing/QA team! Quality is down to everyone - Developers own
     Acceptance Tests!!!
     • Don’t let every Test start and init the app. Optimise for Cycle-Time, be efficient in your use of
     test environments.
     • Don’t include Systems outside of your control in your Acceptance Test Scope
     • Don’t Put ‘wait()’ instructions in your tests hoping it will solve intermittency
     

     View Slide

128. Tricks for Success
     

     View Slide

129. Tricks for Success
     • Do Ensure That Developers Own the Tests
     

     View Slide

130. Tricks for Success
     • Do Ensure That Developers Own the Tests
     • Do Focus Your Tests on “What” not “How”
     

     View Slide

131. Tricks for Success
     • Do Ensure That Developers Own the Tests
     • Do Focus Your Tests on “What” not “How”
     • Do Think of Your Tests as “Executable Specifications”
     

     View Slide

132. Tricks for Success
     • Do Ensure That Developers Own the Tests
     • Do Focus Your Tests on “What” not “How”
     • Do Think of Your Tests as “Executable Specifications”
     • Do Make Acceptance Testing Part of your “Definition of Done”
     

     View Slide

133. Tricks for Success
     • Do Ensure That Developers Own the Tests
     • Do Focus Your Tests on “What” not “How”
     • Do Think of Your Tests as “Executable Specifications”
     • Do Make Acceptance Testing Part of your “Definition of Done”
     • Do Keep Tests Isolated from one-another
     

     View Slide

134. Tricks for Success
     • Do Ensure That Developers Own the Tests
     • Do Focus Your Tests on “What” not “How”
     • Do Think of Your Tests as “Executable Specifications”
     • Do Make Acceptance Testing Part of your “Definition of Done”
     • Do Keep Tests Isolated from one-another
     • Do Keep Your Tests Repeatable
     

     View Slide

135. Tricks for Success
     • Do Ensure That Developers Own the Tests
     • Do Focus Your Tests on “What” not “How”
     • Do Think of Your Tests as “Executable Specifications”
     • Do Make Acceptance Testing Part of your “Definition of Done”
     • Do Keep Tests Isolated from one-another
     • Do Keep Your Tests Repeatable
     • Do Use the Language of the Problem Domain - Do try the DSL approach, whatever your tech.
     

     View Slide

136. Tricks for Success
     • Do Ensure That Developers Own the Tests
     • Do Focus Your Tests on “What” not “How”
     • Do Think of Your Tests as “Executable Specifications”
     • Do Make Acceptance Testing Part of your “Definition of Done”
     • Do Keep Tests Isolated from one-another
     • Do Keep Your Tests Repeatable
     • Do Use the Language of the Problem Domain - Do try the DSL approach, whatever your tech.
     • Do Stub External Systems
     

     View Slide

137. Tricks for Success
     • Do Ensure That Developers Own the Tests
     • Do Focus Your Tests on “What” not “How”
     • Do Think of Your Tests as “Executable Specifications”
     • Do Make Acceptance Testing Part of your “Definition of Done”
     • Do Keep Tests Isolated from one-another
     • Do Keep Your Tests Repeatable
     • Do Use the Language of the Problem Domain - Do try the DSL approach, whatever your tech.
     • Do Stub External Systems
     • Do Test in “Production-Like” Environments
     

     View Slide

138. Tricks for Success
     • Do Ensure That Developers Own the Tests
     • Do Focus Your Tests on “What” not “How”
     • Do Think of Your Tests as “Executable Specifications”
     • Do Make Acceptance Testing Part of your “Definition of Done”
     • Do Keep Tests Isolated from one-another
     • Do Keep Your Tests Repeatable
     • Do Use the Language of the Problem Domain - Do try the DSL approach, whatever your tech.
     • Do Stub External Systems
     • Do Test in “Production-Like” Environments
     • Do Make Instructions Appear Synchronous at the Level of the Test Case
     

     View Slide

139. Tricks for Success
     • Do Ensure That Developers Own the Tests
     • Do Focus Your Tests on “What” not “How”
     • Do Think of Your Tests as “Executable Specifications”
     • Do Make Acceptance Testing Part of your “Definition of Done”
     • Do Keep Tests Isolated from one-another
     • Do Keep Your Tests Repeatable
     • Do Use the Language of the Problem Domain - Do try the DSL approach, whatever your tech.
     • Do Stub External Systems
     • Do Test in “Production-Like” Environments
     • Do Make Instructions Appear Synchronous at the Level of the Test Case
     • Do Test for ANY change
     

     View Slide

140. Tricks for Success
     • Do Ensure That Developers Own the Tests
     • Do Focus Your Tests on “What” not “How”
     • Do Think of Your Tests as “Executable Specifications”
     • Do Make Acceptance Testing Part of your “Definition of Done”
     • Do Keep Tests Isolated from one-another
     • Do Keep Your Tests Repeatable
     • Do Use the Language of the Problem Domain - Do try the DSL approach, whatever your tech.
     • Do Stub External Systems
     • Do Test in “Production-Like” Environments
     • Do Make Instructions Appear Synchronous at the Level of the Test Case
     • Do Test for ANY change
     • Do Keep your Tests Efficient
     

     View Slide

141. Q&A
     http://www.continuous-delivery.co.uk
     Dave Farley
     http://www.davefarley.net
     @davefarley77
     

     View Slide