EXTENT-2019: Learning from Failure is not just for Humans

Learning from Failure is not just for Humans
Eugene Litvinenko
QA Project Manager, Exactpro

EXTENT-2019
17 September 2019, London

Video: https://youtu.be/UJNz_9DwtF8

To learn more about Exactpro, visit our website https://exactpro.com/
EXTENT Conference website: https://extent.exactpro.com/
Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro
Facebook https://www.facebook.com/exactpro/
Instagram https://www.instagram.com/exactpro/

Subscribe to Exactpro YouTube channel https://www.youtube.com/c/exactprosystems

Transcript

1. Learning from Failure is not Just
   for Humans
   Eugene Litvinenko,
   QA Project Manager, Exactpro
   

   View Slide

2. Mistakes make us better
   2
   

   View Slide

3. Mistakes make us better
   SUCCESS
   Failure
   Failure
   Failure
   Failure
   Failure
   Failure
   Failure
   3
   

   View Slide

4. Mistakes make us better
   SUCCESS
   Failure
   Failure
   Failure
   Failure
   Failure
   Failure
   Failure
   4
   

   View Slide

5. Silly fish vs. Sailfish
   5
   

   View Slide

6. What is Sailfish?
   Web
   Binary
   FIX
   Market Data
   Database
   6
   

   View Slide

7. What is Sailfish?
   #step #description #user_name #action #message_type SecurityID OrdType Side OrderQty Price Status
   Test Case 1: Trade test case start
   1 Send BUY Order with Qty = 1000, Price = 10 (O1) User_1 send NewOrderSingle 12345 LIMIT BUY 1000 10
   2 Receive Execution Report with Status = NEW (O1) User_1 receive ExecutionReport 12345 LIMIT BUY 1000 10 NEW
   3 Send SELL Order with Qty = 1000, Price = 10 (O2) User_2 send NewOrderSingle 12345 LIMIT SELL 1000 10
   4 Receive Execution Report with Status = FILLED (O2) User_2 receive ExecutionReport 12345 LIMIT SELL 1000 10 FILLED
   5 Receive Execution Report with Status = FILLED (O1) User_1 receive ExecutionReport 12345 LIMIT BUY 1000 10 FILLED
   test case end
   7
   

   View Slide

8. The main actions of Sailfish
   #step #description #user_name #action #message_type SecurityID OrdType Side OrderQty Price Status
   Test Case 1: Trade test case start
   1 Send BUY Order with Qty = 1000, Price = 10 (O1) User_1 send NewOrderSingle 12345 LIMIT BUY 1000 10
   2 Receive Execution Report with Status = NEW (O1) User_1 receive ExecutionReport 12345 LIMIT BUY 1000 10 NEW
   3 Send SELL Order with Qty = 1000, Price = 10 (O2) User_2 send NewOrderSingle 12345 LIMIT SELL 1000 10
   4 Receive Execution Report with Status = FILLED (O2) User_2 receive ExecutionReport 12345 LIMIT SELL 1000 10 FILLED
   5 Receive Execution Report with Status = FILLED (O1) User_1 receive ExecutionReport 12345 LIMIT BUY 1000 10 FILLED
   test case end
   8
   

   View Slide

9. Passed and Failed test steps
   send NewOrderSingle (PASSED) [0.001s]
   - receive ExecutionReport (PASSED) [0.001s]:
   + Input Parameters
   - Verification: Message (PASSED):
   Field Expected Result Actual Result Status
   ... ... ... ...
   OrdType LIMIT LIMIT PASSED
   Status NEW NEW PASSED
   ... ... ... ...
   9
   

   View Slide

10. Passed and Failed test steps
    send NewOrderSingle (PASSED) [0.001s]
    - receive ExecutionReport (FAILED) [0.421s]:
    + Input Parameters
    + Verification: Similar message [1]. Failed/Passed/Conditionally Passed/NA: 4/26/0/6 (FAILED)
    + Verification: Similar message [2]. Failed/Passed/Conditionally Passed/NA: 2/28/0/6 (FAILED)
    + Verification: Similar message [3]. Failed/Passed/Conditionally Passed/NA: 1/29/0/6 (FAILED)
    - Verification: Similar message [4]. Failed/Passed/Conditionally Passed/NA: 1/29/0/6 (FAILED):
    Field Expected Result Actual Result Status
    ... ... ... ...
    OrdType LIMIT LIMIT PASSED
    Status NEW REJECTED FAILED
    ... ... ... ...
    10
    

    View Slide

11. Supervised Machine Learning through
    unpleasant experience
    send NewOrderSingle (PASSED) [0.001s]
    - receive ExecutionReport (FAILED) [0.421s]:
    + Input Parameters
    + Verification: Similar message [1]. Failed/Passed/Conditionally Passed/NA: 4/26/0/6 (FAILED)
    + Verification: Similar message [2]. Failed/Passed/Conditionally Passed/NA: 2/28/0/6 (FAILED)
    + Verification: Similar message [3]. Failed/Passed/Conditionally Passed/NA: 1/29/0/6 (FAILED)
    - Verification: Similar message [4]. Failed/Passed/Conditionally Passed/NA: 1/29/0/6 (FAILED):
    Field Expected Result Actual Result Status
    ... ... ... ...
    OrdType LIMIT LIMIT PASSED
    Status NEW REJECTED FAILED
    ... ... ... ...
    11
    

    View Slide

12. Supervised Machine Learning through
    unpleasant experience
    send NewOrderSingle (PASSED) [0.001s]
    - receive ExecutionReport (FAILED) [0.421s]:
    + Input Parameters
    - Verification: Similar message [1]. Failed/Passed/Conditionally Passed/NA: 1/29/0/6 (FAILED): 89%
    + Verification: Similar message [2]. Failed/Passed/Conditionally Passed/NA: 4/26/0/6 (FAILED) 81%
    + Verification: Similar message [3]. Failed/Passed/Conditionally Passed/NA: 2/28/0/6 (FAILED) 54%
    + Verification: Similar message [4]. Failed/Passed/Conditionally Passed/NA: 1/29/0/6 (FAILED) 42%
    Field Expected Result Actual Result Status
    ... ... ... ...
    OrdType LIMIT LIMIT PASSED
    Status NEW REJECTED FAILED
    ... ... ... ...
    12
    

    View Slide

13. Supervised Machine Learning: going forward
    tens of thousands of tests
    thousands of raised bugs
    13
    

    View Slide

14. Supervised Machine Learning: going forward
    This test step failed
    with the following
    results...
    This test step failed
    with the following
    results…
    And the reason is...
    14
    

    View Slide

15. Thank you!
    15
    

    View Slide