EXTENT-2019: Learning from Failure is not just for Humans

Learning from Failure is not just for Humans
Eugene Litvinenko
QA Project Manager, Exactpro

EXTENT-2019
17 September 2019, London

Video: https://youtu.be/UJNz_9DwtF8

To learn more about Exactpro, visit our website https://exactpro.com/
EXTENT Conference website: https://extent.exactpro.com/
Follow us on
LinkedIn https://www.linkedin.com/company/exactpro-systems-llc
Twitter https://twitter.com/exactpro
Facebook https://www.facebook.com/exactpro/
Instagram https://www.instagram.com/exactpro/

Subscribe to Exactpro YouTube channel https://www.youtube.com/c/exactprosystems

Transcript

1. Learning from Failure is not Just for Humans Eugene Litvinenko, QA Project Manager, Exactpro
2. Mistakes make us better 2
3. Mistakes make us better SUCCESS Failure Failure Failure Failure Failure Failure Failure 3
4. Mistakes make us better SUCCESS Failure Failure Failure Failure Failure Failure Failure 4
5. Silly fish vs. Sailfish 5
6. What is Sailfish? Web Binary FIX Market Data Database 6
7. What is Sailfish? #step #description #user_name #action #message_type SecurityID OrdType Side OrderQty Price Status Test Case 1: Trade test case start 1 Send BUY Order with Qty = 1000, Price = 10 (O1) User_1 send NewOrderSingle 12345 LIMIT BUY 1000 10 2 Receive Execution Report with Status = NEW (O1) User_1 receive ExecutionReport 12345 LIMIT BUY 1000 10 NEW 3 Send SELL Order with Qty = 1000, Price = 10 (O2) User_2 send NewOrderSingle 12345 LIMIT SELL 1000 10 4 Receive Execution Report with Status = FILLED (O2) User_2 receive ExecutionReport 12345 LIMIT SELL 1000 10 FILLED 5 Receive Execution Report with Status = FILLED (O1) User_1 receive ExecutionReport 12345 LIMIT BUY 1000 10 FILLED test case end 7
8. The main actions of Sailfish #step #description #user_name #action #message_type SecurityID OrdType Side OrderQty Price Status Test Case 1: Trade test case start 1 Send BUY Order with Qty = 1000, Price = 10 (O1) User_1 send NewOrderSingle 12345 LIMIT BUY 1000 10 2 Receive Execution Report with Status = NEW (O1) User_1 receive ExecutionReport 12345 LIMIT BUY 1000 10 NEW 3 Send SELL Order with Qty = 1000, Price = 10 (O2) User_2 send NewOrderSingle 12345 LIMIT SELL 1000 10 4 Receive Execution Report with Status = FILLED (O2) User_2 receive ExecutionReport 12345 LIMIT SELL 1000 10 FILLED 5 Receive Execution Report with Status = FILLED (O1) User_1 receive ExecutionReport 12345 LIMIT BUY 1000 10 FILLED test case end 8
9. Passed and Failed test steps send NewOrderSingle (PASSED) [0.001s] - receive ExecutionReport (PASSED) [0.001s]: + Input Parameters - Verification: Message (PASSED): Field Expected Result Actual Result Status ... ... ... ... OrdType LIMIT LIMIT PASSED Status NEW NEW PASSED ... ... ... ... 9
10. Passed and Failed test steps send NewOrderSingle (PASSED) [0.001s] - receive ExecutionReport (FAILED) [0.421s]: + Input Parameters + Verification: Similar message [1]. Failed/Passed/Conditionally Passed/NA: 4/26/0/6 (FAILED) + Verification: Similar message [2]. Failed/Passed/Conditionally Passed/NA: 2/28/0/6 (FAILED) + Verification: Similar message [3]. Failed/Passed/Conditionally Passed/NA: 1/29/0/6 (FAILED) - Verification: Similar message [4]. Failed/Passed/Conditionally Passed/NA: 1/29/0/6 (FAILED): Field Expected Result Actual Result Status ... ... ... ... OrdType LIMIT LIMIT PASSED Status NEW REJECTED FAILED ... ... ... ... 10
11. Supervised Machine Learning through unpleasant experience send NewOrderSingle (PASSED) [0.001s] - receive ExecutionReport (FAILED) [0.421s]: + Input Parameters + Verification: Similar message [1]. Failed/Passed/Conditionally Passed/NA: 4/26/0/6 (FAILED) + Verification: Similar message [2]. Failed/Passed/Conditionally Passed/NA: 2/28/0/6 (FAILED) + Verification: Similar message [3]. Failed/Passed/Conditionally Passed/NA: 1/29/0/6 (FAILED) - Verification: Similar message [4]. Failed/Passed/Conditionally Passed/NA: 1/29/0/6 (FAILED): Field Expected Result Actual Result Status ... ... ... ... OrdType LIMIT LIMIT PASSED Status NEW REJECTED FAILED ... ... ... ... 11
12. Supervised Machine Learning through unpleasant experience send NewOrderSingle (PASSED) [0.001s] - receive ExecutionReport (FAILED) [0.421s]: + Input Parameters - Verification: Similar message [1]. Failed/Passed/Conditionally Passed/NA: 1/29/0/6 (FAILED): 89% + Verification: Similar message [2]. Failed/Passed/Conditionally Passed/NA: 4/26/0/6 (FAILED) 81% + Verification: Similar message [3]. Failed/Passed/Conditionally Passed/NA: 2/28/0/6 (FAILED) 54% + Verification: Similar message [4]. Failed/Passed/Conditionally Passed/NA: 1/29/0/6 (FAILED) 42% Field Expected Result Actual Result Status ... ... ... ... OrdType LIMIT LIMIT PASSED Status NEW REJECTED FAILED ... ... ... ... 12
13. Supervised Machine Learning: going forward tens of thousands of tests thousands of raised bugs 13
14. Supervised Machine Learning: going forward This test step failed with the following results... This test step failed with the following results… And the reason is... 14
15. Thank you! 15