TMPA-2017: A Survey on Model-Based Testing Tools for Test Case Generation

TMPA-2017: Tools and Methods of Program Analysis
3-4 March, 2017, Hotel Holiday Inn Moscow Vinogradovo, Moscow

A Survey on Model-Based Testing Tools for Test Case Generation
Wenbin Li, Franck Le Gall, Naum Spaseski, Easy Global Market

For video follow the link: https://youtu.be/Q5wvOX5cdrM
Would like to know more?
Visit our website:
www.tmpaconf.org
www.exactprosystems.com/events/tmpa

Follow us:
https://www.linkedin.com/company/exactpro-systems-llc?trk=biz-companies-cym
https://twitter.com/exactpro

Transcript

1. MAKE YOUR GLOBAL MARKET EASY www.eglobalmark.com A SURVEY ON MODEL-BASED TESTING TOOLS FOR TEST CASE GENERATION Wenbin Li, Easy Global Market, France
2. OUTLINE TMPA 2017, Wenbin Li (Easy Global Market) Page 2 Easy Global Market (EGM) ˃ Activities ˃ Projects Model-Based Testing (MBT) ˃ Concept and Workflow ˃ Aspects Survey Criteria ˃ Model Specification ˃ Test Generation ˃ Test Description ˃ Overall Support Survey MBT Tools Future Directions
3. EASY GLOBAL MARKET: ACTIVITIES TMPA 2017, Wenbin Li (Easy Global Market) Page 3
4. EU H2020: PHANTOM TMPA 2017, Wenbin Li (Easy Global Market) Page 4 • Multi-Objective Programming for Parallel Computing System Model-based Testing for
5. MODEL BASED TESTING WORKFLOW TMPA 2017, Wenbin Li (Easy Global Market) Page 5 (1) Creation of MBT Models (2) Generation of Test Cases (3) Concretization of Test Cases (4) Execution of Test Cases (5) Results Analysis
6. MODEL BASED TESTING ASPECTS TMPA 2017, Wenbin Li (Easy Global Market) Page 6 Three Aspects ˃ Test Case Generation ˃ Test Data Generation ˃ Test Script Generation Survey: ˃ Overall Support ˃ Special Focus on Test Case Gen. Why Test Cases Generation? ˃ Complicated Selection Criteria ˃ Significant Benefits
7. SURVEY CRITERIA TMPA 2017, Wenbin Li (Easy Global Market) Page 7 “A taxonomy of model-based testing approaches” by Utting, M., Pretschner, A., Legeard, B. Extension ˃ Test Description ˃ Overall Support
8. SURVEY CRITERIA TMPA 2017, Wenbin Li (Easy Global Market) Page 8
9. SURVEY CRITERIA: MODEL SPECIFICATION TMPA 2017, Wenbin Li (Easy Global Market) Page 9 Model Specification Scope Characteristics Paradigm Input(I)/Input&Ouput(IO) Untimed(U)/Timed(T) Deterministic(D)/Non-Deterministic(N)/Both(B) Discrete(D)/Continuous(C)/Hybrid(H) Pre-Post Domains Transition Based History Based Functional Operational Stochastic Data Flow
10. SURVEY CRITERIA: TEST GENERATION TMPA 2017, Wenbin Li (Easy Global Market) Page 10 Test Generation Test Selection Criteria Technology Structural Coverage Data Coverage Requirement Coverage Test Case Specification Random&Stochastic Fault Based Random Generation Search Based Model Checking Symbolic Execution Theorem Proving Constraint Solving Process Online/Offline/Both
11. SURVEY CRITERIA: TEST DESCRIPTION TMPA 2017, Wenbin Li (Easy Global Market) Page 11 Test Description Test Traceability Test Oracle Format Yes/No Yes/No Testing Languages: TTCN-3, Junit, etc. Programing Languages: Java, C++, etc. Mark Up Languages: XML, HTML, etc. Test Frameworks: Selenium, Robot, etc. Other Formats
12. SURVEY CRITERIA: OVERALL SUPPORT TMPA 2017, Wenbin Li (Easy Global Market) Page 12 Overall Support Step Coverage Tool Type Open Source(O) Academic(A) Commercial(C) Step 1, 2, 3, 4, 5 Aspect Coverage 1. Test Case Generation 2. Test Data Generation 3. Test Script Generation
13. SURVEY ON MBT TOOLS: MODELS AND TEST TMPA 2017, Wenbin Li (Easy Global Market) Page 13 Tools Model Specification Test Description Scope Characteristics Paradigm Traceability Oracles Formats DIVERSITY IO U/B/D Transition Based No Yes TTCN-3 FMBT IO U/D/D Pre-Post Domains No No AAL/Python Modbat IO U/B/D Transition based Yes Yes Other formats TCG IO U/B/D Transition based No No Other formats Tcases I U/D/D Pre-Post Domains No No XML, JUnit MISTA IO U/B/D Operational, Transition Based No No Python, HTML, Selenium, etc. MoMuT IO T/D/H Transition Based Yes No Other formats HTG IO T/D/H Transition Based, Functional No No C/C++ Lurette IO T/B/H Functional Yes Yes Lutin VERA IO U/D/D Transition based No Yes XML CompleteTest IO T/D/H Data Flow Yes Yes C CertifyIT IO U/D/D Pre-Post Domains, Transition Based Yes Yes JUnit, HTML, Perl, HP quality center, etc. PragmaDev IO T/B/H Transition-based, History Based Yes Yes TTCN-3, C/C++
14. SURVEY ON MBT TOOLS: TEST AND OVERALL SUPPORT TMPA 2017, Wenbin Li (Easy Global Market) Page 14 Tools Test Generation Overall Support Test Selection Technology Process Step Aspect Type DIVERSITY Structural Coverage, Random&Stochastic, Test Case Specification Symbolic Execution Offline 1-2 1,2 O FMBT Structural Coverage, Test Case Specification Search Based Both 1-5 1,3 O Modbat Random&Stochastic, Random Generation, Search Based Both 1-5 1 O TCG Structural Coverage, Random&Stochastic Search Based, Random Generation Offline 1-2 1,2 O Tcases Data Coverage Constraint Solving Offline 1-3 1 O MISTA Structural Coverage, Random&Stochastic Search Based, Random Generation Both 1-4 1,3 O MoMuT Fault Based Search Based Offline 1-2 1 A HTG Data Coverage Random Generation, Search Based Offline 1-3 1 A Lurette Random&Stochastic Random Generation Online 1-3 1,2 A VERA Structural Coverage Search based Both 1-5 1,3 A CompleteTest Structural Coverage Model Checking, Search Based Offline 1-3 1 A CertifyIT Structural Coverage, Test Case Specification Search Based, Model Checking Offline 1-3 1,3 C PragmaDev Structural Coverage, Random&Stochastic Symbolic Execution Offline 1-5 1,2 C
15. DIVERSITY TMPA 2017, Wenbin Li (Easy Global Market) Page 15
16. SURVEY ON MBT TOOLS: MODELS AND TEST TMPA 2017, Wenbin Li (Easy Global Market) Page 16 Tools Model Specification Test Description Scope Characteristics Paradigm Traceability Oracles Formats DIVERSITY IO U/B/D Transition Based No Yes TTCN-3 FMBT IO U/D/D Pre-Post Domains No No AAL/Python Modbat IO U/B/D Transition based Yes Yes Other formats TCG IO U/B/D Transition based No No Other formats Tcases I U/D/D Pre-Post Domains No No XML, JUnit MISTA IO U/B/D Operational, Transition Based No No Python, HTML, Selenium, etc. MoMuT IO T/D/H Transition Based Yes No Other formats HTG IO T/D/H Transition Based, Functional No No C/C++ Lurette IO T/B/H Functional Yes Yes Lutin VERA IO U/D/D Transition based No Yes XML CompleteTest IO T/D/H Data Flow Yes Yes C CertifyIT IO U/D/D Pre-Post Domains, Transition Based Yes Yes JUnit, HTML, Perl, HP quality center, etc. PragmaDev IO T/B/H Transition-based, History Based Yes Yes TTCN-3, C/C++
17. SURVEY ON MBT TOOLS: TEST AND OVERALL SUPPORT TMPA 2017, Wenbin Li (Easy Global Market) Page 17 Tools Test Generation Overall Support Test Selection Technology Process Step Aspect Type DIVERSITY Structural Coverage, Random&Stochastic, Test Case Specification Symbolic Execution Offline 1-2 1,2 O FMBT Structural Coverage, Test Case Specification Search Based Both 1-5 1,3 O Modbat Random&Stochastic, Random Generation, Search Based Both 1-5 1 O TCG Structural Coverage, Random&Stochastic Search Based, Random Generation Offline 1-2 1,2 O Tcases Data Coverage Constraint Solving Offline 1-3 1 O MISTA Structural Coverage, Random&Stochastic Search Based, Random Generation Both 1-4 1,3 O MoMuT Fault Based Search Based Offline 1-2 1 A HTG Data Coverage Random Generation, Search Based Offline 1-3 1 A Lurette Random&Stochastic Random Generation Online 1-3 1,2 A VERA Structural Coverage Search based Both 1-5 1,3 A CompleteTest Structural Coverage Model Checking, Search Based Offline 1-3 1 A CertifyIT Structural Coverage, Test Case Specification Search Based, Model Checking Offline 1-3 1,3 C PragmaDev Structural Coverage, Random&Stochastic Symbolic Execution Offline 1-5 1,2 C
18. FUTURE DIRECTIONS TMPA 2017, Wenbin Li (Easy Global Market) Page 18 Support of online testing for real time systems Generation of test cases in standard testing language Improvement of MBT tool interfaces Record of traceability Automatic generation of test oracle
19. THANK YOU Tel: E.mail: www.eglobalmark.com • Dr. Wenbin Li • Research Engineer • +33-(0)9 82 33 06 72 • wenbin.li@eglobalmark.com