MAKE YOUR GLOBAL MARKET EASY www.eglobalmark.com A SURVEY ON MODEL-BASED TESTING TOOLS FOR TEST CASE GENERATION Wenbin Li, Easy Global Market, France

OUTLINE TMPA 2017, Wenbin Li (Easy Global Market) Page 2 Easy Global Market (EGM) ˃ Activities ˃ Projects Model-Based Testing (MBT) ˃ Concept and Workflow ˃ Aspects Survey Criteria ˃ Model Specification ˃ Test Generation ˃ Test Description ˃ Overall Support Survey MBT Tools Future Directions

EASY GLOBAL MARKET: ACTIVITIES TMPA 2017, Wenbin Li (Easy Global Market) Page 3

EU H2020: PHANTOM TMPA 2017, Wenbin Li (Easy Global Market) Page 4 • Multi-Objective Programming for Parallel Computing System Model-based Testing for

MODEL BASED TESTING WORKFLOW TMPA 2017, Wenbin Li (Easy Global Market) Page 5 (1) Creation of MBT Models (2) Generation of Test Cases (3) Concretization of Test Cases (4) Execution of Test Cases (5) Results Analysis

MODEL BASED TESTING ASPECTS TMPA 2017, Wenbin Li (Easy Global Market) Page 6 Three Aspects ˃ Test Case Generation ˃ Test Data Generation ˃ Test Script Generation Survey: ˃ Overall Support ˃ Special Focus on Test Case Gen. Why Test Cases Generation? ˃ Complicated Selection Criteria ˃ Significant Benefits

SURVEY CRITERIA TMPA 2017, Wenbin Li (Easy Global Market) Page 7 “A taxonomy of model-based testing approaches” by Utting, M., Pretschner, A., Legeard, B. Extension ˃ Test Description ˃ Overall Support

SURVEY CRITERIA TMPA 2017, Wenbin Li (Easy Global Market) Page 8

SURVEY CRITERIA: MODEL SPECIFICATION TMPA 2017, Wenbin Li (Easy Global Market) Page 9 Model Specification Scope Characteristics Paradigm Input(I)/Input&Ouput(IO) Untimed(U)/Timed(T) Deterministic(D)/Non-Deterministic(N)/Both(B) Discrete(D)/Continuous(C)/Hybrid(H) Pre-Post Domains Transition Based History Based Functional Operational Stochastic Data Flow

SURVEY CRITERIA: TEST GENERATION TMPA 2017, Wenbin Li (Easy Global Market) Page 10 Test Generation Test Selection Criteria Technology Structural Coverage Data Coverage Requirement Coverage Test Case Specification Random&Stochastic Fault Based Random Generation Search Based Model Checking Symbolic Execution Theorem Proving Constraint Solving Process Online/Offline/Both

SURVEY CRITERIA: TEST DESCRIPTION TMPA 2017, Wenbin Li (Easy Global Market) Page 11 Test Description Test Traceability Test Oracle Format Yes/No Yes/No Testing Languages: TTCN-3, Junit, etc. Programing Languages: Java, C++, etc. Mark Up Languages: XML, HTML, etc. Test Frameworks: Selenium, Robot, etc. Other Formats

SURVEY CRITERIA: OVERALL SUPPORT TMPA 2017, Wenbin Li (Easy Global Market) Page 12 Overall Support Step Coverage Tool Type Open Source(O) Academic(A) Commercial(C) Step 1, 2, 3, 4, 5 Aspect Coverage 1. Test Case Generation 2. Test Data Generation 3. Test Script Generation

SURVEY ON MBT TOOLS: MODELS AND TEST TMPA 2017, Wenbin Li (Easy Global Market) Page 13 Tools Model Specification Test Description Scope Characteristics Paradigm Traceability Oracles Formats DIVERSITY IO U/B/D Transition Based No Yes TTCN-3 FMBT IO U/D/D Pre-Post Domains No No AAL/Python Modbat IO U/B/D Transition based Yes Yes Other formats TCG IO U/B/D Transition based No No Other formats Tcases I U/D/D Pre-Post Domains No No XML, JUnit MISTA IO U/B/D Operational, Transition Based No No Python, HTML, Selenium, etc. MoMuT IO T/D/H Transition Based Yes No Other formats HTG IO T/D/H Transition Based, Functional No No C/C++ Lurette IO T/B/H Functional Yes Yes Lutin VERA IO U/D/D Transition based No Yes XML CompleteTest IO T/D/H Data Flow Yes Yes C CertifyIT IO U/D/D Pre-Post Domains, Transition Based Yes Yes JUnit, HTML, Perl, HP quality center, etc. PragmaDev IO T/B/H Transition-based, History Based Yes Yes TTCN-3, C/C++

SURVEY ON MBT TOOLS: TEST AND OVERALL SUPPORT TMPA 2017, Wenbin Li (Easy Global Market) Page 14 Tools Test Generation Overall Support Test Selection Technology Process Step Aspect Type DIVERSITY Structural Coverage, Random&Stochastic, Test Case Specification Symbolic Execution Offline 1-2 1,2 O FMBT Structural Coverage, Test Case Specification Search Based Both 1-5 1,3 O Modbat Random&Stochastic, Random Generation, Search Based Both 1-5 1 O TCG Structural Coverage, Random&Stochastic Search Based, Random Generation Offline 1-2 1,2 O Tcases Data Coverage Constraint Solving Offline 1-3 1 O MISTA Structural Coverage, Random&Stochastic Search Based, Random Generation Both 1-4 1,3 O MoMuT Fault Based Search Based Offline 1-2 1 A HTG Data Coverage Random Generation, Search Based Offline 1-3 1 A Lurette Random&Stochastic Random Generation Online 1-3 1,2 A VERA Structural Coverage Search based Both 1-5 1,3 A CompleteTest Structural Coverage Model Checking, Search Based Offline 1-3 1 A CertifyIT Structural Coverage, Test Case Specification Search Based, Model Checking Offline 1-3 1,3 C PragmaDev Structural Coverage, Random&Stochastic Symbolic Execution Offline 1-5 1,2 C

DIVERSITY TMPA 2017, Wenbin Li (Easy Global Market) Page 15

SURVEY ON MBT TOOLS: MODELS AND TEST TMPA 2017, Wenbin Li (Easy Global Market) Page 16 Tools Model Specification Test Description Scope Characteristics Paradigm Traceability Oracles Formats DIVERSITY IO U/B/D Transition Based No Yes TTCN-3 FMBT IO U/D/D Pre-Post Domains No No AAL/Python Modbat IO U/B/D Transition based Yes Yes Other formats TCG IO U/B/D Transition based No No Other formats Tcases I U/D/D Pre-Post Domains No No XML, JUnit MISTA IO U/B/D Operational, Transition Based No No Python, HTML, Selenium, etc. MoMuT IO T/D/H Transition Based Yes No Other formats HTG IO T/D/H Transition Based, Functional No No C/C++ Lurette IO T/B/H Functional Yes Yes Lutin VERA IO U/D/D Transition based No Yes XML CompleteTest IO T/D/H Data Flow Yes Yes C CertifyIT IO U/D/D Pre-Post Domains, Transition Based Yes Yes JUnit, HTML, Perl, HP quality center, etc. PragmaDev IO T/B/H Transition-based, History Based Yes Yes TTCN-3, C/C++

SURVEY ON MBT TOOLS: TEST AND OVERALL SUPPORT TMPA 2017, Wenbin Li (Easy Global Market) Page 17 Tools Test Generation Overall Support Test Selection Technology Process Step Aspect Type DIVERSITY Structural Coverage, Random&Stochastic, Test Case Specification Symbolic Execution Offline 1-2 1,2 O FMBT Structural Coverage, Test Case Specification Search Based Both 1-5 1,3 O Modbat Random&Stochastic, Random Generation, Search Based Both 1-5 1 O TCG Structural Coverage, Random&Stochastic Search Based, Random Generation Offline 1-2 1,2 O Tcases Data Coverage Constraint Solving Offline 1-3 1 O MISTA Structural Coverage, Random&Stochastic Search Based, Random Generation Both 1-4 1,3 O MoMuT Fault Based Search Based Offline 1-2 1 A HTG Data Coverage Random Generation, Search Based Offline 1-3 1 A Lurette Random&Stochastic Random Generation Online 1-3 1,2 A VERA Structural Coverage Search based Both 1-5 1,3 A CompleteTest Structural Coverage Model Checking, Search Based Offline 1-3 1 A CertifyIT Structural Coverage, Test Case Specification Search Based, Model Checking Offline 1-3 1,3 C PragmaDev Structural Coverage, Random&Stochastic Symbolic Execution Offline 1-5 1,2 C

FUTURE DIRECTIONS TMPA 2017, Wenbin Li (Easy Global Market) Page 18 Support of online testing for real time systems Generation of test cases in standard testing language Improvement of MBT tool interfaces Record of traceability Automatic generation of test oracle

THANK YOU Tel: E.mail: www.eglobalmark.com • Dr. Wenbin Li • Research Engineer • +33-(0)9 82 33 06 72 • [email protected]