Technology and Tools for Developing Industrial Software Test Suites Based on Formal Models and Implementing Scalable Testing Process on Supercomputer

TMPA-2017: Tools and Methods of Program Analysis
3-4 March, 2017, Hotel Holiday Inn Moscow Vinogradovo, Moscow

Technology and Tools for Developing Industrial Software Test Suites Based on Formal Models and Implementing Scalable Testing Process on Supercomputer
Vsevolod Kotlyarov, Pavel Drobintsev, Nikita Voinov, Ivan Selin, Alexey Tolstoles, St.Petersburg Polytechnic University

For video follow the link: https://youtu.be/H2wWJSke9CY
Would like to know more?
Visit our website:
www.tmpaconf.org
www.exactprosystems.com/events/tmpa

Follow us:
https://www.linkedin.com/company/exactpro-systems-llc?trk=biz-companies-cym
https://twitter.com/exactpro

Transcript

1. Pavel Drobintsev, Vsevolod Kotlyarov, Ivan Selin, Alexey Tolstoles, Nikita Voinov Peter the Great Saint Petersburg Polytechnic University Saint Petersburg, Russia Scalable testing process based on formal models
2. Model Oriented Approach • Multilevel model of application during design process • Behavioral model is built from requirements • Model is iteratively specified • Test cases are generated from this model in an automated way 03.03.2017 2
3. VRS/TAT • Software verification and testing toolset • Requirements are formalized in UCM language • VRS verifier runs reachability and consistency checks • Iterative process of model modification until all checks are passed • Generation of symbolic traces • Concretization (symbolic traces – traces with specific values) • Adapting tests to run them on real system • Testing 03.03.2017 3
4. Use Case Maps (UCM) • High-level graphical language for requirements modelling • ITU Z.151 • Can be used for creating structured behavioral diagrams of interacting agents • UCM notation doesn’t provide enough information to generate traces 03.03.2017 4
5. UCM Metadata • Auxiliary text field linked to a UCM responsibility • Stores information about variables and signals • Signal is a way of message transfer between agents 03.03.2017 5
6. VRS Verifier • During the initial model development: • Reachability and consistency checks • During testing process: • Generation of symbolic traces based on UCM model control flow • Concretization (substitution of left, right and middle values from tolerance range) 03.03.2017 6
7. VRS/TAT Levels of Abstraction 03.03.2017 7 Automated Automated Not automated
8. Challenges • Data mapping problem: • Abstract models which need to be extended in order to run against real system • Extension may require additional information • New information must comply with what’s already in the model • Amount of generated test cases may be too big to run using common methods 03.03.2017 8
9. Data Structures Conversion Problem • Structure of concrete test signals does not correspond with real system signals • Manual creation of detailed test scenarios takes too much time and resources 03.03.2017 9
10. Data Structures Conversion. Approach • Process called “Lowering” • The name comes from descending on lower levels of abstraction • In general, “Lowering” can be described as creating processing rules for each signal and application of these rules to concrete scenarios • An editor was made to restrict user from making incorrect structures 03.03.2017 10
11. Data Structures Conversion. Restrictions • There are several restrictions to ensure the correctness of test scenarios: • If you separate concrete value into several independent parts, it is prohibited to change them in a way, when joining them back together will give another result than it was before separation • Only structures similar to SUT interfaces could be used • Only constant values from concretization and templates values are allowed 03.03.2017 11
12. Data Structures Conversion. Results • Before introduction of “lowering”, test mapping was made by hand after each tests generation • After automating the tests mapping there are only 2 manual steps left in VRS/TAT process: • UCM model development • Specifying “lowering” rules • Both of them only needed to be done once 03.03.2017 12
13. UPD: VRS/TAT Levels of Abstraction 03.03.2017 13 Automated Automated Automated Huge amount
14. Execution problem • Huge test amount • A lot of time needed for execution • Costs increase • Run tools in parallel 03.03.2017 14
15. VRS Guided Search • VRS has an option to perform guided search • Guide is a marked sequence on a model, which must be traversed • All traces that don’t apply the criteria to travel through guide are cut off by VRS during traces generation • The idea is to set guides on UCM model and launch several VRS instances with corresponding guides 03.03.2017 15
16. VRS Guide 03.03.2017 16
17. Execution process • User sets guides in the most “thick” places of UCM model • Several VRS instances run, each on a different compute node with a unique set of guides • As a result, whole test suite is divided into several parts • Each part can be processed independently in an isolated container (node, VM, etc.) 03.03.2017 17
18. Scalability • Theoretically, scalability is linear • In real life, it is not achievable because of irregular test distribution and different execution complexity of test cases 03.03.2017 18
19. Conclusion • Tests are generated from the verified model and can be ran against real systems • Presented approach allows to speed up the testing process • It speeds up both tests generation and tests execution • Scalable process, performance increase is limited by the model and/or available computational resources 03.03.2017 19
20. Thanks for your attention! 03.03.2017 20
21. Backup 1. New features in lowering. TDL • There is an ability to use SUT interfaces files (*.tdl) in Lowering Editor to obtain parameters signal structure 03.03.2017 21
22. Backup 2. Few words on VRS • VRS works with inner model in basic protocols (Hoare triples) • Each one has: • Pre-condition • Process • Post-condition 03.03.2017 22
23. Backup 2. Few words on VRS • UCM model is converted into basic protocols model using UCM elements location and UCM metadata (and inverse) • VRS matches pre and post conditions of different elements and checks tolerance ranges to find all possible traces • Guided search builds traces from guide forward and backward 03.03.2017 23