Pavel Drobintsev, Vsevolod Kotlyarov, Ivan Selin, Alexey Tolstoles, Nikita Voinov Peter the Great Saint Petersburg Polytechnic University Saint Petersburg, Russia Scalable testing process based on formal models

Model Oriented Approach • Multilevel model of application during design process • Behavioral model is built from requirements • Model is iteratively specified • Test cases are generated from this model in an automated way 03.03.2017 2

VRS/TAT • Software verification and testing toolset • Requirements are formalized in UCM language • VRS verifier runs reachability and consistency checks • Iterative process of model modification until all checks are passed • Generation of symbolic traces • Concretization (symbolic traces – traces with specific values) • Adapting tests to run them on real system • Testing 03.03.2017 3

Use Case Maps (UCM) • High-level graphical language for requirements modelling • ITU Z.151 • Can be used for creating structured behavioral diagrams of interacting agents • UCM notation doesn’t provide enough information to generate traces 03.03.2017 4

UCM Metadata • Auxiliary text field linked to a UCM responsibility • Stores information about variables and signals • Signal is a way of message transfer between agents 03.03.2017 5

VRS Verifier • During the initial model development: • Reachability and consistency checks • During testing process: • Generation of symbolic traces based on UCM model control flow • Concretization (substitution of left, right and middle values from tolerance range) 03.03.2017 6

VRS/TAT Levels of Abstraction 03.03.2017 7 Automated Automated Not automated

Challenges • Data mapping problem: • Abstract models which need to be extended in order to run against real system • Extension may require additional information • New information must comply with what’s already in the model • Amount of generated test cases may be too big to run using common methods 03.03.2017 8

Data Structures Conversion Problem • Structure of concrete test signals does not correspond with real system signals • Manual creation of detailed test scenarios takes too much time and resources 03.03.2017 9

Data Structures Conversion. Approach • Process called “Lowering” • The name comes from descending on lower levels of abstraction • In general, “Lowering” can be described as creating processing rules for each signal and application of these rules to concrete scenarios • An editor was made to restrict user from making incorrect structures 03.03.2017 10

Data Structures Conversion. Restrictions • There are several restrictions to ensure the correctness of test scenarios: • If you separate concrete value into several independent parts, it is prohibited to change them in a way, when joining them back together will give another result than it was before separation • Only structures similar to SUT interfaces could be used • Only constant values from concretization and templates values are allowed 03.03.2017 11

Data Structures Conversion. Results • Before introduction of “lowering”, test mapping was made by hand after each tests generation • After automating the tests mapping there are only 2 manual steps left in VRS/TAT process: • UCM model development • Specifying “lowering” rules • Both of them only needed to be done once 03.03.2017 12

UPD: VRS/TAT Levels of Abstraction 03.03.2017 13 Automated Automated Automated Huge amount

Execution problem • Huge test amount • A lot of time needed for execution • Costs increase • Run tools in parallel 03.03.2017 14

VRS Guided Search • VRS has an option to perform guided search • Guide is a marked sequence on a model, which must be traversed • All traces that don’t apply the criteria to travel through guide are cut off by VRS during traces generation • The idea is to set guides on UCM model and launch several VRS instances with corresponding guides 03.03.2017 15

VRS Guide 03.03.2017 16

Execution process • User sets guides in the most “thick” places of UCM model • Several VRS instances run, each on a different compute node with a unique set of guides • As a result, whole test suite is divided into several parts • Each part can be processed independently in an isolated container (node, VM, etc.) 03.03.2017 17

Scalability • Theoretically, scalability is linear • In real life, it is not achievable because of irregular test distribution and different execution complexity of test cases 03.03.2017 18

Conclusion • Tests are generated from the verified model and can be ran against real systems • Presented approach allows to speed up the testing process • It speeds up both tests generation and tests execution • Scalable process, performance increase is limited by the model and/or available computational resources 03.03.2017 19

Thanks for your attention! 03.03.2017 20

Backup 1. New features in lowering. TDL • There is an ability to use SUT interfaces files (*.tdl) in Lowering Editor to obtain parameters signal structure 03.03.2017 21

Backup 2. Few words on VRS • VRS works with inner model in basic protocols (Hoare triples) • Each one has: • Pre-condition • Process • Post-condition 03.03.2017 22

Backup 2. Few words on VRS • UCM model is converted into basic protocols model using UCM elements location and UCM metadata (and inverse) • VRS matches pre and post conditions of different elements and checks tolerance ranges to find all possible traces • Guided search builds traces from guide forward and backward 03.03.2017 23