A regression testing with semi-automatic test selection for auditing of IMS database Alexey Ruchay 1,2 Ivan Kliavin 2 Tatiana Kotova 3 Julia Ivanova 2 1 Chelyabinsk state university, Chelyabinsk, Russia, 2 Applied technologies, Chelyabinsk, Russia, 3 Rocket software, Waltham, USA, {aruchay,ikliavin,tryzhova,jivanova}@rocketsoftware.com TOOLS AND METHODS OF PROGRAM ANALYSIS March, 3th, Moscow

Regression testing of audit system for IMS database on z/OS Regression testing is important for checking that software changes did not break previously working functionality. However, regression testing is costly as it runs many tests for many revisions. Our manual regression testing requires 3 months to run. For testing of audit system for IMS database we use 50 online z/OS jobs and 50 batch z/OS jobs with 1000 transactions per second, 10 stress testing scripts with 100 000 transactions per second, 1000 IMS LOG transactions and 3000 SMF transactions for simulation of data access information from IMS database. For our testing there are 120 automated test, 350 test cases, 4 versions of audit system for IMS database, 4 versions of IMS databases, and 3 z/OS subsytems. Duration of an automated test is about 10-20 minutes. One automated test has about 100 000 transactions.

Test Automation Framework

Our framework satisfies A regression testing requires 3 weeks to run by our test automation framework instead of 3 months of manual testing. • High Level Requirements. The framework executes test cases automatically, is easy for using without programming skills, simple for support, doesn’t need permanent control while executing tests. It allows to run tests manually or schedule automatic execution at predefined time or after certain events. Non-fatal errors caused by test environment are handled and reported without stopping test execution. Test results are verified. Every executed test case is marked as Passed or Failed and failed test cases have a short error message. Test execution is logged using different configurable logging levels. Test report is created and published automatically. • Ease of Use. The framework uses keyword driven approach, supports creating user keywords. It provides functionality that selects and groups tests for particular task. • Maintainability. The framework is modular and has coding and naming conventions. It is implemented using high level scripting languages. The testware in the framework is under version control (Git) and is adequately documented.

Technologies that we used for developing Test Automation Framework Programming languages, Frameworks: • Java — basic functionality. • JavaScript, PHP, HTML, Qooxdoo (JS framework) — web interface. Java Libraries • S3270 — for integration TN3270 system on z/OS. • Selenium WebDriver — for browser instance. • Ganymed SSH-2 — implements the SSH-2 protocol. • Apache Commons Net — library implements the client side of many basic Internet protocols. • Apache FreeMarker — library to generate text output based on templates and changing data. • Jsoup — library for working with real-world HTML. • Gson — library to serialize and deserialize Java objects to JSON. • Apache log4j — logging utility. • httpclient — package that implements the client side of the most recent HTTP standards and recommendations.

How we can select sequence of tests for regression testing. Regression test selection • Retest-All technique. • Random/Ad-Hoc technique. • Dataflow technique. • Test Case Prioritization technique. • Regression Test Minimization technique.

Semi-automatic test selection. Criteria and rules • Total efforts, the number of required tests, • Required time for tests, total time execution, • Required number of particular test executions, • Availability of necessary IMS databases, z/OS subsystems, • Priority of tests; • Results of previous tests.

Criteria j for any test case ti Consider a test suite T containing n test cases, {t1 , t2 , . . . , tn }. For any test case ti , define set of attributes ai ⊆ A. Let A represent the set of the functionalities of auditing IMS database that is manually specified by tester. For any test case ti , let pi = ∏ k j=1 tj i denote priority of using test cases, where tj i — assessment of factor of using test case ti with the criteria j. • t1 = {0, 1} — factor of availability of necessary z/OS subsystem, version of audit system for IMS database, version of IMS databases. t1 = 1 is available, and t1 = 0 is not available. • t2 = [1, 10] — priority of test. Tester sets this factor for each test case ti . For example, t2 = 10 for high priority. • t3 = [1, d] — factor of result of previous tests. This factor is calculated for all previous test executions. d is a number of program failures revealed by the test cases ti . • t4 = [1, 20] — required number of test executions. Tester sets this factor for each test cases ti . For example, t4 = 10 for jobs with high risk of failure. • t5 = [0, 10] — factor of specification requirements based on information from developers. • t6 = [1, 10] — factor of execution time of tests. For example, t6 = 1 for long and t6 = 10 for short test case.

An algorithm for priority of using test cases • Specify z/OS subsystem, version of audit system for IMS database on z/OS, version of IMS databases. • Specify criteria tj (some of them should be specified manually). • Specify subset T′ of test cases in accordance with subset A manually (optional, not necessary for automatic test selection). • Estimate priority of using test cases {p1 , . . . , pn } by evaluating all criteria tj. • Order (sort) test cases ti in subset T′ according to the value of pi .

Conclusion • The presented test automation framework satisfies most important lower level requirements in terms of ease-of-use and maintainability. • Regression testing with our approach requires 1 day for smoke testing and 1 weeks for detailed regression testing to meet the requirements of our management and customers. • Semi-automatic test selection approach allows more detailed manual configuration of regression test sequence. • Our approach provides a significant reduction of regression testing time, balances the time for the analysis, collection and the execution phases, rather than focusing just on reducing the number of selected tests. • The execution experience was positive and the framework successfully implemented for testing of IMS database auditing system on z/OS.

Thank you for attention!