Advanced Monitoring and Analysis Solution for Performance Testing of Wireless Networks Author: Parijat Bhattacharjee Software Testing Conference 2012 

Target Audience and Takeaway Who can benefit from this topic and how? Testers in Performance Testing Test &Tool The topic will resonate with performance testing engineers who spend a lot of time in analysis of system and tool logs. The topic can help Test and Tool Architects identify targeted areas of improvement in Author: Parijat Bhattacharjee email: [email protected] Advanced Monitoring & Analysis Solution for Performance Testing of Wireless Networks December 2012 2 Test &Tool Architects Managers identify targeted areas of improvement in their test setup and tools. The managers can use the information to identify which features add more value and give higher returns on investment. 

Performance Testing Why, What and How of performance testing What sort of tests are needed? How can the system be tested? Why performance testing? To ensure that the Network Elements meet industry standards Should have five-nines availability Should meet customer specified load requirements five-nines 99.999% availability translates to approximately 5 minutes and 15 seconds of down time in a year! Author: Parijat Bhattacharjee email: [email protected] Advanced Monitoring & Analysis Solution for Performance Testing of Wireless Networks December 2012 3 What sort of tests are needed? Performance testing therefore includes testing of system availabil ity and reliability We therefore run longer load tests such as: Extended Load Tests Peak Traffic Tests How can the system be tested? Can real devices be used? Cost and Availability are two constraints Use of emulators circumvent these issues They are cheaper than real devices They can be created and made ready faster availability reliability Cost Availability emulators 

Emulated Device C Emulated Device A Emulated Device A Emulated Device A Device Under Test Performance Testing What makes a good emulator? Question: Which is more important – an easy mechanism for entering the load profile or an easy mechanism for analyzing the result? Facts: 1. Load profiles are generally modeled by System Engineering and do not change frequently 2. A test may run for days at a time and the Author: Parijat Bhattacharjee email: [email protected] Advanced Monitoring & Analysis Solution for Performance Testing of Wireless Networks December 2012 4 Performance Test Environment Emulated Device B Emulated Device B Emulated Device B Central Controller Answer: Providing better mechanisms for monitoring and analysis of load test results adds greater value than an elaborate GUI for input. 2. A test may run for days at a time and the size of the logs generated is often huge 3. Results need to be analyzed for every test 4. Statistics do not provide a temporal dimension and therefore provide limited information 5. Log files contain information relevant to developers – not end users 

Basic Output Requirements Design Considerations for Load Emulators The new concept is a paradigm shift that turns the older approach on its head in order to generate more value for the same quantum of tool development effort. Recurring costs with original design • Time to validate test results • Time to identify the cause of an issue • Product team support resources • Tools team on support resources • Cycle time detecting and fixing defects Information segregation Information presentation Author: Parijat Bhattacharjee email: [email protected] Advanced Monitoring & Analysis Solution for Performance Testing of Wireless Networks December 2012 5 defects segregation presentation 

Basic Output Requirements New Requirements for the New Paradigm Three laws of emulator output 1. Eliminate the need for users to refer to log files that are meant for developers 2. Provide adequate information to the user to enable analysis of results from the domain perspective Information segregation Author: Parijat Bhattacharjee email: [email protected] Advanced Monitoring & Analysis Solution for Performance Testing of Wireless Networks December 2012 6 3. Provide features that speed up analysis of results and enable user to make inferences and extrapolate Fundamental Constraint The performance (load generation capability) of the tool should not be impacted adversely by these changes. Information presentation 

Event Data Logging Terminology and Definitions and framework An Event (in this context) is defined as “any action performed by the tool that is of significance from a domain perspective”. An EDL Record is a line of pipe- delimited ASCII text terminated by a new line representing a single event. | | | | | | | | | | | … Author: Parijat Bhattacharjee email: [email protected] Advanced Monitoring & Analysis Solution for Performance Testing of Wireless Networks December 2012 7 An EDL log is a file containing EDL records for events that are occurring within the load tool while executing a load profile. Events occur at each Tool instance and the EDL records are generated into EDL logs on the individual machines. These are collected, collated and processed on the EDL server which also hosts a web server to provide a browser based UI for the user. 

Event Data Logging Dataflow Through the Framework Author: Parijat Bhattacharjee email: [email protected] Advanced Monitoring & Analysis Solution for Performance Testing of Wireless Networks December 2012 8 

EDL Benefits Sample output for a 48 hour call load – assume any technology Tester need not look into logs for analysis EDL analysis is easier and analysis time for each test run is reduced EDL provides temporal view of the data The graphs help the tester to easily Author: Parijat Bhattacharjee email: [email protected] Advanced Monitoring & Analysis Solution for Performance Testing of Wireless Networks December 2012 9 Tester can drill down into the EDL data to access the records at the point of failure EDLs contain information relevant to testers; other logs can be correlated with EDLs based on time stamp. pinpoint the time when the problem occurred. 

Thank You Author: Parijat Bhattacharjee email: [email protected] Advanced Monitoring & Analysis Solution for Performance Testing of Wireless Networks December 2012 10 Thank You