Advanced Monitoring and Analysis Methods for Performance Testing of Wireless Networks

This presentation is among the Top 27 Best Papers/Practice/Tutorials selected, out of 460+ submissions received, to be presented @STC 2012.

Presentation Abstract

One of the important aspects of testing various Wireless Networks is verifying their ability to handle peak traffic and high loads over extended durations. Test engineers often use emulators to load the real elements with a large number of calls or data sessions.

Given the extended duration of the tests and the number of calls generated, analyzing the results of a load test can be a daunting task. The established mechanism for detecting and analyzing failures is based on statistics. Though statistics provide a snapshot of the system to the tester, analysis of the problem is extremely difficult since the statistics provide no time reference.
To analyze a failure, testers then need to fall back to the analysis of product and emulator logs. While this provides additional information such as the time reference, they are usually huge and complex to read. Therefore, to understand the logs, the tester needs to contact the tools and the product teams for support, adding to the cycle time for execution of load test cases.

This paper discusses the possibility of developing an approach that allows the users to monitor the test setup using graphical analysis tools that are built into the emulated components. This is shown to provide a number of advantages such as providing a time reference and freeing the tester from the need to analyze log files in order to ascertain the nature of a problem.

About the Author

Parijat Bhattacharjee is currently working as a R&D engineer with Nokia Siemens Networks Pvt. Ltd. Prior to this he was working with Motorola Solutions India Pvt. Ltd. as a Senior Staff Engineer at their Tools - Center of Excellence. His total experience in telecommunications software is about 12 years and during this time, he has been closely involved in the architecture, design and development of performance evaluation solutions for various wireless technologies.

Transcript

1. Advanced Monitoring and Analysis Solution for Performance Testing of Wireless Networks Author: Parijat Bhattacharjee Software Testing Conference 2012
2. Target Audience and Takeaway Who can benefit from this topic and how? Testers in Performance Testing Test &Tool The topic will resonate with performance testing engineers who spend a lot of time in analysis of system and tool logs. The topic can help Test and Tool Architects identify targeted areas of improvement in Author: Parijat Bhattacharjee email: parijat.bhattacharjee@nsn.com Advanced Monitoring & Analysis Solution for Performance Testing of Wireless Networks December 2012 2 Test &Tool Architects Managers identify targeted areas of improvement in their test setup and tools. The managers can use the information to identify which features add more value and give higher returns on investment.
3. Performance Testing Why, What and How of performance testing What sort of tests are needed? How can the system be tested? Why performance testing? To ensure that the Network Elements meet industry standards Should have five-nines availability Should meet customer specified load requirements five-nines 99.999% availability translates to approximately 5 minutes and 15 seconds of down time in a year! Author: Parijat Bhattacharjee email: parijat.bhattacharjee@nsn.com Advanced Monitoring & Analysis Solution for Performance Testing of Wireless Networks December 2012 3 What sort of tests are needed? Performance testing therefore includes testing of system availabil ity and reliability We therefore run longer load tests such as: Extended Load Tests Peak Traffic Tests How can the system be tested? Can real devices be used? Cost and Availability are two constraints Use of emulators circumvent these issues They are cheaper than real devices They can be created and made ready faster availability reliability Cost Availability emulators
4. Emulated Device C Emulated Device A Emulated Device A Emulated Device A Device Under Test Performance Testing What makes a good emulator? Question: Which is more important – an easy mechanism for entering the load profile or an easy mechanism for analyzing the result? Facts: 1. Load profiles are generally modeled by System Engineering and do not change frequently 2. A test may run for days at a time and the Author: Parijat Bhattacharjee email: parijat.bhattacharjee@nsn.com Advanced Monitoring & Analysis Solution for Performance Testing of Wireless Networks December 2012 4 Performance Test Environment Emulated Device B Emulated Device B Emulated Device B Central Controller Answer: Providing better mechanisms for monitoring and analysis of load test results adds greater value than an elaborate GUI for input. 2. A test may run for days at a time and the size of the logs generated is often huge 3. Results need to be analyzed for every test 4. Statistics do not provide a temporal dimension and therefore provide limited information 5. Log files contain information relevant to developers – not end users
5. Basic Output Requirements Design Considerations for Load Emulators The new concept is a paradigm shift that turns the older approach on its head in order to generate more value for the same quantum of tool development effort. Recurring costs with original design • Time to validate test results • Time to identify the cause of an issue • Product team support resources • Tools team on support resources • Cycle time detecting and fixing defects Information segregation Information presentation Author: Parijat Bhattacharjee email: parijat.bhattacharjee@nsn.com Advanced Monitoring & Analysis Solution for Performance Testing of Wireless Networks December 2012 5 defects segregation presentation
6. Basic Output Requirements New Requirements for the New Paradigm Three laws of emulator output 1. Eliminate the need for users to refer to log files that are meant for developers 2. Provide adequate information to the user to enable analysis of results from the domain perspective Information segregation Author: Parijat Bhattacharjee email: parijat.bhattacharjee@nsn.com Advanced Monitoring & Analysis Solution for Performance Testing of Wireless Networks December 2012 6 3. Provide features that speed up analysis of results and enable user to make inferences and extrapolate Fundamental Constraint The performance (load generation capability) of the tool should not be impacted adversely by these changes. Information presentation
7. Event Data Logging Terminology and Definitions and framework An Event (in this context) is defined as “any action performed by the tool that is of significance from a domain perspective”. An EDL Record is a line of pipe- delimited ASCII text terminated by a new line representing a single event. <Dbg Lvl> | <Tool Id> | <Seq Num> | <Sess id> | <CCB id> | <Start time> | <End time> | <Class> | <Type> | <Result> | <Cause> | … Author: Parijat Bhattacharjee email: parijat.bhattacharjee@nsn.com Advanced Monitoring & Analysis Solution for Performance Testing of Wireless Networks December 2012 7 An EDL log is a file containing EDL records for events that are occurring within the load tool while executing a load profile. Events occur at each Tool instance and the EDL records are generated into EDL logs on the individual machines. These are collected, collated and processed on the EDL server which also hosts a web server to provide a browser based UI for the user.
8. Event Data Logging Dataflow Through the Framework Author: Parijat Bhattacharjee email: parijat.bhattacharjee@nsn.com Advanced Monitoring & Analysis Solution for Performance Testing of Wireless Networks December 2012 8
9. EDL Benefits Sample output for a 48 hour call load – assume any technology Tester need not look into logs for analysis EDL analysis is easier and analysis time for each test run is reduced EDL provides temporal view of the data The graphs help the tester to easily Author: Parijat Bhattacharjee email: parijat.bhattacharjee@nsn.com Advanced Monitoring & Analysis Solution for Performance Testing of Wireless Networks December 2012 9 Tester can drill down into the EDL data to access the records at the point of failure EDLs contain information relevant to testers; other logs can be correlated with EDLs based on time stamp. pinpoint the time when the problem occurred.
10. Thank You Author: Parijat Bhattacharjee email: parijat.bhattacharjee@nsn.com Advanced Monitoring & Analysis Solution for Performance Testing of Wireless Networks December 2012 10 Thank You