SAP Concur - Making Development Test Reporting Less Expensive

Testing at functional and performance levels involves a lot of data that needs to be correlated by environment, by build, etc., all over the same periods of time. The Elastic stack enables us to cost-effectively cross-analyze data combined from several vendor solutions.

Wilson Mar | Elastic{ON} Tour Seattle | December 3, 2015

Transcript

2. By Wilson Mar, SAP Concur Seattle December 3, 2015 1   Making Development Test Reporting Less Expensive
3. Topics • Maturity levels • Spatial arrangement • Early warning leading indicators • Ratios • Thresholds for action • Automation 2
4. Hello 3 wilsonmar@gmail.com wilson.mar@concur.com @wilsonmar Performance Architect
5. The usual suspects 4 h%p://www.nasa.gov/mission_pages/msl/mul7media/edl_room20120810.html  
6. Regular Content Page 5 Predic7on?   Ac7on?  
7. Maturity in Analytics (Gartner) 6 h%p://7moellio%.com/blog/2013/02/gartnerbi-­‐emea-­‐2013-­‐part-­‐1-­‐analy7cs-­‐moves-­‐to-­‐the-­‐core.html   Difficulty   Value   What   Happened?   Why  did  it   Happen?   What  will   Happen?   How  can  we     make  It  happen?   Hindsight   Insight   Foresight   Descrip7ve   Diagnos7c   Predic7ve   Prescrip7ve   Informa1on   Op1miza1on   highest   payoff   Tes$ng   Scaling   Tuning   Scanning  
8. Overlay similar metrics in same location 7 Load:   *  Connec7ons   *  Sessions   *  Opera7ons   Queues:   *  Login  wai7ng   *  Server  wai7ng   *  DB  wai7ng   Rate/Second:   *  Hits   *  Errors   *  Swapping   Available  RAM:   *  App  servers   *  Back  e.  servers   *  Test  servers   Latency:   *  Landing   *  Login   *  Transac7ons   *  Network   *  Back-­‐end   Ra1os:   *  HTTP  200/all   *  Error/event   *  Memory/user   Data  Thruput:   *  F5   *  bps  network         *  storage     Percentages:   *  CPU/server       *  Free  space       inputs   outputs   processing  
9. Statistical anomaly (disruption) detection 8 h%ps://www.elas7c.co/blog/implemen7ng-­‐a-­‐sta7s7cal-­‐anomaly-­‐detector-­‐part-­‐1   Zachary  Tang:   h%p://www.ebaytechblog.com/2015/08/19/sta7s7cal-­‐anomaly-­‐detec7on/   h%ps://www.usenix.org/system/files/conference/hotcloud15/ hotcloud15-­‐goldberg.pdf   “post-mortem analysis of disruptions has always revealed that at least one monitored signal exhibited unusual behavior before the disruption began to significantly affect end- users, but these anomalies failed to be detected.” - David Goldberg & Yinan Shan (eBay Atlas system)
10. h%p://wwwper_est.com  (no  longer  available)   Scalability
11. Ratio for early warning = threshold for action 10 Light   load   Heavy     load   Overload   Time   #  users  (imposed  load)   %  CPU  u7liza7on,  Memory   #  Seconds  response  7me   Wai$ng  for  others   Capacity   Limit   Swapping   Queuing   Memory     per  user   Threshold   rate  of  increase   Narra7on  by  Wilson  Mar  at  h%ps://www.youtube.com/watch?v=I6EzGV0GE2A  
12. Kibana Aggregations h%ps://speakerdeck.com/elas7c/a-­‐technical-­‐overview-­‐of-­‐the-­‐elas7c-­‐plaiorm  
13. Time series + forecast 12 Pipeline aggregation in v2.0   h%ps://www.elas7c.co/webinars/elas7csearch-­‐2-­‐0-­‐overview   Forecast:   h%ps://www.elas7c.co/blog/elas7csearch-­‐as-­‐a-­‐7me-­‐series-­‐data-­‐store  
14. github.com/wilsonmar/oss-perf internet browsers mobile apps API ES vendor APIs load gen. Logstash Elastic- search Kibana app server Swagger API req. results & logs load scripts codegen git Sonar Qube server & app logs monitor stream Browser Mob Proxy Selenium code JMeter ref. data agents Appium code alerts https://goo.gl/aeWSvH thresholds HAR files network sham data-gen timings export/import/compare data provision Docker Puppet watcher DS & ML Artifactory perf. run variations configs Wiremock Jenkins CI/CD consumers Selenium WebDriver Appium Driver profiler schedule Narra7on  by  Wilson  Mar  at  h%ps://www.youtube.com/watch?v=WnVCUzCO1CI  
15.     Prod   Average   90th     percen7le   Prior  Release   Dev.   New  Release  >   BVT   EVT   RQA   Staging   Par7al   Prod   Full   Prod  
16. Index to filter Environment under test Environment ID: •  Data center region 15 App under test •  Version & Release ID •  Jenkins job ID •  Node ID •  Component •  Module •  Class •  Developer •  Log level Load Test rig •  Test rig version •  Server host name •  Load script name •  Load script version
17. Recap • Reach for higher maturity • Group similar metrics in same location. • Identify early warning leading indicators. • Calculate ratios. • Identify thresholds to act on early warning signals. • Automate manual “air traffic control”. 16
18. https://speakerdeck.com/elastic/ YouTube: http://bit.ly/1jyOiKa 17  
19. Questions? 18 wilsonmar@gmail.com wilson.mar@concur.com wilsonmar on LinkedIn wilsonmar4 on Skype 310.320-7878 @wilsonmar
20. www.elas7c.co   19