Upgrade to Pro — share decks privately, control downloads, hide ads and more …

SAP Concur - Making Development Test Reporting ...

Sponsored · Ship Features Fearlessly Turn features on and off without deploys. Used by thousands of Ruby developers. →
Elastic Co
December 03, 2015

SAP Concur - Making Development Test Reporting Less Expensive

Testing at functional and performance levels involves a lot of data that needs to be correlated by environment, by build, etc., all over the same periods of time. The Elastic stack enables us to cost-effectively cross-analyze data combined from several vendor solutions.

Wilson Mar | Elastic{ON} Tour Seattle | December 3, 2015

Elastic Co

December 03, 2015

Transcript

20 slides
  1. By Wilson Mar, SAP Concur Seattle December 3, 2015 1

      Making Development Test Reporting Less Expensive
  2. Maturity in Analytics (Gartner) 6 h%p://7moellio%.com/blog/2013/02/gartnerbi-­‐emea-­‐2013-­‐part-­‐1-­‐analy7cs-­‐moves-­‐to-­‐the-­‐core.html   Difficulty   Value

      What   Happened?   Why  did  it   Happen?   What  will   Happen?   How  can  we     make  It  happen?   Hindsight   Insight   Foresight   Descrip7ve   Diagnos7c   Predic7ve   Prescrip7ve   Informa1on   Op1miza1on   highest   payoff   Tes$ng   Scaling   Tuning   Scanning  
  3. Overlay similar metrics in same location 7 Load:   *

     Connec7ons   *  Sessions   *  Opera7ons   Queues:   *  Login  wai7ng   *  Server  wai7ng   *  DB  wai7ng   Rate/Second:   *  Hits   *  Errors   *  Swapping   Available  RAM:   *  App  servers   *  Back  e.  servers   *  Test  servers   Latency:   *  Landing   *  Login   *  Transac7ons   *  Network   *  Back-­‐end   Ra1os:   *  HTTP  200/all   *  Error/event   *  Memory/user   Data  Thruput:   *  F5   *  bps  network         *  storage     Percentages:   *  CPU/server       *  Free  space       inputs   outputs   processing  
  4. Statistical anomaly (disruption) detection 8 h%ps://www.elas7c.co/blog/implemen7ng-­‐a-­‐sta7s7cal-­‐anomaly-­‐detector-­‐part-­‐1   Zachary  Tang:  

    h%p://www.ebaytechblog.com/2015/08/19/sta7s7cal-­‐anomaly-­‐detec7on/   h%ps://www.usenix.org/system/files/conference/hotcloud15/ hotcloud15-­‐goldberg.pdf   “post-mortem analysis of disruptions has always revealed that at least one monitored signal exhibited unusual behavior before the disruption began to significantly affect end- users, but these anomalies failed to be detected.” - David Goldberg & Yinan Shan (eBay Atlas system)
  5. Ratio for early warning = threshold for action 10 Light

      load   Heavy     load   Overload   Time   #  users  (imposed  load)   %  CPU  u7liza7on,  Memory   #  Seconds  response  7me   Wai$ng  for  others   Capacity   Limit   Swapping   Queuing   Memory     per  user   Threshold   rate  of  increase   Narra7on  by  Wilson  Mar  at  h%ps://www.youtube.com/watch?v=I6EzGV0GE2A  
  6. Time series + forecast 12 Pipeline aggregation in v2.0  

    h%ps://www.elas7c.co/webinars/elas7csearch-­‐2-­‐0-­‐overview   Forecast:   h%ps://www.elas7c.co/blog/elas7csearch-­‐as-­‐a-­‐7me-­‐series-­‐data-­‐store  
  7. github.com/wilsonmar/oss-perf internet browsers mobile apps API ES vendor APIs load

    gen. Logstash Elastic- search Kibana app server Swagger API req. results & logs load scripts codegen git Sonar Qube server & app logs monitor stream Browser Mob Proxy Selenium code JMeter ref. data agents Appium code alerts https://goo.gl/aeWSvH thresholds HAR files network sham data-gen timings export/import/compare data provision Docker Puppet watcher DS & ML Artifactory perf. run variations configs Wiremock Jenkins CI/CD consumers Selenium WebDriver Appium Driver profiler schedule Narra7on  by  Wilson  Mar  at  h%ps://www.youtube.com/watch?v=WnVCUzCO1CI  
  8.     Prod   Average   90th     percen7le

      Prior  Release   Dev.   New  Release  >   BVT   EVT   RQA   Staging   Par7al   Prod   Full   Prod  
  9. Index to filter Environment under test Environment ID: •  Data

    center region 15 App under test •  Version & Release ID •  Jenkins job ID •  Node ID •  Component •  Module •  Class •  Developer •  Log level Load Test rig •  Test rig version •  Server host name •  Load script name •  Load script version
  10. Recap • Reach for higher maturity • Group similar metrics in same

    location. • Identify early warning leading indicators. • Calculate ratios. • Identify thresholds to act on early warning signals. • Automate manual “air traffic control”. 16

More Decks by Elastic Co

Other Decks in Technology