Software Quality Metrics

2e147e13e3cf585a86626062d710a205?s=47 yunglin
June 27, 2012
60

Software Quality Metrics

2e147e13e3cf585a86626062d710a205?s=128

yunglin

June 27, 2012
Tweet

Transcript

  1. ழ᜗ᐶࣖ ࡡᓃ߅Ҧ О͑೙ yho@bluetangstudio.com

  2. Ң݊ም • ࡡᓃ߅Ҧ௴፬ɛ

  3. Ң݊ም • ࡡᓃ߅Ҧ௴፬ࠋபɛ

  4. Ң݊ም • ࡡᓃ߅Ҧ௴፬ࠋபɛ • Server Side Developer for 8 years.

  5. Ң݊ም • ࡡᓃ߅Ҧ௴፬ࠋபɛ • Server Side Developer for 8 years.

    • QA Engineer @ Glu Mobile
  6. Ң݊ም • ࡡᓃ߅Ҧ௴፬ࠋபɛ • Server Side Developer. • QA Engineer

    @ Glu Mobile • SDET @ Real Networks
  7. Ң݊ም • ࡡᓃ߅Ҧ௴፬ࠋபɛ • Server Side Developer for 8 years.

    • QA Engineer @ Glu Mobile • SDET @ Real Networks • SDE @ Medio Systems.
  8. TODAY’S TOPIC

  9. Software Quality

  10. Why Software Quality is Important?

  11. When I am busy working on features

  12. and finding customers

  13. Because it is a feature

  14. Because it is a strategic weapon

  15. Trust me, I can build a data center for you

  16. VMWare: We know our service.

  17. software is all about trust.

  18. Users need to trust the cloud won’t explode and bring

    down their business
  19. slashdot effect - overnight success turns into a disaster.

  20. How to build trust?

  21. Trust is faith, Faith is a religion. But engineering is

    a science not religion.
  22. None
  23. Trust is Quality

  24. What is Quality • Availability • Manageability • Performance •

    Reliability • Scalability • Security
  25. Availability • of your software • of your IT /

    cloud infrastructure • of your network environment • of your external dependencies • of your client side network environment
  26. Calculate Availability source: Economics of High Availability for Telecommunications Systems.

    An Intel® Primer
  27. Network Availability • 2001/02/09 ʕߕऎ᝙ቊ௡୵ʵॎ • 2003/10/02 ʕߕऎ᝙ቊ௡୵ʵॎ • 2006/12/26

    ܩ݆ήቤ • 2009/08/13 ୽זдჱࠬ • 2010/03/04 ৷ඪ̀͠ήቤ • 2011/11/14 ऎ᝙ආБၪࡌʬ඄
  28. Availability of External Dependencies

  29. None
  30. SOA in reality

  31. Availability • Service Level Agreement • Scheduled down time •

    Unscheduled down time
  32. Manageability • Expose Information for System Admin to check the

    condition of the online system. • Configurability • Monitorability
  33. Manageability • Monitoring is must, every cloud Infrastructure fails •

    Amazon • Azure • Google
  34. None
  35. None
  36. None
  37. Monitoring • Service credits are not automatics. • client has

    to file a ticket immediately. • claim downtime of certain time period. • proof / log of outrage and area.
  38. Performance • Response Time. • Throughput • Network Bandwidth

  39. Reliability • Availability v.s. Reliability • Availability: system uptime •

    Reliability: number of errors in a given time.
  40. Reliability • Crash or become unresponsive occasionally. • Output is

    inconsistent.
  41. Scalability • Slashdot Effect. • Scale Up and Scale Out

    • ERP system are much more sticky. • A mid-size website may have ~200 active sessions • But a ERP system used by a company of 200 employees would have 200 active sessions.
  42. Security • Secure communication. • User Authorization and Authentication. •

    Data Management. • Usage/Operation logging.
  43. Trust is tangible evidences.

  44. • Internal Audit •DEV Test •QA Process • External Audit

    • ISO/27001 - security management standard. • yho@bluetangstudio.com :)
  45. Software Metrics

  46. How do we know Service A can handle X amount

    of traffic before we launch the Service?
  47. Load Test

  48. Load Tests • Performance Test • Stress Test • Longevity

    Test
  49. Performance Test • find • best case performance. • baseline

    performance.
  50. Performance Test • Key figures • request per minutes •

    number of concurrent requests • number of machines • overall throughput
  51. Performance Test • Ꮠ͜೻ό࿁ᏀɢٙˀᏐ݊νО • ஈଣඎ throughput ݊щึᎇഹ request ᅰᄣ̋Ͼᄣ̋d݊щί൴

    ཀݔࡈᔷұᓃࣛdึක֐ ˀᔷɨࠥ • ਂމ͊Ը൙П݊щცࠅᄣ̋ዚኜא٫݊؁ॴዚኜٙਿ๟࠽f • ཫПɪᇞ༈Դ͜εˇዚኜ ٙ࣬ኽf
  52. Important Figures • minimum • maximum • mean • standard

    deviation • 75 percentile • 95 percentile • 98 percentile • 99 percentile • 99.9 percentile
  53. A Picture Worth A Thousand Words

  54. None
  55. Stress Test • Test how system perform under extreme situation.

    • ie: iphone5 is on sales.
  56. Longevity Test • evaluates a system's ability to handle a

    constant, moderate work load for a long time. • find memory leak • find hidden routine that exhausts system resource.
  57. Monitoring System Under Test • OS Level: CPU, Memory, Disc

    I/O, and Network Usage • JVM: Memory Usage, Object counts, GC Cycles. • Application Level: request count, method call time, size of queue, cache hit/miss ratio
  58. Load Test is the external view of the system.

  59. we shall measure the code from developer’s point of view

    too.
  60. what we thought != reality

  61. Shopping Cart • SQL Query • Page Rendering • Session

    Replication
  62. Shopping Cart • SQL Query • Page Rendering • Session

    Replication It is slow!
  63. Shopping Cart • SQL Query • Page Rendering • Session

    Replication • Load test result: 300ms.
  64. Shopping Cart • SQL Query • Page Rendering • Session

    Replication we shall cache sql results!
  65. Shopping Cart • SQL Query • Page Rendering • Session

    Replication • 50 ms • 20 ms • 230 ms
  66. We need to measure the code.

  67. To make better decision.

  68. •Obverse ᝈ࿀ •Orient ሜ዆ •Decide Ӕഄ •Act Бਗ

  69. • Perf4j • Java Simon • Google Guava Stopwatch •

    Metrics by Coda Hale. TOOLS
  70. Thank you • Any Questions?