Software Quality Metrics

Transcript

1. ழ᜗ᐶࣖ ࡡᓃ߅Ҧ О͑೙ yho@bluetangstudio.com

2. Ң݊ም • ࡡᓃ߅Ҧ௴፬ɛ

3. Ң݊ም • ࡡᓃ߅Ҧ௴፬ࠋபɛ

4. Ң݊ም • ࡡᓃ߅Ҧ௴፬ࠋபɛ • Server Side Developer for 8 years.

5. Ң݊ም • ࡡᓃ߅Ҧ௴፬ࠋபɛ • Server Side Developer for 8 years.

   • QA Engineer @ Glu Mobile

6. Ң݊ም • ࡡᓃ߅Ҧ௴፬ࠋபɛ • Server Side Developer. • QA Engineer

   @ Glu Mobile • SDET @ Real Networks

7. Ң݊ም • ࡡᓃ߅Ҧ௴፬ࠋபɛ • Server Side Developer for 8 years.

   • QA Engineer @ Glu Mobile • SDET @ Real Networks • SDE @ Medio Systems.

8. TODAY’S TOPIC

9. Software Quality

10. Why Software Quality is Important?

11. When I am busy working on features

12. and finding customers

13. Because it is a feature

14. Because it is a strategic weapon

15. Trust me, I can build a data center for you

16. VMWare: We know our service.

17. software is all about trust.

18. Users need to trust the cloud won’t explode and bring

    down their business

19. slashdot effect - overnight success turns into a disaster.

20. How to build trust?

21. Trust is faith, Faith is a religion. But engineering is

    a science not religion.
22. None

23. Trust is Quality

24. What is Quality • Availability • Manageability • Performance •

    Reliability • Scalability • Security

25. Availability • of your software • of your IT /

    cloud infrastructure • of your network environment • of your external dependencies • of your client side network environment

26. Calculate Availability source: Economics of High Availability for Telecommunications Systems.

    An Intel® Primer

27. Network Availability • 2001/02/09 ʕߕऎ᝙ቊ௡୵ʵॎ • 2003/10/02 ʕߕऎ᝙ቊ௡୵ʵॎ • 2006/12/26

    ܩ݆ήቤ • 2009/08/13 ୽זдჱࠬ • 2010/03/04 ৷ඪ̀͠ήቤ • 2011/11/14 ऎ᝙ආБၪࡌʬ඄

28. Availability of External Dependencies

29. None

30. SOA in reality

31. Availability • Service Level Agreement • Scheduled down time •

    Unscheduled down time

32. Manageability • Expose Information for System Admin to check the

    condition of the online system. • Configurability • Monitorability

33. Manageability • Monitoring is must, every cloud Infrastructure fails •

    Amazon • Azure • Google
34. None
35. None
36. None

37. Monitoring • Service credits are not automatics. • client has

    to file a ticket immediately. • claim downtime of certain time period. • proof / log of outrage and area.

38. Performance • Response Time. • Throughput • Network Bandwidth

39. Reliability • Availability v.s. Reliability • Availability: system uptime •

    Reliability: number of errors in a given time.

40. Reliability • Crash or become unresponsive occasionally. • Output is

    inconsistent.

41. Scalability • Slashdot Effect. • Scale Up and Scale Out

    • ERP system are much more sticky. • A mid-size website may have ~200 active sessions • But a ERP system used by a company of 200 employees would have 200 active sessions.

42. Security • Secure communication. • User Authorization and Authentication. •

    Data Management. • Usage/Operation logging.

43. Trust is tangible evidences.

44. • Internal Audit •DEV Test •QA Process • External Audit

    • ISO/27001 - security management standard. • yho@bluetangstudio.com :)

45. Software Metrics

46. How do we know Service A can handle X amount

    of traffic before we launch the Service?

47. Load Test

48. Load Tests • Performance Test • Stress Test • Longevity

    Test

49. Performance Test • find • best case performance. • baseline

    performance.

50. Performance Test • Key figures • request per minutes •

    number of concurrent requests • number of machines • overall throughput

51. Performance Test • Ꮠ͜೻ό࿁ᏀɢٙˀᏐ݊νО • ஈଣඎ throughput ݊щึᎇഹ request ᅰᄣ̋Ͼᄣ̋d݊щί൴

    ཀݔࡈᔷұᓃࣛdึක֐ ˀᔷɨࠥ • ਂމ͊Ը൙П݊щცࠅᄣ̋ዚኜא٫݊؁ॴዚኜٙਿ๟࠽f • ཫПɪᇞ༈Դ͜εˇዚኜ ٙ࣬ኽf

52. Important Figures • minimum • maximum • mean • standard

    deviation • 75 percentile • 95 percentile • 98 percentile • 99 percentile • 99.9 percentile

53. A Picture Worth A Thousand Words

54. None

55. Stress Test • Test how system perform under extreme situation.

    • ie: iphone5 is on sales.

56. Longevity Test • evaluates a system's ability to handle a

    constant, moderate work load for a long time. • find memory leak • find hidden routine that exhausts system resource.

57. Monitoring System Under Test • OS Level: CPU, Memory, Disc

    I/O, and Network Usage • JVM: Memory Usage, Object counts, GC Cycles. • Application Level: request count, method call time, size of queue, cache hit/miss ratio

58. Load Test is the external view of the system.

59. we shall measure the code from developer’s point of view

    too.

60. what we thought != reality

61. Shopping Cart • SQL Query • Page Rendering • Session

    Replication

62. Shopping Cart • SQL Query • Page Rendering • Session

    Replication It is slow!

63. Shopping Cart • SQL Query • Page Rendering • Session

    Replication • Load test result: 300ms.

64. Shopping Cart • SQL Query • Page Rendering • Session

    Replication we shall cache sql results!

65. Shopping Cart • SQL Query • Page Rendering • Session

    Replication • 50 ms • 20 ms • 230 ms

66. We need to measure the code.

67. To make better decision.

68. •Obverse ᝈ࿀ •Orient ሜ዆ •Decide Ӕഄ •Act Бਗ

69. • Perf4j • Java Simon • Google Guava Stopwatch •

    Metrics by Coda Hale. TOOLS

70. Thank you • Any Questions?