Non-Functional Testing and Tuning - A Scientific Approach

Slide 1

Slide 1 text

@YourTwitterHandle #YourSessionHashtag Kevin Rudland & Andrew Harmel-Law, Capgemini UK http://capgemini.github.io Non-Functional Testing and Tuning: A Scientific Approach @kevinrudland / @al94781 #test_nfrs_with_science

Slide 2

Slide 2 text

Who Are We ? • Andrew Harmel-Law • Kevin Rudland @kevinrudland / @al94781 #test_nfrs_with_science

Slide 3

Slide 3 text

The bit you always write at the end after you’ve done all the work which summarises your findings. ABSTRACT @kevinrudland / @al94781 #test_nfrs_with_science

Slide 4

Slide 4 text

Meta-Hypothesis: H1: We can give ourselves confidence that our hyper-distributed systems will meet our Non Functional Requirements by applying the same method that scientists use to understand and predict the world around us: Science; specifically the Scientific Method. @kevinrudland / @al94781 #test_nfrs_with_science

Slide 5

Slide 5 text

INTRODUCTION (Pt. 1) The bit where we try and get you to agree with us about the nature of the problem we need to solve. @kevinrudland / @al94781 #test_nfrs_with_science

Slide 6

Slide 6 text

Image from: http://www.slideshare.net/InfoQ/migrating-to-cloud-native-with-microservices (slide 66) @kevinrudland / @al94781 #test_nfrs_with_science

Slide 7

Slide 7 text

Have we lost, or are we in the process of losing, predictability in the systems we build? @kevinrudland / @al94781 #test_nfrs_with_science

Slide 8

Slide 8 text

But Wait... “...if you are trying to break new ground and be really innovative, that's where you have to apply first-principle thinking and try to identify the most fundamental truths in any particular arena and you reason up from there.” Elon Musk [Profiles in versatility - http: //www.aps. org/publications/apsnews/201310/profiles. cfm] talking about applying the methods of Physics to problems Musk at the 2015 Tesla Motors Annual Meeting Photo by Steve Jurvetson - https://www.flickr.com/photos/jurvetson/18659265152/, CC BY 2.0, https://commons.wikimedia.org/w/index.php?curid=40974345 @kevinrudland / @al94781 #test_nfrs_with_science

Slide 9

Slide 9 text

COUNTER-PROPOSAL: This proliferation is a good thing Scalability? - Up! Resilience? - Up! Service Availability? - Up! Maintainability? - Up! Monitorability? - Up! Deployment Frequency? - Up! Time-to-change? - Down! Flexibility? - Up! Cost? - Focussed! @kevinrudland / @al94781 #test_nfrs_with_science

Slide 10

Slide 10 text

Rather than having lost predictability, did we ever really have it in the first place? And haven’t we actually gained as a result? @kevinrudland / @al94781 #test_nfrs_with_science

Slide 11

Slide 11 text

So then how do we know we have these qualities? (Given that we’re agreed we've fundamentally relinquished *control* of them.) @kevinrudland / @al94781 #test_nfrs_with_science

Slide 12

Slide 12 text

from “The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win" by Gene Kim, Kevin Behr & George Spafford @kevinrudland / @al94781 #test_nfrs_with_science

Slide 13

Slide 13 text

1. Testing simply to validate is no longer enough 16th-century illustration of Archimedes in the bath, with Hiero's crown at bottom right By Unknown - Historia N° 767 - Novembre 2010 - page 38, Public Domain, https: //commons.wikimedia.org/w/index.php?curid=12080104 @kevinrudland / @al94781 #test_nfrs_with_science

Slide 14

Slide 14 text

@kevinrudland / @al94781 #test_nfrs_with_science 1. Testing simply to validate is no longer enough 2. We must test to investigate and learn 16th-century illustration of Archimedes in the bath, with Hiero's crown at bottom right By Unknown - Historia N° 767 - Novembre 2010 - page 38, Public Domain, https: //commons.wikimedia.org/w/index.php?curid=12080104

Slide 15

Slide 15 text

1. Testing simply to validate is no longer enough 2. We must test to investigate and learn 3. And the best way to do that is to hypothesise, experiment, measure and adapt… 16th-century illustration of Archimedes in the bath, with Hiero's crown at bottom right By Unknown - Historia N° 767 - Novembre 2010 - page 38, Public Domain, https: //commons.wikimedia.org/w/index.php?curid=12080104 @kevinrudland / @al94781 #test_nfrs_with_science

Slide 16

Slide 16 text

1. Testing simply to validate is no longer enough 2. We must test to investigate and learn 3. And the best way to do that is to hypothesise, experiment, measure and adapt… Repeatedly. 16th-century illustration of Archimedes in the bath, with Hiero's crown at bottom right By Unknown - Historia N° 767 - Novembre 2010 - page 38, Public Domain, https: //commons.wikimedia.org/w/index.php?curid=12080104 @kevinrudland / @al94781 #test_nfrs_with_science

Slide 17

Slide 17 text

Meta-Hypothesis #1: H1: We can give ourselves confidence that our hyper-distributed systems will meet our Non Functional Requirements by applying the same method that scientists use to understand and predict the world around us: Science; specifically the Scientific Method. @kevinrudland / @al94781 #test_nfrs_with_science

Slide 18

Slide 18 text

INTRODUCTION (Pt. 2) The after we agree what the problem is and have come to a possible solution but still need some details about the approach @kevinrudland / @al94781 #test_nfrs_with_science

Slide 19

Slide 19 text

“The scientific method is a body of techniques for investigating phenomena [&] acquiring new knowledge" Ibn al-Haytham (Alhazen), considered the father of the modern scientific methodology by Unknown - http://islamicrenaissance.tumblr. com/post/38457296805/ibn-al-haytham-father-of-fiber-optics-born-in, Public Domain, https://commons.wikimedia.org/w/index.php? curid=47132269 Tip: Acquire Knowledge @kevinrudland / @al94781 #test_nfrs_with_science

Slide 20

Slide 20 text

Tip: Theories == NFRs Science is driven by theories, and theories are detailed statements about what we believe to be true. Roger Bacon sometimes credited as being one of the earliest European Advocates of the Modern Scientific Method By Jan Verhas - Britannica upload from a 1867 original, Public Domain, https: //commons.wikimedia.org/w/index.php?curid=44315585 @kevinrudland / @al94781 #test_nfrs_with_science

Slide 21

Slide 21 text

Example: NFR-Driven “Theories” Example 1: “The System will be able to support peak volumes of 100,000 requests per minute. These requests are to have a response time of 0.5 seconds or less, with less than 1% errors per request reported, considering 2000 requests per second.” Example 2: “Scaling Out - Adding additional instances of the API Service on additional servers increases the throughput near-linearly.” @kevinrudland / @al94781 #test_nfrs_with_science

Slide 22

Slide 22 text

Tip: Falsifiability "A scientific hypothesis must be falsifiable [...] otherwise, [it] cannot be meaningfully tested" Johannes Kepler, believed by some to be the archetype of the inductive scientific genius By Unknown - Kopie eines verlorengegangenen Originals von 1610 im Benediktinerkloster in Kremsmünster, Public Domain, https://commons.wikimedia. org/w/index.php?curid=470711 @kevinrudland / @al94781 #test_nfrs_with_science

Slide 23

Slide 23 text

Example: Hypotheses Example 1: H1: WHEN xxx requests per second are sent to the ABC_SERVICE THEN the average will respond successfully within 0.5 seconds and less than 1% will result in an error H0: WHEN … THEN the average will respond in greater than 0.5 seconds OR more than 1% will result in an error @kevinrudland / @al94781 #test_nfrs_with_science

Slide 24

Slide 24 text

Tip: Prioritise Counterexamples "no number of positive outcomes [...] can confirm a scientific theory, but a single counterexample is logically decisive" Karl Popper, By LSE library - http://www.flickr.com/photos/lselibrary/3833724834/in/set- 72157623156680255/, No restrictions, https://commons.wikimedia.org/w/index.php? curid=9694262 @kevinrudland / @al94781 #test_nfrs_with_science

Slide 25

Slide 25 text

never happens Prioritisation : Scalability @kevinrudland / @al94781 #test_nfrs_with_science

Slide 26

Slide 26 text

Prioritisation : SLAs & Resilience @kevinrudland / @al94781 #test_nfrs_with_science

Slide 27

Slide 27 text

Prioritisation : Ohhh, Shiny and New! @kevinrudland / @al94781 #test_nfrs_with_science

Slide 28

Slide 28 text

Example: A Scary Hypotheses : Tolerance of Downstream Failures Example 3: H1: Failure of the ABC service to connect to the XYZ service is handled without manual intervention, logged, and reported back to the ABC service within 100ms of the call to the XYZ service being made. If there are > X errors in a 10 second window, the circuit-breaker in the ABC service opens H0: Failure of the ABC service to connect to the XYZ service causes failures which need manual intervention, which are not logged, and the circuit-breaker in the ABC service does not open, no matter how many, or how frequently errors are seen @kevinrudland / @al94781 #test_nfrs_with_science

Slide 29

Slide 29 text

Tip: Apply Occam’s Razor "the simplest explanation is usually the correct one” William of Ockham, credited with originating Occam’s Razor By self-created (Moscarlop) - Own work, CC BY-SA 3.0, https://commons.wikimedia. org/w/index.php?curid=5523066 @kevinrudland / @al94781 #test_nfrs_with_science

Slide 30

Slide 30 text

METHOD The bit where we discuss how we’re going to get to where we think we want to get @kevinrudland / @al94781 #test_nfrs_with_science

Slide 31

Slide 31 text

Tip: Be Repeatable "If an experiment cannot be repeated to produce the same results, this implies that the original results might have been in error." Marie Curie, conducted pioneering research on radioactivity By Unknown - Christie's, [1], Public Domain, https://commons.wikimedia.org/w/index.php? curid=32735756 @kevinrudland / @al94781 #test_nfrs_with_science

Slide 32

Slide 32 text

• Subjects • Their Environment • The Independent Variables • (The Dependent Variables) Milgram’s Obedience Experiment Image from: https://thoughtmaybe.com/the-milgram-experiment-obedience/ Tip: Experiments Typically Consist of... @kevinrudland / @al94781 #test_nfrs_with_science

Slide 33

Slide 33 text

Subjects Think: ● How many of them? ● How are they connected? ● What do they already know? Milgram’s Obedience Experiment Image from: https://thoughtmaybe.com/the-milgram-experiment-obedience/ @kevinrudland / @al94781 #test_nfrs_with_science

Slide 34

Slide 34 text

Example: Hypothesis - v.2 GIVEN n instances of the ABC_SERVICE with the DEFAULT_CONFIGURATION and empty REQUEST_CACHES, and which are connected to n instances of the downstream XYZ_SERVICE with the DEFAULT_CONFIGURATION via EUREKA over the PROD_NETWORK H1: WHEN xxx requests per second are sent to the ABC_SERVICE THEN the average will respond successfully within 0.5 seconds and less than 1% will result in an error H0: WHEN xxx requests per second are sent to the ABC_SERVICE THEN the average will not respond successfully within 0.5 seconds and more than 1% will result in an error @kevinrudland / @al94781 #test_nfrs_with_science

Slide 35

Slide 35 text

Tip: Sometimes; Keep Looking “Absence of evidence is not evidence of absence.” Chien-Shiung Wu, Chinese American experimental physicist who made significant contributions in the field of nuclear physics. She is best known for conducting the Wu experiment, which contradicted the hypothetical law of conservation of parity By Smithsonian Institution from United States - Chien-shiung Wu (1912-1997)Uploaded by Fæ, No restrictions, https://commons.wikimedia.org/w/index.php?curid=18877882 @kevinrudland / @al94781 #test_nfrs_with_science

Slide 36

Slide 36 text

WARNING! Sometimes things go very well in test and this can give you a false sense of security: Sometimes you can be live-like, but not live-like enough (and didn’t realise). What went wrong? How did we manage to predict things? @kevinrudland / @al94781 #test_nfrs_with_science

Slide 37

Slide 37 text

Meta-Hypothesis #2: H1: the only environment exactly like production is production, therefore only results from testing in production really count. @kevinrudland / @al94781 #test_nfrs_with_science

Slide 38

Slide 38 text

Tip: Fake it “To be ecologically valid, the methods, materials and setting of a study must approximate the real-life situation that is under investigation.” Francis Bacon, philosophical advocate and practitioner of the scientific method By Paul van Somer (1576/1578–1622) - pl.pinterest.com, Public Domain, https: //commons.wikimedia.org/w/index.php?curid=19958108 @kevinrudland / @al94781 #test_nfrs_with_science

Slide 39

Slide 39 text

Tip: Axes of Realism So, if science can do it, what are our axes of realism?: ● Environment ○ Configuration ○ Data ○ Platform and connectivity ○ Monitoring ● Also, are you in; ○ a Normal-Running or a DR scenario? ● And are there; ○ events such as a fail and recover,? ○ or a slow down and speed up? Etc. @kevinrudland / @al94781 #test_nfrs_with_science

Slide 40

Slide 40 text

Tip: Realism Techniques ● Take a copy of prod data ● Take a copy of prod requests ● Close observation of prod followed by replication in the “lab” @kevinrudland / @al94781 #test_nfrs_with_science

Slide 41

Slide 41 text

Method: Data • Of all the environmental elements, data can have a MOST SIGNIFICANT IMPACT • Easiest to change • This is the type of data, its mix and variety • Look at your data model – Examine: 1:n – Examine: nullable @kevinrudland / @al94781 #test_nfrs_with_science

Slide 42

Slide 42 text

Charles R. Drew, developed improved techniques for blood storage, and applied his expert knowledge to developing large-scale blood banks By Associated Photographic Services, Inc - Original Repository: Howard University. Moorland-Spingarn Research Center. Charles R. Drew Papers http://profiles.nlm.nih. gov/ps/retrieve/ResourceMetadata/BGBBCT, https://en.wikipedia.org/w/index.php? curid=47837720 Tip: Manipulate Independent Variables @kevinrudland / @al94781 #test_nfrs_with_science

Slide 43

Slide 43 text

Hard-Coded Successful Requests @kevinrudland / @al94781 #test_nfrs_with_science

Slide 44

Slide 44 text

A Mix of Good and Bad Requests @kevinrudland / @al94781 #test_nfrs_with_science

Slide 45

Slide 45 text

Example: Hypothesis - v.3.1 GIVEN n instances of the ABC_SERVICE with the DEFAULT_CONFIGURATION and empty REQUEST_CACHES, and which are connected to n instances of the downstream XYZ_SERVICE with the DEFAULT_CONFIGURATION via EUREKA over the PROD_NETWORK H1: WHEN a mix of xxx VALID requests per second and yyy INVALID requests per second are sent to the ABC_SERVICE THEN the average will respond correctly within 0.5 seconds and less than 1% will result in an error H0: WHEN a mix of xxx VALID requests per second and yyy INVALID requests per second are sent to the ABC_SERVICE THEN the average will not respond correctly within 0.5 seconds and more than 1% will result in an error @kevinrudland / @al94781 #test_nfrs_with_science

Slide 46

Slide 46 text

@kevinrudland / @al94781 #test_nfrs_with_science

Slide 47

Slide 47 text

JMeter - Dynamic Request Gen @kevinrudland / @al94781 #test_nfrs_with_science

Slide 48

Slide 48 text

@kevinrudland / @al94781 #test_nfrs_with_science

Slide 49

Slide 49 text

Dynamic Request Generation @kevinrudland / @al94781 #test_nfrs_with_science

Slide 50

Slide 50 text

Dynamic Request Generation @kevinrudland / @al94781 #test_nfrs_with_science

Slide 51

Slide 51 text

Test Plus Background Load Mix @kevinrudland / @al94781 #test_nfrs_with_science

Slide 52

Slide 52 text

Example: Hypothesis - v.3.2 GIVEN n instances of the ABC_SERVICE with the DEFAULT_CONFIGURATION and empty REQUEST_CACHES, and which are connected to n instances of the downstream XYZ_SERVICE with the DEFAULT_CONFIGURATION via EUREKA over the PROD_NETWORK with BACKGROUND_LOAD H1: WHEN a mix of xxx VALID requests per second and yyy INVALID requests per second are sent to the ABC_SERVICE THEN the average will respond correctly within 0.5 seconds and less than 1% will result in an error H0: WHEN a mix of xxx VALID requests per second and yyy INVALID requests per second are sent to the ABC_SERVICE THEN the average will not respond correctly within 0.5 seconds and more than 1% will result in an error @kevinrudland / @al94781 #test_nfrs_with_science

Slide 53

Slide 53 text

Method: Events Dear boy… An 1888 lithograph of the 1883 eruption of Krakatoa. By Lithograph: Parker & Coward, Britain; - Image published as Plate 1 in The eruption of Krakatoa, and subsequent phenomena. Report of the Krakatoa Committee of the Royal Society (London, Trubner & Co., 1888)., Public Domain, https://commons.wikimedia. org/w/index.php?curid=7696837 @kevinrudland / @al94781 #test_nfrs_with_science

Slide 54

Slide 54 text

Method: Execute • Manipulate the independent variables: – e.g. req./sec. – e.g. concurrent users – e.g. req. mix • One at a time @kevinrudland / @al94781 #test_nfrs_with_science

Slide 55

Slide 55 text

So, back at Meta-Hypothesis #2: Proven - H0: We can proceed without testing in production @kevinrudland / @al94781 #test_nfrs_with_science

Slide 56

Slide 56 text

The bit where we look at the outputs collected from testing our hypotheses RESULTS @kevinrudland / @al94781 #test_nfrs_with_science

Slide 57

Slide 57 text

Tip: Be Empirical "To be termed scientific, a method of inquiry is commonly based on empirical or measurable evidence" Rosalind Franklin, had a key role (largely unacknowledged during her lifetime) in discovering the helical structure of DNA By Jewish Chronicle Archive/Heritage-Imageshttp://www.britannica.com/EBchecked/topic- art/217394/99712/Rosalind-Franklin, Fair use, https://en.wikipedia.org/w/index.php? curid=24959067 @kevinrudland / @al94781 #test_nfrs_with_science

Slide 58

Slide 58 text

Tip: Measure Dependent Variables Sir C. V. Rahman, winner of the 1030 Nobel Prize for Physics for his work on light scattering By Nobel Foundation - From Nobel Lectures, Physics 1922-1941, Elsevier Publishing Company, Amsterdam, 1965, Public Domain, https://commons.wikimedia.org/w/index. php?curid=4213636 @kevinrudland / @al94781 #test_nfrs_with_science

Slide 59

Slide 59 text

@kevinrudland / @al94781 #test_nfrs_with_science

Slide 60

Slide 60 text

@kevinrudland / @al94781 #test_nfrs_with_science

Slide 61

Slide 61 text

DISCUSSION The bit where we try and make coherent sense of what we found @kevinrudland / @al94781 #test_nfrs_with_science

Slide 62

Slide 62 text

Tip: Investigate ● Tally your counts ● Attribute all noted errors ● and resource utilization The Stanford Prison Experiment Image from http://www.prisonexp.org/the-story/ @kevinrudland / @al94781 #test_nfrs_with_science

Slide 63

Slide 63 text

Tip: Stats Can Help • Why do you care?: • Means mean nothing • The impacts of (statistical) distribution @kevinrudland / @al94781 #test_nfrs_with_science

Slide 64

Slide 64 text

@kevinrudland / @al94781 #test_nfrs_with_science

Slide 65

Slide 65 text

Stats: Data Distribution @kevinrudland / @al94781 #test_nfrs_with_science

Slide 66

Slide 66 text

A Better Example @kevinrudland / @al94781 #test_nfrs_with_science

Slide 67

Slide 67 text

Stats: Percentiles @kevinrudland / @al94781 #test_nfrs_with_science

Slide 68

Slide 68 text

A Better Example (Again) @kevinrudland / @al94781 #test_nfrs_with_science

Slide 69

Slide 69 text

Example: Hypothesis - v.4 GIVEN n instances of the ABC_SERVICE with the DEFAULT_CONFIGURATION and empty REQUEST_CACHES, and which are connected to n instances of the downstream XYZ_SERVICE with the DEFAULT_CONFIGURATION via EUREKA over the PROD_NETWORK with BACKGROUND_LOAD H1: WHEN a mix of xxx VALID requests per second and xxx INVALID requests per second are sent to the ABC_SERVICE THEN 95% will respond correctly within 0.5 seconds and less than 1% will result in an error H0: WHEN a mix of xxx VALID requests per second and xxx INVALID requests per second are sent to the ABC_SERVICE THEN 95% will not respond correctly within 0.5 seconds and more than 1% will result in an error @kevinrudland / @al94781 #test_nfrs_with_science

Slide 70

Slide 70 text

Decide: H1 or H0? Did the test pass or fail? @kevinrudland / @al94781 #test_nfrs_with_science

Slide 71

Slide 71 text

The bit where we sum up. (And also perhaps invite some questions.) CONCLUSION @kevinrudland / @al94781 #test_nfrs_with_science

Slide 72

Slide 72 text

Conclusion: Refine and Retest • We did not verify H1 :( • Do you need to look elsewhere? • Revisit H0 / H1 - Apply Occam’s Razor • Only change 1 thing at a time @kevinrudland / @al94781 #test_nfrs_with_science

Slide 73

Slide 73 text

Tip: Tune & Fix Too ● Change one thing at a time (unless it’s your DVs) ● Think!: ○ Will it have the desired effect? ○ Will it have side-effects? ○ Tuning the slowest won’t always win Image: cc: Kalle Hyttinen - https://www.flickr.com/photos/129560755@N03 @kevinrudland / @al94781 #test_nfrs_with_science

Slide 74

Slide 74 text

Tip: Re-Run (again) Always re-baseline after a change / fix has gone in - make no other changes @kevinrudland / @al94781 #test_nfrs_with_science

Slide 75

Slide 75 text

N.b. Your Targets Don't get carried away @kevinrudland / @al94781 #test_nfrs_with_science

Slide 76

Slide 76 text

The bit you always write at the end after you’ve done all the work which summarises your findings. ABSTRACT @kevinrudland / @al94781 #test_nfrs_with_science

Slide 77

Slide 77 text

Slide 78

Slide 78 text

The bit everyone dreads in case they got something wrong along the way. QUESTIONS? @kevinrudland / @al94781 #test_nfrs_with_science

Slide 79

Slide 79 text

cc: ptrlx - https://www.flickr.com/photos/58615912@N05 Oh, and did we mention we (Capgemini) are hiring? (see bit.ly/cg-jvm-jobs-nfr for lots more info) @kevinrudland / @al94781 #test_nfrs_with_science

Slide 80

Slide 80 text

The bit where you dump things that didn’t fit anywhere else. APPENDICES @kevinrudland / @al94781 #test_nfrs_with_science

Slide 81

Slide 81 text

@kevinrudland / @al94781 #test_nfrs_with_science

Slide 82

Slide 82 text

Marginalia: Record It • Use templates • Visualise • Summarize • Supporting evidence Marginalia Image from: http://spec.lib.miamioh.edu/home/marginalia-from-the-stacks/ @kevinrudland / @al94781 #test_nfrs_with_science