@YourTwitterHandle #YourSessionHashtag Kevin Rudland & Andrew Harmel-Law, Capgemini UK http://capgemini.github.io Non-Functional Testing and Tuning: A Scientific Approach @kevinrudland / @al94781 #test_nfrs_with_science
The bit you always write at the end after you’ve done all the work which summarises your findings. ABSTRACT @kevinrudland / @al94781 #test_nfrs_with_science
Meta-Hypothesis: H1: We can give ourselves confidence that our hyper-distributed systems will meet our Non Functional Requirements by applying the same method that scientists use to understand and predict the world around us: Science; specifically the Scientific Method. @kevinrudland / @al94781 #test_nfrs_with_science
INTRODUCTION (Pt. 1) The bit where we try and get you to agree with us about the nature of the problem we need to solve. @kevinrudland / @al94781 #test_nfrs_with_science
But Wait... “...if you are trying to break new ground and be really innovative, that's where you have to apply first-principle thinking and try to identify the most fundamental truths in any particular arena and you reason up from there.” Elon Musk [Profiles in versatility - http: //www.aps. org/publications/apsnews/201310/profiles. cfm] talking about applying the methods of Physics to problems Musk at the 2015 Tesla Motors Annual Meeting Photo by Steve Jurvetson - https://www.flickr.com/photos/jurvetson/18659265152/, CC BY 2.0, https://commons.wikimedia.org/w/index.php?curid=40974345 @kevinrudland / @al94781 #test_nfrs_with_science
Rather than having lost predictability, did we ever really have it in the first place? And haven’t we actually gained as a result? @kevinrudland / @al94781 #test_nfrs_with_science
So then how do we know we have these qualities? (Given that we’re agreed we've fundamentally relinquished *control* of them.) @kevinrudland / @al94781 #test_nfrs_with_science
from “The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win" by Gene Kim, Kevin Behr & George Spafford @kevinrudland / @al94781 #test_nfrs_with_science
1. Testing simply to validate is no longer enough 16th-century illustration of Archimedes in the bath, with Hiero's crown at bottom right By Unknown - Historia N° 767 - Novembre 2010 - page 38, Public Domain, https: //commons.wikimedia.org/w/index.php?curid=12080104 @kevinrudland / @al94781 #test_nfrs_with_science
@kevinrudland / @al94781 #test_nfrs_with_science 1. Testing simply to validate is no longer enough 2. We must test to investigate and learn 16th-century illustration of Archimedes in the bath, with Hiero's crown at bottom right By Unknown - Historia N° 767 - Novembre 2010 - page 38, Public Domain, https: //commons.wikimedia.org/w/index.php?curid=12080104
1. Testing simply to validate is no longer enough 2. We must test to investigate and learn 3. And the best way to do that is to hypothesise, experiment, measure and adapt… 16th-century illustration of Archimedes in the bath, with Hiero's crown at bottom right By Unknown - Historia N° 767 - Novembre 2010 - page 38, Public Domain, https: //commons.wikimedia.org/w/index.php?curid=12080104 @kevinrudland / @al94781 #test_nfrs_with_science
1. Testing simply to validate is no longer enough 2. We must test to investigate and learn 3. And the best way to do that is to hypothesise, experiment, measure and adapt… Repeatedly. 16th-century illustration of Archimedes in the bath, with Hiero's crown at bottom right By Unknown - Historia N° 767 - Novembre 2010 - page 38, Public Domain, https: //commons.wikimedia.org/w/index.php?curid=12080104 @kevinrudland / @al94781 #test_nfrs_with_science
Meta-Hypothesis #1: H1: We can give ourselves confidence that our hyper-distributed systems will meet our Non Functional Requirements by applying the same method that scientists use to understand and predict the world around us: Science; specifically the Scientific Method. @kevinrudland / @al94781 #test_nfrs_with_science
INTRODUCTION (Pt. 2) The after we agree what the problem is and have come to a possible solution but still need some details about the approach @kevinrudland / @al94781 #test_nfrs_with_science
“The scientific method is a body of techniques for investigating phenomena [&] acquiring new knowledge" Ibn al-Haytham (Alhazen), considered the father of the modern scientific methodology by Unknown - http://islamicrenaissance.tumblr. com/post/38457296805/ibn-al-haytham-father-of-fiber-optics-born-in, Public Domain, https://commons.wikimedia.org/w/index.php? curid=47132269 Tip: Acquire Knowledge @kevinrudland / @al94781 #test_nfrs_with_science
Tip: Theories == NFRs Science is driven by theories, and theories are detailed statements about what we believe to be true. Roger Bacon sometimes credited as being one of the earliest European Advocates of the Modern Scientific Method By Jan Verhas - Britannica upload from a 1867 original, Public Domain, https: //commons.wikimedia.org/w/index.php?curid=44315585 @kevinrudland / @al94781 #test_nfrs_with_science
Example: NFR-Driven “Theories” Example 1: “The System will be able to support peak volumes of 100,000 requests per minute. These requests are to have a response time of 0.5 seconds or less, with less than 1% errors per request reported, considering 2000 requests per second.” Example 2: “Scaling Out - Adding additional instances of the API Service on additional servers increases the throughput near-linearly.” @kevinrudland / @al94781 #test_nfrs_with_science
Tip: Falsifiability "A scientific hypothesis must be falsifiable [...] otherwise, [it] cannot be meaningfully tested" Johannes Kepler, believed by some to be the archetype of the inductive scientific genius By Unknown - Kopie eines verlorengegangenen Originals von 1610 im Benediktinerkloster in Kremsmünster, Public Domain, https://commons.wikimedia. org/w/index.php?curid=470711 @kevinrudland / @al94781 #test_nfrs_with_science
Example: Hypotheses Example 1: H1: WHEN xxx requests per second are sent to the ABC_SERVICE THEN the average will respond successfully within 0.5 seconds and less than 1% will result in an error H0: WHEN … THEN the average will respond in greater than 0.5 seconds OR more than 1% will result in an error @kevinrudland / @al94781 #test_nfrs_with_science
Tip: Prioritise Counterexamples "no number of positive outcomes [...] can confirm a scientific theory, but a single counterexample is logically decisive" Karl Popper, By LSE library - http://www.flickr.com/photos/lselibrary/3833724834/in/set- 72157623156680255/, No restrictions, https://commons.wikimedia.org/w/index.php? curid=9694262 @kevinrudland / @al94781 #test_nfrs_with_science
Example: A Scary Hypotheses : Tolerance of Downstream Failures Example 3: H1: Failure of the ABC service to connect to the XYZ service is handled without manual intervention, logged, and reported back to the ABC service within 100ms of the call to the XYZ service being made. If there are > X errors in a 10 second window, the circuit-breaker in the ABC service opens H0: Failure of the ABC service to connect to the XYZ service causes failures which need manual intervention, which are not logged, and the circuit-breaker in the ABC service does not open, no matter how many, or how frequently errors are seen @kevinrudland / @al94781 #test_nfrs_with_science
Tip: Apply Occam’s Razor "the simplest explanation is usually the correct one” William of Ockham, credited with originating Occam’s Razor By self-created (Moscarlop) - Own work, CC BY-SA 3.0, https://commons.wikimedia. org/w/index.php?curid=5523066 @kevinrudland / @al94781 #test_nfrs_with_science
Tip: Be Repeatable "If an experiment cannot be repeated to produce the same results, this implies that the original results might have been in error." Marie Curie, conducted pioneering research on radioactivity By Unknown - Christie's, [1], Public Domain, https://commons.wikimedia.org/w/index.php? curid=32735756 @kevinrudland / @al94781 #test_nfrs_with_science
Subjects Think: ● How many of them? ● How are they connected? ● What do they already know? Milgram’s Obedience Experiment Image from: https://thoughtmaybe.com/the-milgram-experiment-obedience/ @kevinrudland / @al94781 #test_nfrs_with_science
Example: Hypothesis - v.2 GIVEN n instances of the ABC_SERVICE with the DEFAULT_CONFIGURATION and empty REQUEST_CACHES, and which are connected to n instances of the downstream XYZ_SERVICE with the DEFAULT_CONFIGURATION via EUREKA over the PROD_NETWORK H1: WHEN xxx requests per second are sent to the ABC_SERVICE THEN the average will respond successfully within 0.5 seconds and less than 1% will result in an error H0: WHEN xxx requests per second are sent to the ABC_SERVICE THEN the average will not respond successfully within 0.5 seconds and more than 1% will result in an error @kevinrudland / @al94781 #test_nfrs_with_science
Tip: Sometimes; Keep Looking “Absence of evidence is not evidence of absence.” Chien-Shiung Wu, Chinese American experimental physicist who made significant contributions in the field of nuclear physics. She is best known for conducting the Wu experiment, which contradicted the hypothetical law of conservation of parity By Smithsonian Institution from United States - Chien-shiung Wu (1912-1997)Uploaded by Fæ, No restrictions, https://commons.wikimedia.org/w/index.php?curid=18877882 @kevinrudland / @al94781 #test_nfrs_with_science
WARNING! Sometimes things go very well in test and this can give you a false sense of security: Sometimes you can be live-like, but not live-like enough (and didn’t realise). What went wrong? How did we manage to predict things? @kevinrudland / @al94781 #test_nfrs_with_science
Meta-Hypothesis #2: H1: the only environment exactly like production is production, therefore only results from testing in production really count. @kevinrudland / @al94781 #test_nfrs_with_science
Tip: Fake it “To be ecologically valid, the methods, materials and setting of a study must approximate the real-life situation that is under investigation.” Francis Bacon, philosophical advocate and practitioner of the scientific method By Paul van Somer (1576/1578–1622) - pl.pinterest.com, Public Domain, https: //commons.wikimedia.org/w/index.php?curid=19958108 @kevinrudland / @al94781 #test_nfrs_with_science
Tip: Axes of Realism So, if science can do it, what are our axes of realism?: ● Environment ○ Configuration ○ Data ○ Platform and connectivity ○ Monitoring ● Also, are you in; ○ a Normal-Running or a DR scenario? ● And are there; ○ events such as a fail and recover,? ○ or a slow down and speed up? Etc. @kevinrudland / @al94781 #test_nfrs_with_science
Tip: Realism Techniques ● Take a copy of prod data ● Take a copy of prod requests ● Close observation of prod followed by replication in the “lab” @kevinrudland / @al94781 #test_nfrs_with_science
Method: Data • Of all the environmental elements, data can have a MOST SIGNIFICANT IMPACT • Easiest to change • This is the type of data, its mix and variety • Look at your data model – Examine: 1:n – Examine: nullable @kevinrudland / @al94781 #test_nfrs_with_science
Charles R. Drew, developed improved techniques for blood storage, and applied his expert knowledge to developing large-scale blood banks By Associated Photographic Services, Inc - Original Repository: Howard University. Moorland-Spingarn Research Center. Charles R. Drew Papers http://profiles.nlm.nih. gov/ps/retrieve/ResourceMetadata/BGBBCT, https://en.wikipedia.org/w/index.php? curid=47837720 Tip: Manipulate Independent Variables @kevinrudland / @al94781 #test_nfrs_with_science
Example: Hypothesis - v.3.1 GIVEN n instances of the ABC_SERVICE with the DEFAULT_CONFIGURATION and empty REQUEST_CACHES, and which are connected to n instances of the downstream XYZ_SERVICE with the DEFAULT_CONFIGURATION via EUREKA over the PROD_NETWORK H1: WHEN a mix of xxx VALID requests per second and yyy INVALID requests per second are sent to the ABC_SERVICE THEN the average will respond correctly within 0.5 seconds and less than 1% will result in an error H0: WHEN a mix of xxx VALID requests per second and yyy INVALID requests per second are sent to the ABC_SERVICE THEN the average will not respond correctly within 0.5 seconds and more than 1% will result in an error @kevinrudland / @al94781 #test_nfrs_with_science
Example: Hypothesis - v.3.2 GIVEN n instances of the ABC_SERVICE with the DEFAULT_CONFIGURATION and empty REQUEST_CACHES, and which are connected to n instances of the downstream XYZ_SERVICE with the DEFAULT_CONFIGURATION via EUREKA over the PROD_NETWORK with BACKGROUND_LOAD H1: WHEN a mix of xxx VALID requests per second and yyy INVALID requests per second are sent to the ABC_SERVICE THEN the average will respond correctly within 0.5 seconds and less than 1% will result in an error H0: WHEN a mix of xxx VALID requests per second and yyy INVALID requests per second are sent to the ABC_SERVICE THEN the average will not respond correctly within 0.5 seconds and more than 1% will result in an error @kevinrudland / @al94781 #test_nfrs_with_science
Method: Events Dear boy… An 1888 lithograph of the 1883 eruption of Krakatoa. By Lithograph: Parker & Coward, Britain; - Image published as Plate 1 in The eruption of Krakatoa, and subsequent phenomena. Report of the Krakatoa Committee of the Royal Society (London, Trubner & Co., 1888)., Public Domain, https://commons.wikimedia. org/w/index.php?curid=7696837 @kevinrudland / @al94781 #test_nfrs_with_science
Method: Execute • Manipulate the independent variables: – e.g. req./sec. – e.g. concurrent users – e.g. req. mix • One at a time @kevinrudland / @al94781 #test_nfrs_with_science
Tip: Be Empirical "To be termed scientific, a method of inquiry is commonly based on empirical or measurable evidence" Rosalind Franklin, had a key role (largely unacknowledged during her lifetime) in discovering the helical structure of DNA By Jewish Chronicle Archive/Heritage-Imageshttp://www.britannica.com/EBchecked/topic- art/217394/99712/Rosalind-Franklin, Fair use, https://en.wikipedia.org/w/index.php? curid=24959067 @kevinrudland / @al94781 #test_nfrs_with_science
Tip: Measure Dependent Variables Sir C. V. Rahman, winner of the 1030 Nobel Prize for Physics for his work on light scattering By Nobel Foundation - From Nobel Lectures, Physics 1922-1941, Elsevier Publishing Company, Amsterdam, 1965, Public Domain, https://commons.wikimedia.org/w/index. php?curid=4213636 @kevinrudland / @al94781 #test_nfrs_with_science
Tip: Stats Can Help • Why do you care?: • Means mean nothing • The impacts of (statistical) distribution @kevinrudland / @al94781 #test_nfrs_with_science
Example: Hypothesis - v.4 GIVEN n instances of the ABC_SERVICE with the DEFAULT_CONFIGURATION and empty REQUEST_CACHES, and which are connected to n instances of the downstream XYZ_SERVICE with the DEFAULT_CONFIGURATION via EUREKA over the PROD_NETWORK with BACKGROUND_LOAD H1: WHEN a mix of xxx VALID requests per second and xxx INVALID requests per second are sent to the ABC_SERVICE THEN 95% will respond correctly within 0.5 seconds and less than 1% will result in an error H0: WHEN a mix of xxx VALID requests per second and xxx INVALID requests per second are sent to the ABC_SERVICE THEN 95% will not respond correctly within 0.5 seconds and more than 1% will result in an error @kevinrudland / @al94781 #test_nfrs_with_science
Conclusion: Refine and Retest • We did not verify H1 :( • Do you need to look elsewhere? • Revisit H0 / H1 - Apply Occam’s Razor • Only change 1 thing at a time @kevinrudland / @al94781 #test_nfrs_with_science
Tip: Tune & Fix Too ● Change one thing at a time (unless it’s your DVs) ● Think!: ○ Will it have the desired effect? ○ Will it have side-effects? ○ Tuning the slowest won’t always win Image: cc: Kalle Hyttinen - https://www.flickr.com/photos/129560755@N03 @kevinrudland / @al94781 #test_nfrs_with_science
The bit you always write at the end after you’ve done all the work which summarises your findings. ABSTRACT @kevinrudland / @al94781 #test_nfrs_with_science
Meta-Hypothesis #1: H1: We can give ourselves confidence that our hyper-distributed systems will meet our Non Functional Requirements by applying the same method that scientists use to understand and predict the world around us: Science; specifically the Scientific Method. @kevinrudland / @al94781 #test_nfrs_with_science P R O V E N
cc: ptrlx - https://www.flickr.com/photos/58615912@N05 Oh, and did we mention we (Capgemini) are hiring? (see bit.ly/cg-jvm-jobs-nfr for lots more info) @kevinrudland / @al94781 #test_nfrs_with_science
andrewharmellaw
rigolon
hiashisan
linedevth
kentosuzuki
andpad
k_koheyi
_kensh
hansolo21
kawajiritakayuki
shimacos
riru
.png){kind=icon}
pharma_x_tech
andyhume
sferik
maggiecrowley
reverentgeek
frogandcode
brianwarren
lauravandoore
jeffersonlam
tanoku
palkan
bluesmoon
productmarketing