Lean Startup Conference 2019: 8 Maladies of Rapid Experimentation

During Simply Business’ growth from a small company to the largest ever insurtech exit to Travelers, ‘Dr.’ Lukas confronted each of the 8 deadly maladies personally. He watched as great experiments and expansive ideas withered mercilessly. In this fast paced talk, you can experience the diagnoses and confront the cures. These ailments are common in advanced teams, and need to be monitored and treated carefully. Their cure often involves rework of culture, people and techniques. You will grow to understand Low-hanging-fruit-itis, Caged-in Syndrome, No-rigor-mortis and 5 other debilitating conditions. You needn’t suffer as he did. Apply these lessons liberally (for external use only), at least twice a month, and you will be healed. Lukas is Group CTO at Simply Business and has been practicing BML and similar techniques for well over a decade.

Transcript

1. © Simply Business 2019 .
2. Who are we? Simply Business is an online insurance broker with more than 600,000 micro-business customers, covering over 1,000 types of business Our disruption? Serving customers forgotten by time-pressed insurance agents © Simply Business 2019 .
3. Small business Big impact SMEs account for 99% of all UK and US businesses £1.9tn annual revenue in the UK $8.3tn annual revenue in the US © Simply Business 2019 .
4. Award-winning culture © Simply Business 2019 .
5. Exited to Travelers Insurance (2017) © Simply Business 2019 .
6. • Group CTO @SB for last 9 years (+Agile, +CI/CD, +++) • Lean Startup(like) techniques for well more than a decade • Current heavy focus on Architecture, Innovation and Strategy (and where those things block us) Who am I? © Simply Business 2019 .
7. What is this talk about? ➔ disorders I’ve healed in SB ➔ teams often suffer from more than one ➔ I have cures, but no vaccines This is going to be sick! © Simply Business 2019 .
8. What will I help you with? © Simply Business 2019 Ability to diagnose team/org losing its way A language for the challenges .
9. © Simply Business 2019 It is possible to fail in many ways...while to succeed is possible only in one way. Aristotle ‘‘ ’’ .
10. © Simply Business 2019 .
11. But first, the roadmap to a malady 1. Patients: one or more examples 2. Symptoms: what to watch out for 3. Diagnosis: naming the malady 4. Consequences: what happens when symptoms go unchecked 5. Root causes: the emotional angle 6. Coaching: questions to ask teams 7. Cure © Simply Business 2019 .
12. PATIENT © Simply Business 2019 CASE 01 It did But PayPal is more expensive than current provider We would never switch just because people would use it Tested to see if PayPal would get click-throughs .
13. ❏ Everything must be statistically significant ❏ Doing experiments without success criteria or way forward ✓ ✓ ❏ Choosing tests based familiarity with the technique © Simply Business 2019 CASE 01 ✓ .
14. © Simply Business 2019 CASE 01 .
15. ➔ Time wasted on experiments which don’t matter ➔ Experimenting because we should ➔ Lack of accountability ➔ While A/B tests are critical for incremental changes, they can’t be the only tool in the arsenal © Simply Business 2019 A/B TESTOSTROPHY .
16. ➔ Fear (of damaging something) / lack of courage © Simply Business 2019 A/B TESTOSTROPHY ➔ Doing “what has been asked” ➔ No deep understanding of goals ➔ Intolerance of failure .
17. © Simply Business 2019 A/B TESTOSTROPHY Q: Is this valuable? Q: Are we moving fast enough? Q: What does success look like? .
18. ➔ Decide what you need to learn, then design experiment © Simply Business 2019 A/B TESTOSTROPHY ➔ Teams accountable for results, not experiments ➔ Customer behaviour before treatment ➔ Don’t test if cost to test more than likely worth .
19. © Simply Business 2019 CASE 01 .
20. © Simply Business 2019 .
21. Fixed numerous small niggles in back office for call center PATIENT © Simply Business 2019 CASE 02 To prove value of product team Progress would be next quarter In the end, team’s performance was questioned .
22. ❏ Continuous quick wins to prove we are worth it ✓ ❏ Marginal metric improvements ✓ ❏ Big change is coming (but not yet) ✓ © Simply Business 2019 CASE 02 .
23. © Simply Business 2019 CASE 02 .
24. Low results © Simply Business 2019 LOW-HANGING-FRUIT-ARRHEA .
25. ➔ No vision © Simply Business 2019 LOW-HANGING-FRUIT-ARRHEA ➔ Fear of not proving value ➔ Expectation of immediate results .
26. Q: Are you making progress? © Simply Business 2019 LOW-HANGING-FRUIT-ARRHEA Q: What is it? Q: What does success look like? .
27. ➔ Start with a valuable goal and don’t be distracted © Simply Business 2019 LOW-HANGING-FRUIT-ARRHEA .
28. © Simply Business 2019 CASE 03 .
29. © Simply Business 2019 .
30. PATIENT We A/B tested an improved quote comparison page (toggling) © Simply Business 2019 CASE 03 .
31. PATIENT We A/B tested an improved quote comparison page (toggling) Then did not act on positive results © Simply Business 2019 CASE 03 Because people questioned: “are we sure?” .
32. ❏ Few ideas on how to move forward ✓ ❏ “Nothing different will happen so don’t try” ✓ ❏ Having to prove every step, an inch at a time. Results not accepted. Like a court case against a powerful person ✓ ❏ Major questions unresolved in spite of much activity ✓ © Simply Business 2019 CASE 03 .
33. © Simply Business 2019 CASE 03 .
34. ➔ Business questions do not move forward, or at slow pace ➔ Business failure © Simply Business 2019 CAGED-IN SYNDROME .
35. ➔ Don’t want to look bad (in one’s field of expertise) © Simply Business 2019 CAGED-IN SYNDROME ➔ Lack of understanding of what change is needed ➔ Listening to others, but no point of view ➔ External pressure ➔ Anchored in current world, rather than desired future .
36. ➔ What are your riskiest assumptions? © Simply Business 2019 CAGED-IN SYNDROME ➔ What happens if you are wrong about the way things work? .
37. ➔ Decide / negotiate ahead of time the burden of proof. Then meet that proof © Simply Business 2019 CAGED-IN SYNDROME ➔ Test the riskiest assumptions, regardless of sensitivities ➔ Once proven, questions/learnings are not permitted to be reopened ➔ “You can question the methodology, but you can’t question the outcome” .
38. CASE 03 © Simply Business 2019 .
39. © Simply Business 2019 .
40. PATIENT A Needed to speak to insurance partners about our proposition © Simply Business 2019 CASE 04 Kept enhancing and enhancing our deck But without testing the pitch on ‘customers’ When we finally reached out, we didn’t use the deck Led to weeks of delay .
41. PATIENT B © Simply Business 2019 CASE 04 After choosing to improve product for higher value customers Experimented around which segment could be best served Kept testing opportunity after opportunity Couldn’t decide which one to go after The playing field remained wide open .
42. ❏ Continuously asking: “is this the best opportunity?” ✓ ✓ ❏ No choices made while more tests are run © Simply Business 2019 CASE 04 .
43. © Simply Business 2019 CASE 04 .
44. Hunt for “the best thing” leads to paralysis © Simply Business 2019 THE-BEST POX .
45. ➔ Dipping toes in the water, never jumping in © Simply Business 2019 THE-BEST POX ➔ Fear of mistakes ➔ Questioning of value from senior leadership .
46. Q: Is what you know good enough to move forward? © Simply Business 2019 THE-BEST POX Q: How will you decide the best path? What are the criteria? How hard will it be to get the answer? .
47. ➔ Finding the “best” is hard/expensive. Choose good and go for it © Simply Business 2019 THE-BEST POX ➔ Choosing which opportunity will likely be a heuristic or gut feel ➔ Make the most of that good opportunity ➔ Over time, optimise .
48. © Simply Business 2019 CASE 04 .
49. © Simply Business 2019 .
50. PATIENT © Simply Business 2019 CASE 05 Team executed a series of experiments Always comfortable that they would pan out With a clear path forward They hadn’t solved the tricky problems Having slipped into delivery mode .
51. ❏ No emotional highs and lows as difficult truths learned or overcome ✓ © Simply Business 2019 CASE 06 .
52. © Simply Business 2019 CASE 06 .
53. Building something truly new is never easy. If it appears to be, something is likely wrong. The feeling of running a startup is similar to a roller coaster. Expect that © Simply Business 2019 IT’S-SO-EASY DISEASY .
54. ➔ Going to comfort is easy © Simply Business 2019 IT’S-SO-EASY DISEASY ➔ Lack of time pressure (metered funding) ➔ “Positivity” culture (bad news is punishable) .
55. Q: What could mess you up? © Simply Business 2019 IT’S-SO-EASY DISEASY Q: What concrete data do you have that this will work? What would you do to get it? . .
56. ➔ Ensure actual progress on riskiest assumptions (not excessive positivity) © Simply Business 2019 IT’S-SO-EASY DISEASY ➔ Regular pivot / persevere / kill meetings ➔ Insist on strong (and improving) evidence .
57. © Simply Business 2019 CASE 06 .
58. © Simply Business 2019 .
59. PATIENT Experiments proved gap in the market © Simply Business 2019 CASE 06 Learnings not diligently documented So didn’t stop continued speculation on the existence of good enough products .
60. ❏ Debate is never closed ✓ ❏ Weak documentation of learning (hard to find, if documented) ✓ ❏ Opinions trump data ✓ ❏ Tests run forever, or poorly designed ✓ ❏ No innovation accounting metrics (or not up to date) ✓ ❏ Lack of governance (missing: standups, experiment design, documentation standards, pivot/persevere/kill meetings, prioritisation, retrospectives, communication) ✓ © Simply Business 2019 CASE 06 .
61. © Simply Business 2019 CASE 06 .
62. ➔ What little progress that is made, is quickly unravelled ➔ Learning stuck in individuals’ heads ➔ Learning not in priority order © Simply Business 2019 NO-RIGOR-MORTIS .
63. ➔ People on team fear owning the result (mercenaries) © Simply Business 2019 NO-RIGOR-MORTIS ➔ Lack of leadership ➔ No accountability given or requested ➔ Too many non-dedicated team members .
64. Q: Where can I see what you’ve learned? © Simply Business 2019 NO-RIGOR-MORTIS Q: When did you last finish an experiment? Q: Do you have the resources to succeed? .
65. • Bi-weekly cadence in which riskiest assumptions are reduced. Failure to make progress is pivot situation (accountability) © Simply Business 2019 NO-RIGOR-MORTIS • Track governance carefully. May be only way to know if a team is making progress (innovation accounting) • If needed, team capability must be addressed .
66. © Simply Business 2019 CASE 06 .
67. © Simply Business 2019 .
68. Spent a year researching most valuable way to serve high value customers PATIENT © Simply Business 2019 CASE 07 Led to us not choosing a way forward More and more tests to run, none landing killer conclusion Kept asking: is this an opportunity? .
69. ❏ Understanding from first principles (we know nothing) ✓ ❏ Many tests, few results ✓ ❏ No courage or conviction ✓ ❏ Few sharp puzzle pieces but no overall picture ✓ © Simply Business 2019 CASE 07 .
70. © Simply Business 2019 CASE 07 .
71. Lots of time passes, and the team is on the verge of moving forward, but then… © Simply Business 2019 BOIL-THE-OCEANOMA .
72. ➔ Fear of making a call © Simply Business 2019 BOIL-THE-OCEANOMA ➔ Must know everything to decide what to do ➔ Waiting for the spreadsheet to give the answer .
73. © Simply Business 2019 BOIL-THE-OCEANOMA Q: What progress have you made in the last two weeks? Q: If you had to, what would you build right now? Q: What are the riskiest assumptions still to be identified? .
74. ➔ Make small iterative steps in the right direction, rather than giant leaps © Simply Business 2019 BOIL-THE-OCEANOMA ➔ Track the reduction or change in riskiest assumptions carefully ➔ Expect changes on the order of weeks, not months or years .
75. © Simply Business 2019 CASE 07 .
76. © Simply Business 2019 .
77. We launched a new product for commercial landlords PATIENT A © Simply Business 2019 CASE 08 Before it had traction, we pulled the funding because we had higher short-term value opportunities Pulling the funding meant that the product never reached our target performance metrics .
78. The Board provided money to investigate a new market PATIENT B © Simply Business 2019 CASE 08 But it was unclear what a positive result would be for the Board When asked, the answer was... “Nothing would convince us to move forward” We gave the money back .
79. ❏ Even if the experiment is a success, there are no resources to follow up the success ✓ ❏ Funding gets pulled early © Simply Business 2019 CASE 08 ✓ .
80. © Simply Business 2019 CASE 08 .
81. © Simply Business 2019 Experiments have no value since they won’t be followed up or get the support they need INVESTAPHOBIA .
82. ➔ Leadership never committed to project in the first place © Simply Business 2019 INVESTAPHOBIA ➔ People expect the numbers to look good right away (no innovation metrics) .
83. Q: How important is this to the business? © Simply Business 2019 INVESTAPHOBIA .
84. ➔ This is not a go/no-go situation. Funding should be ring-fenced for projects where opportunity is proven © Simply Business 2019 INVESTAPHOBIA .
85. © Simply Business 2019 CASE 08 .
86. © Simply Business 2019 .
87. © Simply Business 2019 If Simply Business is this bad, how did we do so well? We identified and cured these painful diseases We are never satisfied (self-critical) Our culture is about learning and being a great place for the right kind of people .
88. © Simply Business 2019 It’s all about the people/team. The more entrepreneurial, the better Fear of messing up is powerful – courage is needed Culture, and culture of experimentation are crucial (you can’t fake it) Must be ready to invest in ambiguity Good and bad experimentation can be hard to tease apart. Maladies provide strong early signals Things to remember for your own teams .
89. © Simply Business 2019 Takeaways ➔ A/B TESTOSTROPHY – many similar tests, little learning Do tests that matter ➔ LOW-HANGING-FRUIT-ARRHEA – constant quick wins to prove worth Stop trying to prove you are worth it. Do the important ➔ CAGED-IN SYNDROME – nothing can be changed, so don’t try You have to change the world to succeed ➔ THE-BEST POX – always asking “Is this the BEST opportunity?” Pick a good opportunity. Finding the best takes too long .
90. © Simply Business 2019 Takeaways ➔ IT’S-SO-EASY DISEASY – “We got this, don’t worry” Innovation is a roller-coaster, expect that ➔ NO-RIGOR-MORTIS – No innovation accounting or regular progress on risky assumptions Make sure you measure progress on a bi-weekly basis ➔ BOIL-THE-OCEANOMA – Need to understand it all before any action Small iterations towards your big goal ➔ INVESTAPHOBIA – No funds available even if the test succeeds Without funding, don’t bother with a test .
91. © Simply Business 2019 .