Lean Startup Conference 2019: 8 Maladies of Rapid Experimentation

During Simply Business’ growth from a small company to the largest ever insurtech exit to Travelers, ‘Dr.’ Lukas confronted each of the 8 deadly maladies personally. He watched as great experiments and expansive ideas withered mercilessly. In this fast paced talk, you can experience the diagnoses and confront the cures. These ailments are common in advanced teams, and need to be monitored and treated carefully. Their cure often involves rework of culture, people and techniques. You will grow to understand Low-hanging-fruit-itis, Caged-in Syndrome, No-rigor-mortis and 5 other debilitating conditions. You needn’t suffer as he did. Apply these lessons liberally (for external use only), at least twice a month, and you will be healed. Lukas is Group CTO at Simply Business and has been practicing BML and similar techniques for well over a decade.

Transcript

1. © Simply Business 2019
   .
   

   View Slide

2. Who are we?
   Simply Business is an online insurance broker
   with more than 600,000 micro-business
   customers, covering over 1,000 types of business
   Our disruption? Serving customers forgotten by
   time-pressed insurance agents
   © Simply Business 2019
   .
   

   View Slide

3. Small business
   Big impact
   SMEs account for 99% of
   all UK and US businesses
   £1.9tn annual revenue
   in the UK
   $8.3tn annual revenue
   in the US
   © Simply Business 2019
   .
   

   View Slide

4. Award-winning culture
   © Simply Business 2019
   .
   

   View Slide

5. Exited to Travelers Insurance
   (2017)
   © Simply Business 2019
   .
   

   View Slide

6. ● Group CTO @SB for last 9 years (+Agile, +CI/CD, +++)
   ● Lean Startup(like) techniques for well more than a decade
   ● Current heavy focus on Architecture, Innovation and Strategy
   (and where those things block us)
   Who am I?
   © Simply Business 2019
   .
   

   View Slide

7. What is this talk about?
   ➔ disorders I’ve healed in SB
   ➔ teams often suffer from more than one
   ➔ I have cures, but no vaccines
   This is going to be sick!
   © Simply Business 2019
   .
   

   View Slide

8. What will I help you with?
   © Simply Business 2019
   Ability
   to diagnose
   team/org
   losing its way
   A language
   for the
   challenges
   .
   

   View Slide

9. © Simply Business 2019
   It is possible to fail
   in many ways...while
   to succeed is possible
   only in one way.
   Aristotle
   ‘‘
   ’’
   .
   

   View Slide

10. © Simply Business 2019
    .
    

    View Slide

11. But first, the roadmap to a malady
    1. Patients: one or more examples
    2. Symptoms: what to watch out for
    3. Diagnosis: naming the malady
    4. Consequences: what happens when symptoms go unchecked
    5. Root causes: the emotional angle
    6. Coaching: questions to ask teams
    7. Cure
    © Simply Business 2019
    .
    

    View Slide

12. PATIENT
    © Simply Business 2019
    CASE 01
    It did
    But PayPal is more expensive than current provider
    We would never switch just because people would use it
    Tested to see if PayPal would get click-throughs
    .
    

    View Slide

13. ❏ Everything must be statistically significant
    ❏ Doing experiments without success criteria or way forward
    ✓
    ✓
    ❏ Choosing tests based familiarity with the technique
    © Simply Business 2019
    CASE 01
    ✓
    .
    

    View Slide

14. © Simply Business 2019
    CASE 01
    .
    

    View Slide

15. ➔ Time wasted on experiments which don’t matter
    ➔ Experimenting because we should
    ➔ Lack of accountability
    ➔ While A/B tests are critical for incremental changes, they can’t
    be the only tool in the arsenal
    © Simply Business 2019
    A/B TESTOSTROPHY
    .
    

    View Slide

16. ➔ Fear (of damaging something) / lack of courage
    © Simply Business 2019
    A/B TESTOSTROPHY
    ➔ Doing “what has been asked”
    ➔ No deep understanding of goals
    ➔ Intolerance of failure
    .
    

    View Slide

17. © Simply Business 2019
    A/B TESTOSTROPHY
    Q: Is this valuable?
    Q: Are we moving fast enough?
    Q: What does success
    look like?
    .
    

    View Slide

18. ➔ Decide what you need to learn, then design experiment
    © Simply Business 2019
    A/B TESTOSTROPHY
    ➔ Teams accountable for results, not experiments
    ➔ Customer behaviour before treatment
    ➔ Don’t test if cost to test more than likely worth
    .
    

    View Slide

19. © Simply Business 2019
    CASE 01
    .
    

    View Slide

20. © Simply Business 2019
    .
    

    View Slide

21. Fixed numerous small niggles in back office for call center
    PATIENT
    © Simply Business 2019
    CASE 02
    To prove value of product team
    Progress would be next quarter
    In the end, team’s performance was questioned
    .
    

    View Slide

22. ❏ Continuous quick wins to prove we are worth it
    ✓
    ❏ Marginal metric improvements
    ✓
    ❏ Big change is coming (but not yet)
    ✓
    © Simply Business 2019
    CASE 02
    .
    

    View Slide

23. © Simply Business 2019
    CASE 02
    .
    

    View Slide

24. Low results
    © Simply Business 2019
    LOW-HANGING-FRUIT-ARRHEA
    .
    

    View Slide

25. ➔ No vision
    © Simply Business 2019
    LOW-HANGING-FRUIT-ARRHEA
    ➔ Fear of not proving value
    ➔ Expectation of immediate results
    .
    

    View Slide

26. Q: Are you making progress?
    © Simply Business 2019
    LOW-HANGING-FRUIT-ARRHEA
    Q: What is it?
    Q: What does success look like?
    .
    

    View Slide

27. ➔ Start with a valuable goal and don’t be distracted
    © Simply Business 2019
    LOW-HANGING-FRUIT-ARRHEA
    .
    

    View Slide

28. © Simply Business 2019
    CASE 03
    .
    

    View Slide

29. © Simply Business 2019
    .
    

    View Slide

30. PATIENT
    We A/B tested an improved quote
    comparison page (toggling)
    © Simply Business 2019
    CASE 03
    .
    

    View Slide

31. PATIENT
    We A/B tested an improved quote
    comparison page (toggling)
    Then did not act on positive results
    © Simply Business 2019
    CASE 03
    Because people questioned: “are we sure?”
    .
    

    View Slide

32. ❏ Few ideas on how to move forward
    ✓
    ❏ “Nothing different will happen so don’t try”
    ✓
    ❏ Having to prove every step, an inch at a time. Results not accepted.
    Like a court case against a powerful person
    ✓
    ❏ Major questions unresolved in spite of much activity
    ✓
    © Simply Business 2019
    CASE 03
    .
    

    View Slide

33. © Simply Business 2019
    CASE 03
    .
    

    View Slide

34. ➔ Business questions do not move forward,
    or at slow pace
    ➔ Business failure
    © Simply Business 2019
    CAGED-IN SYNDROME
    .
    

    View Slide

35. ➔ Don’t want to look bad (in one’s field of expertise)
    © Simply Business 2019
    CAGED-IN SYNDROME
    ➔ Lack of understanding of what change is needed
    ➔ Listening to others, but no point of view
    ➔ External pressure
    ➔ Anchored in current world, rather than desired future
    .
    

    View Slide

36. ➔ What are your riskiest assumptions?
    © Simply Business 2019
    CAGED-IN SYNDROME
    ➔ What happens if you are wrong about
    the way things work?
    .
    

    View Slide

37. ➔ Decide / negotiate ahead of time the burden of proof.
    Then meet that proof
    © Simply Business 2019
    CAGED-IN SYNDROME
    ➔ Test the riskiest assumptions, regardless of sensitivities
    ➔ Once proven, questions/learnings are not permitted to be reopened
    ➔ “You can question the methodology, but you can’t question
    the outcome”
    .
    

    View Slide

38. CASE 03
    © Simply Business 2019
    .
    

    View Slide

39. © Simply Business 2019
    .
    

    View Slide

40. PATIENT A
    Needed to speak to insurance partners
    about our proposition
    © Simply Business 2019
    CASE 04
    Kept enhancing and enhancing our deck
    But without testing the pitch on ‘customers’
    When we finally reached out, we didn’t use the deck
    Led to weeks of delay
    .
    

    View Slide

41. PATIENT B
    © Simply Business 2019
    CASE 04
    After choosing to improve product for higher value customers
    Experimented around which segment could be best served
    Kept testing opportunity after opportunity
    Couldn’t decide which one to go after
    The playing field remained wide open
    .
    

    View Slide

42. ❏ Continuously asking: “is this the best opportunity?”
    ✓
    ✓
    ❏ No choices made while more tests are run
    © Simply Business 2019
    CASE 04
    .
    

    View Slide

43. © Simply Business 2019
    CASE 04
    .
    

    View Slide

44. Hunt for “the best thing”
    leads to paralysis
    © Simply Business 2019
    THE-BEST POX
    .
    

    View Slide

45. ➔ Dipping toes in the water, never jumping in
    © Simply Business 2019
    THE-BEST POX
    ➔ Fear of mistakes
    ➔ Questioning of value from senior leadership
    .
    

    View Slide

46. Q: Is what you know good enough to move forward?
    © Simply Business 2019
    THE-BEST POX
    Q: How will you decide the best
    path? What are the criteria?
    How hard will it be to get
    the answer?
    .
    

    View Slide

47. ➔ Finding the “best” is hard/expensive. Choose good and go for it
    © Simply Business 2019
    THE-BEST POX
    ➔ Choosing which opportunity will likely be a heuristic or gut feel
    ➔ Make the most of that good opportunity
    ➔ Over time, optimise
    .
    

    View Slide

48. © Simply Business 2019
    CASE 04
    .
    

    View Slide

49. © Simply Business 2019
    .
    

    View Slide

50. PATIENT
    © Simply Business 2019
    CASE 05
    Team executed a series of experiments
    Always comfortable that they would pan out
    With a clear path forward
    They hadn’t solved the tricky problems
    Having slipped into delivery mode
    .
    

    View Slide

51. ❏ No emotional highs and lows as difficult truths learned or overcome
    ✓
    © Simply Business 2019
    CASE 06
    .
    

    View Slide

52. © Simply Business 2019
    CASE 06
    .
    

    View Slide

53. Building something truly new is never easy.
    If it appears to be, something is likely wrong.
    The feeling of running a startup is
    similar to a roller coaster.
    Expect that
    © Simply Business 2019
    IT’S-SO-EASY DISEASY
    .
    

    View Slide

54. ➔ Going to comfort is easy
    © Simply Business 2019
    IT’S-SO-EASY DISEASY
    ➔ Lack of time pressure (metered funding)
    ➔ “Positivity” culture (bad news is punishable)
    .
    

    View Slide

55. Q: What could mess you up?
    © Simply Business 2019
    IT’S-SO-EASY DISEASY
    Q: What concrete data do you have
    that this will work? What would
    you do to get it? .
    .
    

    View Slide

56. ➔ Ensure actual progress on riskiest assumptions
    (not excessive positivity)
    © Simply Business 2019
    IT’S-SO-EASY DISEASY
    ➔ Regular pivot / persevere / kill meetings
    ➔ Insist on strong (and improving) evidence
    .
    

    View Slide

57. © Simply Business 2019
    CASE 06
    .
    

    View Slide

58. © Simply Business 2019
    .
    

    View Slide

59. PATIENT
    Experiments proved gap in the market
    © Simply Business 2019
    CASE 06
    Learnings not diligently documented
    So didn’t stop continued speculation on
    the existence of good enough products
    .
    

    View Slide

60. ❏ Debate is never closed
    ✓
    ❏ Weak documentation of learning (hard to find, if documented)
    ✓
    ❏ Opinions trump data
    ✓
    ❏ Tests run forever, or poorly designed
    ✓
    ❏ No innovation accounting metrics (or not up to date)
    ✓
    ❏ Lack of governance (missing: standups, experiment design,
    documentation standards, pivot/persevere/kill meetings, prioritisation,
    retrospectives, communication)
    ✓
    © Simply Business 2019
    CASE 06
    .
    

    View Slide

61. © Simply Business 2019
    CASE 06
    .
    

    View Slide

62. ➔ What little progress that is made, is quickly unravelled
    ➔ Learning stuck in individuals’ heads
    ➔ Learning not in priority order
    © Simply Business 2019
    NO-RIGOR-MORTIS
    .
    

    View Slide

63. ➔ People on team fear owning the result (mercenaries)
    © Simply Business 2019
    NO-RIGOR-MORTIS
    ➔ Lack of leadership
    ➔ No accountability given or requested
    ➔ Too many non-dedicated team members
    .
    

    View Slide

64. Q: Where can I see what you’ve learned?
    © Simply Business 2019
    NO-RIGOR-MORTIS
    Q: When did you last finish an experiment?
    Q: Do you have the resources to succeed?
    .
    

    View Slide

65. ● Bi-weekly cadence in which riskiest assumptions
    are reduced. Failure to make progress is pivot
    situation (accountability)
    © Simply Business 2019
    NO-RIGOR-MORTIS
    ● Track governance carefully. May be only way to know
    if a team is making progress (innovation accounting)
    ● If needed, team capability must be addressed
    .
    

    View Slide

66. © Simply Business 2019
    CASE 06
    .
    

    View Slide

67. © Simply Business 2019
    .
    

    View Slide

68. Spent a year researching most valuable way
    to serve high value customers
    PATIENT
    © Simply Business 2019
    CASE 07
    Led to us not choosing a way forward
    More and more tests to run, none landing killer conclusion
    Kept asking: is this an opportunity?
    .
    

    View Slide

69. ❏ Understanding from first principles (we know nothing)
    ✓
    ❏ Many tests, few results
    ✓
    ❏ No courage or conviction
    ✓
    ❏ Few sharp puzzle pieces but no overall picture
    ✓
    © Simply Business 2019
    CASE 07
    .
    

    View Slide

70. © Simply Business 2019
    CASE 07
    .
    

    View Slide

71. Lots of time passes, and the team is on the
    verge of moving forward, but then…
    © Simply Business 2019
    BOIL-THE-OCEANOMA
    .
    

    View Slide

72. ➔ Fear of making a call
    © Simply Business 2019
    BOIL-THE-OCEANOMA
    ➔ Must know everything to
    decide what to do
    ➔ Waiting for the spreadsheet
    to give the answer
    .
    

    View Slide

73. © Simply Business 2019
    BOIL-THE-OCEANOMA
    Q: What progress have you made in the last two weeks?
    Q: If you had to, what would you build right now?
    Q: What are the riskiest assumptions still to be identified?
    .
    

    View Slide

74. ➔ Make small iterative steps in the right direction, rather
    than giant leaps
    © Simply Business 2019
    BOIL-THE-OCEANOMA
    ➔ Track the reduction or change in riskiest assumptions carefully
    ➔ Expect changes on the order of weeks, not months or years
    .
    

    View Slide

75. © Simply Business 2019
    CASE 07
    .
    

    View Slide

76. © Simply Business 2019
    .
    

    View Slide

77. We launched a new product for commercial landlords
    PATIENT A
    © Simply Business 2019
    CASE 08
    Before it had traction, we pulled the funding because we
    had higher short-term value opportunities
    Pulling the funding meant that the product never
    reached our target performance metrics
    .
    

    View Slide

78. The Board provided money to investigate a new market
    PATIENT B
    © Simply Business 2019
    CASE 08
    But it was unclear what a positive result would be for the Board
    When asked, the answer was...
    “Nothing would convince us to move forward”
    We gave the money back
    .
    

    View Slide

79. ❏ Even if the experiment is a success, there are no resources
    to follow up the success
    ✓
    ❏ Funding gets pulled early
    © Simply Business 2019
    CASE 08
    ✓
    .
    

    View Slide

80. © Simply Business 2019
    CASE 08
    .
    

    View Slide

81. © Simply Business 2019
    Experiments have no value since they
    won’t be followed up or get the support they need
    INVESTAPHOBIA
    .
    

    View Slide

82. ➔ Leadership never committed to project in the first place
    © Simply Business 2019
    INVESTAPHOBIA
    ➔ People expect the numbers to look good right away
    (no innovation metrics)
    .
    

    View Slide

83. Q: How important is this to
    the business?
    © Simply Business 2019
    INVESTAPHOBIA
    .
    

    View Slide

84. ➔ This is not a go/no-go situation. Funding should be ring-fenced
    for projects where opportunity is proven
    © Simply Business 2019
    INVESTAPHOBIA
    .
    

    View Slide

85. © Simply Business 2019
    CASE 08
    .
    

    View Slide

86. © Simply Business 2019
    .
    

    View Slide

87. © Simply Business 2019
    If Simply Business is this bad, how did we do so well?
    We identified and cured these painful diseases
    We are never satisfied (self-critical)
    Our culture is about learning and being a great place
    for the right kind of people
    .
    

    View Slide

88. © Simply Business 2019
    It’s all about the people/team. The more entrepreneurial, the better
    Fear of messing up is powerful – courage is needed
    Culture, and culture of experimentation are crucial (you can’t fake it)
    Must be ready to invest in ambiguity
    Good and bad experimentation can be hard to tease apart.
    Maladies provide strong early signals
    Things to remember for your own teams
    .
    

    View Slide

89. © Simply Business 2019
    Takeaways
    ➔ A/B TESTOSTROPHY – many similar tests, little learning
    Do tests that matter
    ➔ LOW-HANGING-FRUIT-ARRHEA – constant quick wins to prove worth
    Stop trying to prove you are worth it. Do the important
    ➔ CAGED-IN SYNDROME – nothing can be changed, so don’t try
    You have to change the world to succeed
    ➔ THE-BEST POX – always asking “Is this the BEST opportunity?”
    Pick a good opportunity. Finding the best takes too long
    .
    

    View Slide

90. © Simply Business 2019
    Takeaways
    ➔ IT’S-SO-EASY DISEASY – “We got this, don’t worry”
    Innovation is a roller-coaster, expect that
    ➔ NO-RIGOR-MORTIS – No innovation accounting or regular
    progress on risky assumptions
    Make sure you measure progress on a bi-weekly basis
    ➔ BOIL-THE-OCEANOMA – Need to understand it all before any action
    Small iterations towards your big goal
    ➔ INVESTAPHOBIA – No funds available even if the test succeeds
    Without funding, don’t bother with a test
    .
    

    View Slide

91. © Simply Business 2019
    .
    

    View Slide