Measuring Hard-to-Measure Things

It's all about humans.

Transcript

1. the best way to build and ship software
   Hard-to-Measure Things
   @chrissiebrodigan, GitHub User Research
   

   View Slide

2. the best way to build and ship software
   A little about me ….
   2
   !
   •Live in Sausalito
   •Trained historian
   •UX designer
   •GitHub’s first researcher
   

   View Slide

3. the best way to build and ship software
   

   View Slide

4. the best way to build and ship software
   

   View Slide

5. the best way to build and ship software
   GitHub 3 years ago …
   Justin & I research things together now!!
   

   View Slide

6. the best way to build and ship software
   What would interest people about
   user research at Monitorama?
   

   View Slide

7. the best way to build and ship software
   

   View Slide

8. “
   the best way to build and ship software 8
   !
   WHAT ARE THE WAYS
   HIDDEN VARIABLES
   SCREW US?
   – @D2FN
   

   View Slide

9. the best way to build and ship software
   All products have
   blind spots
   

   View Slide

10. the best way to build and ship software
    

    View Slide

11. the best way to build and ship software
    Research is
    GitHub’s flashlight
    

    View Slide

12. the best way to build and ship software
    Why?
    User
    research
    What is
    happening
    Monitoring
    tools
    

    View Slide

13. the best way to build and ship software
    WTF are those
    humans doing?
    

    View Slide

14. the best way to build and ship software
    How do you uncover why
    something is happening?
    

    View Slide

15. “
    the best way to build and ship software 15
    !
    GAP FROM 

    METRIC/GRAPH TO
    INSIGHT CAN BE HUGE
    – @RANDOMMOOD
    

    View Slide

16. the best way to build and ship software
    

    View Slide

17. “
    the best way to build and ship software 17
    !
    RESEARCH:
    WITHOUT IT ALL YOU
    HAVE IS LUCK.
    - @SBOAK
    

    View Slide

18. the best way to build and ship software
    Human instrument for
    socially meaningful questions
    

    View Slide

19. the best way to build and ship software
    Get out from behind the
    numbers & in front of people
    

    View Slide

20. the best way to build and ship software
    

    View Slide

21. the best way to build and ship software
    When you talk with people look for …
    21
    !
    1. Goals
    2. Motivations
    3. Workarounds
    

    Humans do really interesting things with software
    when they’re confused & frustrated
    

    View Slide

22. the best way to build and ship software
    Example:
    Someone is running a large job
    

    View Slide

23. the best way to build and ship software
    Solution:
    Limit or block them?
    

    View Slide

24. the best way to build and ship software
    Modest Proposal …
    24
    !
    1. Talk with them about their goals
    2. Look for their motivations
    3. Design a better solution
    *https://en.wikipedia.org/wiki/A_Modest_Proposal
    

    View Slide

25. the best way to build and ship software
    Develop trust in research
    Listen to the right people
    

    View Slide

26. the best way to build and ship software
    Let me show you …
    

    View Slide

27. the best way to build and ship software
    Who uses GitHub.com?
    

    View Slide

28. the best way to build and ship software
    Who uses
    GitHub Enterprise?
    

    View Slide

29. the best way to build and ship software
    Who administers
    GitHub Enterprise?
    

    View Slide

30. the best way to build and ship software
    ♥♥
    ♥ HUG
    

    View Slide

31. the best way to build and ship software
    Product that’s out of our
    hands and on your hardware
    

    View Slide

32. the best way to build and ship software
    Abundance of data
    GitHub.com
    

    View Slide

33. the best way to build and ship software
    Absence of data
    GitHub Enterprise
    

    View Slide

34. the best way to build and ship software
    How do you study a
    BLACK BOX?
    

    View Slide

35. the best way to build and ship software
    Story #1
    System Administrator
    Happiness
    

    View Slide

36. the best way to build and ship software
    Designed a
    bi-annual survey
    

    View Slide

37. the best way to build and ship software
    Instrument measures …
    37
    !
    1. Demographics
    2. Product Experience
    3. Net Promoter Score
    

    View Slide

38. “
    the best way to build and ship software 38
    !
    FASTER, SMALLER PACKAGES
    AND MORE RELIABLE UPDATE
    PROCEDURE (I WET MY PANTS
    EACH TIME I UPGRADE.)
    Where do we need to improve the most?
    

    View Slide

39. the best way to build and ship software
    Instrument is designed to be …
    39
    !
    • Repeatable
    • Longitudinal (company-level)
    • Actionable
    

    View Slide

40. the best way to build and ship software
    Takeaway:
    Increase product focus on the
    administrator experience
    

    View Slide

41. the best way to build and ship software
    Story #2.
    Slow rollout of a
    large feature
    

    View Slide

42. the best way to build and ship software
    Git Large File Storage
    (Git LFS)
    on GitHub
    

    View Slide

43. the best way to build and ship software
    3. Early access program
    

    View Slide

44. the best way to build and ship software
    Things that kept our
    team up at night …
    

    View Slide

45. the best way to build and ship software
    Pricing & Billing
    Fair & easy?
    

    View Slide

46. the best way to build and ship software
    Infrastructure
    Responsible & stable?
    

    View Slide

47. the best way to build and ship software
    

    View Slide

48. the best way to build and ship software
    Support
    Generate volume?
    

    View Slide

49. the best way to build and ship software
    3-part research plan
    

    View Slide

50. the best way to build and ship software
    Research plan …
    50
    !
    1. Survey
    2. Pre-release
    3. Early access program:
    • Entry screener
    • Controlled experiment
    • Exit survey
    

    View Slide

51. the best way to build and ship software
    1. Survey
    Sample size: 1,800
    

    View Slide

52. the best way to build and ship software
    2. Pre-release
    Sample size: 20
    

    View Slide

53. the best way to build and ship software
    3. Early access program
    Sample size: 13, 000+
    

    View Slide

54. the best way to build and ship software
    EAP runs as a
    controlled experiment
    

    View Slide

55. the best way to build and ship software
    

    View Slide

56. the best way to build and ship software
    

    View Slide

57. the best way to build and ship software
    

    View Slide

58. the best way to build and ship software
    Six weeks
    

    View Slide

59. the best way to build and ship software
    Sent two exit surveys
    

    View Slide

60. the best way to build and ship software
    

    View Slide

61. the best way to build and ship software
    If we could grant you
    one wish to improve GitHub,
    what would you wish for?
    

    View Slide

62. the best way to build and ship software
    All 3 data sources changed
    our knowledge of the world
    

    View Slide

63. the best way to build and ship software
    But, we forgot about
    something important
    

    View Slide

64. the best way to build and ship software
    We’ve been telling
    people for years DON’T
    use large files with Git
    

    View Slide

65. the best way to build and ship software
    Reified with
    file size limits on
    GitHub.com
    

    View Slide

66. the best way to build and ship software
    Asked people to leap into
    a new belief system, adopt
    a tool, & change behavior
    

    View Slide

67. the best way to build and ship software
    Create a path to
    change behaviors
    

    View Slide

68. the best way to build and ship software
    Framework for behaviors …
    68
    !
    1. Current – Used to doing
    2. New – Open to try to achieve goals
    3. Aspirational – Future big changes.
    Current
    New
    Aspirational
    Thank you Diane Loviglio for teaching me!
    

    View Slide

69. the best way to build and ship software
    Git LFS
    1.
    2.
    3.
    4.
    Current
    behaviors
    Use Dropbox
    Reality
    Don’t use Git 

    with large
    files
    New
    behaviors
    Use Git LFS
    Aspirational

    behaviors
    Growth
    

    View Slide

70. the best way to build and ship software
    Going from current to
    aspirational behavior is
    kind of crazy
    

    View Slide

71. the best way to build and ship software
    Takeaway:
    Adoption takes time & trust
    especially across team workflows
    

    View Slide

72. the best way to build and ship software
    Next EAP:
    Org/user permissions
    

    View Slide

73. the best way to build and ship software
    

    View Slide

74. the best way to build and ship software
    

    View Slide

75. the best way to build and ship software
    

    View Slide

76. the best way to build and ship software
    Will belief systems
    change with new
    permissions?
    

    View Slide

77. the best way to build and ship software
    Story #3.
    The undo button
    

    View Slide

78. the best way to build and ship software
    

    View Slide

79. the best way to build and ship software
    Branching v. forking
    

    View Slide

80. the best way to build and ship software
    Activity:
    Feature prioritization
    

    View Slide

81. the best way to build and ship software
    Most important to least important:
    81
    !
    •Branch Permissions
    •Automatically syncing forks
    •Sign-off
    •Only merge with passing tests
    •Undo button
    •Disable force push
    •Private forks
    •Prevent merging from the command line
    

    View Slide

82. the best way to build and ship software
    Most important to least important:
    82
    !
    •Branch Permissions
    •Automatically syncing forks
    •Sign-off
    •Only merge with passing tests
    •Undo button
    •Disable force push
    •Private forks
    •Prevent merging from the command line
    

    View Slide

83. the best way to build and ship software
    “What! There’s an
    undo button?”
    

    View Slide

84. the best way to build and ship software
    “What do you mean by
    undo button?”
    

    View Slide

85. the best way to build and ship software
    “You tell me, where would
    you use an undo button?”
    

    View Slide

86. the best way to build and ship software
    “How would an undo
    button help you?”
    

    View Slide

87. the best way to build and ship software
    “Tell me about a story when an
    undo button have helped you?”
    

    View Slide

88. “
    the best way to build and ship software 88
    !
    WHAT ARE THE WAYS
    HIDDEN VARIABLES HELP
    US CONNECT WITH
    HUMANS?
    -@TENACIOUSCB
    

    View Slide

89. the best way to build and ship software
    Hard-to-measure things
    are fun to measure
    

    View Slide

90. the best way to build and ship software
    Guarantee:
    If you think in terms of human behaviors
    & talk with people
    You will become better at
    discovering & solving problems
    

    View Slide

91. the best way to build and ship software
    You are my people,
    I want to hang out, help out
    

    View Slide

92. the best way to build and ship software
    • Pull Requests
    • Issues
    • Admin-UI
    • Permissions
    • New User Experiences
    Are you interested?
    cb@github.com
    

    View Slide

93. the best way to build and ship software
    ♥ Thank You ♥
    cb@github.com @tenacicouscb
    

    View Slide