How may I enhance your developer experience?

Productivity, effectiveness, internal tools, foundation, infrastructure, developer experience - these are just a few names for an area of engineering focused on making developers of an organization happy and productive. In this talk, I present Slack's take on building a better developer experience for engineers working on mobile. I will dissect this vast area through the lens of how we measure it and the kinds of efforts we have chosen to tackle to move those measurements. If you're into developer productivity as well as metrics & surveys, this talk is for you.

Transcript

1. How may I enhance your developer experience? Valera Zakharov
2. https://twitter.com/iamdevloper/status/1019108240632762368
4. I think there's an enormous amount of developer productivity and job satisfaction that could be recaptured by making the systems we depend on more reliable.
5. “We want Slack to be the best place in the world to make software” Cal Henderson, CTO of Slack ~Q3 of 2016
8. “I wouldn’t want to be a mobile developer anywhere else” ~ A Slack Mobile Dev in 5 years
9. Why Developer Experience?
10. Mission Build tools & systems that empower mobile developers to ship code with confidence while enjoying a pleasant and productive engineering experience. Workflow Quality
11. Team, Customers, Stats
12. General 2 platforms/apps 3.5 m Weekly Active Users on Android 5.8 m on iOS 2 github.com repos ~60 mobile devs ~100 PRs merged to master per week per repo 10 required checks on Android (~17 min) 9 required checks on iOS (~28 min)
13. Tests 2234 unit 330 functional UI 1 perf 1 e2e 4914 unit 590 functional UI 4 perf 92 e2e iOS Android
14. CI 1 jenkins instance ~30 jenkins jobs in use ~5500 CI jobs runs per week 60 Android build nodes (AWS) 35 iOS build nodes on Mac Stadium
15. Releases alpha - every night beta - every week GA - every 2 weeks
16. Mobile is Different No fast rollback -> High Risk Huge Test Matrix: [Screen Size, APIs, Devices, Network] Opinionated Platform Tools: [IDEs, Build Systems, Virtual/Physical Devices] Apple doesn’t like developers?
17. Current UDX
24. How do we measure success?
25. How are we doing right now? ? ? ? ?? ? ?
26. Are we trending in the right direction? ? ? ? ?? ? ?
27. Are we prioritizing the right things? ? ? ? ?? ? ?
28. Are we solving what we set out to solve? ? ? ? ?? ? ?
29. Telemetry Surveys Listening Effort Detail Bias Volume of Data The feedback pyramid
30. Instrumentation Example post { always { junit allowEmptyResults: true, testResults: 'testing/test-lab-*/test_result_*.xml' setGitStatus(‘https://github.com/foo/bar', 'E2E Smoke Tests', currentBuild.result, "${params.SLACK_SHA}") uploadMobileBuildInfoToQC() } String cmd = "curl -X POST ${apiUrl + env.API_TOKEN} \ -d \"platform=${platform}\" \ -d \"job_name=${env.JOB_NAME}\" \ -d \"jenkins_build_number=${currentBuild.number}\" \ -d \"build_cause=${buildCause}\" \ -d \"branch_name=${env.BRANCH_NAME}\" \ -d \"duration_ms=${currentBuild.duration}\" \ -d \"author_name=${env.CHANGE_AUTHOR}\" \ -d \"failure_detail=${env.FAILURE_DETAIL}\" \ -d \"status=${currentBuild.currentResult}\""
31. Time To Mergeable
32. Time To Mergeable
33. Master Build Stability
34. Triage Quality
35. Metrics are not very helpful if you don’t look at them
36. Telemetry • Cheap • “Real Time” & Alert-able • Capturing Sentiment • Showing Big Picture
38. Telemetry Surveys Listening Effort Detail Subjectivity Volume of Data
39. Surveys Infrequent, but regular Short (~5 min) Allow for free-form feedback
40. Surveys Infrequent, but regular Short (~5 min) Allow for free-form feedback https://goo.gl/forms/1MrbKSxck6SgBesq2
41. Survey in Slack
42. Trends over Time
43. Qualitative Feedback “With current local build time ~ 10 minutes, I can't say my overall development workflow is pleasant and productive.” “I feel that MY changes generally won't as I'm writing tests to ensure things are working correctly. However, the question is if "any changes" are merged to master (ie: other than my own) and I'm unsure if PRs other than those I'm reviewing are testing things to the extent we desire and I don't think we do much auditing of tests here.”
44. NPS From the standpoint of developer tooling, how likely are you to recommend Slack as a great place to develop mobile applications?
45. Surveys • Big Picture • Long-term Trends • Hard to develop & analyze • Annoying?
46. Telemetry Surveys Listening Effort Detail Subjectivity Volume of Data
47. Telemetry Surveys Listening Effort Detail Subjectivity Volume of Data
48. Telemetry Surveys Listening Effort Detail Subjectivity Volume of Data Micro Surveys
49. Micro-Surveys Very lightweight human feedback Fits within workflow
53. Micro-Surveys • Cheap • Can measure sentiment & other difficult areas • Human error • Engagement
54. Telemetry Surveys Listening Effort Detail Subjectivity Volume of Data Micro Surveys
55. Listening Group sessions & 1-1 conversations where devs can speak and feel heard
56. How are we doing right now? ? ? ? ?? ? ?
58. NET PROMOTER SCORE BENCHMARKS FOR FORTUNE 500 COMPANIES
59. “We don't have a whole lot of tooling that makes development here special.”
63. What’s Going Well • 71% satisfaction on workflow • Triage • TTM on Android • Scaling with AWS & Firebase Test Lab • Having Metrics • iOS & Android Test Council
64. What’s not going well • Master Build Stability • Jenkins • iOS CI environment (Mac Stadium) • Flaky Tests • TTM on iOS • Workflow • Local build times • “nothing special” • Confidence in Quality
65. Are we trending in the right direction? ? ? ? ?? ? ?
67. Are we prioritizing the right things? ? ? ? ?? ? ?
70. Objectives for near future Make the Slack mobile developer workflow world-class Empower mobile developers to own and improve quality