Your Build Pipeline is a Product

Why your Build Pipeline is a Product and how we used that thinking at Dr Me to significantly improve developers' experience and efficiency with UX improvement processes.

Transcript

1. Your Build Pipeline is a Product
2. Hi, I'm chendo • Sometimes known as Jack Chen • @chendo on Github, Twitter • Chief Technology Officer at Dr Me
3. The Product • An app that allows patients to have a consultation with a doctor • Doctor can issue medical certificates, prescriptions, referrals, pathology, imaging • iOS, Android and Web
4. The Stack • Rails backend • Deployed as set of containers to a DC/OS cluster • React frontend • Deployed to mobile with Cordova as hybrid app • ES6, Flow, GraphQL, styled-components • Buildkite for CI
5. Why is a Build Pipeline a Product? • It provides some sort of value • Prevent bugs • Automate process • It has users: you!
6. Why treat it as a Product? • Apply product design principles and processes • Iterate and improve • Keep your developers happy
7. User Experience Flow • Push a change • Wait for build • If broken, figure out what broke and why • Fix it • Ensure broken test(s) are fixed locally • Push the fix ---^
8. Evolution of our Frontend Build Pipeline
9. Integration tests provide the most confidence
10. How hard could it be? — Jeremy Clarkson
11. Iteration #1 • Nightwatch: JS testing tool • Frontend is JS • docker-compose to run backend, frontend, tests and selenium
13. Issues • ! Effort++ to figure out what broke and why • Need to look at test source to find out what happened before it failed • Screenshots were annoying to look at • Need to run the tests locally to see JS errors • ! ! Unable to test video chat flow
14. Issues • ! Bad "Developer Experience" • Manual waits • !!! setValue is actually appendValue
16. Goal: Decrease effort to understand failure
17. Goal: Make writing tests not suck
18. Iteration #2 • ! RSpec/Capybara • ! JS console output • ! Interactive test writing [demo] • ! Video call tests • " Step-by-step output via TracePoint • " Screenshots displayed near error point
21. Issues • ! Error & backtrace at the end of output • ! Need to scroll through a lot of output to find actual issue • ! Current notification methods annoying • Email is noisy enough, Slack integration is noisy
22. Goal: Decrease effort to know the build failed
23. Iteration #3 • ! Hide unnecessary output • https://github.com/chendo/buildkite-rspec-formatter • ! 'Need to know' Slack notifications • Bonus: Slack notification for Github comments • " Move error & backtrace with the others • " Per-step JS console output
24. Collapsed-by-default output
25. Targeted Slack notifications
26. Targeted Slack notifications
27. Per Step JS Log
28. Issues • Still need to do manual testing to verify things work visually • Builds take too long • Concurrent frontend builds on the same machine causes it to grind to a halt • Tests begin failing due to timeouts
29. Goal: Faster feedback cycle
30. Goal: Must be able to trust results
31. Goal: Prevent undesired visual changes
32. Iteration #4 • ! Visual regression with Percy.io • Easy to integrate • " Parallelise tests with Knapsack • " Restructured Buildkite agents • " rspec-retry
33. Visual regression testing
34. Visual regression testing
35. Visual regression testing
36. Parallelise tests
37. Summary • Significantly decreased effort to understand failure • Per-step logging • JS console output • Screenshots • Localised useful info (error, backtrace, screenshot) • Hide irrelevant output • Improved confidence with visual regression testing
38. Summary • Improved feedback cycle • Faster test suite • Notifications rather than polling • Made writing integration tests much easier • Happy developers! !
39. Roadmap • Backend logs in output • Get relevant build output into notification • Per-step screenshots • Video of tests • Rebuild and unblock builds from Slack • Automated on-device integration testing
40. Conclusion • Your build pipeline is a product and you should improve it • Tooling and processes are also products and you should improve them
41. Any questions?
42. Thanks! P.S. we're hiring!