Service Operation Centered Development

DevOpsDaysTokyo 2019

Transcript

1. Service Operation Centered Development 2019/04/09 DevOpsDays Tokyo 2019 Mitsuyuki Shiiba (@bufferings) EC Incubation Development Dept. Rakuten, Inc.
2. @bufferings #devops_b2 "It's their fault, not ours" Photo by Alexander McFeron on Unsplash
3. @bufferings #devops_b2 After all, does it make the service better?
4. @bufferings #devops_b2 Service Operation Development Interaction
5. @bufferings #devops_b2 Mitsuyuki Shiiba Web Application Engineer @Rakuten Osaka from 2010
6. @bufferings #devops_b2 30 countries & regions with local operational presence 70 + services Almost 1.3 B global members ¥ 15.4 T global gross transaction value *FY2018 Figures are from https://global.rakuten.com/corp/ accessed on April 3rd, 2019
7. @bufferings #devops_b2 (Wantedly) https://shiiba.page.link/ecid
8. @bufferings #devops_b2 Service Operation
9. @bufferings #devops_b2 Maintenance Monitoring Trouble shooting Inquiry handling Analysis Improvement Alert handling Version upgrade etc… Service Operation
10. @bufferings #devops_b2 Service Operation is to keep the service stable
11. @bufferings #devops_b2 Service Operation is to the service Photo by Ravi Roshan on Unsplash
12. @bufferings #devops_b2 "It's a lot of fun!"
13. @bufferings #devops_b2 doing both operation and development including the releases The Team
14. @bufferings #devops_b2 that's because knowing != understanding
15. @bufferings #devops_b2 Too many alerts Non-automated tests Long methods Meaningless names Manual deployment etc… Not updated documents Big ball of mud Dirty servers What makes Service Operation tough
16. @bufferings #devops_b2 then I understood in 2011 understanding != doing
17. @bufferings #devops_b2 Photo by Simon Rae on Unsplash
18. @bufferings #devops_b2 We should connect dev & ops in our heads putting Service Operation at the center
19. @bufferings #devops_b2 Good culture, feature flags, chatops, and more. Rakuma team wants you! -> (Wantedly) https://shiiba.page.link/rakuma if current_time <= 7m
20. @bufferings #devops_b2 Development
21. @bufferings #devops_b2 1. Reply to all the alerts immediately
22. @bufferings #devops_b2 It's US who get woken up at midnight, not anyone else. Photo by Andre Mouton on Unsplash
23. @bufferings #devops_b2 Is this really necessary? • Can we check it next morning? • Can we retry it? • Can we develop automatic recovery? What's actually happened? • What's the impact on the users? • What should we do for it?
24. @bufferings #devops_b2 so that everyone can think about the impact & the solution LogMessageBuilder message("What's happened?") .cause("What's the cause?") .impact("What's the user impact?") .solution("What do we have to do?") .build()
25. @bufferings #devops_b2 to know the actual impact on the users (Learned) Handle them at the controller layer Data Access Application Service Controller ← We can know the user impact ← We can't know the user impact
26. @bufferings #devops_b2 2. Safe by Design
27. @bufferings #devops_b2 Fool Proof to use the tools at ease Fail Safe to keep consistency for every single line Idempotence so that we can send a same message multiple times Eventual Consistency for automatic recovery Decoupled Architecture to minimize the user impact
28. @bufferings #devops_b2 3. Services over Projects
29. @bufferings #devops_b2 Think the service narrative including non-systematized area
30. @bufferings #devops_b2 Service quality over project deadline
31. @bufferings #devops_b2 Delivered value over estimation accuracy
32. @bufferings #devops_b2 Reading time over writing time
33. @bufferings #devops_b2 Codes/Tests/Docs Readable • having reason for every single line • meaningful names • small methods • overview → detail Maintainable • keep them updated • just enough quality & quantity
34. @bufferings #devops_b2 Team's growth over resource efficiency
35. @bufferings #devops_b2 • Pair Work, Mob Work • Learning Sessions • Daily Experiments Learning Teams
36. @bufferings #devops_b2 EC Start-up Group is mobbing everyday! -> (Rakuten Careers) https://shiiba.page.link/ecsu if current_time <= 15m
37. @bufferings #devops_b2 Interaction
38. @bufferings #devops_b2 1. Feel the forces
39. @bufferings #devops_b2 "I don't know why they don't do this!"
40. @bufferings #devops_b2 Feel the forces in the org. Respect & trust other people. Doing the right thing in your place doesn't work
41. @bufferings #devops_b2 We only can see what we know Respect other people's expertise
42. @bufferings #devops_b2 Trust everyone feels joy to make the service better. Trust people, doubt Ba(場: environment)
43. @bufferings #devops_b2 or change the forces together Take advantage of the forces for the service
44. @bufferings #devops_b2 2. See around the corner
45. @bufferings #devops_b2 Photo by dan carlson on Unsplash
46. @bufferings #devops_b2 See what would come next for the service, and prepare for them • business • technology • design • organization Everything is changing rapidly
47. @bufferings #devops_b2 Service Operation Development Interaction 1. Reply to all the alerts 2. Safe by Design 3. Services over Projects 1. Feel the forces 2. See around the corner Summary
48. @bufferings #devops_b2 After all, does it make the service better?
49. @bufferings #devops_b2 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr https://conferences.oreilly.com/velocity/velocity2009/public/sc hedule/detail/7641 Appendix
50. @bufferings #devops_b2 "Yutapon coding" copyright © 2007 plx system co. http://net2.system.to/pc/font.html About Font