$30 off During Our Annual Pro Sale. View Details »

Service Operation Centered Development

Service Operation Centered Development

DevOpsDaysTokyo 2019

Mitsuyuki Shiiba

April 05, 2019
Tweet

More Decks by Mitsuyuki Shiiba

Other Decks in Technology

Transcript

  1. Service Operation Centered Development 2019/04/09 DevOpsDays Tokyo 2019 Mitsuyuki Shiiba

    (@bufferings) EC Incubation Development Dept. Rakuten, Inc.
  2. @bufferings #devops_b2 "It's their fault, not ours" Photo by Alexander

    McFeron on Unsplash
  3. @bufferings #devops_b2 After all, does it make the service better?

  4. @bufferings #devops_b2 Service Operation Development Interaction

  5. @bufferings #devops_b2 Mitsuyuki Shiiba Web Application Engineer @Rakuten Osaka from

    2010
  6. @bufferings #devops_b2 30 countries & regions with local operational presence

    70 + services Almost 1.3 B global members ¥ 15.4 T global gross transaction value *FY2018 Figures are from https://global.rakuten.com/corp/ accessed on April 3rd, 2019
  7. @bufferings #devops_b2 (Wantedly) https://shiiba.page.link/ecid

  8. @bufferings #devops_b2 Service Operation

  9. @bufferings #devops_b2 Maintenance Monitoring Trouble shooting Inquiry handling Analysis Improvement

    Alert handling Version upgrade etc… Service Operation
  10. @bufferings #devops_b2 Service Operation is to keep the service stable

  11. @bufferings #devops_b2 Service Operation is to the service Photo by

    Ravi Roshan on Unsplash
  12. @bufferings #devops_b2 "It's a lot of fun!"

  13. @bufferings #devops_b2 doing both operation and development including the releases

    The Team
  14. @bufferings #devops_b2 that's because knowing != understanding

  15. @bufferings #devops_b2 Too many alerts Non-automated tests Long methods Meaningless

    names Manual deployment etc… Not updated documents Big ball of mud Dirty servers What makes Service Operation tough
  16. @bufferings #devops_b2 then I understood in 2011 understanding != doing

  17. @bufferings #devops_b2 Photo by Simon Rae on Unsplash

  18. @bufferings #devops_b2 We should connect dev & ops in our

    heads putting Service Operation at the center
  19. @bufferings #devops_b2 Good culture, feature flags, chatops, and more. Rakuma

    team wants you! -> (Wantedly) https://shiiba.page.link/rakuma if current_time <= 7m
  20. @bufferings #devops_b2 Development

  21. @bufferings #devops_b2 1. Reply to all the alerts immediately

  22. @bufferings #devops_b2 It's US who get woken up at midnight,

    not anyone else. Photo by Andre Mouton on Unsplash
  23. @bufferings #devops_b2 Is this really necessary? • Can we check

    it next morning? • Can we retry it? • Can we develop automatic recovery? What's actually happened? • What's the impact on the users? • What should we do for it?
  24. @bufferings #devops_b2 so that everyone can think about the impact

    & the solution LogMessageBuilder message("What's happened?") .cause("What's the cause?") .impact("What's the user impact?") .solution("What do we have to do?") .build()
  25. @bufferings #devops_b2 to know the actual impact on the users

    (Learned) Handle them at the controller layer Data Access Application Service Controller ← We can know the user impact ← We can't know the user impact
  26. @bufferings #devops_b2 2. Safe by Design

  27. @bufferings #devops_b2 Fool Proof to use the tools at ease

    Fail Safe to keep consistency for every single line Idempotence so that we can send a same message multiple times Eventual Consistency for automatic recovery Decoupled Architecture to minimize the user impact
  28. @bufferings #devops_b2 3. Services over Projects

  29. @bufferings #devops_b2 Think the service narrative including non-systematized area

  30. @bufferings #devops_b2 Service quality over project deadline

  31. @bufferings #devops_b2 Delivered value over estimation accuracy

  32. @bufferings #devops_b2 Reading time over writing time

  33. @bufferings #devops_b2 Codes/Tests/Docs Readable • having reason for every single

    line • meaningful names • small methods • overview → detail Maintainable • keep them updated • just enough quality & quantity
  34. @bufferings #devops_b2 Team's growth over resource efficiency

  35. @bufferings #devops_b2 • Pair Work, Mob Work • Learning Sessions

    • Daily Experiments Learning Teams
  36. @bufferings #devops_b2 EC Start-up Group is mobbing everyday! -> (Rakuten

    Careers) https://shiiba.page.link/ecsu if current_time <= 15m
  37. @bufferings #devops_b2 Interaction

  38. @bufferings #devops_b2 1. Feel the forces

  39. @bufferings #devops_b2 "I don't know why they don't do this!"

  40. @bufferings #devops_b2 Feel the forces in the org. Respect &

    trust other people. Doing the right thing in your place doesn't work
  41. @bufferings #devops_b2 We only can see what we know Respect

    other people's expertise
  42. @bufferings #devops_b2 Trust everyone feels joy to make the service

    better. Trust people, doubt Ba(場: environment)
  43. @bufferings #devops_b2 or change the forces together Take advantage of

    the forces for the service
  44. @bufferings #devops_b2 2. See around the corner

  45. @bufferings #devops_b2 Photo by dan carlson on Unsplash

  46. @bufferings #devops_b2 See what would come next for the service,

    and prepare for them • business • technology • design • organization Everything is changing rapidly
  47. @bufferings #devops_b2 Service Operation Development Interaction 1. Reply to all

    the alerts 2. Safe by Design 3. Services over Projects 1. Feel the forces 2. See around the corner Summary
  48. @bufferings #devops_b2 After all, does it make the service better?

  49. @bufferings #devops_b2 10+ Deploys Per Day: Dev and Ops Cooperation

    at Flickr https://conferences.oreilly.com/velocity/velocity2009/public/sc hedule/detail/7641 Appendix
  50. @bufferings #devops_b2 "Yutapon coding" copyright © 2007 plx system co.

    http://net2.system.to/pc/font.html About Font