$30 off During Our Annual Pro Sale. View Details »

Service Operation Centered Development

Service Operation Centered Development

DevOpsDaysTokyo 2019

Mitsuyuki Shiiba

April 05, 2019

More Decks by Mitsuyuki Shiiba

Other Decks in Technology


  1. Service Operation Centered Development 2019/04/09 DevOpsDays Tokyo 2019 Mitsuyuki Shiiba

    (@bufferings) EC Incubation Development Dept. Rakuten, Inc.
  2. @bufferings #devops_b2 "It's their fault, not ours" Photo by Alexander

    McFeron on Unsplash
  3. @bufferings #devops_b2 After all, does it make the service better?

  4. @bufferings #devops_b2 Service Operation Development Interaction

  5. @bufferings #devops_b2 Mitsuyuki Shiiba Web Application Engineer @Rakuten Osaka from

  6. @bufferings #devops_b2 30 countries & regions with local operational presence

    70 + services Almost 1.3 B global members ¥ 15.4 T global gross transaction value *FY2018 Figures are from https://global.rakuten.com/corp/ accessed on April 3rd, 2019
  7. @bufferings #devops_b2 (Wantedly) https://shiiba.page.link/ecid

  8. @bufferings #devops_b2 Service Operation

  9. @bufferings #devops_b2 Maintenance Monitoring Trouble shooting Inquiry handling Analysis Improvement

    Alert handling Version upgrade etc… Service Operation
  10. @bufferings #devops_b2 Service Operation is to keep the service stable

  11. @bufferings #devops_b2 Service Operation is to the service Photo by

    Ravi Roshan on Unsplash
  12. @bufferings #devops_b2 "It's a lot of fun!"

  13. @bufferings #devops_b2 doing both operation and development including the releases

    The Team
  14. @bufferings #devops_b2 that's because knowing != understanding

  15. @bufferings #devops_b2 Too many alerts Non-automated tests Long methods Meaningless

    names Manual deployment etc… Not updated documents Big ball of mud Dirty servers What makes Service Operation tough
  16. @bufferings #devops_b2 then I understood in 2011 understanding != doing

  17. @bufferings #devops_b2 Photo by Simon Rae on Unsplash

  18. @bufferings #devops_b2 We should connect dev & ops in our

    heads putting Service Operation at the center
  19. @bufferings #devops_b2 Good culture, feature flags, chatops, and more. Rakuma

    team wants you! -> (Wantedly) https://shiiba.page.link/rakuma if current_time <= 7m
  20. @bufferings #devops_b2 Development

  21. @bufferings #devops_b2 1. Reply to all the alerts immediately

  22. @bufferings #devops_b2 It's US who get woken up at midnight,

    not anyone else. Photo by Andre Mouton on Unsplash
  23. @bufferings #devops_b2 Is this really necessary? • Can we check

    it next morning? • Can we retry it? • Can we develop automatic recovery? What's actually happened? • What's the impact on the users? • What should we do for it?
  24. @bufferings #devops_b2 so that everyone can think about the impact

    & the solution LogMessageBuilder message("What's happened?") .cause("What's the cause?") .impact("What's the user impact?") .solution("What do we have to do?") .build()
  25. @bufferings #devops_b2 to know the actual impact on the users

    (Learned) Handle them at the controller layer Data Access Application Service Controller ← We can know the user impact ← We can't know the user impact
  26. @bufferings #devops_b2 2. Safe by Design

  27. @bufferings #devops_b2 Fool Proof to use the tools at ease

    Fail Safe to keep consistency for every single line Idempotence so that we can send a same message multiple times Eventual Consistency for automatic recovery Decoupled Architecture to minimize the user impact
  28. @bufferings #devops_b2 3. Services over Projects

  29. @bufferings #devops_b2 Think the service narrative including non-systematized area

  30. @bufferings #devops_b2 Service quality over project deadline

  31. @bufferings #devops_b2 Delivered value over estimation accuracy

  32. @bufferings #devops_b2 Reading time over writing time

  33. @bufferings #devops_b2 Codes/Tests/Docs Readable • having reason for every single

    line • meaningful names • small methods • overview → detail Maintainable • keep them updated • just enough quality & quantity
  34. @bufferings #devops_b2 Team's growth over resource efficiency

  35. @bufferings #devops_b2 • Pair Work, Mob Work • Learning Sessions

    • Daily Experiments Learning Teams
  36. @bufferings #devops_b2 EC Start-up Group is mobbing everyday! -> (Rakuten

    Careers) https://shiiba.page.link/ecsu if current_time <= 15m
  37. @bufferings #devops_b2 Interaction

  38. @bufferings #devops_b2 1. Feel the forces

  39. @bufferings #devops_b2 "I don't know why they don't do this!"

  40. @bufferings #devops_b2 Feel the forces in the org. Respect &

    trust other people. Doing the right thing in your place doesn't work
  41. @bufferings #devops_b2 We only can see what we know Respect

    other people's expertise
  42. @bufferings #devops_b2 Trust everyone feels joy to make the service

    better. Trust people, doubt Ba(場: environment)
  43. @bufferings #devops_b2 or change the forces together Take advantage of

    the forces for the service
  44. @bufferings #devops_b2 2. See around the corner

  45. @bufferings #devops_b2 Photo by dan carlson on Unsplash

  46. @bufferings #devops_b2 See what would come next for the service,

    and prepare for them • business • technology • design • organization Everything is changing rapidly
  47. @bufferings #devops_b2 Service Operation Development Interaction 1. Reply to all

    the alerts 2. Safe by Design 3. Services over Projects 1. Feel the forces 2. See around the corner Summary
  48. @bufferings #devops_b2 After all, does it make the service better?

  49. @bufferings #devops_b2 10+ Deploys Per Day: Dev and Ops Cooperation

    at Flickr https://conferences.oreilly.com/velocity/velocity2009/public/sc hedule/detail/7641 Appendix
  50. @bufferings #devops_b2 "Yutapon coding" copyright © 2007 plx system co.

    http://net2.system.to/pc/font.html About Font