Service Operation Centered Development

DevOpsDaysTokyo 2019

Transcript

1. Service Operation Centered Development
   2019/04/09 DevOpsDays Tokyo 2019
   Mitsuyuki Shiiba (@bufferings)
   EC Incubation Development Dept.
   Rakuten, Inc.
   

   View Slide

2. @bufferings #devops_b2
   "It's their fault, not ours"
   Photo by Alexander McFeron on Unsplash
   

   View Slide

3. @bufferings #devops_b2
   After all, does it make the service better?
   

   View Slide

4. @bufferings #devops_b2
   Service
   Operation
   Development
   Interaction
   

   View Slide

5. @bufferings #devops_b2
   Mitsuyuki Shiiba
   Web Application Engineer
   @Rakuten Osaka from 2010
   

   View Slide

6. @bufferings #devops_b2
   30
   countries & regions
   with local operational presence
   70 +
   services
   Almost
   1.3 B
   global members
   ¥ 15.4 T
   global gross transaction value
   *FY2018
   Figures are from https://global.rakuten.com/corp/ accessed on April 3rd, 2019
   

   View Slide

7. @bufferings #devops_b2
   (Wantedly) https://shiiba.page.link/ecid
   

   View Slide

8. @bufferings #devops_b2
   Service
   Operation
   

   View Slide

9. @bufferings #devops_b2
   Maintenance
   Monitoring
   Trouble shooting
   Inquiry handling
   Analysis
   Improvement
   Alert handling
   Version upgrade
   etc…
   Service
   Operation
   

   View Slide

10. @bufferings #devops_b2
    Service Operation is
    to keep the service stable
    

    View Slide

11. @bufferings #devops_b2
    Service Operation is to the service
    Photo by Ravi Roshan on Unsplash
    

    View Slide

12. @bufferings #devops_b2
    "It's a lot of fun!"
    

    View Slide

13. @bufferings #devops_b2
    doing both operation and development including the releases
    The Team
    

    View Slide

14. @bufferings #devops_b2
    that's because
    knowing != understanding
    

    View Slide

15. @bufferings #devops_b2
    Too many alerts
    Non-automated tests
    Long methods
    Meaningless names
    Manual deployment
    etc…
    Not updated documents
    Big ball of mud
    Dirty servers
    What makes
    Service
    Operation
    tough
    

    View Slide

16. @bufferings #devops_b2
    then I understood in 2011
    understanding != doing
    

    View Slide

17. @bufferings #devops_b2
    Photo by Simon Rae on Unsplash
    

    View Slide

18. @bufferings #devops_b2
    We should connect dev & ops in our heads
    putting Service Operation at the center
    

    View Slide

19. @bufferings #devops_b2
    Good culture, feature flags, chatops, and more.
    Rakuma team wants you! -> (Wantedly) https://shiiba.page.link/rakuma
    if current_time <= 7m
    
    

    View Slide

20. @bufferings #devops_b2
    Development
    

    View Slide

21. @bufferings #devops_b2
    1. Reply to all the alerts immediately
    

    View Slide

22. @bufferings #devops_b2
    It's US who get woken up at midnight, not anyone else.
    Photo by Andre Mouton on Unsplash
    

    View Slide

23. @bufferings #devops_b2
    Is this really necessary?
    • Can we check it next morning?
    • Can we retry it?
    • Can we develop automatic recovery?
    What's actually happened?
    • What's the impact on the users?
    • What should we do for it?
    

    View Slide

24. @bufferings #devops_b2
    so that everyone can think about the impact & the solution
    LogMessageBuilder
    message("What's happened?")
    .cause("What's the cause?")
    .impact("What's the user impact?")
    .solution("What do we have to do?")
    .build()
    

    View Slide

25. @bufferings #devops_b2
    to know the actual impact on the users
    (Learned) Handle them at the controller layer
    Data Access
    Application Service
    Controller ← We can know the user impact
    ← We can't know the user impact
    

    View Slide

26. @bufferings #devops_b2
    2. Safe by Design
    

    View Slide

27. @bufferings #devops_b2
    Fool Proof
    to use the tools at ease
    Fail Safe
    to keep consistency for
    every single line
    Idempotence
    so that we can send a same
    message multiple times
    Eventual Consistency
    for automatic recovery
    Decoupled Architecture
    to minimize the user impact
    

    View Slide

28. @bufferings #devops_b2
    3. Services over Projects
    

    View Slide

29. @bufferings #devops_b2
    Think the service narrative
    including non-systematized area
    

    View Slide

30. @bufferings #devops_b2
    Service quality over project deadline
    

    View Slide

31. @bufferings #devops_b2
    Delivered value over estimation accuracy
    

    View Slide

32. @bufferings #devops_b2
    Reading time over writing time
    

    View Slide

33. @bufferings #devops_b2
    Codes/Tests/Docs
    Readable
    • having reason for every single line
    • meaningful names
    • small methods
    • overview → detail
    Maintainable
    • keep them updated
    • just enough quality & quantity
    

    View Slide

34. @bufferings #devops_b2
    Team's growth over resource efficiency
    

    View Slide

35. @bufferings #devops_b2
    • Pair Work, Mob Work
    • Learning Sessions
    • Daily Experiments
    Learning Teams
    

    View Slide

36. @bufferings #devops_b2
    EC Start-up Group is mobbing everyday!
    -> (Rakuten Careers) https://shiiba.page.link/ecsu
    if current_time <= 15m
    
    

    View Slide

37. @bufferings #devops_b2
    Interaction
    

    View Slide

38. @bufferings #devops_b2
    1. Feel the forces
    

    View Slide

39. @bufferings #devops_b2
    "I don't know why they don't do this!"
    

    View Slide

40. @bufferings #devops_b2
    Feel the forces in the org. Respect & trust other people.
    Doing the right thing in your place doesn't work
    

    View Slide

41. @bufferings #devops_b2
    We only can see what we know
    Respect other people's expertise
    

    View Slide

42. @bufferings #devops_b2
    Trust everyone feels joy to make the service better.
    Trust people, doubt Ba(場: environment)
    

    View Slide

43. @bufferings #devops_b2
    or change the forces together
    Take advantage of the forces for the service
    

    View Slide

44. @bufferings #devops_b2
    2. See around the corner
    

    View Slide

45. @bufferings #devops_b2
    Photo by dan carlson on Unsplash
    

    View Slide

46. @bufferings #devops_b2
    See what would come next for the service, and prepare for them
    • business
    • technology
    • design
    • organization
    Everything is changing rapidly
    

    View Slide

47. @bufferings #devops_b2
    Service
    Operation
    Development
    Interaction
    1. Reply to all the alerts
    2. Safe by Design
    3. Services over Projects
    1. Feel the forces
    2. See around the corner
    Summary
    

    View Slide

48. @bufferings #devops_b2
    After all, does it make the service better?
    

    View Slide

49. @bufferings #devops_b2
    10+ Deploys Per Day: Dev and Ops Cooperation at Flickr
    https://conferences.oreilly.com/velocity/velocity2009/public/sc
    hedule/detail/7641
    Appendix
    

    View Slide

50. @bufferings #devops_b2
    "Yutapon coding" copyright © 2007 plx system co.
    http://net2.system.to/pc/font.html
    About Font
    

    View Slide