Service Operation Centered Development

Service Operation Centered Development 2019/04/09 DevOpsDays Tokyo 2019 Mitsuyuki Shiiba
(@bufferings) EC Incubation Development Dept. Rakuten, Inc.

@bufferings #devops_b2 "It's their fault, not ours" Photo by Alexander
McFeron on Unsplash

@bufferings #devops_b2 After all, does it make the service better?

@bufferings #devops_b2 Service Operation Development Interaction

@bufferings #devops_b2 Mitsuyuki Shiiba Web Application Engineer @Rakuten Osaka from
2010

@bufferings #devops_b2 30 countries & regions with local operational presence
70 + services Almost 1.3 B global members ¥ 15.4 T global gross transaction value *FY2018 Figures are from https://global.rakuten.com/corp/ accessed on April 3rd, 2019

@bufferings #devops_b2 (Wantedly) https://shiiba.page.link/ecid

@bufferings #devops_b2 Service Operation

@bufferings #devops_b2 Maintenance Monitoring Trouble shooting Inquiry handling Analysis Improvement
Alert handling Version upgrade etc… Service Operation

@bufferings #devops_b2 Service Operation is to keep the service stable

@bufferings #devops_b2 Service Operation is to the service Photo by
Ravi Roshan on Unsplash

@bufferings #devops_b2 "It's a lot of fun!"

@bufferings #devops_b2 doing both operation and development including the releases
The Team

@bufferings #devops_b2 that's because knowing != understanding

@bufferings #devops_b2 Too many alerts Non-automated tests Long methods Meaningless
names Manual deployment etc… Not updated documents Big ball of mud Dirty servers What makes Service Operation tough

@bufferings #devops_b2 then I understood in 2011 understanding != doing

@bufferings #devops_b2 Photo by Simon Rae on Unsplash

@bufferings #devops_b2 We should connect dev & ops in our
heads putting Service Operation at the center

@bufferings #devops_b2 Good culture, feature flags, chatops, and more. Rakuma
team wants you! -> (Wantedly) https://shiiba.page.link/rakuma if current_time <= 7m

@bufferings #devops_b2 Development

@bufferings #devops_b2 1. Reply to all the alerts immediately

@bufferings #devops_b2 It's US who get woken up at midnight,
not anyone else. Photo by Andre Mouton on Unsplash

@bufferings #devops_b2 Is this really necessary? • Can we check
it next morning? • Can we retry it? • Can we develop automatic recovery? What's actually happened? • What's the impact on the users? • What should we do for it?

@bufferings #devops_b2 so that everyone can think about the impact
& the solution LogMessageBuilder message("What's happened?") .cause("What's the cause?") .impact("What's the user impact?") .solution("What do we have to do?") .build()

@bufferings #devops_b2 to know the actual impact on the users
(Learned) Handle them at the controller layer Data Access Application Service Controller ← We can know the user impact ← We can't know the user impact

@bufferings #devops_b2 2. Safe by Design

@bufferings #devops_b2 Fool Proof to use the tools at ease
Fail Safe to keep consistency for every single line Idempotence so that we can send a same message multiple times Eventual Consistency for automatic recovery Decoupled Architecture to minimize the user impact

@bufferings #devops_b2 3. Services over Projects

@bufferings #devops_b2 Think the service narrative including non-systematized area

@bufferings #devops_b2 Service quality over project deadline

@bufferings #devops_b2 Delivered value over estimation accuracy

@bufferings #devops_b2 Reading time over writing time

@bufferings #devops_b2 Codes/Tests/Docs Readable • having reason for every single
line • meaningful names • small methods • overview → detail Maintainable • keep them updated • just enough quality & quantity

@bufferings #devops_b2 Team's growth over resource efficiency

@bufferings #devops_b2 • Pair Work, Mob Work • Learning Sessions
• Daily Experiments Learning Teams

@bufferings #devops_b2 EC Start-up Group is mobbing everyday! -> (Rakuten
Careers) https://shiiba.page.link/ecsu if current_time <= 15m

@bufferings #devops_b2 Interaction

@bufferings #devops_b2 1. Feel the forces

@bufferings #devops_b2 "I don't know why they don't do this!"

@bufferings #devops_b2 Feel the forces in the org. Respect &
trust other people. Doing the right thing in your place doesn't work

@bufferings #devops_b2 We only can see what we know Respect
other people's expertise

@bufferings #devops_b2 Trust everyone feels joy to make the service
better. Trust people, doubt Ba(場: environment)

@bufferings #devops_b2 or change the forces together Take advantage of
the forces for the service

@bufferings #devops_b2 2. See around the corner

@bufferings #devops_b2 Photo by dan carlson on Unsplash

@bufferings #devops_b2 See what would come next for the service,
and prepare for them • business • technology • design • organization Everything is changing rapidly

@bufferings #devops_b2 Service Operation Development Interaction 1. Reply to all
the alerts 2. Safe by Design 3. Services over Projects 1. Feel the forces 2. See around the corner Summary

@bufferings #devops_b2 After all, does it make the service better?

@bufferings #devops_b2 10+ Deploys Per Day: Dev and Ops Cooperation
at Flickr https://conferences.oreilly.com/velocity/velocity2009/public/sc hedule/detail/7641 Appendix

Service Operation Centered Development

Service Operation Centered Development

More Decks by Mitsuyuki Shiiba

Other Decks in Technology

Featured

Transcript