"雑" / Almost Microservices

at RailsDM 2019

1. 雑 "Almost microservices" Tatsuhiro Ujihisa RailsDM 2019 2019-03-22 1/92
2. https://en.wiktionary.or g/wiki/%E9%9B%91 Adjectival noun 雑 (shinjitai kanji, kyūjitai kanji 雜, -na inflection, hiragana ざ つ, rōmaji zatsu) • rough, crude, sloppy, messy • miscellaneous (see note) 2/92
3. ujihisa • https://github.com/ujihisa • https://twitter.com/ujm • VimConf 2018 Speaker & Organizer, RubyKaigi 2019 Speaker • Quipper Limited ◦ Co-workers' talks: @banyan, @mtsmfm ◦ "Distributed Monolith" @kyanny from RailsDM 2018 3/92
4. ujihisa before Quipper • 2010 ~ 2016 ◦ https://hootsuite.com ◦ PHP Monolith ◦ Scala Microservices ◦ Go Microservices • 2017~2018 ◦ https://fril.jp ◦ Rails Distributed Monolith "分断されたモノリス"
5. ujihisa • Cooking • Learned a lot about fish
6. vim-jp
7. Quipper/Quipper, Quipper/Study Sapuri • https://learn.quipper.com https://learn.studysapuri.jp ◦ Rails Distributed Monolith
9. Quipper • Joined in 2018-09 ◦ 7 months so far ▪ 4 months at School Team (*) ▪ 1 month at Unemployed (*) ▪ 2 months at Data Restructuring Team
10. Almost microservices ujihisa 9/92
11. "雑" Almost microservices / @ujihisa • Release is important ◦ Grind it to small pieces for scalability • Human makes mistakes • Feature toggles pattern is powerful ◦ Quipper calls it Darklaunch ◦ Provide strong support to Darklaunch • Apply Circuit Breaker pattern and Canary release if necessary • Pragmatic Microservices' essence
12. Web app development • Research & decide what to make • rails new • infrastructure setup • git, code review, CI setup • design models, views, and controllers • development! of course unit tests too! • refactor • release & debug
13. Web app development • Research & decide what to make • rails new • infrastructure setup • git, code review, CI setup • design models, views, and controllers • development! of course unit tests too! • refactor • 　　　　　　　　　　　　　　　　& debug Release
14. Software release Release gives development meaning
15. What is software release? 俺 サーバ
16. What is software release? 俺 サーバ おまえら
17. Release • Add a new feature and make it available to users • Modify existing features • Delete existing features • Add/modify/delete internal implementation (or Refactor)
18. "Release" typical properties • Release = Code deploy && daemon restart ◦ Not atomic • Code deploy size = n commits ◦ n depends on ▪ num devs ▪ deploy frequency • Quipper: ◦ 636 commits, 155 PRs, 42 devs, once a week
19. Release difficulties • Edge case logical bugs ◦ Works for most users ◦ Breaks for certain users ◦ • Unexpected untested changes ◦ Somewhere far can get changed ◦ • Runtime env specific issues
20. Release difficulties • Edge case logical bugs ◦ Works for most users ◦ Breaks for certain users ◦ More tests! ? • Unexpected untested changes ◦ Somewhere far can get changed ◦ More tests! ? • Runtime env specific issues
21. Testing is hard I even make a mistake exactly both at the implementation and the test code
22. from https://speakerdeck.com/sinamon129/sisutemuzhang-hai-tofalsexiang-kihe-ifang-at-sinamon129-number-tokyogirlsrb TokyoGirls.rb Meetup
23. Tests cost • Cost of a test case ◦ Write ◦ Run ◦ Maintain • Benefit of a test case ◦ Can it prevent new bugs? ◦ Can it explain the behaviour to us well? • Good test = low cost * high benefit
24. Do I write tests? • Yes! • All model public methods ◦ Sometimes even for private methods only if it makes sense ◦ If it's too trivial I sometimes skip • All controller actions ◦ Without very edgy cases • Not really for others Caution: This page can be controversial
25. Safe Release 安心してリリースしたい
26. Solutions? • Integration test ◦ Headless browser ◦ Browser • Static code analysis • Dynamic analysis • Performance benchmarking/simulation • Test manually • Pay somebody to test manually
27. Solutions? • Integration test ◦ Headless browser ◦ Browser • Static code analysis • Dynamic analysis • Performance benchmarking/simulation • Test manually • Pay somebody to test manually → Weekly release
28. Quipper's solution (1): make manual test trivial • Pull request code review • CI deploys your branch into a dedicated cluster ◦ Reviewer can actually try the web app ◦ Masked production data リ
29. Quipper's solution (2): Outsource human tests • Test all the features every week ◦ Product Managers make happy path scenario, and pass it to QA company
30. Remaining problems • Weekly release with giant changes ◦ How to detect which commit broke? • Long life branches ◦ Merge conflicts ◦ Hard to co-work • Nontrivial to rollback specific one ◦ Hotfix to master branch, merge back to develop branch 27/92
31. Release anytime but how? 28/92
32. Release gradually Binary release: 0 → 1 Analogue release: 0 → 0.1 → 0.2 → ... → 0.9 → 1
33. Release gradually • Big bang release (0% -> 100%) • Split features smaller, and release each (0% -> 20% -> 30% -> ... -> 90% -> 100%) • Release to certain people first, and expose to more people gradually (0% -> 1% -> ... -> 99% -> 100%) ◦ Can be randomized ◦ Can be manually specified
34. Release gradually • Big bang release (0% -> 100%) • Split features smaller, and release each (0% -> 20% -> 30% -> ... -> 90% -> 100%) • Release to certain people first, and expose to more people gradually (0% -> 1% -> ... -> 99% -> 100%) ◦ Can be randomized ◦ Can be manually specified MVP Minimum Viable Product
35. Release gradually • Big bang release (0% -> 100%) • Split features smaller, and release each of them (0% -> 20% -> 30% -> ... -> 90% -> 100%) • Release to certain people first, and expose to more people gradually (0% -> 1% -> ... -> 99% -> 100%) ◦ Can be randomized ◦ Can be manually specified MVP Minimum Viable Product https://blog.deming.org/2014/11/minimal-viable-product/
41. in production Iterate and learn 41/93
42. Iterate and learn in production https://www.youtube.com/w atch?v=75wa8Lx4yc4
43. Release gradually • Big bang release (0% -> 100%) • Split features smaller, and release each of them (0% -> 20% -> 30% -> ... -> 90% -> 100%) • Release to certain people first, and expose to more people gradually (0% -> 1% -> ... -> 99% -> 100%) ◦ Can be randomized ◦ Can be manually specified 43/93
44. Release to certain people # dashboard_controller.rb before_action :authenticate def index @user = current_user render :index end
45. Release to certain people # dashboard_controller.rb before_action :authenticate def index @user = current_user if [234, 345, 456].include?(@user.id) render :the_new_fancy_index else render :index end end
46. Release to certain people # dashboard_controller.rb before_action :authenticate def index @user = current_user if ['aaa@quipper.com', 'test@example.com'].include?(@user.email) render :the_new_fancy_index else render :index end end
47. Release to certain people dynamically # dashboard_controller.rb before_action :authenticate def index @user = current_user if UsersToShow.exists?(key: 'fancy-dashboard', email: @user.email) render :the_new_fancy_index else render :index end end Run this manually: UsersToShow.create!(key: 'fancy-dashboard', email: 'aaa@quipper.com')
48. Release to some people (percentage) def index @user = current_user if @user.id % 100 < 31 render :the_new_fancy_index else render :index end end
49. Release to some people (percentage) dynamically def index @user = current_user percentage = PercentageToShow.find(key: 'fancy-dashboard')&.value if @user.id % 100 < (percentage || 0) render :the_new_fancy_index else render :index end end Run this manually PercentageToShow.create!(key: 'fancy-dashboard', value: 31)
50. Dynamic tweaking • More powerful than it sounds like • No code change to release • No code change to rollback • Any time to release ◦ Not at the same time to other commits • No need to have perfect tests for unreachable code
51. Digital (Big bang) release
52. Gradual release (feature)
53. Gradual release (user)
55. Gradual release (user*feature)
58. New problems • Gradual release adds complexity ◦ Manage both old/new code ◦ Behaviour depends on each users ◦ More often to rollback ◦ Tend to forget to release • Hard to hold both old/new code for: ◦ Application-global changes ◦ Infrastructure-layer changes 58/93
59. P. Mange both old/new code • S. Embrace it • Make it pattern for ease to read def index @user = current_user percentage = PercentageToShow.find(key: 'fancy-dashboard')&.value if @user.id.hash % 100 < (percentage || 0) render :the_new_fancy_index else render :index end end
60. P. Mange both old/new code • S. Embrace it • Make it pattern for ease to read def index @user = current_user if FeatureToggles.variation('enable-fancy-dashboard', @user) render :the_new_fancy_index else render :index end end
61. P. Mange both old/new code def index @user = current_user if FeatureToggles.variation('enable-fancy-dashboard', @user) render :the_new_fancy_index else render :index end end • "Feature toggles" There's a name for it ◦ https://martinfowler.com/articles/feature-toggles .html ◦ "modify system without changing code"
62. Darklaunch Feature Toggles
63. P. Mange both old/new code def index @user = current_user if Darklaunch.variation('enable-fancy-dashboard', @user) render :the_new_fancy_index else render :index end end • Quipper uses the word "Darklaunch" ◦ potential confusion with "Feature flags" we already have
64. P. Mange both old/new code def index @user = current_user if Darklaunch.variation('enable-fancy-dashboard', @user) render :the_new_fancy_index else render :index end end • Quipper's Darklaunch stores into MongoDB ◦ Most models are MongoMapper ◦ (Darklaunch was on Redis at first)
66. def index @user = current_user if Darklaunch.variation('enable-fancy-dashboard', {id: @user.id, …}) render :the_new_fancy_index else render :index end end • returns true only for new experimental features ◦ true = new (dangerous), false = old (safe)
67. def index @user = current_user if Darklaunch.variation('enable-fancy-dashboard', {id: @user.id, …}) render :the_new_fancy_index else render :index end end • Pass user as a PORO ◦ Let darklaunch not to know about business logic
68. def index @user = current_user if Darklaunch.v1_variation('enable-fancy-dashboard', {id: @user.id}) render :the_new_fancy_index else render :index end end • Darklaunch can't darklaunch itself ◦ Never change the interface
69. P. More often to rollback • Hypothesis: more often to release = more often to rollback • What's rollback ◦ 0. an issue happens ◦ 1. find where it happens ◦ 2. change app behave as previous ◦ 3. release the change ◦ (4. investigate why and solve it)
70. • S. make it easy to darklaunch/undarklaunch P. More often to rollback
71. P. More often to rollback • Back then ◦ revert commit ◦ deploy (= release) def index @user = current_user render :the_new_fancy_index # boom! RuntimeError! end ↓ rollback def index @user = current_user render :index
72. P. More often to rollback def index @user = current_user if Darklaunch.v1_variation('enable-fancy-dashboard', {id: @user.id}) render :the_new_fancy_index # boom! RuntimeError! else render :index end end • Typical task: ◦ (User reports a bug) ◦ An exception at the new code ◦ Undarklaunch the key
73. P. More often to rollback def index @user = current_user if Darklaunch.v1_variation('enable-fancy-dashboard', {id: @user.id}) begin render :the_new_fancy_index # boom! RuntimeError! rescue => e ... Darklaunch.remove('enable-fancy-dashboard') • Typical task: ◦ (User reports a bug) ◦ An exception at the new code ◦ Undarklaunch the key
74. • feature togglesの説明 • quipperのdarklaunchの説明 ◦ 実例スクショ ◦ slack会話スクショ • back-officeのUIの説明 • 外部サービスの説明 (詳しくないから飛ばすかも)
75. Circuit Breaker A pattern to prevent catastrophic cascade https://martinfowler.com/bliki/CircuitBreaker.html
76. Circuit Breaker (Martin Fowler) • Very common • Stateful gateway to external resources • Toggle between "Open", "Half Open", and "Closed" • Requests timeout immediately in "Closed"
77. Can be used also to toggle behaviour That's what I showed with Darklaunch
78. Special thanks to Hootsuite • https://hootsuite.com • PHP Monolith + Scala Microservices • Darklaunch ◦ Consul • Circuit breaker ◦ https://github.com/hootsuite/scala-circuit-breake r
79. Re: New problems • Gradual release adds complexity ◦ Manage both old/new code ◦ Behaviour depends on each users ◦ More often to rollback ◦ Tend to forget to release • Hard to hold both old/new code for: ◦ Application-global changes ◦ Infrastructure-layer changes
80. Re: New problems • Gradual release adds complexity ◦ Manage both old/new code ◦ Behaviour depends on each users ◦ More often to rollback ◦ Tend to forget to release • Hard to hold both old/new code for: ◦ Application-global changes ◦ Infrastructure-layer changes Solved! Darklaunch module Darklaunch dashboard (*) Circuit breaker Darklaunch stats (*) Canary release Canary release
81. 雑 "Almost microservices" Tatsuhiro Ujihisa RailsDM 2019 2019-03-22 80/93
82. Microservices An application with collection of "services" 81/93
83. Microservices Pros • Team develops/maintains a subset of an app ◦ ownership! • Easy to make a big change ◦ totally independent releases ▪ frequent small releases = trivial to debug production issues • no need to know details of other services
84. Microservices Cons • Distributed ◦ "Networks are unreliable in the worst possible way" ― 7 deadly sin • Requires well-defined service boundaries ◦ is it possible? • Hard to change interface • Well, need to know details of both services
85. Distributed Monolith and Microservices • Monolith ◦ Not scalable with many devs ◦ Different concerns in same code • Distributed Monolith ◦ Monolith + Network partitioning • Microservices ◦ Easier to deploy, and make runtime-wide changes ◦ Hard to make microservices-wide changes
86. Distributed Monolith and Microservices • Monolith ◦ Not scalable with many devs ◦ Different concerns in same code • Distributed Monolith ◦ Monolith + Partitioning • Microservices ◦ Easier to deploy, and make runtime-wide changes ◦ Hard to make microservices-wide changes
87. Monolith + Feature toggles • Different concerns in same code ◦ Modules/classes/methods • Easier to deploy • Manageable to make app-global changes • No network partitioning
88. Monolith + Feature toggles • Different concerns in same code ◦ Modules/classes/methods • Easier to deploy • Manageable to make app-global changes • No network partitioning 雑 "Almost microservices" Tatsuhiro Ujihisa RailsDM 2019 2019-03-22
89. end
90. ujihisa before Quipper • 2010 ~ 2016 ◦ https://hootsuite.com ◦ PHP Monolith ◦ Scala Microservices ◦ Go Microservices • 2017~2018 ◦ https://fril.jp ◦ Rails Distributed Monolith "分断されたモノリス" 89/93
91. ujihisa this year • 2019-03 ◦ RailsDM 2019 "雑" Almost Microservices • 2019-04 ◦ RubyKaigi 2019 "Play with local vars" • 2019-05+ ◦ Move back to Vancouver, BC, Canada 90/93
92. https://www.reddit.com/r/MapPorn/comments/9q4wyj/civilization_vi_earth_map/ 90/92
93. https://www.reddit.com/r/MapPorn/comments/9q4wyj/civilization_vi_earth_map/ 91/92
94. "雑" Almost microservices / @ujihisa • Release is important ◦ Grind it to small pieces for scalability • Human makes mistakes • Feature toggles pattern is powerful ◦ Quipper calls it Darklaunch ◦ Provide strong support to Darklaunch • Apply (somewhat) Circuit Breaker pattern and Canary release if necessary • See you at the RubyKaigi 2019! 93/93