Save 37% off PRO during our Black Friday Sale! »

"雑" / Almost Microservices

D9d0ceb387e3b6de5c4562af78e8a910?s=47 ujihisa
March 22, 2019

"雑" / Almost Microservices

at RailsDM 2019

D9d0ceb387e3b6de5c4562af78e8a910?s=128

ujihisa

March 22, 2019
Tweet

Transcript

  1. 雑 "Almost microservices" Tatsuhiro Ujihisa RailsDM 2019 2019-03-22 1/92

  2. https://en.wiktionary.or g/wiki/%E9%9B%91 Adjectival noun 雑 (shinjitai kanji, kyūjitai kanji 雜,

    -na inflection, hiragana ざ つ, rōmaji zatsu) • rough, crude, sloppy, messy • miscellaneous (see note) 2/92
  3. ujihisa • https://github.com/ujihisa • https://twitter.com/ujm • VimConf 2018 Speaker &

    Organizer, RubyKaigi 2019 Speaker • Quipper Limited ◦ Co-workers' talks: @banyan, @mtsmfm ◦ "Distributed Monolith" @kyanny from RailsDM 2018 3/92
  4. ujihisa before Quipper • 2010 ~ 2016 ◦ https://hootsuite.com ◦

    PHP Monolith ◦ Scala Microservices ◦ Go Microservices • 2017~2018 ◦ https://fril.jp ◦ Rails Distributed Monolith "分断されたモノリス"
  5. ujihisa • Cooking • Learned a lot about fish

  6. vim-jp

  7. Quipper/Quipper, Quipper/Study Sapuri • https://learn.quipper.com https://learn.studysapuri.jp ◦ Rails Distributed Monolith

  8. None
  9. Quipper • Joined in 2018-09 ◦ 7 months so far

    ▪ 4 months at School Team (*) ▪ 1 month at Unemployed (*) ▪ 2 months at Data Restructuring Team
  10. Almost microservices ujihisa 9/92

  11. "雑" Almost microservices / @ujihisa • Release is important ◦

    Grind it to small pieces for scalability • Human makes mistakes • Feature toggles pattern is powerful ◦ Quipper calls it Darklaunch ◦ Provide strong support to Darklaunch • Apply Circuit Breaker pattern and Canary release if necessary • Pragmatic Microservices' essence
  12. Web app development • Research & decide what to make

    • rails new • infrastructure setup • git, code review, CI setup • design models, views, and controllers • development! of course unit tests too! • refactor • release & debug
  13. Web app development • Research & decide what to make

    • rails new • infrastructure setup • git, code review, CI setup • design models, views, and controllers • development! of course unit tests too! • refactor •                 & debug Release
  14. Software release Release gives development meaning

  15. What is software release? 俺 サーバ

  16. What is software release? 俺 サーバ おまえら

  17. Release • Add a new feature and make it available

    to users • Modify existing features • Delete existing features • Add/modify/delete internal implementation (or Refactor)
  18. "Release" typical properties • Release = Code deploy && daemon

    restart ◦ Not atomic • Code deploy size = n commits ◦ n depends on ▪ num devs ▪ deploy frequency • Quipper: ◦ 636 commits, 155 PRs, 42 devs, once a week
  19. Release difficulties • Edge case logical bugs ◦ Works for

    most users ◦ Breaks for certain users ◦ • Unexpected untested changes ◦ Somewhere far can get changed ◦ • Runtime env specific issues
  20. Release difficulties • Edge case logical bugs ◦ Works for

    most users ◦ Breaks for certain users ◦ More tests! ? • Unexpected untested changes ◦ Somewhere far can get changed ◦ More tests! ? • Runtime env specific issues
  21. Testing is hard I even make a mistake exactly both

    at the implementation and the test code
  22. from https://speakerdeck.com/sinamon129/sisutemuzhang-hai-tofalsexiang-kihe-ifang-at-sinamon129-number-tokyogirlsrb TokyoGirls.rb Meetup

  23. Tests cost • Cost of a test case ◦ Write

    ◦ Run ◦ Maintain • Benefit of a test case ◦ Can it prevent new bugs? ◦ Can it explain the behaviour to us well? • Good test = low cost * high benefit
  24. Do I write tests? • Yes! • All model public

    methods ◦ Sometimes even for private methods only if it makes sense ◦ If it's too trivial I sometimes skip • All controller actions ◦ Without very edgy cases • Not really for others Caution: This page can be controversial
  25. Safe Release 安心してリリースしたい

  26. Solutions? • Integration test ◦ Headless browser ◦ Browser •

    Static code analysis • Dynamic analysis • Performance benchmarking/simulation • Test manually • Pay somebody to test manually
  27. Solutions? • Integration test ◦ Headless browser ◦ Browser •

    Static code analysis • Dynamic analysis • Performance benchmarking/simulation • Test manually • Pay somebody to test manually → Weekly release
  28. Quipper's solution (1): make manual test trivial • Pull request

    code review • CI deploys your branch into a dedicated cluster ◦ Reviewer can actually try the web app ◦ Masked production data リ
  29. Quipper's solution (2): Outsource human tests • Test all the

    features every week ◦ Product Managers make happy path scenario, and pass it to QA company
  30. Remaining problems • Weekly release with giant changes ◦ How

    to detect which commit broke? • Long life branches ◦ Merge conflicts ◦ Hard to co-work • Nontrivial to rollback specific one ◦ Hotfix to master branch, merge back to develop branch 27/92
  31. Release anytime but how? 28/92

  32. Release gradually Binary release: 0 → 1 Analogue release: 0

    → 0.1 → 0.2 → ... → 0.9 → 1
  33. Release gradually • Big bang release (0% -> 100%) •

    Split features smaller, and release each (0% -> 20% -> 30% -> ... -> 90% -> 100%) • Release to certain people first, and expose to more people gradually (0% -> 1% -> ... -> 99% -> 100%) ◦ Can be randomized ◦ Can be manually specified
  34. Release gradually • Big bang release (0% -> 100%) •

    Split features smaller, and release each (0% -> 20% -> 30% -> ... -> 90% -> 100%) • Release to certain people first, and expose to more people gradually (0% -> 1% -> ... -> 99% -> 100%) ◦ Can be randomized ◦ Can be manually specified MVP Minimum Viable Product
  35. Release gradually • Big bang release (0% -> 100%) •

    Split features smaller, and release each of them (0% -> 20% -> 30% -> ... -> 90% -> 100%) • Release to certain people first, and expose to more people gradually (0% -> 1% -> ... -> 99% -> 100%) ◦ Can be randomized ◦ Can be manually specified MVP Minimum Viable Product https://blog.deming.org/2014/11/minimal-viable-product/
  36. None
  37. None
  38. None
  39. None
  40. None
  41. in production Iterate and learn 41/93

  42. Iterate and learn in production https://www.youtube.com/w atch?v=75wa8Lx4yc4

  43. Release gradually • Big bang release (0% -> 100%) •

    Split features smaller, and release each of them (0% -> 20% -> 30% -> ... -> 90% -> 100%) • Release to certain people first, and expose to more people gradually (0% -> 1% -> ... -> 99% -> 100%) ◦ Can be randomized ◦ Can be manually specified 43/93
  44. Release to certain people # dashboard_controller.rb before_action :authenticate def index

    @user = current_user render :index end
  45. Release to certain people # dashboard_controller.rb before_action :authenticate def index

    @user = current_user if [234, 345, 456].include?(@user.id) render :the_new_fancy_index else render :index end end
  46. Release to certain people # dashboard_controller.rb before_action :authenticate def index

    @user = current_user if ['aaa@quipper.com', 'test@example.com'].include?(@user.email) render :the_new_fancy_index else render :index end end
  47. Release to certain people dynamically # dashboard_controller.rb before_action :authenticate def

    index @user = current_user if UsersToShow.exists?(key: 'fancy-dashboard', email: @user.email) render :the_new_fancy_index else render :index end end Run this manually: UsersToShow.create!(key: 'fancy-dashboard', email: 'aaa@quipper.com')
  48. Release to some people (percentage) def index @user = current_user

    if @user.id % 100 < 31 render :the_new_fancy_index else render :index end end
  49. Release to some people (percentage) dynamically def index @user =

    current_user percentage = PercentageToShow.find(key: 'fancy-dashboard')&.value if @user.id % 100 < (percentage || 0) render :the_new_fancy_index else render :index end end Run this manually PercentageToShow.create!(key: 'fancy-dashboard', value: 31)
  50. Dynamic tweaking • More powerful than it sounds like •

    No code change to release • No code change to rollback • Any time to release ◦ Not at the same time to other commits • No need to have perfect tests for unreachable code
  51. Digital (Big bang) release

  52. Gradual release (feature)

  53. Gradual release (user)

  54. None
  55. Gradual release (user*feature)

  56. None
  57. None
  58. New problems • Gradual release adds complexity ◦ Manage both

    old/new code ◦ Behaviour depends on each users ◦ More often to rollback ◦ Tend to forget to release • Hard to hold both old/new code for: ◦ Application-global changes ◦ Infrastructure-layer changes 58/93
  59. P. Mange both old/new code • S. Embrace it •

    Make it pattern for ease to read def index @user = current_user percentage = PercentageToShow.find(key: 'fancy-dashboard')&.value if @user.id.hash % 100 < (percentage || 0) render :the_new_fancy_index else render :index end end
  60. P. Mange both old/new code • S. Embrace it •

    Make it pattern for ease to read def index @user = current_user if FeatureToggles.variation('enable-fancy-dashboard', @user) render :the_new_fancy_index else render :index end end
  61. P. Mange both old/new code def index @user = current_user

    if FeatureToggles.variation('enable-fancy-dashboard', @user) render :the_new_fancy_index else render :index end end • "Feature toggles" There's a name for it ◦ https://martinfowler.com/articles/feature-toggles .html ◦ "modify system without changing code"
  62. Darklaunch Feature Toggles

  63. P. Mange both old/new code def index @user = current_user

    if Darklaunch.variation('enable-fancy-dashboard', @user) render :the_new_fancy_index else render :index end end • Quipper uses the word "Darklaunch" ◦ potential confusion with "Feature flags" we already have
  64. P. Mange both old/new code def index @user = current_user

    if Darklaunch.variation('enable-fancy-dashboard', @user) render :the_new_fancy_index else render :index end end • Quipper's Darklaunch stores into MongoDB ◦ Most models are MongoMapper ◦ (Darklaunch was on Redis at first)
  65. None
  66. def index @user = current_user if Darklaunch.variation('enable-fancy-dashboard', {id: @user.id, …})

    render :the_new_fancy_index else render :index end end • returns true only for new experimental features ◦ true = new (dangerous), false = old (safe)
  67. def index @user = current_user if Darklaunch.variation('enable-fancy-dashboard', {id: @user.id, …})

    render :the_new_fancy_index else render :index end end • Pass user as a PORO ◦ Let darklaunch not to know about business logic
  68. def index @user = current_user if Darklaunch.v1_variation('enable-fancy-dashboard', {id: @user.id}) render

    :the_new_fancy_index else render :index end end • Darklaunch can't darklaunch itself ◦ Never change the interface
  69. P. More often to rollback • Hypothesis: more often to

    release = more often to rollback • What's rollback ◦ 0. an issue happens ◦ 1. find where it happens ◦ 2. change app behave as previous ◦ 3. release the change ◦ (4. investigate why and solve it)
  70. • S. make it easy to darklaunch/undarklaunch P. More often

    to rollback
  71. P. More often to rollback • Back then ◦ revert

    commit ◦ deploy (= release) def index @user = current_user render :the_new_fancy_index # boom! RuntimeError! end ↓ rollback def index @user = current_user render :index
  72. P. More often to rollback def index @user = current_user

    if Darklaunch.v1_variation('enable-fancy-dashboard', {id: @user.id}) render :the_new_fancy_index # boom! RuntimeError! else render :index end end • Typical task: ◦ (User reports a bug) ◦ An exception at the new code ◦ Undarklaunch the key
  73. P. More often to rollback def index @user = current_user

    if Darklaunch.v1_variation('enable-fancy-dashboard', {id: @user.id}) begin render :the_new_fancy_index # boom! RuntimeError! rescue => e ... Darklaunch.remove('enable-fancy-dashboard') • Typical task: ◦ (User reports a bug) ◦ An exception at the new code ◦ Undarklaunch the key
  74. • feature togglesの説明 • quipperのdarklaunchの説明 ◦ 実例スクショ ◦ slack会話スクショ •

    back-officeのUIの説明 • 外部サービスの説明 (詳しくないから飛ばすかも)
  75. Circuit Breaker A pattern to prevent catastrophic cascade https://martinfowler.com/bliki/CircuitBreaker.html

  76. Circuit Breaker (Martin Fowler) • Very common • Stateful gateway

    to external resources • Toggle between "Open", "Half Open", and "Closed" • Requests timeout immediately in "Closed"
  77. Can be used also to toggle behaviour That's what I

    showed with Darklaunch
  78. Special thanks to Hootsuite • https://hootsuite.com • PHP Monolith +

    Scala Microservices • Darklaunch ◦ Consul • Circuit breaker ◦ https://github.com/hootsuite/scala-circuit-breake r
  79. Re: New problems • Gradual release adds complexity ◦ Manage

    both old/new code ◦ Behaviour depends on each users ◦ More often to rollback ◦ Tend to forget to release • Hard to hold both old/new code for: ◦ Application-global changes ◦ Infrastructure-layer changes
  80. Re: New problems • Gradual release adds complexity ◦ Manage

    both old/new code ◦ Behaviour depends on each users ◦ More often to rollback ◦ Tend to forget to release • Hard to hold both old/new code for: ◦ Application-global changes ◦ Infrastructure-layer changes Solved! Darklaunch module Darklaunch dashboard (*) Circuit breaker Darklaunch stats (*) Canary release Canary release
  81. 雑 "Almost microservices" Tatsuhiro Ujihisa RailsDM 2019 2019-03-22 80/93

  82. Microservices An application with collection of "services" 81/93

  83. Microservices Pros • Team develops/maintains a subset of an app

    ◦ ownership! • Easy to make a big change ◦ totally independent releases ▪ frequent small releases = trivial to debug production issues • no need to know details of other services
  84. Microservices Cons • Distributed ◦ "Networks are unreliable in the

    worst possible way" ― 7 deadly sin • Requires well-defined service boundaries ◦ is it possible? • Hard to change interface • Well, need to know details of both services
  85. Distributed Monolith and Microservices • Monolith ◦ Not scalable with

    many devs ◦ Different concerns in same code • Distributed Monolith ◦ Monolith + Network partitioning • Microservices ◦ Easier to deploy, and make runtime-wide changes ◦ Hard to make microservices-wide changes
  86. Distributed Monolith and Microservices • Monolith ◦ Not scalable with

    many devs ◦ Different concerns in same code • Distributed Monolith ◦ Monolith + Partitioning • Microservices ◦ Easier to deploy, and make runtime-wide changes ◦ Hard to make microservices-wide changes
  87. Monolith + Feature toggles • Different concerns in same code

    ◦ Modules/classes/methods • Easier to deploy • Manageable to make app-global changes • No network partitioning
  88. Monolith + Feature toggles • Different concerns in same code

    ◦ Modules/classes/methods • Easier to deploy • Manageable to make app-global changes • No network partitioning 雑 "Almost microservices" Tatsuhiro Ujihisa RailsDM 2019 2019-03-22
  89. end

  90. ujihisa before Quipper • 2010 ~ 2016 ◦ https://hootsuite.com ◦

    PHP Monolith ◦ Scala Microservices ◦ Go Microservices • 2017~2018 ◦ https://fril.jp ◦ Rails Distributed Monolith "分断されたモノリス" 89/93
  91. ujihisa this year • 2019-03 ◦ RailsDM 2019 "雑" Almost

    Microservices • 2019-04 ◦ RubyKaigi 2019 "Play with local vars" • 2019-05+ ◦ Move back to Vancouver, BC, Canada 90/93
  92. https://www.reddit.com/r/MapPorn/comments/9q4wyj/civilization_vi_earth_map/ 90/92

  93. https://www.reddit.com/r/MapPorn/comments/9q4wyj/civilization_vi_earth_map/ 91/92

  94. "雑" Almost microservices / @ujihisa • Release is important ◦

    Grind it to small pieces for scalability • Human makes mistakes • Feature toggles pattern is powerful ◦ Quipper calls it Darklaunch ◦ Provide strong support to Darklaunch • Apply (somewhat) Circuit Breaker pattern and Canary release if necessary • See you at the RubyKaigi 2019! 93/93