"雑" / Almost Microservices

D9d0ceb387e3b6de5c4562af78e8a910?s=47 ujihisa
March 22, 2019

"雑" / Almost Microservices

at RailsDM 2019

D9d0ceb387e3b6de5c4562af78e8a910?s=128

ujihisa

March 22, 2019
Tweet

Transcript

  1. 2.

    https://en.wiktionary.or g/wiki/%E9%9B%91 Adjectival noun 雑 (shinjitai kanji, kyūjitai kanji 雜,

    -na inflection, hiragana ざ つ, rōmaji zatsu) • rough, crude, sloppy, messy • miscellaneous (see note) 2/92
  2. 3.

    ujihisa • https://github.com/ujihisa • https://twitter.com/ujm • VimConf 2018 Speaker &

    Organizer, RubyKaigi 2019 Speaker • Quipper Limited ◦ Co-workers' talks: @banyan, @mtsmfm ◦ "Distributed Monolith" @kyanny from RailsDM 2018 3/92
  3. 4.

    ujihisa before Quipper • 2010 ~ 2016 ◦ https://hootsuite.com ◦

    PHP Monolith ◦ Scala Microservices ◦ Go Microservices • 2017~2018 ◦ https://fril.jp ◦ Rails Distributed Monolith "分断されたモノリス"
  4. 6.
  5. 8.
  6. 9.

    Quipper • Joined in 2018-09 ◦ 7 months so far

    ▪ 4 months at School Team (*) ▪ 1 month at Unemployed (*) ▪ 2 months at Data Restructuring Team
  7. 11.

    "雑" Almost microservices / @ujihisa • Release is important ◦

    Grind it to small pieces for scalability • Human makes mistakes • Feature toggles pattern is powerful ◦ Quipper calls it Darklaunch ◦ Provide strong support to Darklaunch • Apply Circuit Breaker pattern and Canary release if necessary • Pragmatic Microservices' essence
  8. 12.

    Web app development • Research & decide what to make

    • rails new • infrastructure setup • git, code review, CI setup • design models, views, and controllers • development! of course unit tests too! • refactor • release & debug
  9. 13.

    Web app development • Research & decide what to make

    • rails new • infrastructure setup • git, code review, CI setup • design models, views, and controllers • development! of course unit tests too! • refactor •                 & debug Release
  10. 17.

    Release • Add a new feature and make it available

    to users • Modify existing features • Delete existing features • Add/modify/delete internal implementation (or Refactor)
  11. 18.

    "Release" typical properties • Release = Code deploy && daemon

    restart ◦ Not atomic • Code deploy size = n commits ◦ n depends on ▪ num devs ▪ deploy frequency • Quipper: ◦ 636 commits, 155 PRs, 42 devs, once a week
  12. 19.

    Release difficulties • Edge case logical bugs ◦ Works for

    most users ◦ Breaks for certain users ◦ • Unexpected untested changes ◦ Somewhere far can get changed ◦ • Runtime env specific issues
  13. 20.

    Release difficulties • Edge case logical bugs ◦ Works for

    most users ◦ Breaks for certain users ◦ More tests! ? • Unexpected untested changes ◦ Somewhere far can get changed ◦ More tests! ? • Runtime env specific issues
  14. 21.

    Testing is hard I even make a mistake exactly both

    at the implementation and the test code
  15. 23.

    Tests cost • Cost of a test case ◦ Write

    ◦ Run ◦ Maintain • Benefit of a test case ◦ Can it prevent new bugs? ◦ Can it explain the behaviour to us well? • Good test = low cost * high benefit
  16. 24.

    Do I write tests? • Yes! • All model public

    methods ◦ Sometimes even for private methods only if it makes sense ◦ If it's too trivial I sometimes skip • All controller actions ◦ Without very edgy cases • Not really for others Caution: This page can be controversial
  17. 26.

    Solutions? • Integration test ◦ Headless browser ◦ Browser •

    Static code analysis • Dynamic analysis • Performance benchmarking/simulation • Test manually • Pay somebody to test manually
  18. 27.

    Solutions? • Integration test ◦ Headless browser ◦ Browser •

    Static code analysis • Dynamic analysis • Performance benchmarking/simulation • Test manually • Pay somebody to test manually → Weekly release
  19. 28.

    Quipper's solution (1): make manual test trivial • Pull request

    code review • CI deploys your branch into a dedicated cluster ◦ Reviewer can actually try the web app ◦ Masked production data リ
  20. 29.

    Quipper's solution (2): Outsource human tests • Test all the

    features every week ◦ Product Managers make happy path scenario, and pass it to QA company
  21. 30.

    Remaining problems • Weekly release with giant changes ◦ How

    to detect which commit broke? • Long life branches ◦ Merge conflicts ◦ Hard to co-work • Nontrivial to rollback specific one ◦ Hotfix to master branch, merge back to develop branch 27/92
  22. 32.
  23. 33.

    Release gradually • Big bang release (0% -> 100%) •

    Split features smaller, and release each (0% -> 20% -> 30% -> ... -> 90% -> 100%) • Release to certain people first, and expose to more people gradually (0% -> 1% -> ... -> 99% -> 100%) ◦ Can be randomized ◦ Can be manually specified
  24. 34.

    Release gradually • Big bang release (0% -> 100%) •

    Split features smaller, and release each (0% -> 20% -> 30% -> ... -> 90% -> 100%) • Release to certain people first, and expose to more people gradually (0% -> 1% -> ... -> 99% -> 100%) ◦ Can be randomized ◦ Can be manually specified MVP Minimum Viable Product
  25. 35.

    Release gradually • Big bang release (0% -> 100%) •

    Split features smaller, and release each of them (0% -> 20% -> 30% -> ... -> 90% -> 100%) • Release to certain people first, and expose to more people gradually (0% -> 1% -> ... -> 99% -> 100%) ◦ Can be randomized ◦ Can be manually specified MVP Minimum Viable Product https://blog.deming.org/2014/11/minimal-viable-product/
  26. 36.
  27. 37.
  28. 38.
  29. 39.
  30. 40.
  31. 43.

    Release gradually • Big bang release (0% -> 100%) •

    Split features smaller, and release each of them (0% -> 20% -> 30% -> ... -> 90% -> 100%) • Release to certain people first, and expose to more people gradually (0% -> 1% -> ... -> 99% -> 100%) ◦ Can be randomized ◦ Can be manually specified 43/93
  32. 45.

    Release to certain people # dashboard_controller.rb before_action :authenticate def index

    @user = current_user if [234, 345, 456].include?(@user.id) render :the_new_fancy_index else render :index end end
  33. 46.

    Release to certain people # dashboard_controller.rb before_action :authenticate def index

    @user = current_user if ['aaa@quipper.com', 'test@example.com'].include?(@user.email) render :the_new_fancy_index else render :index end end
  34. 47.

    Release to certain people dynamically # dashboard_controller.rb before_action :authenticate def

    index @user = current_user if UsersToShow.exists?(key: 'fancy-dashboard', email: @user.email) render :the_new_fancy_index else render :index end end Run this manually: UsersToShow.create!(key: 'fancy-dashboard', email: 'aaa@quipper.com')
  35. 48.

    Release to some people (percentage) def index @user = current_user

    if @user.id % 100 < 31 render :the_new_fancy_index else render :index end end
  36. 49.

    Release to some people (percentage) dynamically def index @user =

    current_user percentage = PercentageToShow.find(key: 'fancy-dashboard')&.value if @user.id % 100 < (percentage || 0) render :the_new_fancy_index else render :index end end Run this manually PercentageToShow.create!(key: 'fancy-dashboard', value: 31)
  37. 50.

    Dynamic tweaking • More powerful than it sounds like •

    No code change to release • No code change to rollback • Any time to release ◦ Not at the same time to other commits • No need to have perfect tests for unreachable code
  38. 54.
  39. 56.
  40. 57.
  41. 58.

    New problems • Gradual release adds complexity ◦ Manage both

    old/new code ◦ Behaviour depends on each users ◦ More often to rollback ◦ Tend to forget to release • Hard to hold both old/new code for: ◦ Application-global changes ◦ Infrastructure-layer changes 58/93
  42. 59.

    P. Mange both old/new code • S. Embrace it •

    Make it pattern for ease to read def index @user = current_user percentage = PercentageToShow.find(key: 'fancy-dashboard')&.value if @user.id.hash % 100 < (percentage || 0) render :the_new_fancy_index else render :index end end
  43. 60.

    P. Mange both old/new code • S. Embrace it •

    Make it pattern for ease to read def index @user = current_user if FeatureToggles.variation('enable-fancy-dashboard', @user) render :the_new_fancy_index else render :index end end
  44. 61.

    P. Mange both old/new code def index @user = current_user

    if FeatureToggles.variation('enable-fancy-dashboard', @user) render :the_new_fancy_index else render :index end end • "Feature toggles" There's a name for it ◦ https://martinfowler.com/articles/feature-toggles .html ◦ "modify system without changing code"
  45. 63.

    P. Mange both old/new code def index @user = current_user

    if Darklaunch.variation('enable-fancy-dashboard', @user) render :the_new_fancy_index else render :index end end • Quipper uses the word "Darklaunch" ◦ potential confusion with "Feature flags" we already have
  46. 64.

    P. Mange both old/new code def index @user = current_user

    if Darklaunch.variation('enable-fancy-dashboard', @user) render :the_new_fancy_index else render :index end end • Quipper's Darklaunch stores into MongoDB ◦ Most models are MongoMapper ◦ (Darklaunch was on Redis at first)
  47. 65.
  48. 66.

    def index @user = current_user if Darklaunch.variation('enable-fancy-dashboard', {id: @user.id, …})

    render :the_new_fancy_index else render :index end end • returns true only for new experimental features ◦ true = new (dangerous), false = old (safe)
  49. 67.

    def index @user = current_user if Darklaunch.variation('enable-fancy-dashboard', {id: @user.id, …})

    render :the_new_fancy_index else render :index end end • Pass user as a PORO ◦ Let darklaunch not to know about business logic
  50. 68.

    def index @user = current_user if Darklaunch.v1_variation('enable-fancy-dashboard', {id: @user.id}) render

    :the_new_fancy_index else render :index end end • Darklaunch can't darklaunch itself ◦ Never change the interface
  51. 69.

    P. More often to rollback • Hypothesis: more often to

    release = more often to rollback • What's rollback ◦ 0. an issue happens ◦ 1. find where it happens ◦ 2. change app behave as previous ◦ 3. release the change ◦ (4. investigate why and solve it)
  52. 71.

    P. More often to rollback • Back then ◦ revert

    commit ◦ deploy (= release) def index @user = current_user render :the_new_fancy_index # boom! RuntimeError! end ↓ rollback def index @user = current_user render :index
  53. 72.

    P. More often to rollback def index @user = current_user

    if Darklaunch.v1_variation('enable-fancy-dashboard', {id: @user.id}) render :the_new_fancy_index # boom! RuntimeError! else render :index end end • Typical task: ◦ (User reports a bug) ◦ An exception at the new code ◦ Undarklaunch the key
  54. 73.

    P. More often to rollback def index @user = current_user

    if Darklaunch.v1_variation('enable-fancy-dashboard', {id: @user.id}) begin render :the_new_fancy_index # boom! RuntimeError! rescue => e ... Darklaunch.remove('enable-fancy-dashboard') • Typical task: ◦ (User reports a bug) ◦ An exception at the new code ◦ Undarklaunch the key
  55. 74.

    • feature togglesの説明 • quipperのdarklaunchの説明 ◦ 実例スクショ ◦ slack会話スクショ •

    back-officeのUIの説明 • 外部サービスの説明 (詳しくないから飛ばすかも)
  56. 76.

    Circuit Breaker (Martin Fowler) • Very common • Stateful gateway

    to external resources • Toggle between "Open", "Half Open", and "Closed" • Requests timeout immediately in "Closed"
  57. 78.

    Special thanks to Hootsuite • https://hootsuite.com • PHP Monolith +

    Scala Microservices • Darklaunch ◦ Consul • Circuit breaker ◦ https://github.com/hootsuite/scala-circuit-breake r
  58. 79.

    Re: New problems • Gradual release adds complexity ◦ Manage

    both old/new code ◦ Behaviour depends on each users ◦ More often to rollback ◦ Tend to forget to release • Hard to hold both old/new code for: ◦ Application-global changes ◦ Infrastructure-layer changes
  59. 80.

    Re: New problems • Gradual release adds complexity ◦ Manage

    both old/new code ◦ Behaviour depends on each users ◦ More often to rollback ◦ Tend to forget to release • Hard to hold both old/new code for: ◦ Application-global changes ◦ Infrastructure-layer changes Solved! Darklaunch module Darklaunch dashboard (*) Circuit breaker Darklaunch stats (*) Canary release Canary release
  60. 83.

    Microservices Pros • Team develops/maintains a subset of an app

    ◦ ownership! • Easy to make a big change ◦ totally independent releases ▪ frequent small releases = trivial to debug production issues • no need to know details of other services
  61. 84.

    Microservices Cons • Distributed ◦ "Networks are unreliable in the

    worst possible way" ― 7 deadly sin • Requires well-defined service boundaries ◦ is it possible? • Hard to change interface • Well, need to know details of both services
  62. 85.

    Distributed Monolith and Microservices • Monolith ◦ Not scalable with

    many devs ◦ Different concerns in same code • Distributed Monolith ◦ Monolith + Network partitioning • Microservices ◦ Easier to deploy, and make runtime-wide changes ◦ Hard to make microservices-wide changes
  63. 86.

    Distributed Monolith and Microservices • Monolith ◦ Not scalable with

    many devs ◦ Different concerns in same code • Distributed Monolith ◦ Monolith + Partitioning • Microservices ◦ Easier to deploy, and make runtime-wide changes ◦ Hard to make microservices-wide changes
  64. 87.

    Monolith + Feature toggles • Different concerns in same code

    ◦ Modules/classes/methods • Easier to deploy • Manageable to make app-global changes • No network partitioning
  65. 88.

    Monolith + Feature toggles • Different concerns in same code

    ◦ Modules/classes/methods • Easier to deploy • Manageable to make app-global changes • No network partitioning 雑 "Almost microservices" Tatsuhiro Ujihisa RailsDM 2019 2019-03-22
  66. 89.

    end

  67. 90.

    ujihisa before Quipper • 2010 ~ 2016 ◦ https://hootsuite.com ◦

    PHP Monolith ◦ Scala Microservices ◦ Go Microservices • 2017~2018 ◦ https://fril.jp ◦ Rails Distributed Monolith "分断されたモノリス" 89/93
  68. 91.

    ujihisa this year • 2019-03 ◦ RailsDM 2019 "雑" Almost

    Microservices • 2019-04 ◦ RubyKaigi 2019 "Play with local vars" • 2019-05+ ◦ Move back to Vancouver, BC, Canada 90/93
  69. 94.

    "雑" Almost microservices / @ujihisa • Release is important ◦

    Grind it to small pieces for scalability • Human makes mistakes • Feature toggles pattern is powerful ◦ Quipper calls it Darklaunch ◦ Provide strong support to Darklaunch • Apply (somewhat) Circuit Breaker pattern and Canary release if necessary • See you at the RubyKaigi 2019! 93/93