"雑" / Almost Microservices

Transcript

1. 雑 "Almost microservices" Tatsuhiro Ujihisa RailsDM 2019 2019-03-22 1/92

2. https://en.wiktionary.or g/wiki/%E9%9B%91 Adjectival noun 雑 (shinjitai kanji, kyūjitai kanji 雜,

   -na inflection, hiragana ざ つ, rōmaji zatsu) • rough, crude, sloppy, messy • miscellaneous (see note) 2/92

3. ujihisa • https://github.com/ujihisa • https://twitter.com/ujm • VimConf 2018 Speaker &

   Organizer, RubyKaigi 2019 Speaker • Quipper Limited ◦ Co-workers' talks: @banyan, @mtsmfm ◦ "Distributed Monolith" @kyanny from RailsDM 2018 3/92

4. ujihisa before Quipper • 2010 ~ 2016 ◦ https://hootsuite.com ◦

   PHP Monolith ◦ Scala Microservices ◦ Go Microservices • 2017~2018 ◦ https://fril.jp ◦ Rails Distributed Monolith "分断されたモノリス"

5. ujihisa • Cooking • Learned a lot about fish

6. vim-jp

7. Quipper/Quipper, Quipper/Study Sapuri • https://learn.quipper.com https://learn.studysapuri.jp ◦ Rails Distributed Monolith

8. None

9. Quipper • Joined in 2018-09 ◦ 7 months so far

   ▪ 4 months at School Team (*) ▪ 1 month at Unemployed (*) ▪ 2 months at Data Restructuring Team

10. Almost microservices ujihisa 9/92

11. "雑" Almost microservices / @ujihisa • Release is important ◦

    Grind it to small pieces for scalability • Human makes mistakes • Feature toggles pattern is powerful ◦ Quipper calls it Darklaunch ◦ Provide strong support to Darklaunch • Apply Circuit Breaker pattern and Canary release if necessary • Pragmatic Microservices' essence

12. Web app development • Research & decide what to make

    • rails new • infrastructure setup • git, code review, CI setup • design models, views, and controllers • development! of course unit tests too! • refactor • release & debug

13. Web app development • Research & decide what to make

    • rails new • infrastructure setup • git, code review, CI setup • design models, views, and controllers • development! of course unit tests too! • refactor • 　　　　　　　　　　　　　　　　& debug Release

14. Software release Release gives development meaning

15. What is software release? 俺 サーバ

16. What is software release? 俺 サーバ おまえら

17. Release • Add a new feature and make it available

    to users • Modify existing features • Delete existing features • Add/modify/delete internal implementation (or Refactor)

18. "Release" typical properties • Release = Code deploy && daemon

    restart ◦ Not atomic • Code deploy size = n commits ◦ n depends on ▪ num devs ▪ deploy frequency • Quipper: ◦ 636 commits, 155 PRs, 42 devs, once a week

19. Release difficulties • Edge case logical bugs ◦ Works for

    most users ◦ Breaks for certain users ◦ • Unexpected untested changes ◦ Somewhere far can get changed ◦ • Runtime env specific issues

20. Release difficulties • Edge case logical bugs ◦ Works for

    most users ◦ Breaks for certain users ◦ More tests! ? • Unexpected untested changes ◦ Somewhere far can get changed ◦ More tests! ? • Runtime env specific issues

21. Testing is hard I even make a mistake exactly both

    at the implementation and the test code

22. from https://speakerdeck.com/sinamon129/sisutemuzhang-hai-tofalsexiang-kihe-ifang-at-sinamon129-number-tokyogirlsrb TokyoGirls.rb Meetup

23. Tests cost • Cost of a test case ◦ Write

    ◦ Run ◦ Maintain • Benefit of a test case ◦ Can it prevent new bugs? ◦ Can it explain the behaviour to us well? • Good test = low cost * high benefit

24. Do I write tests? • Yes! • All model public

    methods ◦ Sometimes even for private methods only if it makes sense ◦ If it's too trivial I sometimes skip • All controller actions ◦ Without very edgy cases • Not really for others Caution: This page can be controversial

25. Safe Release 安心してリリースしたい

26. Solutions? • Integration test ◦ Headless browser ◦ Browser •

    Static code analysis • Dynamic analysis • Performance benchmarking/simulation • Test manually • Pay somebody to test manually

27. Solutions? • Integration test ◦ Headless browser ◦ Browser •

    Static code analysis • Dynamic analysis • Performance benchmarking/simulation • Test manually • Pay somebody to test manually → Weekly release

28. Quipper's solution (1): make manual test trivial • Pull request

    code review • CI deploys your branch into a dedicated cluster ◦ Reviewer can actually try the web app ◦ Masked production data リ

29. Quipper's solution (2): Outsource human tests • Test all the

    features every week ◦ Product Managers make happy path scenario, and pass it to QA company

30. Remaining problems • Weekly release with giant changes ◦ How

    to detect which commit broke? • Long life branches ◦ Merge conflicts ◦ Hard to co-work • Nontrivial to rollback specific one ◦ Hotfix to master branch, merge back to develop branch 27/92

31. Release anytime but how? 28/92

32. Release gradually Binary release: 0 → 1 Analogue release: 0

    → 0.1 → 0.2 → ... → 0.9 → 1

33. Release gradually • Big bang release (0% -> 100%) •

    Split features smaller, and release each (0% -> 20% -> 30% -> ... -> 90% -> 100%) • Release to certain people first, and expose to more people gradually (0% -> 1% -> ... -> 99% -> 100%) ◦ Can be randomized ◦ Can be manually specified

34. Release gradually • Big bang release (0% -> 100%) •

    Split features smaller, and release each (0% -> 20% -> 30% -> ... -> 90% -> 100%) • Release to certain people first, and expose to more people gradually (0% -> 1% -> ... -> 99% -> 100%) ◦ Can be randomized ◦ Can be manually specified MVP Minimum Viable Product

35. Release gradually • Big bang release (0% -> 100%) •

    Split features smaller, and release each of them (0% -> 20% -> 30% -> ... -> 90% -> 100%) • Release to certain people first, and expose to more people gradually (0% -> 1% -> ... -> 99% -> 100%) ◦ Can be randomized ◦ Can be manually specified MVP Minimum Viable Product https://blog.deming.org/2014/11/minimal-viable-product/
36. None
37. None
38. None
39. None
40. None

41. in production Iterate and learn 41/93

42. Iterate and learn in production https://www.youtube.com/w atch?v=75wa8Lx4yc4

43. Release gradually • Big bang release (0% -> 100%) •

    Split features smaller, and release each of them (0% -> 20% -> 30% -> ... -> 90% -> 100%) • Release to certain people first, and expose to more people gradually (0% -> 1% -> ... -> 99% -> 100%) ◦ Can be randomized ◦ Can be manually specified 43/93

44. Release to certain people # dashboard_controller.rb before_action :authenticate def index

    @user = current_user render :index end

45. Release to certain people # dashboard_controller.rb before_action :authenticate def index

    @user = current_user if [234, 345, 456].include?(@user.id) render :the_new_fancy_index else render :index end end

46. Release to certain people # dashboard_controller.rb before_action :authenticate def index

    @user = current_user if ['aaa@quipper.com', 'test@example.com'].include?(@user.email) render :the_new_fancy_index else render :index end end

47. Release to certain people dynamically # dashboard_controller.rb before_action :authenticate def

    index @user = current_user if UsersToShow.exists?(key: 'fancy-dashboard', email: @user.email) render :the_new_fancy_index else render :index end end Run this manually: UsersToShow.create!(key: 'fancy-dashboard', email: 'aaa@quipper.com')

48. Release to some people (percentage) def index @user = current_user

    if @user.id % 100 < 31 render :the_new_fancy_index else render :index end end

49. Release to some people (percentage) dynamically def index @user =

    current_user percentage = PercentageToShow.find(key: 'fancy-dashboard')&.value if @user.id % 100 < (percentage || 0) render :the_new_fancy_index else render :index end end Run this manually PercentageToShow.create!(key: 'fancy-dashboard', value: 31)

50. Dynamic tweaking • More powerful than it sounds like •

    No code change to release • No code change to rollback • Any time to release ◦ Not at the same time to other commits • No need to have perfect tests for unreachable code

51. Digital (Big bang) release

52. Gradual release (feature)

53. Gradual release (user)

54. None

55. Gradual release (user*feature)

56. None
57. None

58. New problems • Gradual release adds complexity ◦ Manage both

    old/new code ◦ Behaviour depends on each users ◦ More often to rollback ◦ Tend to forget to release • Hard to hold both old/new code for: ◦ Application-global changes ◦ Infrastructure-layer changes 58/93

59. P. Mange both old/new code • S. Embrace it •

    Make it pattern for ease to read def index @user = current_user percentage = PercentageToShow.find(key: 'fancy-dashboard')&.value if @user.id.hash % 100 < (percentage || 0) render :the_new_fancy_index else render :index end end

60. P. Mange both old/new code • S. Embrace it •

    Make it pattern for ease to read def index @user = current_user if FeatureToggles.variation('enable-fancy-dashboard', @user) render :the_new_fancy_index else render :index end end

61. P. Mange both old/new code def index @user = current_user

    if FeatureToggles.variation('enable-fancy-dashboard', @user) render :the_new_fancy_index else render :index end end • "Feature toggles" There's a name for it ◦ https://martinfowler.com/articles/feature-toggles .html ◦ "modify system without changing code"

62. Darklaunch Feature Toggles

63. P. Mange both old/new code def index @user = current_user

    if Darklaunch.variation('enable-fancy-dashboard', @user) render :the_new_fancy_index else render :index end end • Quipper uses the word "Darklaunch" ◦ potential confusion with "Feature flags" we already have

64. P. Mange both old/new code def index @user = current_user

    if Darklaunch.variation('enable-fancy-dashboard', @user) render :the_new_fancy_index else render :index end end • Quipper's Darklaunch stores into MongoDB ◦ Most models are MongoMapper ◦ (Darklaunch was on Redis at first)
65. None

66. def index @user = current_user if Darklaunch.variation('enable-fancy-dashboard', {id: @user.id, …})

    render :the_new_fancy_index else render :index end end • returns true only for new experimental features ◦ true = new (dangerous), false = old (safe)

67. def index @user = current_user if Darklaunch.variation('enable-fancy-dashboard', {id: @user.id, …})

    render :the_new_fancy_index else render :index end end • Pass user as a PORO ◦ Let darklaunch not to know about business logic

68. def index @user = current_user if Darklaunch.v1_variation('enable-fancy-dashboard', {id: @user.id}) render

    :the_new_fancy_index else render :index end end • Darklaunch can't darklaunch itself ◦ Never change the interface

69. P. More often to rollback • Hypothesis: more often to

    release = more often to rollback • What's rollback ◦ 0. an issue happens ◦ 1. find where it happens ◦ 2. change app behave as previous ◦ 3. release the change ◦ (4. investigate why and solve it)

70. • S. make it easy to darklaunch/undarklaunch P. More often

    to rollback

71. P. More often to rollback • Back then ◦ revert

    commit ◦ deploy (= release) def index @user = current_user render :the_new_fancy_index # boom! RuntimeError! end ↓ rollback def index @user = current_user render :index

72. P. More often to rollback def index @user = current_user

    if Darklaunch.v1_variation('enable-fancy-dashboard', {id: @user.id}) render :the_new_fancy_index # boom! RuntimeError! else render :index end end • Typical task: ◦ (User reports a bug) ◦ An exception at the new code ◦ Undarklaunch the key

73. P. More often to rollback def index @user = current_user

    if Darklaunch.v1_variation('enable-fancy-dashboard', {id: @user.id}) begin render :the_new_fancy_index # boom! RuntimeError! rescue => e ... Darklaunch.remove('enable-fancy-dashboard') • Typical task: ◦ (User reports a bug) ◦ An exception at the new code ◦ Undarklaunch the key

74. • feature togglesの説明 • quipperのdarklaunchの説明 ◦ 実例スクショ ◦ slack会話スクショ •

    back-officeのUIの説明 • 外部サービスの説明 (詳しくないから飛ばすかも)

75. Circuit Breaker A pattern to prevent catastrophic cascade https://martinfowler.com/bliki/CircuitBreaker.html

76. Circuit Breaker (Martin Fowler) • Very common • Stateful gateway

    to external resources • Toggle between "Open", "Half Open", and "Closed" • Requests timeout immediately in "Closed"

77. Can be used also to toggle behaviour That's what I

    showed with Darklaunch

78. Special thanks to Hootsuite • https://hootsuite.com • PHP Monolith +

    Scala Microservices • Darklaunch ◦ Consul • Circuit breaker ◦ https://github.com/hootsuite/scala-circuit-breake r

79. Re: New problems • Gradual release adds complexity ◦ Manage

    both old/new code ◦ Behaviour depends on each users ◦ More often to rollback ◦ Tend to forget to release • Hard to hold both old/new code for: ◦ Application-global changes ◦ Infrastructure-layer changes

80. Re: New problems • Gradual release adds complexity ◦ Manage

    both old/new code ◦ Behaviour depends on each users ◦ More often to rollback ◦ Tend to forget to release • Hard to hold both old/new code for: ◦ Application-global changes ◦ Infrastructure-layer changes Solved! Darklaunch module Darklaunch dashboard (*) Circuit breaker Darklaunch stats (*) Canary release Canary release

81. 雑 "Almost microservices" Tatsuhiro Ujihisa RailsDM 2019 2019-03-22 80/93

82. Microservices An application with collection of "services" 81/93

83. Microservices Pros • Team develops/maintains a subset of an app

    ◦ ownership! • Easy to make a big change ◦ totally independent releases ▪ frequent small releases = trivial to debug production issues • no need to know details of other services

84. Microservices Cons • Distributed ◦ "Networks are unreliable in the

    worst possible way" ― 7 deadly sin • Requires well-defined service boundaries ◦ is it possible? • Hard to change interface • Well, need to know details of both services

85. Distributed Monolith and Microservices • Monolith ◦ Not scalable with

    many devs ◦ Different concerns in same code • Distributed Monolith ◦ Monolith + Network partitioning • Microservices ◦ Easier to deploy, and make runtime-wide changes ◦ Hard to make microservices-wide changes

86. Distributed Monolith and Microservices • Monolith ◦ Not scalable with

    many devs ◦ Different concerns in same code • Distributed Monolith ◦ Monolith + Partitioning • Microservices ◦ Easier to deploy, and make runtime-wide changes ◦ Hard to make microservices-wide changes

87. Monolith + Feature toggles • Different concerns in same code

    ◦ Modules/classes/methods • Easier to deploy • Manageable to make app-global changes • No network partitioning

88. Monolith + Feature toggles • Different concerns in same code

    ◦ Modules/classes/methods • Easier to deploy • Manageable to make app-global changes • No network partitioning 雑 "Almost microservices" Tatsuhiro Ujihisa RailsDM 2019 2019-03-22

89. end

90. ujihisa before Quipper • 2010 ~ 2016 ◦ https://hootsuite.com ◦

    PHP Monolith ◦ Scala Microservices ◦ Go Microservices • 2017~2018 ◦ https://fril.jp ◦ Rails Distributed Monolith "分断されたモノリス" 89/93

91. ujihisa this year • 2019-03 ◦ RailsDM 2019 "雑" Almost

    Microservices • 2019-04 ◦ RubyKaigi 2019 "Play with local vars" • 2019-05+ ◦ Move back to Vancouver, BC, Canada 90/93

92. https://www.reddit.com/r/MapPorn/comments/9q4wyj/civilization_vi_earth_map/ 90/92

93. https://www.reddit.com/r/MapPorn/comments/9q4wyj/civilization_vi_earth_map/ 91/92

94. "雑" Almost microservices / @ujihisa • Release is important ◦

    Grind it to small pieces for scalability • Human makes mistakes • Feature toggles pattern is powerful ◦ Quipper calls it Darklaunch ◦ Provide strong support to Darklaunch • Apply (somewhat) Circuit Breaker pattern and Canary release if necessary • See you at the RubyKaigi 2019! 93/93