Slide 1
Slide 1 text
雑
"Almost microservices"
Tatsuhiro Ujihisa
RailsDM 2019
2019-03-22
1/92
Slide 2
Slide 2 text
https://en.wiktionary.or
g/wiki/%E9%9B%91
Adjectival noun
雑 (shinjitai kanji,
kyūjitai kanji 雜, -na
inflection, hiragana ざ
つ, rōmaji zatsu)
● rough, crude,
sloppy, messy
● miscellaneous
(see note)
2/92
Slide 3
Slide 3 text
ujihisa
● https://github.com/ujihisa
● https://twitter.com/ujm
● VimConf 2018 Speaker & Organizer,
RubyKaigi 2019 Speaker
● Quipper Limited
○ Co-workers' talks: @banyan, @mtsmfm
○ "Distributed Monolith" @kyanny from RailsDM
2018
3/92
Slide 4
Slide 4 text
ujihisa before Quipper
● 2010 ~ 2016
○ https://hootsuite.com
○ PHP Monolith
○ Scala Microservices
○ Go Microservices
● 2017~2018
○ https://fril.jp
○ Rails Distributed Monolith "分断されたモノリス"
Slide 5
Slide 5 text
ujihisa
● Cooking
● Learned a lot about fish
Slide 6
Slide 6 text
vim-jp
Slide 7
Slide 7 text
Quipper/Quipper, Quipper/Study Sapuri
● https://learn.quipper.com
https://learn.studysapuri.jp
○ Rails Distributed Monolith
Slide 8
Slide 8 text
No content
Slide 9
Slide 9 text
Quipper
● Joined in 2018-09
○ 7 months so far
■ 4 months at School Team (*)
■ 1 month at Unemployed (*)
■ 2 months at Data Restructuring Team
Slide 10
Slide 10 text
Almost microservices
ujihisa
9/92
Slide 11
Slide 11 text
"雑" Almost microservices / @ujihisa
● Release is important
○ Grind it to small pieces for scalability
● Human makes mistakes
● Feature toggles pattern is powerful
○ Quipper calls it Darklaunch
○ Provide strong support to Darklaunch
● Apply Circuit Breaker pattern and
Canary release if necessary
● Pragmatic Microservices' essence
Slide 12
Slide 12 text
Web app development
● Research & decide what to make
● rails new
● infrastructure setup
● git, code review, CI setup
● design models, views, and controllers
● development! of course unit tests too!
● refactor
● release & debug
Slide 13
Slide 13 text
Web app development
● Research & decide what to make
● rails new
● infrastructure setup
● git, code review, CI setup
● design models, views, and controllers
● development! of course unit tests too!
● refactor
● & debug
Release
Slide 14
Slide 14 text
Software
release
Release gives development meaning
Slide 15
Slide 15 text
What is software release?
俺 サーバ
Slide 16
Slide 16 text
What is software release?
俺 サーバ おまえら
Slide 17
Slide 17 text
Release
● Add a new feature and make it available to users
● Modify existing features
● Delete existing features
● Add/modify/delete internal implementation
(or Refactor)
Slide 18
Slide 18 text
"Release" typical properties
● Release = Code deploy && daemon restart
○ Not atomic
● Code deploy size = n commits
○ n depends on
■ num devs
■ deploy frequency
● Quipper:
○ 636 commits, 155 PRs, 42 devs, once a week
Slide 19
Slide 19 text
Release difficulties
● Edge case logical bugs
○ Works for most users
○ Breaks for certain users
○
● Unexpected untested changes
○ Somewhere far can get changed
○
● Runtime env specific issues
Slide 20
Slide 20 text
Release difficulties
● Edge case logical bugs
○ Works for most users
○ Breaks for certain users
○ More tests! ?
● Unexpected untested changes
○ Somewhere far can get changed
○ More tests! ?
● Runtime env specific issues
Slide 21
Slide 21 text
Testing is hard
I even make a mistake
exactly both at the implementation and the test code
Slide 22
Slide 22 text
from https://speakerdeck.com/sinamon129/sisutemuzhang-hai-tofalsexiang-kihe-ifang-at-sinamon129-number-tokyogirlsrb
TokyoGirls.rb Meetup
Slide 23
Slide 23 text
Tests cost
● Cost of a test case
○ Write
○ Run
○ Maintain
● Benefit of a test case
○ Can it prevent new bugs?
○ Can it explain the behaviour to us well?
● Good test = low cost * high benefit
Slide 24
Slide 24 text
Do I write tests?
● Yes!
● All model public methods
○ Sometimes even for private methods only if it
makes sense
○ If it's too trivial I sometimes skip
● All controller actions
○ Without very edgy cases
● Not really for others
Caution: This page can be
controversial
Slide 25
Slide 25 text
Safe Release
安心してリリースしたい
Slide 26
Slide 26 text
Solutions?
● Integration test
○ Headless browser
○ Browser
● Static code analysis
● Dynamic analysis
● Performance benchmarking/simulation
● Test manually
● Pay somebody to test manually
Slide 27
Slide 27 text
Solutions?
● Integration test
○ Headless browser
○ Browser
● Static code analysis
● Dynamic analysis
● Performance benchmarking/simulation
● Test manually
● Pay somebody to test manually
→ Weekly release
Slide 28
Slide 28 text
Quipper's solution (1): make manual test trivial
● Pull request code review
● CI deploys your branch into a dedicated cluster
○ Reviewer can actually try the web app
○ Masked production data
リ
Slide 29
Slide 29 text
Quipper's solution (2): Outsource human tests
● Test all the features every week
○ Product Managers make happy path scenario,
and pass it to QA company
Slide 30
Slide 30 text
Remaining problems
● Weekly release with giant changes
○ How to detect which commit broke?
● Long life branches
○ Merge conflicts
○ Hard to co-work
● Nontrivial to rollback specific one
○ Hotfix to master branch, merge back to develop
branch
27/92
Slide 31
Slide 31 text
Release
anytime
but how?
28/92
Slide 32
Slide 32 text
Release gradually
Binary release: 0 → 1
Analogue release: 0 → 0.1 → 0.2 → ... → 0.9 → 1
Slide 33
Slide 33 text
Release gradually
● Big bang release (0% -> 100%)
● Split features smaller, and release each
(0% -> 20% -> 30% -> ... -> 90% -> 100%)
● Release to certain people first, and expose to
more people gradually
(0% -> 1% -> ... -> 99% -> 100%)
○ Can be randomized
○ Can be manually specified
Slide 34
Slide 34 text
Release gradually
● Big bang release (0% -> 100%)
● Split features smaller, and release each
(0% -> 20% -> 30% -> ... -> 90% -> 100%)
● Release to certain people first, and expose to more
people gradually
(0% -> 1% -> ... -> 99% -> 100%)
○ Can be randomized
○ Can be manually specified
MVP
Minimum Viable Product
Slide 35
Slide 35 text
Release gradually
● Big bang release (0% -> 100%)
● Split features smaller, and release each of them
(0% -> 20% -> 30% -> ... -> 90% -> 100%)
● Release to certain people first, and expose to more
people gradually
(0% -> 1% -> ... -> 99% -> 100%)
○ Can be randomized
○ Can be manually specified
MVP
Minimum Viable Product
https://blog.deming.org/2014/11/minimal-viable-product/
Slide 36
Slide 36 text
No content
Slide 37
Slide 37 text
No content
Slide 38
Slide 38 text
No content
Slide 39
Slide 39 text
No content
Slide 40
Slide 40 text
No content
Slide 41
Slide 41 text
in production
Iterate and learn
41/93
Slide 42
Slide 42 text
Iterate and
learn
in production
https://www.youtube.com/w
atch?v=75wa8Lx4yc4
Slide 43
Slide 43 text
Release gradually
● Big bang release (0% -> 100%)
● Split features smaller, and release each of them
(0% -> 20% -> 30% -> ... -> 90% -> 100%)
● Release to certain people first, and expose to more people
gradually
(0% -> 1% -> ... -> 99% -> 100%)
○ Can be randomized
○ Can be manually specified
43/93
Slide 44
Slide 44 text
Release to certain people
# dashboard_controller.rb
before_action :authenticate
def index
@user = current_user
render :index
end
Slide 45
Slide 45 text
Release to certain people
# dashboard_controller.rb
before_action :authenticate
def index
@user = current_user
if [234, 345, 456].include?(@user.id)
render :the_new_fancy_index
else
render :index
end
end
Slide 46
Slide 46 text
Release to certain people
# dashboard_controller.rb
before_action :authenticate
def index
@user = current_user
if ['[email protected]', '[email protected]'].include?(@user.email)
render :the_new_fancy_index
else
render :index
end
end
Slide 47
Slide 47 text
Release to certain people dynamically
# dashboard_controller.rb
before_action :authenticate
def index
@user = current_user
if UsersToShow.exists?(key: 'fancy-dashboard', email: @user.email)
render :the_new_fancy_index
else
render :index
end
end
Run this manually:
UsersToShow.create!(key: 'fancy-dashboard', email: '[email protected]')
Slide 48
Slide 48 text
Release to some people (percentage)
def index
@user = current_user
if @user.id % 100 < 31
render :the_new_fancy_index
else
render :index
end
end
Slide 49
Slide 49 text
Release to some people (percentage) dynamically
def index
@user = current_user
percentage = PercentageToShow.find(key: 'fancy-dashboard')&.value
if @user.id % 100 < (percentage || 0)
render :the_new_fancy_index
else
render :index
end
end
Run this manually
PercentageToShow.create!(key: 'fancy-dashboard', value: 31)
Slide 50
Slide 50 text
Dynamic tweaking
● More powerful than it sounds like
● No code change to release
● No code change to rollback
● Any time to release
○ Not at the same time to other commits
● No need to have perfect tests for
unreachable code
Slide 51
Slide 51 text
Digital (Big bang) release
Slide 52
Slide 52 text
Gradual release (feature)
Slide 53
Slide 53 text
Gradual release (user)
Slide 54
Slide 54 text
No content
Slide 55
Slide 55 text
Gradual release (user*feature)
Slide 56
Slide 56 text
No content
Slide 57
Slide 57 text
No content
Slide 58
Slide 58 text
New problems
● Gradual release adds complexity
○ Manage both old/new code
○ Behaviour depends on each users
○ More often to rollback
○ Tend to forget to release
● Hard to hold both old/new code for:
○ Application-global changes
○ Infrastructure-layer changes
58/93
Slide 59
Slide 59 text
P. Mange both old/new code
● S. Embrace it
● Make it pattern for ease to read
def index
@user = current_user
percentage = PercentageToShow.find(key: 'fancy-dashboard')&.value
if @user.id.hash % 100 < (percentage || 0)
render :the_new_fancy_index
else
render :index
end
end
Slide 60
Slide 60 text
P. Mange both old/new code
● S. Embrace it
● Make it pattern for ease to read
def index
@user = current_user
if FeatureToggles.variation('enable-fancy-dashboard', @user)
render :the_new_fancy_index
else
render :index
end
end
Slide 61
Slide 61 text
P. Mange both old/new code
def index
@user = current_user
if FeatureToggles.variation('enable-fancy-dashboard', @user)
render :the_new_fancy_index
else
render :index
end
end
● "Feature toggles" There's a name for it
○ https://martinfowler.com/articles/feature-toggles
.html
○ "modify system without changing code"
Slide 62
Slide 62 text
Darklaunch
Feature Toggles
Slide 63
Slide 63 text
P. Mange both old/new code
def index
@user = current_user
if Darklaunch.variation('enable-fancy-dashboard', @user)
render :the_new_fancy_index
else
render :index
end
end
● Quipper uses the word "Darklaunch"
○ potential confusion with "Feature flags"
we already have
Slide 64
Slide 64 text
P. Mange both old/new code
def index
@user = current_user
if Darklaunch.variation('enable-fancy-dashboard', @user)
render :the_new_fancy_index
else
render :index
end
end
● Quipper's Darklaunch stores into MongoDB
○ Most models are MongoMapper
○ (Darklaunch was on Redis at first)
Slide 65
Slide 65 text
No content
Slide 66
Slide 66 text
def index
@user = current_user
if Darklaunch.variation('enable-fancy-dashboard', {id: @user.id, …})
render :the_new_fancy_index
else
render :index
end
end
● returns true only for new experimental features
○ true = new (dangerous), false = old (safe)
Slide 67
Slide 67 text
def index
@user = current_user
if Darklaunch.variation('enable-fancy-dashboard', {id: @user.id, …})
render :the_new_fancy_index
else
render :index
end
end
● Pass user as a PORO
○ Let darklaunch not to know about business
logic
Slide 68
Slide 68 text
def index
@user = current_user
if Darklaunch.v1_variation('enable-fancy-dashboard', {id: @user.id})
render :the_new_fancy_index
else
render :index
end
end
● Darklaunch can't darklaunch itself
○ Never change the interface
Slide 69
Slide 69 text
P. More often to rollback
● Hypothesis:
more often to release = more often to rollback
● What's rollback
○ 0. an issue happens
○ 1. find where it happens
○ 2. change app behave as previous
○ 3. release the change
○ (4. investigate why and solve it)
Slide 70
Slide 70 text
● S. make it easy to darklaunch/undarklaunch
P. More often to rollback
Slide 71
Slide 71 text
P. More often to rollback
● Back then
○ revert commit
○ deploy (= release)
def index
@user = current_user
render :the_new_fancy_index # boom! RuntimeError!
end
↓ rollback
def index
@user = current_user
render :index
Slide 72
Slide 72 text
P. More often to rollback
def index
@user = current_user
if Darklaunch.v1_variation('enable-fancy-dashboard', {id: @user.id})
render :the_new_fancy_index # boom! RuntimeError!
else
render :index
end
end
● Typical task:
○ (User reports a bug)
○ An exception at the new code
○ Undarklaunch the key
Slide 73
Slide 73 text
P. More often to rollback
def index
@user = current_user
if Darklaunch.v1_variation('enable-fancy-dashboard', {id: @user.id})
begin
render :the_new_fancy_index # boom! RuntimeError!
rescue => e
...
Darklaunch.remove('enable-fancy-dashboard')
● Typical task:
○ (User reports a bug)
○ An exception at the new code
○ Undarklaunch the key
Slide 74
Slide 74 text
● feature togglesの説明
● quipperのdarklaunchの説明
○ 実例スクショ
○ slack会話スクショ
● back-officeのUIの説明
● 外部サービスの説明 (詳しくないから飛ばすかも)
Slide 75
Slide 75 text
Circuit Breaker
A pattern to prevent catastrophic cascade
https://martinfowler.com/bliki/CircuitBreaker.html
Slide 76
Slide 76 text
Circuit Breaker (Martin Fowler)
● Very common
● Stateful gateway to
external resources
● Toggle between
"Open", "Half Open",
and "Closed"
● Requests timeout
immediately in "Closed"
Slide 77
Slide 77 text
Can be used also to
toggle behaviour
That's what I showed with Darklaunch
Slide 78
Slide 78 text
Special thanks to Hootsuite
● https://hootsuite.com
● PHP Monolith + Scala Microservices
● Darklaunch
○ Consul
● Circuit breaker
○ https://github.com/hootsuite/scala-circuit-breake
r
Slide 79
Slide 79 text
Re: New problems
● Gradual release adds complexity
○ Manage both old/new code
○ Behaviour depends on each users
○ More often to rollback
○ Tend to forget to release
● Hard to hold both old/new code for:
○ Application-global changes
○ Infrastructure-layer changes
Slide 80
Slide 80 text
Re: New problems
● Gradual release adds complexity
○ Manage both old/new code
○ Behaviour depends on each users
○ More often to rollback
○ Tend to forget to release
● Hard to hold both old/new code for:
○ Application-global changes
○ Infrastructure-layer changes
Solved!
Darklaunch module
Darklaunch dashboard (*)
Circuit breaker
Darklaunch stats (*)
Canary release
Canary release
Slide 81
Slide 81 text
雑
"Almost microservices"
Tatsuhiro Ujihisa
RailsDM 2019
2019-03-22
80/93
Slide 82
Slide 82 text
Microservices
An application with collection of "services"
81/93
Slide 83
Slide 83 text
Microservices
Pros
● Team develops/maintains a subset of an app
○ ownership!
● Easy to make a big change
○ totally independent releases
■ frequent small releases =
trivial to debug production issues
● no need to know details of other services
Slide 84
Slide 84 text
Microservices
Cons
● Distributed
○ "Networks are unreliable in the worst possible
way" ― 7 deadly sin
● Requires well-defined service boundaries
○ is it possible?
● Hard to change interface
● Well, need to know details of both services
Slide 85
Slide 85 text
Distributed Monolith and Microservices
● Monolith
○ Not scalable with many devs
○ Different concerns in same code
● Distributed Monolith
○ Monolith + Network partitioning
● Microservices
○ Easier to deploy, and make runtime-wide changes
○ Hard to make microservices-wide changes
Slide 86
Slide 86 text
Distributed Monolith and Microservices
● Monolith
○ Not scalable with many devs
○ Different concerns in same code
● Distributed Monolith
○ Monolith + Partitioning
● Microservices
○ Easier to deploy, and make runtime-wide
changes
○ Hard to make microservices-wide changes
Slide 87
Slide 87 text
Monolith + Feature toggles
● Different concerns in same code
○ Modules/classes/methods
● Easier to deploy
● Manageable to make app-global changes
● No network partitioning
Slide 88
Slide 88 text
Monolith + Feature toggles
● Different concerns in same code
○ Modules/classes/methods
● Easier to deploy
● Manageable to make app-global changes
● No network partitioning
雑
"Almost microservices"
Tatsuhiro Ujihisa
RailsDM 2019
2019-03-22
Slide 89
Slide 89 text
end
Slide 90
Slide 90 text
ujihisa before Quipper
● 2010 ~ 2016
○ https://hootsuite.com
○ PHP Monolith
○ Scala Microservices
○ Go Microservices
● 2017~2018
○ https://fril.jp
○ Rails Distributed Monolith "分断されたモノリス" 89/93
Slide 91
Slide 91 text
ujihisa this year
● 2019-03
○ RailsDM 2019
"雑" Almost Microservices
● 2019-04
○ RubyKaigi 2019
"Play with local vars"
● 2019-05+
○ Move back to Vancouver, BC, Canada
90/93
Slide 92
Slide 92 text
https://www.reddit.com/r/MapPorn/comments/9q4wyj/civilization_vi_earth_map/
90/92
Slide 93
Slide 93 text
https://www.reddit.com/r/MapPorn/comments/9q4wyj/civilization_vi_earth_map/
91/92
Slide 94
Slide 94 text
"雑" Almost microservices / @ujihisa
● Release is important
○ Grind it to small pieces for scalability
● Human makes mistakes
● Feature toggles pattern is powerful
○ Quipper calls it Darklaunch
○ Provide strong support to Darklaunch
● Apply
(somewhat)
Circuit Breaker pattern and Canary
release if necessary
● See you at the RubyKaigi 2019! 93/93