Grind it to small pieces for scalability • Human makes mistakes • Feature toggles pattern is powerful ◦ Quipper calls it Darklaunch ◦ Provide strong support to Darklaunch • Apply Circuit Breaker pattern and Canary release if necessary • Pragmatic Microservices' essence
restart ◦ Not atomic • Code deploy size = n commits ◦ n depends on ▪ num devs ▪ deploy frequency • Quipper: ◦ 636 commits, 155 PRs, 42 devs, once a week
most users ◦ Breaks for certain users ◦ More tests! ? • Unexpected untested changes ◦ Somewhere far can get changed ◦ More tests! ? • Runtime env specific issues
methods ◦ Sometimes even for private methods only if it makes sense ◦ If it's too trivial I sometimes skip • All controller actions ◦ Without very edgy cases • Not really for others Caution: This page can be controversial
to detect which commit broke? • Long life branches ◦ Merge conflicts ◦ Hard to co-work • Nontrivial to rollback specific one ◦ Hotfix to master branch, merge back to develop branch 27/92
Split features smaller, and release each (0% -> 20% -> 30% -> ... -> 90% -> 100%) • Release to certain people first, and expose to more people gradually (0% -> 1% -> ... -> 99% -> 100%) ◦ Can be randomized ◦ Can be manually specified
Split features smaller, and release each (0% -> 20% -> 30% -> ... -> 90% -> 100%) • Release to certain people first, and expose to more people gradually (0% -> 1% -> ... -> 99% -> 100%) ◦ Can be randomized ◦ Can be manually specified MVP Minimum Viable Product
Split features smaller, and release each of them (0% -> 20% -> 30% -> ... -> 90% -> 100%) • Release to certain people first, and expose to more people gradually (0% -> 1% -> ... -> 99% -> 100%) ◦ Can be randomized ◦ Can be manually specified MVP Minimum Viable Product https://blog.deming.org/2014/11/minimal-viable-product/
Split features smaller, and release each of them (0% -> 20% -> 30% -> ... -> 90% -> 100%) • Release to certain people first, and expose to more people gradually (0% -> 1% -> ... -> 99% -> 100%) ◦ Can be randomized ◦ Can be manually specified 43/93
index @user = current_user if UsersToShow.exists?(key: 'fancy-dashboard', email: @user.email) render :the_new_fancy_index else render :index end end Run this manually: UsersToShow.create!(key: 'fancy-dashboard', email: 'aaa@quipper.com')
No code change to release • No code change to rollback • Any time to release ◦ Not at the same time to other commits • No need to have perfect tests for unreachable code
old/new code ◦ Behaviour depends on each users ◦ More often to rollback ◦ Tend to forget to release • Hard to hold both old/new code for: ◦ Application-global changes ◦ Infrastructure-layer changes 58/93
Make it pattern for ease to read def index @user = current_user percentage = PercentageToShow.find(key: 'fancy-dashboard')&.value if @user.id.hash % 100 < (percentage || 0) render :the_new_fancy_index else render :index end end
Make it pattern for ease to read def index @user = current_user if FeatureToggles.variation('enable-fancy-dashboard', @user) render :the_new_fancy_index else render :index end end
if FeatureToggles.variation('enable-fancy-dashboard', @user) render :the_new_fancy_index else render :index end end • "Feature toggles" There's a name for it ◦ https://martinfowler.com/articles/feature-toggles .html ◦ "modify system without changing code"
if Darklaunch.variation('enable-fancy-dashboard', @user) render :the_new_fancy_index else render :index end end • Quipper uses the word "Darklaunch" ◦ potential confusion with "Feature flags" we already have
if Darklaunch.variation('enable-fancy-dashboard', @user) render :the_new_fancy_index else render :index end end • Quipper's Darklaunch stores into MongoDB ◦ Most models are MongoMapper ◦ (Darklaunch was on Redis at first)
release = more often to rollback • What's rollback ◦ 0. an issue happens ◦ 1. find where it happens ◦ 2. change app behave as previous ◦ 3. release the change ◦ (4. investigate why and solve it)
if Darklaunch.v1_variation('enable-fancy-dashboard', {id: @user.id}) render :the_new_fancy_index # boom! RuntimeError! else render :index end end • Typical task: ◦ (User reports a bug) ◦ An exception at the new code ◦ Undarklaunch the key
if Darklaunch.v1_variation('enable-fancy-dashboard', {id: @user.id}) begin render :the_new_fancy_index # boom! RuntimeError! rescue => e ... Darklaunch.remove('enable-fancy-dashboard') • Typical task: ◦ (User reports a bug) ◦ An exception at the new code ◦ Undarklaunch the key
both old/new code ◦ Behaviour depends on each users ◦ More often to rollback ◦ Tend to forget to release • Hard to hold both old/new code for: ◦ Application-global changes ◦ Infrastructure-layer changes
both old/new code ◦ Behaviour depends on each users ◦ More often to rollback ◦ Tend to forget to release • Hard to hold both old/new code for: ◦ Application-global changes ◦ Infrastructure-layer changes Solved! Darklaunch module Darklaunch dashboard (*) Circuit breaker Darklaunch stats (*) Canary release Canary release
◦ ownership! • Easy to make a big change ◦ totally independent releases ▪ frequent small releases = trivial to debug production issues • no need to know details of other services
worst possible way" ― 7 deadly sin • Requires well-defined service boundaries ◦ is it possible? • Hard to change interface • Well, need to know details of both services
many devs ◦ Different concerns in same code • Distributed Monolith ◦ Monolith + Network partitioning • Microservices ◦ Easier to deploy, and make runtime-wide changes ◦ Hard to make microservices-wide changes
many devs ◦ Different concerns in same code • Distributed Monolith ◦ Monolith + Partitioning • Microservices ◦ Easier to deploy, and make runtime-wide changes ◦ Hard to make microservices-wide changes
Grind it to small pieces for scalability • Human makes mistakes • Feature toggles pattern is powerful ◦ Quipper calls it Darklaunch ◦ Provide strong support to Darklaunch • Apply (somewhat) Circuit Breaker pattern and Canary release if necessary • See you at the RubyKaigi 2019! 93/93
ujihisa
smokeymonkey
tsukubachallenge
yoshiakiyamasaki
cygames
wkm2
kawaguti
mu2in
toricls
cyber_unfold
siroitori0413
cchanabo
yokawasa
tanoku
jmmastey
addyosmani
speakerdeck
roundedbygravity
reverentgeek
rasmusluckow
samanthasiow
cromwellryan
holman
lynnandtonic
jcasabona
