Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Easing into continuous deployment
Search
Chris Keathley
July 28, 2017
Programming
2
340
Easing into continuous deployment
How we moved our team from static deployments into continuous deployment.
Chris Keathley
July 28, 2017
Tweet
Share
More Decks by Chris Keathley
See All by Chris Keathley
Solid code isn't flexible
keathley
4
1k
Building Adaptive Systems
keathley
40
2.4k
Contracts for building reliable systems
keathley
5
820
Kafka, the hard parts
keathley
2
1.6k
Building Resilient Elixir Systems
keathley
6
2.1k
Consistent, Distributed Elixir
keathley
5
1.5k
Telling stories with data visualization
keathley
0
580
Leveling up your git skills
keathley
0
710
Generative Testing in Elixir
keathley
0
480
Other Decks in Programming
See All in Programming
GoとPHPのインターフェイスの違い
shimabox
2
190
ファインディの テックブログ爆誕までの軌跡
starfish719
2
1.1k
法律の脱レガシーに学ぶフロントエンド刷新
oguemon
5
740
ペアーズでの、Langfuseを中心とした評価ドリブンなリリースサイクルのご紹介
fukubaka0825
2
320
2024年のkintone API振り返りと2025年 / kintone API look back in 2024
tasshi
0
220
GitHub Actions × RAGでコードレビューの検証の結果
sho_000
0
260
CDK開発におけるコーディング規約の運用
yamanashi_ren01
2
110
データの整合性を保つ非同期処理アーキテクチャパターン / Async Architecture Patterns
mokuo
47
17k
XStateを用いた堅牢なReact Components設計~複雑なClient Stateをシンプルに~ @React Tokyo ミートアップ #2
kfurusho
1
900
SwiftUIで単方向アーキテクチャを導入して得られた成果
takuyaosawa
0
270
Lottieアニメーションをカスタマイズしてみた
tahia910
0
130
Domain-Driven Transformation
hschwentner
2
1.9k
Featured
See All Featured
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
7
630
Rebuilding a faster, lazier Slack
samanthasiow
80
8.8k
Fontdeck: Realign not Redesign
paulrobertlloyd
83
5.4k
Music & Morning Musume
bryan
46
6.3k
Visualization
eitanlees
146
15k
A Tale of Four Properties
chriscoyier
158
23k
Making the Leap to Tech Lead
cromwellryan
133
9.1k
Side Projects
sachag
452
42k
What's in a price? How to price your products and services
michaelherold
244
12k
Reflections from 52 weeks, 52 projects
jeffersonlam
348
20k
StorybookのUI Testing Handbookを読んだ
zakiyama
28
5.5k
Fireside Chat
paigeccino
34
3.2k
Transcript
Continuous Deployment Chris Keathley / @ChrisKeathley /
[email protected]
I work with a distributed team
I work with a distributed team
None
None
None
None
None
Warehouse
Warehouse API
Warehouse API Apps
The problem
Slow Iteration Cycle Deployment Deployment 2 weeks
Slow Iteration Cycle Deployment Deployment Deployment 2 weeks 2 weeks
Slow Iteration Cycle Deployment Deployment Deployment 3 weeks
Slow Iteration Cycle Deployment Deployment Deployment Hopefully someday
Large PRs
Unsure about state of the application
Unsure about state of the application
Unsure about state of the application
Unsure about state of the application
Unsure about state of the application
Rollbacks are a scam
None
None
Data Migration
Data Migration ?
Don’t do this
always Move forward
always Move forward
always Move forward
The goal should never be to roll back a deployment
The goal is to minimize the damage done by any
given deployment
There are bugs in your system
Solutions
We needed to deploy more often
So we did
Automated Deployment
What do you deploy?
Commit Sha
None
Jars
Artifacts
Git Tags
Containers
Your App
Your App Server
None
Master Branch How we merge our code PR
CI Github Registry Container Slack PR Notification
CI Kubernetes Deploy Auto-deploy Green builds of master
CI Kubernetes Deploy Auto-deploy Green builds of master Service A
Service B
CI Kubernetes Deploy Auto-deploy Green builds of master Service B
CI Kubernetes Deploy Auto-deploy Green builds of master Service B
Service A
CI Kubernetes Deploy Auto-deploy Green builds of master Service A
CI Kubernetes Deploy Auto-deploy Green builds of master Service A
Service B
Tests Metrics &
Integration Tests + Property Tests
Integration Tests TEst App DB Service
Modeling Users as FSMs logged_out logged_in login logout vote
Property Tests Add Todo Edit Todo Delete Todo
Property Tests Add Todo Edit Todo Delete Todo
Property Tests Add Todo Edit Todo Delete Todo
Property Tests Add Todo Edit Todo Delete Todo
Generate Commands
Generated Commands [{:add_todo, “Test Todo”, 1}, {:edit_todo, "Edited", 2}, {:delete_todo,
"", 1}, {:add_todo, “New Todo", 3}, {:delete_todo, "", 2} {:edit_todo, “Edited Todo”, 2}]
Generate Commands
Generate Commands
Generate Commands
Generate Commands
Generate Commands
Generated Commands [{:add_todo, “Test Todo”, 1}, {:edit_todo, "Edited", 2}, {:delete_todo,
"", 1}, {:add_todo, “New Todo", 3}, {:delete_todo, "", 2} {:edit_todo, “Edited Todo”, 2}]
Generated Commands [{:add_todo, “Test Todo”, 1}, {:delete_todo, "", 2}] [{:add_todo,
“Test Todo”, 1}, {:edit_todo, "Edited", 2}, {:delete_todo, "", 1}, {:add_todo, “New Todo", 3}, {:delete_todo, "", 2} {:edit_todo, “Edited Todo”, 2}]
Prometheus Service A Grafana Service B Service C
Prometheus Service A Grafana Service B Service C Slack
# Alert for any instance that have a 95th percentile
> 200ms. ALERT APIHighRequestLatency IF api_http_request_latencies_second{quantile="0.95"} > 0.2 FOR 5m ANNOTATIONS { summary = "High request latency on {{ $labels.instance }}", description = "{{ $labels.instance }} has a median request latency above 1s (current value: {{ $value }}s)", }
Track “Business” Metrics
None
Feature releases and flags
None
Features aren’t all or nothing
Features != Deployments
Deployment
Deployment Features
Deployment Features
User
User staff?(user) == true
User staff?(user) == false
User staff?(user) == false
defmodule MyApp.FeatureFlags do alias MyApp.User def foo_enabled?(%User{staff: is_staff}), do: is_staff
def foo_enabled?(_), do: false def bar_enabled?(%User{staff: is_staff}), do: is_staff def bar_enabled?(_), do: false end
Browser Feature Service
Feature Service Feature Service Feature Service
Feature Service Feature Service Feature Service
Feature Service Feature Service Feature Service
You have updates ready! Reset
None
With larger Traffic numbers you could use percentages
Alchemy
“Transmute lead code into gold in production”
Prior Art: https://github.com/github/scientist
Users_Controller DB User.all
DB User.all UserService.all
User.all UserService.all ==
def index(conn) do users = old_query() render(conn, "index.json", users: users)
end
def index(conn) do users = experiment("users-query") |> control(&old_query/0) |> candidate(&new_query/0)
|> run render(conn, "index.json", users: users) end
def index(conn) do users = experiment("users-query") |> control(&old_query/0) |> candidate(&new_query/0)
|> candidate(&fancy_query/0) |> run render(conn, "index.json", users: users) end
1) Shuffles test order 2) Runs Each test in parallel
3) exports the data Alchemy
DB User.all UserService.all Control Candidate Control UserController
None
1) Do the results match? 2) How long does each
test take to return? Measure
No more cutovers
DB User.all UserService.all
DB User.all UserService.all User service
Migrations
http://blog.datomic.com/2017/01/the-ten-rules-of-schema-growth.html
DB Schema App Application Coupling
Your application knows about your schema
Lets remove a column
Lets remove a column 1) all application code needs to
stop using that column
Lets remove a column 1) all application code needs to
stop using that column 2) Update all ETL processes
Lets remove a column 1) all application code needs to
stop using that column 2) Update all ETL processes 3) Update Reporting
Lets remove a column 1) all application code needs to
stop using that column 2) Update all ETL processes 3) Update Reporting 4) Remove the column
Lets remove a column 1) all application code needs to
stop using that column 2) Update all ETL processes 3) Update Reporting 4) Remove the column Split all of these up
Lets Add a column
Lets Add a column 1) Add the column
Lets Add a column 1) Add the column 2) Eventually
start using it
Prefer Additive Migrations
CI Kubernetes Deploy Auto-deploy Green builds of master
CI Kubernetes Deploy Auto-deploy Green builds of master Migration
CI Kubernetes Deploy Auto-deploy Green builds of master Migration DB
Chat-Ops
None
Chat is…
Chat is… Centralized
Chat is… Centralized Transparent
Chat is… Centralized Transparent Open
Try to do operational tasks in chat
None
defmodule Hedwig.Responders.Ping do use Hedwig.Responder @usage """ hedwig: ping -
Responds with 'pong' """ respond ~r/ping$/i, msg do reply msg, "pong" end end
None
Generate grafana graphs
None
Deploy
None
Team Building
Conclusion
These are tools at our disposal
Deploy more often, safely
Thanks Chris Keathley / @ChrisKeathley /
[email protected]