Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Easing into continuous deployment
Search
Chris Keathley
July 28, 2017
Programming
2
380
Easing into continuous deployment
How we moved our team from static deployments into continuous deployment.
Chris Keathley
July 28, 2017
Tweet
Share
More Decks by Chris Keathley
See All by Chris Keathley
Solid code isn't flexible
keathley
5
1k
Building Adaptive Systems
keathley
43
2.7k
Contracts for building reliable systems
keathley
6
900
Kafka, the hard parts
keathley
3
1.7k
Building Resilient Elixir Systems
keathley
7
2.2k
Consistent, Distributed Elixir
keathley
6
1.6k
Telling stories with data visualization
keathley
1
630
Leveling up your git skills
keathley
0
760
Generative Testing in Elixir
keathley
0
520
Other Decks in Programming
See All in Programming
コードの90%をAIが書く世界で何が待っているのか / What awaits us in a world where 90% of the code is written by AI
rkaga
52
33k
Railsアプリケーションと パフォーマンスチューニング ー 秒間5万リクエストの モバイルオーダーシステムを支える事例 ー Rubyセミナー 大阪
falcon8823
5
1.1k
10 Costly Database Performance Mistakes (And How To Fix Them)
andyatkinson
0
320
Deep Dive into ~/.claude/projects
hiragram
14
2.5k
Webの外へ飛び出せ NativePHPが切り拓くPHPの未来
takuyakatsusa
2
550
PostgreSQLのRow Level SecurityをPHPのORMで扱う Eloquent vs Doctrine #phpcon #track2
77web
2
530
効率的な開発手段として VRTを活用する
ishkawa
0
140
すべてのコンテキストを、 ユーザー価値に変える
applism118
3
1.2k
明示と暗黙 ー PHPとGoの インターフェイスの違いを知る
shimabox
2
510
20250628_非エンジニアがバイブコーディングしてみた
ponponmikankan
0
680
High-Level Programming Languages in AI Era -Human Thought and Mind-
hayat01sh1da
PRO
0
770
Team operations that are not burdened by SRE
kazatohiei
1
310
Featured
See All Featured
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
46
9.6k
A designer walks into a library…
pauljervisheath
207
24k
YesSQL, Process and Tooling at Scale
rocio
173
14k
Art, The Web, and Tiny UX
lynnandtonic
299
21k
Why Our Code Smells
bkeepers
PRO
336
57k
Bash Introduction
62gerente
613
210k
Side Projects
sachag
455
42k
StorybookのUI Testing Handbookを読んだ
zakiyama
30
5.9k
Intergalactic Javascript Robots from Outer Space
tanoku
271
27k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
10
950
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
138
34k
Transcript
Continuous Deployment Chris Keathley / @ChrisKeathley /
[email protected]
I work with a distributed team
I work with a distributed team
None
None
None
None
None
Warehouse
Warehouse API
Warehouse API Apps
The problem
Slow Iteration Cycle Deployment Deployment 2 weeks
Slow Iteration Cycle Deployment Deployment Deployment 2 weeks 2 weeks
Slow Iteration Cycle Deployment Deployment Deployment 3 weeks
Slow Iteration Cycle Deployment Deployment Deployment Hopefully someday
Large PRs
Unsure about state of the application
Unsure about state of the application
Unsure about state of the application
Unsure about state of the application
Unsure about state of the application
Rollbacks are a scam
None
None
Data Migration
Data Migration ?
Don’t do this
always Move forward
always Move forward
always Move forward
The goal should never be to roll back a deployment
The goal is to minimize the damage done by any
given deployment
There are bugs in your system
Solutions
We needed to deploy more often
So we did
Automated Deployment
What do you deploy?
Commit Sha
None
Jars
Artifacts
Git Tags
Containers
Your App
Your App Server
None
Master Branch How we merge our code PR
CI Github Registry Container Slack PR Notification
CI Kubernetes Deploy Auto-deploy Green builds of master
CI Kubernetes Deploy Auto-deploy Green builds of master Service A
Service B
CI Kubernetes Deploy Auto-deploy Green builds of master Service B
CI Kubernetes Deploy Auto-deploy Green builds of master Service B
Service A
CI Kubernetes Deploy Auto-deploy Green builds of master Service A
CI Kubernetes Deploy Auto-deploy Green builds of master Service A
Service B
Tests Metrics &
Integration Tests + Property Tests
Integration Tests TEst App DB Service
Modeling Users as FSMs logged_out logged_in login logout vote
Property Tests Add Todo Edit Todo Delete Todo
Property Tests Add Todo Edit Todo Delete Todo
Property Tests Add Todo Edit Todo Delete Todo
Property Tests Add Todo Edit Todo Delete Todo
Generate Commands
Generated Commands [{:add_todo, “Test Todo”, 1}, {:edit_todo, "Edited", 2}, {:delete_todo,
"", 1}, {:add_todo, “New Todo", 3}, {:delete_todo, "", 2} {:edit_todo, “Edited Todo”, 2}]
Generate Commands
Generate Commands
Generate Commands
Generate Commands
Generate Commands
Generated Commands [{:add_todo, “Test Todo”, 1}, {:edit_todo, "Edited", 2}, {:delete_todo,
"", 1}, {:add_todo, “New Todo", 3}, {:delete_todo, "", 2} {:edit_todo, “Edited Todo”, 2}]
Generated Commands [{:add_todo, “Test Todo”, 1}, {:delete_todo, "", 2}] [{:add_todo,
“Test Todo”, 1}, {:edit_todo, "Edited", 2}, {:delete_todo, "", 1}, {:add_todo, “New Todo", 3}, {:delete_todo, "", 2} {:edit_todo, “Edited Todo”, 2}]
Prometheus Service A Grafana Service B Service C
Prometheus Service A Grafana Service B Service C Slack
# Alert for any instance that have a 95th percentile
> 200ms. ALERT APIHighRequestLatency IF api_http_request_latencies_second{quantile="0.95"} > 0.2 FOR 5m ANNOTATIONS { summary = "High request latency on {{ $labels.instance }}", description = "{{ $labels.instance }} has a median request latency above 1s (current value: {{ $value }}s)", }
Track “Business” Metrics
None
Feature releases and flags
None
Features aren’t all or nothing
Features != Deployments
Deployment
Deployment Features
Deployment Features
User
User staff?(user) == true
User staff?(user) == false
User staff?(user) == false
defmodule MyApp.FeatureFlags do alias MyApp.User def foo_enabled?(%User{staff: is_staff}), do: is_staff
def foo_enabled?(_), do: false def bar_enabled?(%User{staff: is_staff}), do: is_staff def bar_enabled?(_), do: false end
Browser Feature Service
Feature Service Feature Service Feature Service
Feature Service Feature Service Feature Service
Feature Service Feature Service Feature Service
You have updates ready! Reset
None
With larger Traffic numbers you could use percentages
Alchemy
“Transmute lead code into gold in production”
Prior Art: https://github.com/github/scientist
Users_Controller DB User.all
DB User.all UserService.all
User.all UserService.all ==
def index(conn) do users = old_query() render(conn, "index.json", users: users)
end
def index(conn) do users = experiment("users-query") |> control(&old_query/0) |> candidate(&new_query/0)
|> run render(conn, "index.json", users: users) end
def index(conn) do users = experiment("users-query") |> control(&old_query/0) |> candidate(&new_query/0)
|> candidate(&fancy_query/0) |> run render(conn, "index.json", users: users) end
1) Shuffles test order 2) Runs Each test in parallel
3) exports the data Alchemy
DB User.all UserService.all Control Candidate Control UserController
None
1) Do the results match? 2) How long does each
test take to return? Measure
No more cutovers
DB User.all UserService.all
DB User.all UserService.all User service
Migrations
http://blog.datomic.com/2017/01/the-ten-rules-of-schema-growth.html
DB Schema App Application Coupling
Your application knows about your schema
Lets remove a column
Lets remove a column 1) all application code needs to
stop using that column
Lets remove a column 1) all application code needs to
stop using that column 2) Update all ETL processes
Lets remove a column 1) all application code needs to
stop using that column 2) Update all ETL processes 3) Update Reporting
Lets remove a column 1) all application code needs to
stop using that column 2) Update all ETL processes 3) Update Reporting 4) Remove the column
Lets remove a column 1) all application code needs to
stop using that column 2) Update all ETL processes 3) Update Reporting 4) Remove the column Split all of these up
Lets Add a column
Lets Add a column 1) Add the column
Lets Add a column 1) Add the column 2) Eventually
start using it
Prefer Additive Migrations
CI Kubernetes Deploy Auto-deploy Green builds of master
CI Kubernetes Deploy Auto-deploy Green builds of master Migration
CI Kubernetes Deploy Auto-deploy Green builds of master Migration DB
Chat-Ops
None
Chat is…
Chat is… Centralized
Chat is… Centralized Transparent
Chat is… Centralized Transparent Open
Try to do operational tasks in chat
None
defmodule Hedwig.Responders.Ping do use Hedwig.Responder @usage """ hedwig: ping -
Responds with 'pong' """ respond ~r/ping$/i, msg do reply msg, "pong" end end
None
Generate grafana graphs
None
Deploy
None
Team Building
Conclusion
These are tools at our disposal
Deploy more often, safely
Thanks Chris Keathley / @ChrisKeathley /
[email protected]