Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Easing into continuous deployment
Search
Chris Keathley
July 28, 2017
Programming
1
260
Easing into continuous deployment
How we moved our team from static deployments into continuous deployment.
Chris Keathley
July 28, 2017
Tweet
Share
More Decks by Chris Keathley
See All by Chris Keathley
Solid code isn't flexible
keathley
3
910
Building Adaptive Systems
keathley
33
1.9k
Contracts for building reliable systems
keathley
5
670
Kafka, the hard parts
keathley
2
1.4k
Building Resilient Elixir Systems
keathley
6
1.9k
Consistent, Distributed Elixir
keathley
5
1.4k
Telling stories with data visualization
keathley
0
490
Leveling up your git skills
keathley
0
610
Generative Testing in Elixir
keathley
0
400
Other Decks in Programming
See All in Programming
アクセシビリティの観点からみたFluent UI Blazorのすすめ
tomokusaba
1
210
Datadogのグラフにデプロイタイミングを表示する / deploy timing on datadog graph
sheepland
1
220
Optiver参戦記&銀メダル解法
tonic
0
250
1BRC – Nerd Sniping the Java Community (Keynote JCon Slovenia)
gunnarmorling
1
140
RubyKaigi Effect 2023, and ...
tagomoris
1
200
計算量オーダーの話
tsudaahr
1
170
App Router に移行して良かったところを語る
andpad
2
350
deadcode超解剖
kuro_kurorrr
1
250
自作HTTPルーターから新しいServeMuxへ
bmf_san
3
1.2k
Why Spring Matters to Jakarta EE - And Vice Versa
ivargrimstad
0
920
Containerization Tips and Tricks for PHP apps
dunglas
3
4.1k
Polarsの現状
daikikatsuragawa
1
1.5k
Featured
See All Featured
Java REST API Framework Comparison - PWX 2021
mraible
PRO
19
7k
Raft: Consensus for Rubyists
vanstee
133
6.3k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
27
2.4k
Automating Front-end Workflow
addyosmani
1357
200k
Reflections from 52 weeks, 52 projects
jeffersonlam
345
19k
Producing Creativity
orderedlist
PRO
338
39k
4 Signs Your Business is Dying
shpigford
176
21k
The Straight Up "How To Draw Better" Workshop
denniskardys
228
130k
Navigating Team Friction
lara
179
13k
Designing for humans not robots
tammielis
247
25k
Git: the NoSQL Database
bkeepers
PRO
423
64k
Testing 201, or: Great Expectations
jmmastey
32
6.8k
Transcript
Continuous Deployment Chris Keathley / @ChrisKeathley /
[email protected]
I work with a distributed team
I work with a distributed team
None
None
None
None
None
Warehouse
Warehouse API
Warehouse API Apps
The problem
Slow Iteration Cycle Deployment Deployment 2 weeks
Slow Iteration Cycle Deployment Deployment Deployment 2 weeks 2 weeks
Slow Iteration Cycle Deployment Deployment Deployment 3 weeks
Slow Iteration Cycle Deployment Deployment Deployment Hopefully someday
Large PRs
Unsure about state of the application
Unsure about state of the application
Unsure about state of the application
Unsure about state of the application
Unsure about state of the application
Rollbacks are a scam
None
None
Data Migration
Data Migration ?
Don’t do this
always Move forward
always Move forward
always Move forward
The goal should never be to roll back a deployment
The goal is to minimize the damage done by any
given deployment
There are bugs in your system
Solutions
We needed to deploy more often
So we did
Automated Deployment
What do you deploy?
Commit Sha
None
Jars
Artifacts
Git Tags
Containers
Your App
Your App Server
None
Master Branch How we merge our code PR
CI Github Registry Container Slack PR Notification
CI Kubernetes Deploy Auto-deploy Green builds of master
CI Kubernetes Deploy Auto-deploy Green builds of master Service A
Service B
CI Kubernetes Deploy Auto-deploy Green builds of master Service B
CI Kubernetes Deploy Auto-deploy Green builds of master Service B
Service A
CI Kubernetes Deploy Auto-deploy Green builds of master Service A
CI Kubernetes Deploy Auto-deploy Green builds of master Service A
Service B
Tests Metrics &
Integration Tests + Property Tests
Integration Tests TEst App DB Service
Modeling Users as FSMs logged_out logged_in login logout vote
Property Tests Add Todo Edit Todo Delete Todo
Property Tests Add Todo Edit Todo Delete Todo
Property Tests Add Todo Edit Todo Delete Todo
Property Tests Add Todo Edit Todo Delete Todo
Generate Commands
Generated Commands [{:add_todo, “Test Todo”, 1}, {:edit_todo, "Edited", 2}, {:delete_todo,
"", 1}, {:add_todo, “New Todo", 3}, {:delete_todo, "", 2} {:edit_todo, “Edited Todo”, 2}]
Generate Commands
Generate Commands
Generate Commands
Generate Commands
Generate Commands
Generated Commands [{:add_todo, “Test Todo”, 1}, {:edit_todo, "Edited", 2}, {:delete_todo,
"", 1}, {:add_todo, “New Todo", 3}, {:delete_todo, "", 2} {:edit_todo, “Edited Todo”, 2}]
Generated Commands [{:add_todo, “Test Todo”, 1}, {:delete_todo, "", 2}] [{:add_todo,
“Test Todo”, 1}, {:edit_todo, "Edited", 2}, {:delete_todo, "", 1}, {:add_todo, “New Todo", 3}, {:delete_todo, "", 2} {:edit_todo, “Edited Todo”, 2}]
Prometheus Service A Grafana Service B Service C
Prometheus Service A Grafana Service B Service C Slack
# Alert for any instance that have a 95th percentile
> 200ms. ALERT APIHighRequestLatency IF api_http_request_latencies_second{quantile="0.95"} > 0.2 FOR 5m ANNOTATIONS { summary = "High request latency on {{ $labels.instance }}", description = "{{ $labels.instance }} has a median request latency above 1s (current value: {{ $value }}s)", }
Track “Business” Metrics
None
Feature releases and flags
None
Features aren’t all or nothing
Features != Deployments
Deployment
Deployment Features
Deployment Features
User
User staff?(user) == true
User staff?(user) == false
User staff?(user) == false
defmodule MyApp.FeatureFlags do alias MyApp.User def foo_enabled?(%User{staff: is_staff}), do: is_staff
def foo_enabled?(_), do: false def bar_enabled?(%User{staff: is_staff}), do: is_staff def bar_enabled?(_), do: false end
Browser Feature Service
Feature Service Feature Service Feature Service
Feature Service Feature Service Feature Service
Feature Service Feature Service Feature Service
You have updates ready! Reset
None
With larger Traffic numbers you could use percentages
Alchemy
“Transmute lead code into gold in production”
Prior Art: https://github.com/github/scientist
Users_Controller DB User.all
DB User.all UserService.all
User.all UserService.all ==
def index(conn) do users = old_query() render(conn, "index.json", users: users)
end
def index(conn) do users = experiment("users-query") |> control(&old_query/0) |> candidate(&new_query/0)
|> run render(conn, "index.json", users: users) end
def index(conn) do users = experiment("users-query") |> control(&old_query/0) |> candidate(&new_query/0)
|> candidate(&fancy_query/0) |> run render(conn, "index.json", users: users) end
1) Shuffles test order 2) Runs Each test in parallel
3) exports the data Alchemy
DB User.all UserService.all Control Candidate Control UserController
None
1) Do the results match? 2) How long does each
test take to return? Measure
No more cutovers
DB User.all UserService.all
DB User.all UserService.all User service
Migrations
http://blog.datomic.com/2017/01/the-ten-rules-of-schema-growth.html
DB Schema App Application Coupling
Your application knows about your schema
Lets remove a column
Lets remove a column 1) all application code needs to
stop using that column
Lets remove a column 1) all application code needs to
stop using that column 2) Update all ETL processes
Lets remove a column 1) all application code needs to
stop using that column 2) Update all ETL processes 3) Update Reporting
Lets remove a column 1) all application code needs to
stop using that column 2) Update all ETL processes 3) Update Reporting 4) Remove the column
Lets remove a column 1) all application code needs to
stop using that column 2) Update all ETL processes 3) Update Reporting 4) Remove the column Split all of these up
Lets Add a column
Lets Add a column 1) Add the column
Lets Add a column 1) Add the column 2) Eventually
start using it
Prefer Additive Migrations
CI Kubernetes Deploy Auto-deploy Green builds of master
CI Kubernetes Deploy Auto-deploy Green builds of master Migration
CI Kubernetes Deploy Auto-deploy Green builds of master Migration DB
Chat-Ops
None
Chat is…
Chat is… Centralized
Chat is… Centralized Transparent
Chat is… Centralized Transparent Open
Try to do operational tasks in chat
None
defmodule Hedwig.Responders.Ping do use Hedwig.Responder @usage """ hedwig: ping -
Responds with 'pong' """ respond ~r/ping$/i, msg do reply msg, "pong" end end
None
Generate grafana graphs
None
Deploy
None
Team Building
Conclusion
These are tools at our disposal
Deploy more often, safely
Thanks Chris Keathley / @ChrisKeathley /
[email protected]