Practicing Deployment - Speaker Deck

Slide 1

Slide 1 text

Practicing Deployment Laura Thomson laura@mozilla.com @lxt 1

Slide 2

Slide 2 text

Disclaimers Not about tools Not prescriptive Recognize where you are, and where you want to be 2

Slide 3

Slide 3 text

Models 3

Slide 4

Slide 4 text

Maturity model (after Capability Maturity Model, CMU) 1. Initial: “chaotic”, “individual heroics” 2. Repeatable: documented 3. Deﬁned: standard process, some tuning 4. Managed: measured 5. Optimizing: continual improvement, innovation 4

Slide 5

Slide 5 text

Initial Startup phase of many projects Long term Push code whenever you feel like it Devs push code Not a lot of tests, automation, or veriﬁcation 5

Slide 6

Slide 6 text

Repeatable Often after a 1.0, first non-beta ship, or first ship with a significant number of users Some kind of documented/known process Push when a feature is done: less often than initially, typically 6

Slide 7

Slide 7 text

Deﬁned Procedural documentation Start of automation Often done by a sysadmin 7

Slide 8

Slide 8 text

Managed Automation Tools: packaging Veriﬁcation post-push Measurement: How often do we push? How long does it take? How did that push affect performance? 8

Slide 9

Slide 9 text

Optimized Take the drama out of deployment Often - not essentially - continuous deployment Typically a lot of test automation Lightweight 9

Slide 10

Slide 10 text

How much do we ship? (Size of a release) Start with per-patch pushes Move to features Then to releases Then back to features The back to per-patch pushes 10

Slide 11

Slide 11 text

Per-patch Features Releases Features 11

Slide 12

Slide 12 text

Velocity models (Frequency of a release) Critical mass Single hard deadline Train model Continuous deployment 12

Slide 13

Slide 13 text

Critical mass “enough stuff to release” MVP smallest quantum with user value 13

Slide 14

Slide 14 text

Single hard deadline Support for X by date Y Shipping to a marketing plan Hard deadlines are hard 14

Slide 15

Slide 15 text

Train model Release e.g. every Wednesday Whatever’s ready to ship, ships Anything else catches the next train 15

Slide 16

Slide 16 text

Continuous deployment Ship each change as soon as it’s done Continuous is kind of a misnomer; deployment is discrete 16

Slide 17

Slide 17 text

Tools and practices 17

Slide 18

Slide 18 text

Source control Stable vs unstable Branch per bug, branch per feature “git ﬂow” is overkill, but you need a process If it’s not per-patch-push, tag what you push Open source needs ESRs even if you’re high velocity 18

Slide 19

Slide 19 text

Dev Envs Dev’s laptop is a horrible environment VMs can be hard to maintain Development databases are hard: fake data, minidbs Development API sandbox Lightweight set up and tear down VMs “Development” staging server (unstable) “Try” servers for branches 19

Slide 20

Slide 20 text

Staging Staging environment MUST REFLECT PRODUCTION Same versions, same proportions: a scale model Realistic trafﬁc and load (scale) Staging must be monitored Staging must have managed conﬁguration 20

Slide 21

Slide 21 text

One Box Fail Staging needs to be more than one box If you have multiple databases or webheads or whatever in prod...you need that in staging 21

Slide 22

Slide 22 text

Continuous Integration Build-on-commit VM-per-build Leeroy/Travis (PR automation) Run all unit tests (Auto) push build to staging Run more tests (acceptance/UI) 22

Slide 23

Slide 23 text

Testing Unit tests: run locally, run on build Acceptance/User tests: run against browser (Selenium, humans) Load test: how does it perform under prod load? Smoke test: what’s the maximum load we can support with this build? 23

Slide 24

Slide 24 text

Deployment tools It doesn’t really matter what you use Automate it Do it the same way in staging and production Use conﬁguration management to deploy conﬁg changes and manage your platform...the same way in staging and production 24

Slide 25

Slide 25 text

QA Feature tests on unstable Full tests on stage Full tests on production (veriﬁcation) 25

Slide 26

Slide 26 text

Measurement Monitoring Performance testing Instrument, instrument, instrument Is it actually possible to have too much data? (Hint: yes. But only if no insight) 26

Slide 27

Slide 27 text

Postmortems What went right What went wrong Blameless: scapegoats only hurt you 27

Slide 28

Slide 28 text

When things go wrong 28

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Quantum of deployment (via Erik Kastner) “What’s the smallest number of steps, with the smallest number of people and the smallest amount of ceremony required to get new code running on your servers?” http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/, 30

Slide 31

Slide 31 text

Chemspills Even if you have heavyweight/non-automated deployments, what does a chemspill look like? 31

Slide 32

Slide 32 text

THIS IS NOT A DRILL 32

Slide 33

Slide 33 text

Fail forward Fail forward: the premise that Mean Time To Repair is the key measure, not MTBF 33

Slide 34

Slide 34 text

Fail Sometimes you can’t fail forward Example: intractable/unforeseen performance problem, hardware failures, datacenter migrations Hit upper time limit (failing forward is taking too long) 34

Slide 35

Slide 35 text

Rollback Going back to the last known good Having a known process for rollback is just as important as having a known process for deployment Practice rollbacks 35

Slide 36

Slide 36 text

Decision points When shipping something new, deﬁne some rules and decision points If it passes this test/performance criteria we’ll ship it If these things go wrong we’ll roll back Make these rules beforehand, while heads are calm 36

Slide 37

Slide 37 text

Feature switches A nicer alternative to rollback Turn a feature on for a subset of users: beta users, developers, n% of users Turn it on for everybody Turn things off if you’re having problems or unexpected load: “load shedding” 37

Slide 38

Slide 38 text

Continuous Deployment 38

Slide 39

Slide 39 text

What is CD? Total misnomer Not continuous, discrete Automated not automatic, generally Intention is push-per-change Usually driven by a Big Red Button 39

Slide 40

Slide 40 text

Technical recommendations Continuous integration with build-on-commit Tests with good coverage, and a good feel for the holes in coverage A staging environment that reﬂects production Managed conﬁguration Scripted single button deployment to a large number of machines 40

Slide 41

Slide 41 text

People and process High levels of trust Realistic risk assessment and tolerance Excellent code review Excellent source code management Tracking, trending, monitoring 41

Slide 42

Slide 42 text

Testing vs monitoring Run tests against production Continuous testing = one kind of monitoring Testing is an important monitor You need other monitors You need tests too 42

Slide 43

Slide 43 text

You should build the capability for continuous deployment even if you never intend to do continuous deployment. 43

Slide 44

Slide 44 text

The only way to get good at deployment is to deploy a lot. 44

Slide 45

Slide 45 text

Questions? laura@mozilla.com 45