Single Artifact Deployments

Slide 1

Slide 1 text

SINGLE ARTIFACT DEPLOYMENTS For Python Applications Jason Filipe @jfilipe

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

DEVELOPER ON THE PLATFORM TEAM

Slide 5

Slide 5 text

WHAT WE’LL COVER • Typical deployment process • Single artifact deployment, obstacles encountered and tools used

Slide 6

Slide 6 text

TYPICAL DEPLOYMENT Fabric script $ fab -‐-‐set target=staging deploy command

Slide 7

Slide 7 text

FABRIC SCRIPT On each app server + worker I. Download tarball of git tag II. Install requirements in a new virtualenv III. Static asset compilation (css/js) IV. Run database migrations (South) V. Swap symlinks (code and virtualenv) and restart processes

Slide 8

Slide 8 text

SYMLINK BASED source code /srv/releases/2012-12-31--00-00-00 /srv/releases/2013-01-01--00-00-00 /srv/releases/current -> 2013-01-01--00-00-00 /srv/releases/previous -> 2012-12-31-00-00-00

Slide 9

Slide 9 text

SYMLINK BASED virtualenv /srv/virtualenv/2012-12-31--00-00-00 /srv/virtualenv/2013-01-01--00-00-00 /srv/virtualenv/default -> 2013-01-01--00-00-00 /srv/virtualenv/previous -> 2012-12-31-00-00-00

Slide 10

Slide 10 text

WHY IT SUCKS

Slide 11

Slide 11 text

DEVELOPER MACHINE DEPENDENCY • Spotty internet connectivity • Addition local deploy dependencies • Fear

Slide 12

Slide 12 text

DUPLICATION • Download tarball of git tag • Install requirements • Static asset minification • etc.. app-01 app-*

Slide 13

Slide 13 text

SLOW 18+ minutes

Slide 14

Slide 14 text

WE WANTED SOMETHING BETTER

Slide 15

Slide 15 text

SINGLE ARTIFACT DEPLOYS

Slide 16

Slide 16 text

Create a single tarball that contains the source code, compiled virtualenv and all static assets minified and ready to go.

Slide 17

Slide 17 text

TOOLS

Slide 18

Slide 18 text

HAPROXY

Slide 19

Slide 19 text

No content

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

SUPERVISORD

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

COMPILED VIRTUALENV PAIN • Hardcoded paths in pyc files • https://github.com/fireteam/ virtualenv-tools • Doesn’t work with python dependencies that have C extensions (e.g. gevent)

Slide 24

Slide 24 text

Compile the virtualenv in the target directory.

Slide 25

Slide 25 text

BUILD PHASE • Jenkins job • Part of build pipeline • accounting-‐unit accounting-‐integration accounting-‐build

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

No content

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

#!/bin/bash cd $WORKSPACE virtualenv_hash = `md5sum requirements.txt` if [ -‐d "/srv/virtualenv/$virtualenv_hash" ]; then echo "Reusing existing virtualenv ${virtualenv_hash}" else echo "Cached virtualenv not found, creating new..." virtualenv "/srv/virtualenv/$virtualenv_hash" pip install -‐r requirements.txt fi cp -‐r "/srv/virtualenv/$virtualenv_hash" "$WORKSPACE/virtualenv" # static asset compilation goes here # grunt release # virtualenv/bin/python python manage.py collectstatic -‐-‐noinput # virtualenv/bin/python manage.py compress tar zcvf accounting-‐$BUILD_NUMBER.tgz -‐-‐exclude='.git' *

Slide 32

Slide 32 text

Build server must match production. python version, distro, etc

Slide 33

Slide 33 text

/srv/virtualenv/ef332e984ecf9044f35da82adaa182f7 /srv/virtualenv/default -> ef332e984ecf9044f35da82adaa182f7

Slide 34

Slide 34 text

/srv/releases/1337 /srv/releases/current -> 1337 /srv/releases/1337/virtualenv -> /srv/virtualenv/ef332e984ecf9044f35da82adaa182f7

Slide 35

Slide 35 text

NEW FABRIC SCRIPT $ fab -‐-‐set target=staging deploy:accounting-‐1337.tgz command

Slide 36

Slide 36 text

NEW FABRIC SCRIPT On each app server + worker I. Download single artifact tarball II. Untar artifact III. List out any pending db migrations IV. Swap symlinks (code and virtualenv) and restart continued

Slide 37

Slide 37 text

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

Nathan Duthoit The path to smoother database migrations

Slide 40

Slide 40 text

Slide 41

Slide 41 text

SUPERVISORD # ... stopwaitsecs=35 stopsignal=QUIT # .... conﬁg $ supervisorctl stop command

Slide 42

Slide 42 text

FAST ~ 2 minutes majority is waiting for inflight requests

Slide 43

Slide 43 text

MINIMIZED SPOF • If things fail it’s during build time

Slide 44

Slide 44 text

ALL DONE? • Much better than initial deploy process • Still have developer machine dependency But we have Jenkins!

Slide 45

Slide 45 text

NEW DEPLOY JOB • accounting-‐deploy • Executes Fabric script • Need to manage Jenkins credentials and click a button We can still do better!

Slide 46

Slide 46 text

Slide 47

Slide 47 text

DEPLOY ALL THE THINGS

Slide 48

Slide 48 text

DEPLOYS VIA HIPCHAT

Slide 49

Slide 49 text

ALTERNATIVES • Wheel PEP 427 • pip support as of 1.4 to build and install wheels command $ pip wheel -‐-‐wheel-‐dir=/tmp/accounting/ wheelhouse -‐r requirements.txt $ pip install -‐-‐use-‐wheel -‐-‐no-‐index -‐-‐find-‐ links=/tmp/accounting/wheelhouse -‐r requirements.txt

Slide 50

Slide 50 text

IMPROVEMENTS • Build native packages (rpm/deb) from single artifact • Optimizations during build phase

Slide 51

Slide 51 text

FUTURE • Docker and packer • Immutable servers

Slide 52

Slide 52 text

TL;DR • Typically longest step of deployments is installing requirements • Precompile your virtualenv in the same directory it will be used in production to avoid hardcoded paths in pyc files using hash of requirements as virtualenv name • Enjoy faster deploys!

Slide 53

Slide 53 text

THANKS! @jﬁlipe filipe.ca