Tweet
Tweet

Slide 1

Slide 1 text

deploying distributed systems

Slide 2

Slide 2 text

Hi, I’m Matt @MATTHEATH

Slide 3

Slide 3 text

promo code: matt (for a £5 discount ;)

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

HOW DO WE DEPLOY?

Slide 10

Slide 10 text

PICK VERSION lets try this one Genuine network graph

Slide 11

Slide 11 text

UPDATE CODE

Slide 12

Slide 12 text

UPDATE CODE

Slide 13

Slide 13 text

UPDATE CODE

Slide 14

Slide 14 text

UPDATE CODE

Slide 15

Slide 15 text

UPDATE CODE

Slide 16

Slide 16 text

DRINK BEER (and wait for alerts...)

Slide 17

Slide 17 text

WHAT ARE WE TRYING TO DO?

Slide 18

Slide 18 text

HOW DO WE DO THIS? FTP SCP HOT EDIT SERVER MAGNETIZED NEEDLE & STEADY HANDS?!

Slide 19

Slide 19 text

NON ATOMIC!

Slide 20

Slide 20 text

NON ATOMIC!

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

SERVER CODE

Slide 23

Slide 23 text

CODE

Slide 24

Slide 24 text

1362393420 SYMLINK

Slide 25

Slide 25 text

1362393420 1362499200 SYMLINK

Slide 26

Slide 26 text

1362393420 1362499200 SYMLINK

Slide 27

Slide 27 text

1362393420 1362499200 1362610174 SYMLINK

Slide 28

Slide 28 text

1362393420 1362499200 1362610174 SYMLINK

Slide 29

Slide 29 text

WHAT ELSE? COMPRESS & MINIFY RESTART SERVERS ASSET VERSIONING DATABASE MIGRATIONS LOAD NEW CONFIGS LINT CHECKS CLEAR CACHES SPAWN WORKERS WARM CACHES

Slide 30

Slide 30 text

ENTER CAPISTRANO!

Slide 31

Slide 31 text

“Capistrano is a utility and framework for executing commands in parallel on multiple remote machines, via SSH”

Slide 32

Slide 32 text

$ cap deploy

Slide 33

Slide 33 text

• Identify version to deploy

Slide 34

Slide 34 text

• Identify version to deploy • Update remotely cached git repository

Slide 35

Slide 35 text

• Identify version to deploy • Update remotely cached git repository • Checkout specific version

Slide 36

Slide 36 text

• Identify version to deploy • Update remotely cached git repository • Checkout specific version • Copy code to a new timestamped folder

Slide 37

Slide 37 text

• Identify version to deploy • Update remotely cached git repository • Checkout specific version • Copy code to a new timestamped folder • Final preflight tasks

Slide 38

Slide 38 text

• Identify version to deploy • Update remotely cached git repository • Checkout specific version • Copy code to a new timestamped folder • Final preflight tasks • Switch symlink to point to new code

Slide 39

Slide 39 text

• Identify version to deploy • Update remotely cached git repository • Checkout specific version • Copy code to a new timestamped folder • Final preflight tasks • Switch symlink to point to new code • Restart web servers

Slide 40

Slide 40 text

• Identify version to deploy • Update remotely cached git repository • Checkout specific version • Copy code to a new timestamped folder • Final preflight tasks • Switch symlink to point to new code • Restart web servers • Post deploy tasks

Slide 41

Slide 41 text

• Identify version to deploy • Update remotely cached git repository • Checkout specific version • Copy code to a new timestamped folder • Final preflight tasks • Switch symlink to point to new code • Restart web servers • Post deploy tasks • Clean up old releases

Slide 42

Slide 42 text

GREAT SUCCESS!

Slide 43

Slide 43 text

$ capify .

Slide 44

Slide 44 text

$ capify . [add] writing './Capfile' [add] making directory './config' [add] writing './config/deploy.rb' [done] capified!

Slide 45

Slide 45 text

No content

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

No content

Slide 48

Slide 48 text

MULTI-STAGE

Slide 49

Slide 49 text

MULTI-STAGE cap dev deploy

Slide 50

Slide 50 text

MULTI-STAGE cap dev deploy cap staging deploy

Slide 51

Slide 51 text

MULTI-STAGE cap dev deploy cap staging deploy cap production deploy

Slide 52

Slide 52 text

No content

Slide 53

Slide 53 text

MULTI-CONFIGURATION?

Slide 54

Slide 54 text

cap londonstaging deploy MULTI-CONFIGURATION?

Slide 55

Slide 55 text

cap londonstaging deploy cap dublinstaging deploy MULTI-CONFIGURATION?

Slide 56

Slide 56 text

cap londonstaging deploy cap dublinstaging deploy cap torontostaging deploy MULTI-CONFIGURATION?

Slide 57

Slide 57 text

cap londonstaging deploy cap dublinstaging deploy cap torontostaging deploy cap chicagostaging deploy MULTI-CONFIGURATION?

Slide 58

Slide 58 text

cap londonstaging deploy cap dublinstaging deploy cap torontostaging deploy cap chicagostaging deploy cap nycstaging deploy MULTI-CONFIGURATION?

Slide 59

Slide 59 text

cap londonstaging deploy cap dublinstaging deploy cap torontostaging deploy cap chicagostaging deploy cap nycstaging deploy cap bostonstaging deploy MULTI-CONFIGURATION?

Slide 60

Slide 60 text

cap londonstaging deploy cap dublinstaging deploy cap torontostaging deploy cap chicagostaging deploy cap nycstaging deploy cap bostonstaging deploy cap ...staging deploy MULTI-CONFIGURATION?

Slide 61

Slide 61 text

cap londonstaging deploy cap dublinstaging deploy cap torontostaging deploy cap chicagostaging deploy cap nycstaging deploy cap bostonstaging deploy cap ...staging deploy cap ...staging deploy MULTI-CONFIGURATION?

Slide 62

Slide 62 text

cap londonstaging deploy cap dublinstaging deploy cap torontostaging deploy cap chicagostaging deploy cap nycstaging deploy cap bostonstaging deploy cap ...staging deploy cap ...staging deploy ... MULTI-CONFIGURATION?

Slide 63

Slide 63 text

No content

Slide 64

Slide 64 text

SOA = COMPLEX

Slide 65

Slide 65 text

INTERNAL TOOLS & APIS HAILOCAB.COM DRIVER APIS CUSTOMER SERVICES CREDIT SERVICES COMMUNICATION SERVICES AUTHENTICATION SERVICES CONFIGURATION SERVICES CRM SERVICES PAYMENT SERVICES EVENTS SERVICES STATS SERVICES ALLOCATION SERVICES BACKGROUND QUEUES CUSTOMER APIS MESSAGING SERVICES APPLICATIONS

Slide 66

Slide 66 text

3 geographic regions

Slide 67

Slide 67 text

9 availability zones 3 geographic regions

Slide 68

Slide 68 text

problem?

Slide 69

Slide 69 text

OVER 9000 DEPLOY TARGETS OVER 9000 DEPLOY TARGETS lol/jk lol/jk

Slide 70

Slide 70 text

EASILY MAKE MISTAKES

Slide 71

Slide 71 text

SOA, SCALING & RESILIENCE VASTLY INCREASE COMPLEXITY

Slide 72

Slide 72 text

USE ROBOTS FOR GOOD

Slide 73

Slide 73 text

UNTIL THEY BECOME SELF AWARE...

Slide 74

Slide 74 text

SO...

Slide 75

Slide 75 text

the deployer

Slide 76

Slide 76 text

Allow anyone...

Slide 77

Slide 77 text

to deploy from anywhere

Slide 78

Slide 78 text

to everywhere

Slide 79

Slide 79 text

One application to deploy all applications Deploy in parallel to n servers in distributed geographic regions with the correct configuration Dynamic server lists Add additional functionality in one central place Play a shipping horn sound when deploying

Slide 80

Slide 80 text

DEMO!

Slide 81

Slide 81 text

PHP, APC, NO STAT, REAL PATH CACHING, SYMLINKS & YOU SRSLY WTF PHP?!?!

Slide 82

Slide 82 text

"echo 'APC Cache Blitz in '`let n= $(php -i | grep realpath_cache_ttl | sed -n 's/.* \([0-9]*\).*/\1/p') +5;echo $n`' seconds... ('`date`')'; sleep `let n=$(php -i | grep realpath_cache_ttl | sed -n 's/.* \([0-9]*\).*/\1/p')+5;echo $n`; echo 'Blitzing... ('`date`')'; blitz"

Slide 83

Slide 83 text

SUCCESSFUL BUILDS AUTO DEPLOY

Slide 84

Slide 84 text

DID THE TESTS PASS? IS THE BRANCH UP TO DATE? SHIP IT!

Slide 85

Slide 85 text

OMG YOU OVERWROTE MY BUILD

Slide 86

Slide 86 text

LOCK ENVIRONMENTS

Slide 87

Slide 87 text

LOCK ENVIRONMENTS CUSTOM STAGING ENDPOINTS

Slide 88

Slide 88 text

WITH ROBOTS AUTOMATE

Slide 89

Slide 89 text

WHAT COULD GO WRONG?

Slide 90

Slide 90 text

WELL...

Slide 91

Slide 91 text

MONITOR HEALTH AFTER DEPLOY

Slide 92

Slide 92 text

ROLLBACK ON FAILURE AUTOMATICALLY

Slide 93

Slide 93 text

KEEP HUMANS UP TO DATE

Slide 94

Slide 94 text

IRC / CHAT [03:14:58] mobile-api/intlrelease (6957bb87) is being deployed to staging by Matt Heath [03:15:25] mobile-api/intlrelease successfully deployed to staging [12:29:20] elasticride/master (390eae4c) is being deployed to staging by Saul Howard

Slide 95

Slide 95 text

TRACK DEPLOYS GRAPHITE+STATSD=—

Slide 96

Slide 96 text

OH SHIT

Slide 97

Slide 97 text

I DON’T USE CAPISTRANO: YOUR ARGUMENT IS INVALID

Slide 98

Slide 98 text

ALTERNATIVES?

Slide 99

Slide 99 text

.deb PACKAGES PUPPET / CHEF AMIs BITTORRENT

Slide 100

Slide 100 text

fin.

Slide 101

Slide 101 text

Image credits: Conan ‘8-bit Marvels’ - Paulo Capdeville Network diagram - Elijah Meeks, dhs.stanford.edu Creative Commons images from sxc.hu thanks to Miguel Saavedra, @rbval, G. Marcus Still interested? Caphub - Multistage capistrano: http://bit.ly/Tkkn4W Deploying at Github: http://git.io/FlaFhA Asgard - Web based deployment from Netflix: http://nflx.it/NsJVtD