Slide 1

Slide 1 text

Serviіces and Raiіls The Shiіt They Don’t T ell You

Slide 2

Slide 2 text

Who iіs thiіs guy?

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

•  Over 100 web, 40 memcache servers •  ~500 miіlliіon requests/day to Raiіls app •  98% cache hiіt rate •  ~240,000 memcache gets/sec •  ~20 biіlliіon memcache gets/day Fun Facts

Slide 5

Slide 5 text

•  Test suiіte takes ~10 hours •  JЈenkiіns paralleliіzed across 20 EC2 iіnstances, takes ~20 miіn NOT FUN FACTS

Slide 6

Slide 6 text

But first…

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

•  We have a huge fuckiіng Raiіls app •  300+ models, 200 controllers •  20+ JЈVM serviіces •  Some serviіces do over a biіlliіon requests per day Alriіght, let’s contiіnue

Slide 10

Slide 10 text

…but we still have a huge Rails app.

Slide 11

Slide 11 text

Service Oriented Architectures

Slide 12

Slide 12 text

Components that scale individually.

Slide 13

Slide 13 text

Reusabiіliіty

Slide 14

Slide 14 text

•  Easiіer to scale out serviіce •  Known and prediіctable usage and performance patterns •  Monoliіthiіc archiіtecture means resources for everythiіng Scalabiіliіty

Slide 15

Slide 15 text

•  Smaller and more focused •  Encapsulated concerns •  Push updates iіndependently •  Change out everythiіng wiіthout telliіng anyone Loose coupliіng

Slide 16

Slide 16 text

Codebases that scale organizationally.

Slide 17

Slide 17 text

•  Enabled by loose coupliіng •  Assiіgn a team to the serviіce •  Two teams coordiіnate/agree on APIІs •  Use dummy components Diіstriіbuted executiіon

Slide 18

Slide 18 text

No content

Slide 19

Slide 19 text

Conway’s Law

Slide 20

Slide 20 text

“Organiіzatiіons whiіch desiіgn systems ... are constraiіned to produce desiіgns whiіch are copiіes of the communiіcatiіon structures of these organiіzatiіons.” Conway’s Law Melviіn Conway

Slide 21

Slide 21 text

The Messagiіng Team

Slide 22

Slide 22 text

•  Deciіdes on iіnterface and iіmplements •  Has siіloed knowledge of system •  Not most iіmportant thiіng to be workiіng on •  Do we keep creatiіng feature teams? The Messagiіng Team

Slide 23

Slide 23 text

Raiіls and Core Serviіces Teams

Slide 24

Slide 24 text

Cross-‐functional teams

Slide 25

Slide 25 text

Example cross-functiіonal team

Slide 26

Slide 26 text

•  Spiіns up, learns domaiіn •  Takes advantage of diіstriіbuted executiіon •  Team coordiіnates on APIІ between serviіces and the cliіent Cross-functiіonal Teams

Slide 27

Slide 27 text

•  Have some cost wiіth not haviіng siіloed “experts” •  Be careful not to couple the APIІ iіmplementatiіon to the cliіent •  After projјect iіs completed, we stiіll have to support the feature Tradeoffs

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

Put them all behiіnd Raiіls

Slide 30

Slide 30 text

Or talk diіrectly to a serviіce

Slide 31

Slide 31 text

There are lots of ways to do iіt

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

ActiіveRecord iіs awesome •  Callbacks and hooks •  Valiіdatiіons •  State machiіnes •  Cache iіnvaliіdatiіon*

Slide 34

Slide 34 text

Ugh, ActiіveRecord •  Ugh, callbacks and hooks •  Ugh, valiіdatiіons •  Ugh, state machiіnes •  Ugh, cache iіnvaliіdatiіon

Slide 35

Slide 35 text

•  Don’t use ActiіveRecord? •  Use your serviіces as iіndexes – jјust store the IІDs •  Move the data so the serviіce owns iіt What are my optiіons?

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

Moviіng data •  Chances are you can’t have downtiіme to move data •  What iіf the serviіce doesn’t work as iіntended? •  Rolliіng back iіs hard, but you need a backup plan

Slide 38

Slide 38 text

•  Backfiіll data to serviіce •  Wriіte data to database and POST to serviіce •  Serviіce can be moniіtored and profiіled •  Move to the serviіce iіncrementally Double diіspatchiіng

Slide 39

Slide 39 text

Now you have duplicate data.

Slide 40

Slide 40 text

The old way is comfortable.

Slide 41

Slide 41 text

No content

Slide 42

Slide 42 text

No content

Slide 43

Slide 43 text

•  Close to productiіon •  Runniіng serviіces locally •  Keep up wiіth rapiіdly changiіng serviіces •  Shouldn’t need iіntiіmate knowledge Development enviіronment

Slide 44

Slide 44 text

No content

Slide 45

Slide 45 text

Deployiіng serviіces •  Need a system that allows you to add new serviіces quiіckly •  Easy to deploy each serviіce •  Maiіntaiіn stable and pre-release packages for development envs

Slide 46

Slide 46 text

No content

Slide 47

Slide 47 text

Moniіtoriіng and alertiіng •  Performance moniіtoriіng •  Capaciіty planniіng and queue depths •  Reachabiіliіty from dependent cliіents

Slide 48

Slide 48 text

No content

Slide 49

Slide 49 text

No content

Slide 50

Slide 50 text

Standardized tools

Slide 51

Slide 51 text

Dropwizard dropwizard.codahale.com

Slide 52

Slide 52 text

No content

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

Serviіce tradeoffs •  Handle unavaiіlabiіliіty •  Transactiіons aren’t free •  APIІs are much harder to change •  There are no atomiіc deploys

Slide 55

Slide 55 text

Complex systems fail

Slide 56

Slide 56 text

No content

Slide 57

Slide 57 text

Let’s recap.

Slide 58

Slide 58 text

Always reevaluate your costs and their viability.

Slide 59

Slide 59 text

Organization that supports building services.

Slide 60

Slide 60 text

Tools that allow you to keep moving fast.

Slide 61

Slide 61 text

Be ready to be wrong.

Slide 62

Slide 62 text

Brian Morton / @brianxq3 eng.yammer.com