Quipper on Containers - Speaker Deck

Slide 1

Slide 1 text

on Containers )

Slide 2

Slide 2 text

@hakobera DevOps Engineer

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Mexico City London Tokyo Manila Jakarta Sales team only Sales and dev team International Offices No one here (yet).

Slide 5

Slide 5 text

Products

Slide 6

Slide 6 text

Container != Docker

Slide 7

Slide 7 text

Docker is one of the container impls. Ubuntu LXD Microsoft® Windows® Server Container Hyper-V® Container

Slide 8

Slide 8 text

When do you use container on your system?

Slide 9

Slide 9 text

Now!

Slide 10

Slide 10 text

Why do you use container?

Slide 11

Slide 11 text

Portability and Beyond Your Cloud Limits

Slide 12

Slide 12 text

Docker on Elastic Beanstalk ● For short batch using worker tier ● Avoid darkness of .ebextension ● Avoid Amazon Linux ○ We want to use Ubuntu

Slide 13

Slide 13 text

Load Test on Elastic Container Service ● Easy to setup locust cluster ○ Consists of 2 Tasks ■ 1 master ■ Multiple slave ○ Manage instances using Spot Fleet ■ Cheap but scalable

Slide 14

Slide 14 text

nginx-build on CircleCI ● Easy to build custom nginx build ○ Bundle custom modules such as ngx_mruby ○ Patch for security issue ■ Update only openssl but not nginx

Slide 15

Slide 15 text

Ansible playbook test on CircleCI ● Run playbook to clean environment ● Run playboot test in parallel

Slide 16

Slide 16 text

Do you also use container for Web application?

Slide 17

Slide 17 text

Yes! But it’s not so easy

Slide 18

Slide 18 text

History of Quipper and Container

Slide 19

Slide 19 text

Quipper is Container Native Company

Slide 20

Slide 20 text

Quipper use container from beginning

Slide 21

Slide 21 text

Back to 2012 First Product First Container is ...

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

It’s too early

Slide 24

Slide 24 text

No source repository Only CTO only knows

Slide 25

Slide 25 text

From 2013 Second Container Choice is ...

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

Heroku is great!

Slide 28

Slide 28 text

Awesome features of Heroku ● Deploy via git ● Buildpack ● Multiple Dyno Size ● One-off dyno ● Review Apps

Slide 29

Slide 29 text

At April, 2015 We are acquired by Recruit Co. Ltd

Slide 30

Slide 30 text

What problem do we have to resolve? ● Performance ○ Latency from Japan and USA (400ms/req) ○ Needed more powerful dyno ● Security ○ At least we need fixed outbound IP address ● Cost

Slide 31

Slide 31 text

We want to keep awesome features ● Deploy using git from CircleCI ● Buildpack ● Multiple Dyno Size ● One-off dyno ● CLI compatibility

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

Your PaaS, Your Rules Open Source Application Platform

Slide 34

Slide 34 text

Similarity to Heroku ● Deploy via git ● Buildpack ● CLI sub command ○ Ex: heroku conifg:set <-> deis config:set ● Interactive console not yet supported ○ Create own tool using Docker API

Slide 35

Slide 35 text

Similarity to Heroku is important Because we’re deploying one source

Slide 36

Slide 36 text

Source: http://docs.deis.io/en/latest/understanding_deis/architecture/

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

Components of PaaS for Web app ● Router ● Controller ● Slug Builder ● Container Registry ● Object Storage ● Scheduler ● Key Value Store ● Database ● Git Repository ● Logger

Slide 41

Slide 41 text

Where is the best place to deploy each component?

Slide 42

Slide 42 text

Late 2015

Slide 43

Slide 43 text

Elastic Load Balancing Router Controller Builder Applications etcd AZ #1 AZ #2 Production Cluster 1st Gen. PosggreSQL On Amazon RDS Docker Registry On Amazon S3

Slide 44

Slide 44 text

Hard to Scale Out

Slide 45

Slide 45 text

April 2016

Slide 46

Slide 46 text

Elastic Load Balancing Router/Applications AZ #1 AZ #2 Controller /Builder etcd Production Cluster 2nd Gen. PosggreSQL On Amazon RDS Docker Registry On Amazon S3

Slide 47

Slide 47 text

Noisy Neighbor AND Cost Increase

Slide 48

Slide 48 text

Fleet is too simple ● Fleet is not resource aware scheduler ● Fleet deploy new container in instance which has the least containers ● Sometimes make it unbalanced workload of clusters

Slide 49

Slide 49 text

Bad scheduling of fleet High High Normal Low Low Low Instance 1 Instance 2 Instance 3

Slide 50

Slide 50 text

We hope resource aware scheduling High High Normal Low Low Instance 1 Instance 2 Instance 3 Low

Slide 51

Slide 51 text

Aug 2016 〜

Slide 52

Slide 52 text

Elastic Load Balancing Router/Apps (Shared) AZ #1 AZ #2 Controller /Builder High-Load Apps (Dedicated) etcd Production Cluster 3rd Gen PosggreSQL On Amazon RDS Docker Registry On Amazon S3

Slide 53

Slide 53 text

Tag based Deployment High HighI Normal Low Low Low high=false high=true

Slide 54

Slide 54 text

Tag based Deployment ● Tag = Fleet metadata ● Set fleet metadata by cloud-init #cloud-config Coreos: ... fleet: public-ip: $private_ipv4 metadata: dataPlane=true,high=true #cloud-config Coreos: ... fleet: public-ip: $private_ipv4 metadata: dataPlane=true,routerMesh=true,high=false For dedicated For shared

Slide 55

Slide 55 text

Elastic Load Balancing Router AZ #1 AZ #2 Controller /Builder Applications etcd Staging Cluster PosggreSQL On Amazon RDS Docker Registry On Amazon S3

Slide 56

Slide 56 text

Other Awesome Features

Slide 57

Slide 57 text

Blue Green Deployment 1. Start new containers 2. Notify to etcd 3. Update nginx.conf by confd 4. Reload router (nginx) 5. Update routing 6. Destroy old containers

Slide 58

Slide 58 text

Use nginx as reverse proxy for PaaS ● Easy to handle naked domain ● Resolve slow client issue ● Can support HTTP/2 ● Flexible routing ○ A/B test, 10% release, etc ...

Slide 59

Slide 59 text

Problem as of now

Slide 60

Slide 60 text

Fleet is too simple (again) ● Fleet is not resource aware scheduler ● Fleet deploy new container in instance which has the least containers ● Sometimes make it unbalanced workload of clusters

Slide 61

Slide 61 text

etcd stability ● etcd sometimes get high CPU load

Slide 62

Slide 62 text

In the future ...

Slide 63

Slide 63 text

Deis Workflow (Deis v2) ● https://github.com/deis/workflow ● Based on Kubernates

Slide 64

Slide 64 text

End

Slide 65

Slide 65 text

We’re hiring! https://www.quipper.com/career/Japan/ https://www.wantedly.com/companies/quipper