Slide 1
Slide 1 text
on Containers
)
Slide 2
Slide 2 text
@hakobera
DevOps Engineer
Slide 3
Slide 3 text
No content
Slide 4
Slide 4 text
Mexico City
London
Tokyo
Manila
Jakarta
Sales team only
Sales and dev team
International Offices
No one here (yet).
Slide 5
Slide 5 text
Products
Slide 6
Slide 6 text
Container != Docker
Slide 7
Slide 7 text
Docker is one of the container impls.
Ubuntu LXD
Microsoft®
Windows® Server Container
Hyper-V® Container
Slide 8
Slide 8 text
When do you use container
on your system?
Slide 9
Slide 9 text
Now!
Slide 10
Slide 10 text
Why do you use container?
Slide 11
Slide 11 text
Portability
and
Beyond Your Cloud Limits
Slide 12
Slide 12 text
Docker on Elastic Beanstalk
● For short batch using worker tier
● Avoid darkness of .ebextension
● Avoid Amazon Linux
○ We want to use Ubuntu
Slide 13
Slide 13 text
Load Test on Elastic Container Service
● Easy to setup locust cluster
○ Consists of 2 Tasks
■ 1 master
■ Multiple slave
○ Manage instances using Spot Fleet
■ Cheap but scalable
Slide 14
Slide 14 text
nginx-build on CircleCI
● Easy to build custom nginx build
○ Bundle custom modules such as ngx_mruby
○ Patch for security issue
■ Update only openssl but not nginx
Slide 15
Slide 15 text
Ansible playbook test on CircleCI
● Run playbook to clean environment
● Run playboot test in parallel
Slide 16
Slide 16 text
Do you also use container
for Web application?
Slide 17
Slide 17 text
Yes!
But it’s not so easy
Slide 18
Slide 18 text
History of
Quipper and Container
Slide 19
Slide 19 text
Quipper is
Container Native Company
Slide 20
Slide 20 text
Quipper use container
from beginning
Slide 21
Slide 21 text
Back to 2012
First Product
First Container is ...
Slide 22
Slide 22 text
No content
Slide 23
Slide 23 text
It’s too early
Slide 24
Slide 24 text
No source repository
Only CTO only knows
Slide 25
Slide 25 text
From 2013
Second Container Choice is ...
Slide 26
Slide 26 text
No content
Slide 27
Slide 27 text
Heroku is great!
Slide 28
Slide 28 text
Awesome features of Heroku
● Deploy via git
● Buildpack
● Multiple Dyno Size
● One-off dyno
● Review Apps
Slide 29
Slide 29 text
At April, 2015
We are acquired by Recruit Co. Ltd
Slide 30
Slide 30 text
What problem do we have to resolve?
● Performance
○ Latency from Japan and USA (400ms/req)
○ Needed more powerful dyno
● Security
○ At least we need fixed outbound IP address
● Cost
Slide 31
Slide 31 text
We want to keep awesome features
● Deploy using git from CircleCI
● Buildpack
● Multiple Dyno Size
● One-off dyno
● CLI compatibility
Slide 32
Slide 32 text
No content
Slide 33
Slide 33 text
Your PaaS, Your Rules
Open Source
Application Platform
Slide 34
Slide 34 text
Similarity to Heroku
● Deploy via git
● Buildpack
● CLI sub command
○ Ex: heroku conifg:set <-> deis config:set
● Interactive console not yet supported
○ Create own tool using Docker API
Slide 35
Slide 35 text
Similarity to Heroku is important
Because we’re deploying one source
Slide 36
Slide 36 text
Source: http://docs.deis.io/en/latest/understanding_deis/architecture/
Slide 37
Slide 37 text
No content
Slide 38
Slide 38 text
No content
Slide 39
Slide 39 text
No content
Slide 40
Slide 40 text
Components of PaaS for Web app
● Router
● Controller
● Slug Builder
● Container Registry
● Object Storage
● Scheduler
● Key Value Store
● Database
● Git Repository
● Logger
Slide 41
Slide 41 text
Where is the best
place to deploy
each component?
Slide 42
Slide 42 text
Late 2015
Slide 43
Slide 43 text
Elastic Load
Balancing
Router
Controller
Builder
Applications
etcd
AZ #1
AZ #2
Production Cluster 1st Gen. PosggreSQL
On Amazon
RDS
Docker Registry
On Amazon S3
Slide 44
Slide 44 text
Hard to Scale Out
Slide 45
Slide 45 text
April 2016
Slide 46
Slide 46 text
Elastic Load
Balancing
Router/Applications
AZ #1
AZ #2
Controller
/Builder
etcd
Production Cluster 2nd Gen. PosggreSQL
On Amazon
RDS
Docker Registry
On Amazon S3
Slide 47
Slide 47 text
Noisy Neighbor
AND
Cost Increase
Slide 48
Slide 48 text
Fleet is too simple
● Fleet is not resource aware scheduler
● Fleet deploy new container in instance
which has the least containers
● Sometimes make it unbalanced workload
of clusters
Slide 49
Slide 49 text
Bad scheduling of fleet
High
High Normal Low
Low
Low
Instance 1 Instance 2 Instance 3
Slide 50
Slide 50 text
We hope resource aware scheduling
High High
Normal
Low
Low
Instance 1 Instance 2 Instance 3
Low
Slide 51
Slide 51 text
Aug 2016 〜
Slide 52
Slide 52 text
Elastic Load
Balancing
Router/Apps (Shared)
AZ #1
AZ #2
Controller
/Builder
High-Load Apps (Dedicated) etcd
Production Cluster 3rd Gen PosggreSQL
On Amazon
RDS
Docker Registry
On Amazon S3
Slide 53
Slide 53 text
Tag based Deployment
High HighI
Normal
Low
Low
Low
high=false
high=true
Slide 54
Slide 54 text
Tag based Deployment
● Tag = Fleet metadata
● Set fleet metadata by cloud-init
#cloud-config
Coreos:
...
fleet:
public-ip: $private_ipv4
metadata: dataPlane=true,high=true
#cloud-config
Coreos:
...
fleet:
public-ip: $private_ipv4
metadata: dataPlane=true,routerMesh=true,high=false
For dedicated For shared
Slide 55
Slide 55 text
Elastic Load
Balancing
Router
AZ #1
AZ #2
Controller
/Builder
Applications etcd
Staging Cluster PosggreSQL
On Amazon
RDS
Docker Registry
On Amazon S3
Slide 56
Slide 56 text
Other Awesome Features
Slide 57
Slide 57 text
Blue Green Deployment
1. Start new containers
2. Notify to etcd
3. Update nginx.conf by confd
4. Reload router (nginx)
5. Update routing
6. Destroy old containers
Slide 58
Slide 58 text
Use nginx as reverse proxy for PaaS
● Easy to handle naked domain
● Resolve slow client issue
● Can support HTTP/2
● Flexible routing
○ A/B test, 10% release, etc ...
Slide 59
Slide 59 text
Problem as of now
Slide 60
Slide 60 text
Fleet is too simple (again)
● Fleet is not resource aware scheduler
● Fleet deploy new container in instance
which has the least containers
● Sometimes make it unbalanced workload
of clusters
Slide 61
Slide 61 text
etcd stability
● etcd sometimes get high CPU load
Slide 62
Slide 62 text
In the future ...
Slide 63
Slide 63 text
Deis Workflow (Deis v2)
● https://github.com/deis/workflow
● Based on Kubernates
Slide 64
Slide 64 text
End
Slide 65
Slide 65 text
We’re hiring!
https://www.quipper.com/career/Japan/
https://www.wantedly.com/companies/quipper