Slide 1
Slide 1 text
Pains and Gains of
Architecting Microservices
on Local Dev Environment
Go Conference 2018 Autumn @creasty @izumin5210
Slide 2
Slide 2 text
Complication of setting up applications
Service discovery
Slide 3
Slide 3 text
Complication of setting up applications
Service discovery
Slide 4
Slide 4 text
Yuki Iwanaga
SOFTWARE ENGINEER AT WANTEDLY, INC
A lover of all things beautiful from typography to well-designed software.
Currently leading product development of International Exapansion team.
Used to be a core member of Wantedly Chat and Wantedly People.
github.com/creasty
Slide 5
Slide 5 text
Complication of setting up applications
Golang, Ruby, Python, Rust, Node.js
~70 services
Need to run multiple services
even for a single feature development
Slide 6
Slide 6 text
In the case of Rails
$ brew install ...
$ cp .env{.sample,}
$ bundle install
$ rails db:create
$ rails db:schema:load
$ rails server
When you had good luck
Slide 7
Slide 7 text
In the case of Rails
When you had bad luck
$ brew install ...
some error occurred
$ cp .env{.sample,}
$ bundle install
failed to install nokogiri
try solution #1 - didn’t work
try solution #2 - got different errors
try solution #3 — finally got it right!
$ rails db:create
server not running
$ rails db:schema:load
$ rails server
error due to an invalid configuration
Slide 8
Slide 8 text
Things that stand in our way
Middleware and system libraries
The I-don’t-speak-your-language situation
Bootstrap scripts being outdated
Which strongly depend on OS
Need to figure out how each language works
Difficult to handle edge cases. Hard to maintain
Slide 9
Slide 9 text
Dockerization
Slide 10
Slide 10 text
Dockerization — Goal
Easy to use
Environment independent
Absorbing a difference
in a setup procedure among services
Slide 11
Slide 11 text
Dockerization
Is that all?
Slide 12
Slide 12 text
Dockerization — Experience
Usability
Transparency of the interface
Not changing the existing workflow
Not needing to know about Docker
Slide 13
Slide 13 text
Dockerization — Experience
When they’re not aware of Docker,
the good experience is achieved
Slide 14
Slide 14 text
The tool I made
Slide 15
Slide 15 text
rid
github.com/creasty/rid
Slide 16
Slide 16 text
rid — In the case of Rails
$ brew install ...
$ cp .env{.sample,}
$ bundle install
$ rails db:create
$ rails db:schema:load
$ rails server
good luck
Slide 17
Slide 17 text
rid — In the case of Rails
$ brew install ...
$ cp .env{.sample,}
$ bundle install
$ rails db:create
$ rails db:schema:load
$ rails server
good luck bad luck
$ brew install ...
some error occurred
$ cp .env{.sample,}
$ bundle install
failed to install nokogiri
try solution #1
try solution #2
try solution #3
$ rails db:create
server not running
$ rails db:schema:load
$ rails server
one last error
Slide 18
Slide 18 text
rid — In the case of Rails
$ brew install ...
$ cp .env{.sample,}
$ bundle install
$ rails db:create
$ rails db:schema:load
$ rails server
good luck
$ rid cp .env{.sample,}
$ rid bundle install
$ rid rails db:create
$ rid rails db:schema:load
$ rid rails server
rid ain’t luck
bad luck
$ brew install ...
some error occurred
$ cp .env{.sample,}
$ bundle install
failed to install nokogiri
try solution #1
try solution #2
try solution #3
$ rails db:create
server not running
$ rails db:schema:load
$ rails server
one last error
Slide 19
Slide 19 text
rid — In a nutshell
rid runs any commands in a container
just by prefixing it
Slide 20
Slide 20 text
rid — In the case of Rails
$ brew install ...
$ cp .env{.sample,}
$ bundle install
$ rails db:create
$ rails db:schema:load
$ rails server
good luck
$ rid cp .env{.sample,}
$ rid bundle install
$ rid rails db:create
$ rid rails db:schema:load
$ rid rails server
rid ain’t luck
bad luck
$ brew install ...
some error occurred
$ cp .env{.sample,}
$ bundle install
failed to install nokogiri
try solution #1
try solution #2
try solution #3
$ rails db:create
server not running
$ rails db:schema:load
$ rails server
one last error
Slide 21
Slide 21 text
rid — In the case of Rails
$ brew install ...
$ cp .env{.sample,}
$ bundle install
$ rails db:create
$ rails db:schema:load
$ rails server
good luck
$ rid cp .env{.sample,}
$ rid bundle install
$ rid rails db:create
$ rid rails db:schema:load
$ rid rails server
rid ain’t luck
bad luck
$ brew install ...
some error occurred
$ cp .env{.sample,}
$ bundle install
failed to install nokogiri
try solution #1
try solution #2
try solution #3
$ rails db:create
server not running
$ rails db:schema:load
$ rails server
one last error
No system library installation
No enviroment-dependent errors
Automatic container creation
Slide 22
Slide 22 text
Usage
Slide 23
Slide 23 text
Usage — Initialization
Files to prepare
rid
Dockerfile
docker-compose.yml
Create a directory at the root directory of a project,
and add Dockerfile and docker-compose.yml
Slide 24
Slide 24 text
Usage — Initialization
docker-compose.yml
• Specify an identifier of the project
in x-rid.project_name
• services.app is the main service,
where commands are executed via rid
version: '3.4'
x-rid:
project_name: creasty/rid-sample
services:
app:
build: .
...
Slide 25
Slide 25 text
Usage — Initialization
Dockerfile
• Specify the working directory as /app
• The rest is the same as usual.
Install system dependencies and configure
FROM ruby:2.5.1-alpine
...
WORKDIR /app
Slide 26
Slide 26 text
Usage — CLI
$ rid -h
$ rid -v
$ rid compose ...
$ rid anything ...
$ rid FOO=envvar1 BAR=2 anything ...
Shows help and version
Executes docker-compose commands
Executes any commands
With arbitrary env vars
CLI
Slide 27
Slide 27 text
Under the hood
Slide 28
Slide 28 text
Under the hood
Basically,
rid is a tiny script
that wraps docker-compose
Nevertheless, rid uses docker-compose in a bit unusual way
Slide 29
Slide 29 text
Under the hood — Execution
Share a single container
$ docker-compose run app CMD1...
$ docker-compose run app CMD2... Can’t run simultaneously
If any service in docker-compose.yml has port fowarding with fixed ports,
only one container can exist at the same time.
Slide 30
Slide 30 text
Under the hood — Execution
Share a single container
$ docker-compose up -d
$ docker-compose exec app CMD1...
$ docker-compose exec app CMD2... Execute the command in an existing container
Start a single container with detach mode
So, instead of creating one-shot containers for each execution,
rid keeps the single container running and executes every commands in it.
Slide 31
Slide 31 text
Under the hood — Execution
Difference detection and migration
docker-compose up -d can be run multiple times. In that case, it automatically detects difference
between a manifest file and the current configuration, and migrates the system if necessary.
• If the image doesn’t exist, it either pull or build
• If a container isn’t running, it creates one
• If a container does exist, it compares the current stack with YAML
• If there’s diff to apply, it recreates the container
Slide 32
Slide 32 text
No pain, no gain
Slide 33
Slide 33 text
No pain, No gain
1 Build an image
2 Install dependencies
3 Start the server
4 Connect to other services
Slide 34
Slide 34 text
No pain, no gain — Build an image
Create a Dockerfile designed specifically
for dev enviroment
• Requisities of development env and that of production env are different
• Production: What’s essential to run
• Development: What’s necessary to develop
• Do not include an application code base
• Use volume mount and manage it on host
• Images for dev environment can be public
• Able to pull those images without authentication
Slide 35
Slide 35 text
No pain, no gain
1 Build an image
2 Install dependencies
3 Start the server
4 Connect to other services
Slide 36
Slide 36 text
Unable to install private libraries!
PAIN!
Slide 37
Slide 37 text
No pain, no gain — Install dependencies
Configure SSH inside a container
• Want to reuse the same keys and configurations as the host
• No additional setup
• Credentials cannot be stored in an image itself
Slide 38
Slide 38 text
No pain, no gain — Install dependencies
$ docker run --rm -it -v ~/.ssh:/root/.ssh:ro alpine:latest
/ # apk add --no-cache -U openssh-client
/ # ssh [email protected]
PTY allocation request failed on channel 0
Hi creasty! You've successfully authenticated, but GitHub does not provide shell access.
Connection to github.com closed.
FROM alpine:latest
RUN apk add --no-cache -U openssh-client git
RUN git config --global --add '[email protected]:.insteadof' https://github.com/
RUN git config --global --add '[email protected]:.insteadof' git://github.com/
Mount the host’s ~/
.ssh directly to a container’s /root/
.ssh with readonly mode
Force git to use SSH for git and https protocols.
Slide 39
Slide 39 text
No pain, no gain
1 Build an image
2 Install dependencies
3 Start the server
4 Connect to other services
Slide 40
Slide 40 text
Volume mount being
painfully slow on macOS
PAIN!
File access in mounted volumes extremely slow, CPU bound - Docker for Mac - Docker Forums
https://forums.docker.com/t/file-access-in-mounted-volumes-extremely-slow-cpu-bound/8076
Slide 41
Slide 41 text
No pain, no gain — Start the server
• Rails server initialization
• 2 sec on the host
• 30 min on Docker
• Rsync/Unison didn’t work either
• Monitoring filesystem events and synchronization take time
Slide 42
Slide 42 text
No pain, no gain — Install dependencies
Manage vendor directories in data volumes
• A vendor directory is a directory where third-party code base is being managed
• Such as vendor/ and node_modules/
• Stopped mounting these directories from the host
• As we seldom edit these files, there’s no point in having the entity of files on the host
Slide 43
Slide 43 text
No pain, no gain — Install dependencies
services:
app:
volumes:
- ..:/app
- vendor:/app/vendor
volumes:
vendor:
Mount the root directory of a project at /app and mount a data volume at /app/vendor
Overlaying a data volume on a sub directory
Slide 44
Slide 44 text
No pain, no gain
1 Build an image
2 Install dependencies
3 Start the server
4 Connect to other services
Slide 45
Slide 45 text
Unable to discover other services
that are running on host
PAIN!
Slide 46
Slide 46 text
No pain, no gain — Connect to other services
Override localhost in a container
with the host's local IP
$ DOCKER_HOST_IP="$(ipconfig getifaddr en0 || ipconfig getifaddr en1)"
$ docker run --rm -it --add-host="localhost:$DOCKER_HOST_IP" alpine:latest
/ # apk add --no-cache -U busybox busybox-extra
/ # telnet localhost 3000
^D
Retrive a local IP address of the host and add new entry to /etc/hosts by specifying --add-host option.
In a container, localhost is resolved to the host.
Slide 47
Slide 47 text
No pain, no gain — Connect to other services
Override localhost in a container
with the host's local IP
$ DOCKER_HOST_IP="$(ipconfig getifaddr en0 || ipconfig getifaddr en1)"
$ docker run --rm -it --add-host="localhost:$DOCKER_HOST_IP" alpine:latest
/ # apk add --no-cache -U busybox busybox-extra
/ # telnet localhost 3000
^D
Retrive a local IP address of the host and add new entry to /etc/hosts by specifying --add-host option.
In a container, localhost is resolved to the host.
Slide 48
Slide 48 text
No pain, no gain — Connect to other services
Use a special DNS name
which Docker officially provides
• host.docker
.internal (v18.03)
• docker
.for
.mac.host.internal (Docker for Mac v17.12)
• docker
.for
.mac.localhost (Docker for Mac v17.06)
Slide 49
Slide 49 text
Who should manage middleware?
PAIN!
Slide 50
Slide 50 text
No pain, no gain — Connect to other services
services:
app:
...
networks:
- default
- sample_db
sample_db:
...
networks:
- sample_db
networks:
default:
sample_db:
The DB is accessible as sample_db from the app container.
Slide 51
Slide 51 text
How to avoid port conflicts?
PAIN!
Slide 52
Slide 52 text
No pain, no gain — Connect to other services
• Private repo at Wantedly
• List of ~70 ports that are used locally
Becoming out of control...
Slide 53
Slide 53 text
Who tf is localhost:6021!?
PAIN!
Slide 54
Slide 54 text
No pain, no gain — Connect to other services
Local DNS!
» @izumin5210
Slide 55
Slide 55 text
izumin5210
Engineer at Wantedly, Inc.
‣ Wantedly People
- Backend - Golang, Ruby, etc.
- Web Frontend - Advisor, Reviewer, Frontend Ops, etc.
‣ ͠Μͦͭʢ18ଔʣ
‣ Lead Gopher ʢ※ࣗশʣ
‣ Wrote github.com/izumin5210/grapi
Slide 56
Slide 56 text
No content
Slide 57
Slide 57 text
ery daemon
...
Docker Event
Watcher
Proxy Server
Manager
DNS server
Proxy
Server
API
Server
$ ery go run ./cmd/web
POST /mappings
Proxy
Server
Start new proxy server!
Register new domain!
proxy
Slide 58
Slide 58 text
ery daemon
...
Docker Event
Watcher
Proxy Server
Manager
DNS server
Proxy
Server
API
Server
$ rid rais s
docker run
Container created!
Proxy
Server
Register new domain!
Proxy
Server
Start new proxy serv
Slide 59
Slide 59 text
ͳΜ͓ͯߦّͷѱ͍πʔϧ
Slide 60
Slide 60 text
ͳΜ͓ͯߦّͷѱ͍πʔϧ
Slide 61
Slide 61 text
‣ Q. ͜Ε k8s / servicemesh ΛϩʔΧϧʹཱͯΔͷͱҧ͏ͷʁ
‣ A. ͨͿΜ͕ͪ͏
- ࣗҰॠ servicemesh ࡞ͬͯΔؾʹͳͬͨ
- શ Backend Engineer ʹ k8s ͷࣝΛཁٻ͢Δͷ͔ʁ
- Backend ઐ͡Όͳ͍ਓ͕αʔόΛॻ͘͜ͱ͋Δ
- ಁաੑͷ
- Application Engineer k8s Λӡ༻͍ͨ͠Μ͡Όͳ͘ɼϓϩμΫτΛϢʔβʹಧ͚͍ͨ
Slide 62
Slide 62 text
‣ Q. ͳʹ͕͏Ε͍͠ʁ
‣ A. ϙʔτ൪߸ͷ͜ͱΕΒΕΔ
- 3000൪ΛϝΠϯͷ Rails ͕͍ͬͯΔͷͰɼ
దʹ `rails new` ͢Δͱࢮ͵
- ෳՕॴಉ࣌ଟൃతʹ microservice ͕Ͱ͖Δͱ
port ͷऔΓ߹͍ʹͳΔ
‣ A. αʔϏε͝ͱʹ middleware ݐͯΒΕΔ
- e.g. Redis ͷ DB 3 ൪୭͕ͬͯΔΜ͚ͩͬʁ
Slide 63
Slide 63 text
‣ Q. ͬͯΈͯͲ͏ʁ
‣ A. ·ͩ୭͔ͭͬͯͳ͍Ͱ͢
- Proof of Concept ͱ Experimental ͷڱؒ
- ಋೖ͕ͪΐͬͱΊΜͲ͏
- loopback address ͷ alias షͬͨΓ
- /etc/resolver Λฤूͨ͠Γ…
- ؆୯ʹ͍ͨ͠ͳͱ͍͏ؾ࣋ͪ͋Δ
Slide 64
Slide 64 text
‣ Proxy server
- `net/http/httputil.ReverseProxy` Λ͍ͬͯΔ
- TCP Proxy ʹࠩ͠ସ͑ͳ͍ͱ PostgreSQL Redis Λ͍͍ײ͡ʹͰ͖ͳ͍
‣ ಋೖͷোนΛԼ͛Δ
- loopback address ͷ alias షͬͨΓɼ/etc/resolver Λฤूͨ͠Γ…
- ͨ͘͠ͳ͍
‣ ଞͷΞϓϩʔνͳ͍ʁ
- Docker / Kubernetes ʹڧ͘ͳͯ͑͘Δɼ΄͔ͷ࣮खஈͳ͍ʁ
Future work - ͍͖ͬͯ
Slide 65
Slide 65 text
Զͨͪͷઓ͍͜Ε͔Βͩʂ
Slide 66
Slide 66 text
͓ΘΓ
Slide 67
Slide 67 text
ಛผฤू൛ TECHBOOK ͋Γ·͢
- Writing JSON API in Go and gRPC
- ࣮ྫͰֶͿ Microservices
- ػցֶशΛαʔϏεΠϯ͢ΔͨΊʹΜͩ͜ͱ
- มԽʹڧ͍Πϯϑϥͱࣄྫ
- A/BςετΛΔͱ͖ʹؾΛ͚͍ͭͯΔ͜ͱ
- ඒຯ͍͓͠ञͷબͼํ
Slide 68
Slide 68 text
RejectCon ʢඇެࣜʣΓ·͢
Slide 69
Slide 69 text
Kubernetes ΛಡΜͩΓ
݄༵ʹ microservices ͷΛͨ͠Γͯ͠·͢
Go ͷѪΛޠͬͨΓ͠·͢