Pains and Gains of Architecting Microservices on Local Dev Environment

Wantedly では解決する課題に応じた最適な技術選定をするために、機能毎に小さくサービスが分けられ、Golang, Ruby,Python, Rust, Node.jsなど多様な言語が使われています。この様な状況においては、増大する開発の複雑さに立ち向かい、「気持ちよく」開発を継続できる環境が重要となります。
本セッションでは「言語が異なることによる開発環境構築の複雑化」と「複数サービス間通信時のサービスディスカバリ」の2つの問題に焦点を当て、ローカル環境での開発体験をいかに向上させたかについてお話します。

Go Conference 2018 Autumn

Transcript

1. Pains and Gains of Architecting Microservices on Local Dev Environment Go Conference 2018 Autumn @creasty @izumin5210
2. Complication of setting up applications Service discovery
3. Complication of setting up applications Service discovery
4. Yuki Iwanaga SOFTWARE ENGINEER AT WANTEDLY, INC A lover of all things beautiful from typography to well-designed software. Currently leading product development of International Exapansion team. Used to be a core member of Wantedly Chat and Wantedly People. github.com/creasty
5. Complication of setting up applications Golang, Ruby, Python, Rust, Node.js ~70 services Need to run multiple services even for a single feature development
6. In the case of Rails $ brew install ... $ cp .env{.sample,} $ bundle install $ rails db:create $ rails db:schema:load $ rails server When you had good luck
7. In the case of Rails When you had bad luck $ brew install ... some error occurred $ cp .env{.sample,} $ bundle install failed to install nokogiri try solution #1 - didn’t work try solution #2 - got different errors try solution #3 — finally got it right! $ rails db:create server not running $ rails db:schema:load $ rails server error due to an invalid configuration
8. Things that stand in our way Middleware and system libraries The I-don’t-speak-your-language situation Bootstrap scripts being outdated Which strongly depend on OS Need to figure out how each language works Difficult to handle edge cases. Hard to maintain
9. Dockerization
10. Dockerization — Goal Easy to use Environment independent Absorbing a difference in a setup procedure among services
11. Dockerization Is that all?
12. Dockerization — Experience Usability Transparency of the interface Not changing the existing workflow Not needing to know about Docker
13. Dockerization — Experience When they’re not aware of Docker, the good experience is achieved
14. The tool I made
15. rid github.com/creasty/rid
16. rid — In the case of Rails $ brew install ... $ cp .env{.sample,} $ bundle install $ rails db:create $ rails db:schema:load $ rails server good luck
17. rid — In the case of Rails $ brew install ... $ cp .env{.sample,} $ bundle install $ rails db:create $ rails db:schema:load $ rails server good luck bad luck $ brew install ... some error occurred $ cp .env{.sample,} $ bundle install failed to install nokogiri try solution #1 try solution #2 try solution #3 $ rails db:create server not running $ rails db:schema:load $ rails server one last error
18. rid — In the case of Rails $ brew install ... $ cp .env{.sample,} $ bundle install $ rails db:create $ rails db:schema:load $ rails server good luck $ rid cp .env{.sample,} $ rid bundle install $ rid rails db:create $ rid rails db:schema:load $ rid rails server rid ain’t luck bad luck $ brew install ... some error occurred $ cp .env{.sample,} $ bundle install failed to install nokogiri try solution #1 try solution #2 try solution #3 $ rails db:create server not running $ rails db:schema:load $ rails server one last error
19. rid — In a nutshell rid runs any commands in a container just by prefixing it
20. rid — In the case of Rails $ brew install ... $ cp .env{.sample,} $ bundle install $ rails db:create $ rails db:schema:load $ rails server good luck $ rid cp .env{.sample,} $ rid bundle install $ rid rails db:create $ rid rails db:schema:load $ rid rails server rid ain’t luck bad luck $ brew install ... some error occurred $ cp .env{.sample,} $ bundle install failed to install nokogiri try solution #1 try solution #2 try solution #3 $ rails db:create server not running $ rails db:schema:load $ rails server one last error
21. rid — In the case of Rails $ brew install ... $ cp .env{.sample,} $ bundle install $ rails db:create $ rails db:schema:load $ rails server good luck $ rid cp .env{.sample,} $ rid bundle install $ rid rails db:create $ rid rails db:schema:load $ rid rails server rid ain’t luck bad luck $ brew install ... some error occurred $ cp .env{.sample,} $ bundle install failed to install nokogiri try solution #1 try solution #2 try solution #3 $ rails db:create server not running $ rails db:schema:load $ rails server one last error No system library installation No enviroment-dependent errors Automatic container creation
22. Usage
23. Usage — Initialization Files to prepare rid Dockerfile docker-compose.yml Create a directory at the root directory of a project, and add Dockerfile and docker-compose.yml
24. Usage — Initialization docker-compose.yml • Specify an identifier of the project in x-rid.project_name • services.app is the main service, where commands are executed via rid version: '3.4' x-rid: project_name: creasty/rid-sample services: app: build: . ...
25. Usage — Initialization Dockerfile • Specify the working directory as /app • The rest is the same as usual. Install system dependencies and configure FROM ruby:2.5.1-alpine ... WORKDIR /app
26. Usage — CLI $ rid -h $ rid -v $ rid compose ... $ rid anything ... $ rid FOO=envvar1 BAR=2 anything ... Shows help and version Executes docker-compose commands Executes any commands With arbitrary env vars CLI
27. Under the hood
28. Under the hood Basically, rid is a tiny script that wraps docker-compose Nevertheless, rid uses docker-compose in a bit unusual way
29. Under the hood — Execution Share a single container $ docker-compose run app CMD1... $ docker-compose run app CMD2... Can’t run simultaneously If any service in docker-compose.yml has port fowarding with fixed ports, only one container can exist at the same time.
30. Under the hood — Execution Share a single container $ docker-compose up -d $ docker-compose exec app CMD1... $ docker-compose exec app CMD2... Execute the command in an existing container Start a single container with detach mode So, instead of creating one-shot containers for each execution, rid keeps the single container running and executes every commands in it.
31. Under the hood — Execution Difference detection and migration docker-compose up -d can be run multiple times. In that case, it automatically detects difference between a manifest file and the current configuration, and migrates the system if necessary. • If the image doesn’t exist, it either pull or build • If a container isn’t running, it creates one • If a container does exist, it compares the current stack with YAML • If there’s diff to apply, it recreates the container
32. No pain, no gain
33. No pain, No gain 1 Build an image 2 Install dependencies 3 Start the server 4 Connect to other services
34. No pain, no gain — Build an image Create a Dockerfile designed specifically for dev enviroment • Requisities of development env and that of production env are different • Production: What’s essential to run • Development: What’s necessary to develop • Do not include an application code base • Use volume mount and manage it on host • Images for dev environment can be public • Able to pull those images without authentication
35. No pain, no gain 1 Build an image 2 Install dependencies 3 Start the server 4 Connect to other services
36. Unable to install private libraries! PAIN!
37. No pain, no gain — Install dependencies Configure SSH inside a container • Want to reuse the same keys and configurations as the host • No additional setup • Credentials cannot be stored in an image itself
38. No pain, no gain — Install dependencies $ docker run --rm -it -v ~/.ssh:/root/.ssh:ro alpine:latest / # apk add --no-cache -U openssh-client / # ssh git@github.com PTY allocation request failed on channel 0 Hi creasty! You've successfully authenticated, but GitHub does not provide shell access. Connection to github.com closed. FROM alpine:latest RUN apk add --no-cache -U openssh-client git RUN git config --global --add 'url.git@github.com:.insteadof' https://github.com/ RUN git config --global --add 'url.git@github.com:.insteadof' git://github.com/ Mount the host’s ~/ .ssh directly to a container’s /root/ .ssh with readonly mode Force git to use SSH for git and https protocols.
39. No pain, no gain 1 Build an image 2 Install dependencies 3 Start the server 4 Connect to other services
40. Volume mount being painfully slow on macOS PAIN! File access in mounted volumes extremely slow, CPU bound - Docker for Mac - Docker Forums https://forums.docker.com/t/file-access-in-mounted-volumes-extremely-slow-cpu-bound/8076
41. No pain, no gain — Start the server • Rails server initialization • 2 sec on the host • 30 min on Docker • Rsync/Unison didn’t work either • Monitoring filesystem events and synchronization take time
42. No pain, no gain — Install dependencies Manage vendor directories in data volumes • A vendor directory is a directory where third-party code base is being managed • Such as vendor/ and node_modules/ • Stopped mounting these directories from the host • As we seldom edit these files, there’s no point in having the entity of files on the host
43. No pain, no gain — Install dependencies services: app: volumes: - ..:/app - vendor:/app/vendor volumes: vendor: Mount the root directory of a project at /app and mount a data volume at /app/vendor Overlaying a data volume on a sub directory
44. No pain, no gain 1 Build an image 2 Install dependencies 3 Start the server 4 Connect to other services
45. Unable to discover other services that are running on host PAIN!
46. No pain, no gain — Connect to other services Override localhost in a container with the host's local IP $ DOCKER_HOST_IP="$(ipconfig getifaddr en0 || ipconfig getifaddr en1)" $ docker run --rm -it --add-host="localhost:$DOCKER_HOST_IP" alpine:latest / # apk add --no-cache -U busybox busybox-extra / # telnet localhost 3000 ^D Retrive a local IP address of the host and add new entry to /etc/hosts by specifying --add-host option. In a container, localhost is resolved to the host.
47. No pain, no gain — Connect to other services Override localhost in a container with the host's local IP $ DOCKER_HOST_IP="$(ipconfig getifaddr en0 || ipconfig getifaddr en1)" $ docker run --rm -it --add-host="localhost:$DOCKER_HOST_IP" alpine:latest / # apk add --no-cache -U busybox busybox-extra / # telnet localhost 3000 ^D Retrive a local IP address of the host and add new entry to /etc/hosts by specifying --add-host option. In a container, localhost is resolved to the host.
48. No pain, no gain — Connect to other services Use a special DNS name which Docker officially provides • host.docker .internal (v18.03) • docker .for .mac.host.internal (Docker for Mac v17.12) • docker .for .mac.localhost (Docker for Mac v17.06)
49. Who should manage middleware? PAIN!
50. No pain, no gain — Connect to other services services: app: ... networks: - default - sample_db sample_db: ... networks: - sample_db networks: default: sample_db: The DB is accessible as sample_db from the app container.
51. How to avoid port conflicts? PAIN!
52. No pain, no gain — Connect to other services • Private repo at Wantedly • List of ~70 ports that are used locally Becoming out of control...
53. Who tf is localhost:6021!? PAIN!
54. No pain, no gain — Connect to other services Local DNS! » @izumin5210
55. izumin5210 Engineer at Wantedly, Inc. ‣ Wantedly People - Backend - Golang, Ruby, etc. - Web Frontend - Advisor, Reviewer, Frontend Ops, etc. ‣ ͠Μͦͭʢ18ଔʣ ‣ Lead Gopher ʢ※ࣗশʣ ‣ Wrote github.com/izumin5210/grapi
57. ery daemon ... Docker Event Watcher Proxy Server 
 Manager DNS server Proxy Server API Server $ ery go run ./cmd/web POST /mappings Proxy Server Start new proxy server! Register new domain! proxy
58. ery daemon ... Docker Event Watcher Proxy Server 
 Manager DNS server Proxy Server API Server $ rid rais s docker run Container created! Proxy Server Register new domain! Proxy Server Start new proxy serv
59. ͳΜ͓ͯߦّͷѱ͍πʔϧ
60. ͳΜ͓ͯߦّͷѱ͍πʔϧ
61. ‣ Q. ͜Ε͸ k8s / servicemesh ΛϩʔΧϧʹཱͯΔͷͱ͸ҧ͏ͷʁ ‣ A. ͨͿΜ͕ͪ͏ - ࣗ෼΋Ұॠ servicemesh ࡞ͬͯΔؾ෼ʹͳͬͨ - શ Backend Engineer ʹ k8s ͷ஌ࣝΛཁٻ͢Δͷ͔ʁ - Backend ઐ໳͡Όͳ͍ਓ͕αʔόΛॻ͘͜ͱ΋͋Δ - ಁաੑͷ໰୊ - Application Engineer ͸ k8s Λӡ༻͍ͨ͠Μ͡Όͳ͘ɼϓϩμΫτΛϢʔβʹಧ͚͍ͨ
62. ‣ Q. ͳʹ͕͏Ε͍͠ʁ ‣ A. ϙʔτ൪߸ͷ͜ͱ๨ΕΒΕΔ - 3000൪ΛϝΠϯͷ Rails ͕࢖͍ͬͯΔͷͰɼ
 ద౰ʹ `rails new` ͢Δͱࢮ͵ - ෳ਺Օॴಉ࣌ଟൃతʹ microservice ͕Ͱ͖Δͱ
 port ͷऔΓ߹͍ʹͳΔ ‣ A. αʔϏε͝ͱʹ middleware ݐͯΒΕΔ - e.g. Redis ͷ DB 3 ൪͸୭͕࢖ͬͯΔΜ͚ͩͬʁ
63. ‣ Q. ࢖ͬͯΈͯͲ͏ʁ ‣ A. ·ͩ୭΋͔ͭͬͯͳ͍Ͱ͢ - Proof of Concept ͱ Experimental ͷڱؒ - ಋೖ͕ͪΐͬͱΊΜͲ͏ - loopback address ͷ alias షͬͨΓ - /etc/resolver Λฤूͨ͠Γ… - ؆୯ʹ͍ͨ͠ͳͱ͍͏ؾ࣋ͪ͸͋Δ
64. ‣ Proxy server - `net/http/httputil.ReverseProxy` Λ࢖͍ͬͯΔ - TCP Proxy ʹࠩ͠ସ͑ͳ͍ͱ PostgreSQL ΍ Redis Λ͍͍ײ͡ʹͰ͖ͳ͍ ‣ ಋೖͷোนΛԼ͛Δ - loopback address ͷ alias షͬͨΓɼ/etc/resolver Λฤूͨ͠Γ… - ͨ͘͠ͳ͍ ‣ ଞͷΞϓϩʔν͸ͳ͍ʁ - Docker / Kubernetes ʹڧ͘ͳͯ͘΋࢖͑Δɼ΄͔ͷ࣮૷खஈ͸ͳ͍ʁ Future work - ΍͍͖ͬͯ
65. Զͨͪͷઓ͍͸͜Ε͔Βͩʂ
66. ͓ΘΓ
67. ಛผฤू൛ TECHBOOK ͋Γ·͢ - Writing JSON API in Go and gRPC - ࣮ྫͰֶͿ Microservices - ػցֶशΛαʔϏεΠϯ͢ΔͨΊʹ೰Μͩ͜ͱ - มԽʹڧ͍Πϯϑϥͱࣄྫ - A/BςετΛ΍Δͱ͖ʹؾΛ͚͍ͭͯΔ͜ͱ - ඒຯ͍͓͠ञͷબͼํ
68. RejectCon ʢඇެࣜʣ΍Γ·͢
69. Kubernetes ΛಡΜͩΓ ݄༵໷ʹ microservices ͷ࿩Λͨ͠Γͯ͠·͢ Go ΁ͷѪΛޠͬͨΓ΋͠·͢