Alchemist and D/OX (developer / operation experience) ElixirConfJP2019 kokurajo Tsunenori Ohara

agenda ● about me ● about us ● goal of this talk Part 1. D/OX ● what is D/OX? ● development cycle ● how to improve D/OX? ● 4 automations ○ testing ○ scaling ○ deployment ☆ ○ logging/monitoring Part 2. Elixir Deployment ● modern web app infra ● elixir impedance mismatch ● hot code swap or CD? ● our deployment journey ● elixir deployment tao Part3. Summary ● conclusion

about me ● Tsunenori Ohara ○ Twitter: @ohrdev ○ Github: ohr486 ● Community ○ beam-lang.tokyo, tokyo.ex, Erlang & Elixir Fest organizer ○ meguro.rb, meguro.es, etc ● Work ○ enza platform division, platform development, general manager ○ technology officer ○ architect, infra/app developer, SRE ○ using elixir since Elixir v1.0 in production ● Hobby ○ making buddha statue ○ log collection (wooden logs) ○ sutra copying

about us ● Drecom Co., Ltd. ● using Elixir since v1.0 was released (2014〜 ● we use elixir with ○ Framework: maru, phoenix ○ Infra/Cloud: AWS ○ Compute: EC2, kubernetes/EKS, lambda ○ Error Tracking: sentry ○ APM: newrelic, appsignal, prometheus ○ DB/KVS: Dynamodb, Redis, MySQL ○ Stream/Messaging: Kafka, Kinesis ● [AD] Tech Inside Drecom (our tech blog) ○ https://tech.drecom.co.jp

goal of this talk ● understand DX/OX ● understand how to improve DX/OX ● understand how to deploy elixir apps ● understand how to automate development works with elixir ● introducing our trial and error of elixir app’s deployment attention! - This talk has little elixir source code.

part 1. D/OX

what is D/OX? ● Developer Experience ○ An indicator of how comfortable the engineer can develop the system ● Operation Experience ○ An indicator of how comfortable the engineer can operate the system ● Good D/OX ○ Development/Operation is fun! ○ Small technical debt ○ Automation ● Bad D/OX ○ Development/Operation is painful! ○ Big/Huge technical debt ○ Manual operations ○ D/OX will deteriorate if left unattended

business quarity cost delivery development cycle development operation product value Good D/OX Bad D/OX anxiery/worry afford/margin bad bihaviour KAIZEN behaviour tao tech debt refactor redesign resources - engineer - money - time

how to improve D/OX ● Good DX? ○ automate manual tasks ■ testing ■ code analyze ○ running on dev env == running on prod env ○ NOT legacy ● Good OX? ○ automate manual tasks ■ delivery ● deployment tactics ○ blue green, canary, hot code swap, etc ■ scaling ○ error handling / trouble shooting ○ easy middleware version up Automation and Good archtecture

4 automations ● testing ● scaling ● deployment ● log tracking

automation: testing ● adopt in initial development phase ● run tests with CI ○ unit test: exunit ○ lint: credo/dogma ○ static analysis: dialyxir ○ coverage: excoveralls ● load test / stress test ○ setup “stress” MIX_ENV ○ same level as dev env, test env and prod env ○ stress branch => hard to cherry-pick ○ stress MIX_ENV => easy to manage ● frequent load testing reduce performance problems ○ maintain load test scenario like unit tests ○ easy to run load tests

automation: scaling ● auto scaling ○ cloud: Auto Scaling Group ○ k8s: horizontal pod autoscaling ● monitoring ○ APM(application performance monitoring): newrelic, prometheus, appsignal ● deployment ○ deployment methods that support auto scaling => Part 2.

automation: deployment ● support auto scaling ● => part 2.

automation: logging ● not place on server ● use log collector ○ to S3, GCS ● check the price for cloud service ○ cloudwatch logs ○ stackdriver ● easy to bulk search ○ AWS athena ○ bigquery ● monitoring ○ APM: newrelic, appsignal, prometheus ■ BEAM metrics ○ Error: sentry appsignal newrelic

part 2. Elixir Deployment

modern web app infra ● public cloud (AWS/GCP/Azure/etc) ○ The server stops. ○ The network is going down. ○ Replace the server if it breaks. ■ AWS: EC2 gacha ● cloud native ○ devops ○ continuous delivery ○ containers ○ microservice ● running on docker/k8s ● serverless

elixir impedance mismatch ● elixir/erlang architecture ○ high cpu efficiency ○ hot code swap ○ umbrella apps on VM ● modern infra/k8s architecture ○ run many (small cpu/mem) pods ○ immutable infrastructure ○ microservices

[erlang vm architecture] Linux Server Erlang VM CPU CPU CPU CPU CPU CPU as many schedulers as the number of CPU erlang process != OS process vm migrate processes between schedulers scheduler process

elixir impedance mismatch Node(Linux Server) Erlang VM CPU CPU CPU CPU sche duler CPU sche duler sche duler sche duler sche duler P P P P P P P Node master Node Node k8s cluster pod pod pod pod pod pod container container container Erlang VM CPU CPU scheduler scheduler high CPU efficiency run many (small CPU/Mem) Pods k8s allocate CPU&Mem high infra cost efficiency

[process architecture] Erlang VM application controller application master application process ・elixir, iex, mix, etc ・user apps ・lib apps

Node(Linux Server) Node(Linux Server) elixir impedance mismatch Node(Linux Server) Erlang VM Node master Node Node k8s cluster app1 pod container container container Erlang VM application controller application master app1 app2 common common app1 app1 app2 app3 app3 app3 app2

elixir impedance mismatch ● VM resource with : large or small resource(CPU,Mem)? ○ running on Instance : => large resource ○ running on k8s : => small resource ● load test and monitoring ○ DON’T GUESS, MEASURE ● easy to (load) test ● monitoring BEAM metrics

hot code swap or CD? ● running on instance ○ hot code swap ■ hard way ● auto scaling ● deployment task ○ CD / Immutable infrastructure is easy to apply ● running on k8s ○ CD / Immutable infrastructure

our deployment journey ● phase0 : local PC / iex -S mix ● phase1 : local PC / daemonize, mix release ● phase2 : single server / ssh login, git pull, mix release ● phase3 : single server / deploy with CI ● phase4 : multi server / package on deploy server, use S3 for delivery ● phase5 : multi server / deploy with autoscaling ● phase6 : multi server / deploy with edeliver/distirelly ● phase7 : multi server / deploy with autoscaling ● phase8 : k8s cluster

phase 0. local PC / iex -S mix local PC $ mix new xxx $ MIX_ENV=prod iex -S mix

phase 1. local PC / daemonize local PC $ mix new test_app $ # Elixir 1.9 $ MIX_ENV=prod mix release $ _build/prod/rel/test_app/bin/test_app daemon

phase 2. single server / ssh, git, mix release local PC Server $ git pull $ _build/prod/rel/test_app/bin/test_app stop $ MIX_ENV=prod mix release $ _build/prod/rel/test_app/bin/test_app start 1 git push 2 ssh login 3 git pull 4 release & restart

phase 3. single server / update with CI local PC Server $ git pull $ _build/prod/rel/test_app/bin/test_app stop $ MIX_ENV=prod mix release $ _build/prod/rel/test_app/bin/test_app start 1 git push 4 git pull 5 release & restart 2 trigger 3 ssh loging

phase 4. multi server / upload release file to S3 local PC Build Server 1 git push 4 git pull 2 trigger 3 ssh login $ git pull $ MIX_ENV=prod mix release $ s3mcd put _build/prod/rel 5 release & upload Server 6 ssh login 7 download rel & restart $ _build/prod/rel/test_app/bin/test_app stop $ s3cmd get $ _build/prod/rel/test_app/bin/test_app start

phase 5. multi server / deploy with auto scaling local PC Build Server 1 git push 4 git pull 2 trigger 3 ssh login $ git pull $ MIX_ENV=prod mix release $ s3mcd put _build/prod/rel 5 release & upload Server 6 ssh login 7 download rel & restart $ _build/prod/rel/test_app/bin/test_app stop $ s3cmd get $ _build/prod/rel/test_app/bin/test_app start NEW Server! 8 autoscaling 9 [in init script] download rel & start

phase 6. multi server / deploy with edeliver/distirelly local PC Build Server 1 git push 4 git pull 2 trigger 3 edeliver build $ git pull $ mix edeliver build $ mix edeliver deploy $ mix edeliver restart 5 release & upload Server 6 develiver deploy 7 edeliver restart tar file 5 download tar tar file

phase 7. multi server / deploy with auto scaling local PC Build Server 1 git push 4 git pull 2 trigger 3 edeliver build $ git pull $ s3cmd put tar $ mix edeliver build $ mix edeliver deploy $ mix edeliver restart Server 7 develiver deploy 8 edeliver restart tar file 5 download tar NEW Server! 9 autoscaling 6 upload tar tar file 10 [in init script] download tar & start $ s3cmd get $ /bin/test_app start

phase 8. k8s cluster local PC 1 git push 2 trigger $ git pull $ docker build --build-arg MIX_ENV=prod $ docker tag test_app xxx $ docker push xxx 3 docker build&tag 4 docker push 5 kubectl apply -f xxx.yaml 6 docker image pull 7 k8s update pods 8 scaling by HPA simple! $ MIX_ENV=prod mix release $ cp _build/prod/rel/test_app / $ /bin/test_app start

elixir deployment tao ● use CI for deployment ● use convenient infrastructure ○ k8s hides deployment complexity ● don’t solve the problem with elixir alone ○ we have awesome cloud services

part 3. Summary

conclusion ● Good D/OX improve product quality ● Good D/OX == Easy to use infrastructure / apps ● Automation! ● deployment is big automation issue ● choose the deployment method that matches your architecture