Alchemist and D/OX

Transcript

1. Alchemist and D/OX
   (developer / operation experience)
   ElixirConfJP2019 kokurajo
   Tsunenori Ohara
   

   View Slide

2. agenda
   ● about me
   ● about us
   ● goal of this talk
   Part 1. D/OX
   ● what is D/OX?
   ● development cycle
   ● how to improve D/OX?
   ● 4 automations
   ○ testing
   ○ scaling
   ○ deployment ☆
   ○ logging/monitoring
   Part 2. Elixir Deployment
   ● modern web app infra
   ● elixir impedance mismatch
   ● hot code swap or CD?
   ● our deployment journey
   ● elixir deployment tao
   Part3. Summary
   ● conclusion
   

   View Slide

3. about me
   ● Tsunenori Ohara
   ○ Twitter: @ohrdev
   ○ Github: ohr486
   ● Community
   ○ beam-lang.tokyo, tokyo.ex, Erlang & Elixir Fest organizer
   ○ meguro.rb, meguro.es, etc
   ● Work
   ○ enza platform division, platform development, general manager
   ○ technology officer
   ○ architect, infra/app developer, SRE
   ○ using elixir since Elixir v1.0 in production
   ● Hobby
   ○ making buddha statue
   ○ log collection (wooden logs)
   ○ sutra copying
   

   View Slide

4. about us
   ● Drecom Co., Ltd.
   ● using Elixir since v1.0 was released (2014〜
   ● we use elixir with
   ○ Framework: maru, phoenix
   ○ Infra/Cloud: AWS
   ○ Compute: EC2, kubernetes/EKS, lambda
   ○ Error Tracking: sentry
   ○ APM: newrelic, appsignal, prometheus
   ○ DB/KVS: Dynamodb, Redis, MySQL
   ○ Stream/Messaging: Kafka, Kinesis
   ● [AD] Tech Inside Drecom (our tech blog)
   ○ https://tech.drecom.co.jp
   

   View Slide

5. goal of this talk
   ● understand DX/OX
   ● understand how to improve DX/OX
   ● understand how to deploy elixir apps
   ● understand how to automate development works with elixir
   ● introducing our trial and error of elixir app’s deployment
   attention!
   - This talk has little elixir source code.
   

   View Slide

6. part 1. D/OX
   

   View Slide

7. what is D/OX?
   ● Developer Experience
   ○ An indicator of how comfortable the engineer can develop the system
   ● Operation Experience
   ○ An indicator of how comfortable the engineer can operate the system
   ● Good D/OX
   ○ Development/Operation is fun!
   ○ Small technical debt
   ○ Automation
   ● Bad D/OX
   ○ Development/Operation is painful!
   ○ Big/Huge technical debt
   ○ Manual operations
   ○ D/OX will deteriorate if left unattended
   

   View Slide

8. business quarity cost delivery
   development cycle
   development
   operation
   product value
   Good D/OX Bad D/OX
   anxiery/worry
   afford/margin bad bihaviour
   KAIZEN
   behaviour
   tao
   tech debt
   refactor
   redesign
   resources
   - engineer
   - money
   - time
   

   View Slide

9. how to improve D/OX
   ● Good DX?
   ○ automate manual tasks
   ■ testing
   ■ code analyze
   ○ running on dev env == running on prod env
   ○ NOT legacy
   ● Good OX?
   ○ automate manual tasks
   ■ delivery
   ● deployment tactics
   ○ blue green, canary, hot code swap, etc
   ■ scaling
   ○ error handling / trouble shooting
   ○ easy middleware version up
   Automation
   and
   Good archtecture
   

   View Slide

10. 4 automations
    ● testing
    ● scaling
    ● deployment
    ● log tracking
    

    View Slide

11. automation: testing
    ● adopt in initial development phase
    ● run tests with CI
    ○ unit test: exunit
    ○ lint: credo/dogma
    ○ static analysis: dialyxir
    ○ coverage: excoveralls
    ● load test / stress test
    ○ setup “stress” MIX_ENV
    ○ same level as dev env, test env and prod env
    ○ stress branch => hard to cherry-pick
    ○ stress MIX_ENV => easy to manage
    ● frequent load testing reduce performance problems
    ○ maintain load test scenario like unit tests
    ○ easy to run load tests
    

    View Slide

12. automation: scaling
    ● auto scaling
    ○ cloud: Auto Scaling Group
    ○ k8s: horizontal pod autoscaling
    ● monitoring
    ○ APM(application performance monitoring): newrelic, prometheus, appsignal
    ● deployment
    ○ deployment methods that support auto scaling => Part 2.
    

    View Slide

13. automation: deployment
    ● support auto scaling
    ● => part 2.
    

    View Slide

14. automation: logging
    ● not place on server
    ● use log collector
    ○ to S3, GCS
    ● check the price for cloud service
    ○ cloudwatch logs
    ○ stackdriver
    ● easy to bulk search
    ○ AWS athena
    ○ bigquery
    ● monitoring
    ○ APM: newrelic, appsignal, prometheus
    ■ BEAM metrics
    ○ Error: sentry
    appsignal
    newrelic
    

    View Slide

15. part 2. Elixir Deployment
    

    View Slide

16. modern web app infra
    ● public cloud (AWS/GCP/Azure/etc)
    ○ The server stops.
    ○ The network is going down.
    ○ Replace the server if it breaks.
    ■ AWS: EC2 gacha
    ● cloud native
    ○ devops
    ○ continuous delivery
    ○ containers
    ○ microservice
    ● running on docker/k8s
    ● serverless
    

    View Slide

17. elixir impedance mismatch
    ● elixir/erlang architecture
    ○ high cpu efficiency
    ○ hot code swap
    ○ umbrella apps on VM
    ● modern infra/k8s architecture
    ○ run many (small cpu/mem) pods
    ○ immutable infrastructure
    ○ microservices
    

    View Slide

18. [erlang vm architecture]
    Linux Server
    Erlang VM
    CPU CPU CPU CPU CPU CPU
    as many schedulers as the number of CPU
    erlang process != OS process
    vm migrate processes between schedulers
    scheduler process
    

    View Slide

19. elixir impedance mismatch
    Node(Linux Server)
    Erlang VM
    CPU CPU CPU CPU
    sche
    duler
    CPU
    sche
    duler
    sche
    duler
    sche
    duler
    sche
    duler
    P
    P
    P
    P
    P
    P
    P
    Node
    master
    Node Node
    k8s cluster
    pod
    pod
    pod
    pod
    pod
    pod
    container
    container
    container
    Erlang VM
    CPU CPU
    scheduler scheduler
    high CPU efficiency
    run many (small
    CPU/Mem) Pods
    k8s allocate
    CPU&Mem
    high infra cost
    efficiency
    

    View Slide

20. [process architecture]
    Erlang VM
    application controller application master
    application
    process
    ・elixir, iex, mix, etc
    ・user apps
    ・lib apps
    

    View Slide

21. Node(Linux Server)
    Node(Linux Server)
    elixir impedance mismatch
    Node(Linux Server)
    Erlang VM
    Node
    master
    Node Node
    k8s cluster
    app1
    pod
    container
    container
    container
    Erlang VM
    application controller
    application
    master
    app1 app2 common
    common
    app1
    app1
    app2
    app3
    app3
    app3
    app2
    

    View Slide

22. elixir impedance mismatch
    ● VM resource with : large or small resource(CPU,Mem)?
    ○ running on Instance : => large resource
    ○ running on k8s : => small resource
    ● load test and monitoring
    ○ DON’T GUESS, MEASURE
    ● easy to (load) test
    ● monitoring BEAM metrics
    

    View Slide

23. hot code swap or CD?
    ● running on instance
    ○ hot code swap
    ■ hard way
    ● auto scaling
    ● deployment task
    ○ CD / Immutable infrastructure is easy to apply
    ● running on k8s
    ○ CD / Immutable infrastructure
    

    View Slide

24. our deployment journey
    ● phase0 : local PC / iex -S mix
    ● phase1 : local PC / daemonize, mix release
    ● phase2 : single server / ssh login, git pull, mix release
    ● phase3 : single server / deploy with CI
    ● phase4 : multi server / package on deploy server, use S3 for delivery
    ● phase5 : multi server / deploy with autoscaling
    ● phase6 : multi server / deploy with edeliver/distirelly
    ● phase7 : multi server / deploy with autoscaling
    ● phase8 : k8s cluster
    

    View Slide

25. phase 0. local PC / iex -S mix
    local PC
    $ mix new xxx
    $ MIX_ENV=prod iex -S mix
    

    View Slide

26. phase 1. local PC / daemonize
    local PC
    $ mix new test_app
    $
    # Elixir 1.9
    $ MIX_ENV=prod mix release
    $ _build/prod/rel/test_app/bin/test_app daemon
    

    View Slide

27. phase 2. single server / ssh, git, mix release
    local PC
    Server
    $ git pull
    $ _build/prod/rel/test_app/bin/test_app stop
    $ MIX_ENV=prod mix release
    $ _build/prod/rel/test_app/bin/test_app start
    1 git push
    2 ssh login
    3 git pull
    4 release & restart
    

    View Slide

28. phase 3. single server / update with CI
    local PC
    Server
    $ git pull
    $ _build/prod/rel/test_app/bin/test_app stop
    $ MIX_ENV=prod mix release
    $ _build/prod/rel/test_app/bin/test_app start
    1 git push
    4 git pull
    5 release & restart
    2 trigger
    3 ssh loging
    

    View Slide

29. phase 4. multi server / upload release file to S3
    local PC
    Build Server
    1 git push 4 git pull
    2 trigger
    3 ssh login
    $ git pull
    $ MIX_ENV=prod mix release
    $ s3mcd put _build/prod/rel
    5 release & upload
    Server
    6 ssh login
    7 download rel & restart
    $ _build/prod/rel/test_app/bin/test_app stop
    $ s3cmd get
    $ _build/prod/rel/test_app/bin/test_app start
    

    View Slide

30. phase 5. multi server / deploy with auto scaling
    local PC
    Build Server
    1 git push 4 git pull
    2 trigger
    3 ssh login
    $ git pull
    $ MIX_ENV=prod mix release
    $ s3mcd put _build/prod/rel
    5 release & upload
    Server
    6 ssh login
    7 download rel & restart
    $ _build/prod/rel/test_app/bin/test_app stop
    $ s3cmd get
    $ _build/prod/rel/test_app/bin/test_app start
    NEW
    Server!
    8 autoscaling
    9 [in init script]
    download rel
    & start
    

    View Slide

31. phase 6. multi server / deploy with edeliver/distirelly
    local PC
    Build
    Server
    1 git push 4 git pull
    2 trigger
    3 edeliver build
    $ git pull
    $ mix edeliver build
    $ mix edeliver deploy
    $ mix edeliver restart
    5 release & upload
    Server
    6 develiver deploy
    7 edeliver restart
    tar file
    5 download tar
    tar file
    

    View Slide

32. phase 7. multi server / deploy with auto scaling
    local PC
    Build
    Server
    1 git push 4 git pull
    2 trigger
    3 edeliver build
    $ git pull
    $ s3cmd put tar
    $ mix edeliver build
    $ mix edeliver deploy
    $ mix edeliver restart
    Server
    7 develiver deploy
    8 edeliver restart
    tar file
    5 download tar
    NEW
    Server!
    9 autoscaling
    6 upload tar
    tar file
    10 [in init script]
    download tar
    & start
    $ s3cmd get
    $ /bin/test_app start
    

    View Slide

33. phase 8. k8s cluster
    local PC
    1 git push
    2 trigger
    $ git pull
    $ docker build --build-arg MIX_ENV=prod
    $ docker tag test_app xxx
    $ docker push xxx
    3 docker build&tag
    4 docker push
    5 kubectl apply -f xxx.yaml
    6 docker image pull
    7 k8s update pods
    8 scaling by HPA
    simple!
    $ MIX_ENV=prod mix release
    $ cp _build/prod/rel/test_app /
    $ /bin/test_app start
    

    View Slide

34. elixir deployment tao
    ● use CI for deployment
    ● use convenient infrastructure
    ○ k8s hides deployment complexity
    ● don’t solve the problem with elixir alone
    ○ we have awesome cloud services
    

    View Slide

35. part 3. Summary
    

    View Slide

36. conclusion
    ● Good D/OX improve product quality
    ● Good D/OX == Easy to use infrastructure / apps
    ● Automation!
    ● deployment is big automation issue
    ● choose the deployment method that matches your architecture
    

    View Slide