Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

SysadminDays - Nestor

Saâd Dif
November 19, 2019

SysadminDays - Nestor

Déploiement et gestion du cycle de vie de micro-services sur Kubernetes avec un outil maison

Kapten's tech team is looking for a tool to deploy and manage their microservices. The quest to find a sharp tool looked gloomy at that time without any existing tool answering all the pressing needs for our knights. Knights of the round table gathered and embraced the challenge to give birth to Nestor. One tool to rule them all!

Nestor is our valiant tool allowing every soul in the tech kingdom to interact with microservices without the need to be a container or orchestration wizard.

Our fearless tech team uses it daily, its goal is simple yet complex: Manage, configure and deploy our audacious microservices stack, which is growing at the speed of light.

Saâd Dif

November 19, 2019
Tweet

Other Decks in Programming

Transcript

  1. Agenda_ 3 Kingdom of Kapten_ Once upon a time_ Uh

    oh_ Nestor_ They lived happily_
  2. 2016 180 160 140 120 100 80 60 40 20

    0 2017 2018 48,6 100 160 Revenue 2016 – 2017 – 2018 Kapten_Growth 7 3 millions customers 400 collaborators 50 000 partner drivers 3000 client companies french ride hailing leader +60%
  3. Kapten_Production 8 1TB storage 60k metrics 15k inc / min

    100TB storage 780k logs / min 50B lines of log 100+ Nodes 130+ µ-services 20+ deploy / day 1k+ processes
  4. Agenda_ 9 Kingdom of Kapten_ Once upon a time_ Uh

    oh_ Nestor_ They lived happily_
  5. Heroku_ 10 • Why ? - PaaS over IaaS -

    Easy to handle and scale for dev - Ready to use - AutoBuild for various languages - Everything as a Service • Configuration management ? - Git repositories letsdeploy/letsdeploy-config - Hand provisioned servers • Deployments ? - Letsdeploy
  6. Letsdeploy-config_ 12 • One file per micro service • Three

    sections for each configuration file { "source": "[email protected]:transcovo/nestor-charybde.git", "target": { "heroku_app_name": "prod-nestor-charybde-cp-eu", "type": "heroku" }, "variables": { "AWS_ACCESS_KEY_ID": "*****", "AWS_SECRET_ACCESS_KEY": "*****", "LOGGER_LEVEL": "info", "LOGGER_NAME": "production.nestor-charybde", "NODE_ENV": "production", "NPM_TOKEN": "*****", "SLACK_TOKEN": "*****" } }
  7. Agenda_ 15 Kingdom of Kapten_ Once upon a time_ Uh

    oh_ Nestor_ They lived happily_
  8. Get out of Heroku_ 18 • Ending contract • Move

    to Kubernetes ◦ Scalability ◦ Costs ◦ Real infra ◦ Security
  9. What we need_ 19 • Simple and easy deployment tool

    ◦ Command wrapper ◦ Same tool for every environments ◦ End users: Our developers
  10. What we need_ 20 12 Factors (12Factor.net) • Build, release,

    run • Processes • Port Binding • Concurrency • Disposability • Dev/Prod parity • Logs • Admin processes • Codebase • Dependencies • Config • Backing services
  11. What we need_ 21 12Factor.net • Our microservices should respect

    those principles • The tool should help us doing so
  12. The clock is ticking_ 23 • Time constraint • Need

    to scale fast • Ansible / Terraform
  13. Agenda_ 24 Kingdom of Kapten_ Once upon a time_ Uh

    oh_ Nestor_ They lived happily_
  14. Nestor_ 25 • Rewrite existing tool • Knowledge already present

    in all teams • First usecase: dev environments
  15. Nestor_ 28 CLI: • Workflow management • Release related commands

    • Configuration • Datastore management
  16. Nestor_ 30 API: • NodeJS • Manage workflow • Triggered

    by CI to initiate builds • Push images to DockerHUB
  17. Nestor_ 33 CRON: • All environments updated from “staging” apps

    versions • Releases on Staging every 30 minutes • Datastore snapshot and reset of all dev environments • Apps configuration from staging sync to dev environments
  18. Nestor_ 34 Configuration built dynamically: • Kubernetes templates • “project.yaml”

    file for global configuration • File for each micro service • Children environments • Merge of files
  19. Nestor_ 35 Kubernetes templates: $ tree nestor-config/templates nestor-config/templates ├── anti-affinity-node.yaml

    ├── anti-affinity-zone.yaml ├── config-map.yaml ├── cronjob.yaml ├── deployment.yaml ├── hpa.yaml ├── ingress-app.yaml ├── ingress-global.yaml ├── ingress-nginx-global.yaml ├── job.yaml ├── namespace.yaml ├── nginx.conf ├── secret-tls.yaml └── service.yaml 0 directories, 14 files $ cat nestor-config/templates/service.yaml apiVersion: v1 kind: Service metadata: name: '{{name}}' labels: app: '{{app}}' spec: ports: - port: 80 targetPort: {{target_port}} selector: app: '{{app}}' process: web type: ClusterIP
  20. Nestor_ 36 project.yaml file: $ cat nestor-config/project.yaml project: kapten env:

    production domain: production.kapten.com docker: build: variables: NPM_TOKEN: $NPM_TOKEN registries: docker.com: - id: docker organization: kapten ... ... deployments: kubernetes: - cluster_name: kapten_production_eu-w1 hpa_replicas: true scales: web: minReplicas: 3 maxReplicas: 3 resources: web: limits: memory: 256Mi cpu: 0.2 nodeSelector: default: tier: app ... ... variables: ope: NODE_ENV: 'production' METRICS_DESTINATION: metrics.kapten.com slack: token: '**********' channels: info: tech-release-prod error: tech-release-prod ...
  21. Nestor_ 37 App configuration file: app: mary git: origin: [email protected]:transcovo/mary.git

    is_enabled: true resources: web: limits: cpu: 0.2 memory: 250Mi requests: cpu: 0.2 memory: 250Mi scales: web: maxReplicas: 9 minReplicas: 3 targetCPUUtilizationPercentage: 75 teams: - security variables: app: {} ope: SENTRY_DSN: "********"
  22. Nestor_ 38 Monitoring and alerting in configuration files: • Routing

    • Threshold Config validation templateVars: tplCriticity: high tplTeam: security tplWeb2xxThreshold: "0" tplWeb50thLatencyThreshold: "0.30" tplWeb95thLatencyThreshold: "2"
  23. Nestor_ 39 History management • Nestor history • History saved

    on specific repository • Used for rollbacks
  24. Nestor_ 41 Most used features by developers: • Deploy specific

    branch • Port forwarding • Switch between environments • Datastore management
  25. From code... Workflow with Nestor_ 42 e2e tests Load tests

    Unit tests Testing env. Monitoring To prod... ...in minutes with nestor_!
  26. Workflow_ dev 1 dev X . . . dev 2

    Master Staging GreenLight Production Peer Review Terminator Shadow CircleCI success (master branch) • Create a git tag • Build docker Image • Rebase Greenlight branch from Master Legend • Rebase Terminator, Staging and Shadow from Greenlight Nestor-api GL success • Rebase Production from Shadow Rundeck Deploy Nestor-api Rebase and move git tag Nestor-api Call nestor-api
  27. Agenda_ 44 Kingdom of Kapten_ Once upon a time_ Uh

    oh_ Nestor_ They lived happily_
  28. To sum up_ 45 • From PaaS to IaaS •

    From deployment workflow to release and config management • Migration took 4 months
  29. Thoughts_ 46 • We needed Nestor • Making your own

    tool / wrapper is not a shame • Answered a specific need at a specific time
  30. Next steps_ 48 Helm ? • Community based • No

    need to rewrite what is already here • New V3 • Stable and reliable
  31. Next steps_ 49 To study: • Debug pod live (Telepresence

    / Monday) • Make it available for everyone ?