Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Code Review goes Cloud Native - Gerrit on Kubernetes managed by Gardener

msohn
November 16, 2018

Code Review goes Cloud Native - Gerrit on Kubernetes managed by Gardener

Gerrit User Summit 2018 Palo Alto on Nov 16, 2018
https://gerrit.googlesource.com/summit/2018/+/master/index.md

msohn

November 16, 2018
Tweet

More Decks by msohn

Other Decks in Technology

Transcript

  1. Code Review goes Cloud Native
    Gerrit on Kubernetes
    managed by Gardener

    View Slide

  2. Gerrit Master
    MySQL
    Master
    Fetch
    Gerrit Master
    git gc
    Push
    Git Repositories
    read
    write
    gc

    View Slide

  3. Add Slaves
    MySQL
    Master
    Fetch
    Gerrit Master
    git gc
    Push
    Git Repositories
    read
    write
    gc

    View Slide

  4. Add Slaves
    MySQL
    Master
    Fetch
    Gerrit Master
    git gc
    Replication
    Plugin
    Push
    Git Repositories
    read
    write
    gc

    View Slide

  5. Add Slaves
    MySQL
    Master
    Apache-Git-
    backend
    Fetch
    Gerrit Master
    git gc
    Replication
    Plugin
    Push
    Git Repositories
    read
    write
    gc

    View Slide

  6. Add Slaves
    MySQL
    Master
    Apache-Git-
    backend
    Fetch
    Gerrit Master
    git gc
    Replication
    Plugin
    Push
    Git Repositories
    read
    write
    gc
    read

    View Slide

  7. Add Slaves
    MySQL
    Master
    Apache-Git-
    backend
    Git Repositories
    write
    Fetch
    Gerrit Master
    git gc
    Replication
    Plugin
    Push
    Git Repositories
    read
    write
    gc
    read
    ssh: create repos

    View Slide

  8. Add Slaves
    MySQL
    Master
    Apache-Git-
    backend
    https: git push
    Git Repositories
    write
    Fetch
    Gerrit Master
    git gc
    Replication
    Plugin
    Push
    Git Repositories
    read
    write
    gc
    read
    ssh: create repos

    View Slide

  9. Add Slaves
    MySQL
    Master
    Apache-Git-
    backend
    MySQL
    Slave
    https: git push
    Git Repositories
    write
    Fetch
    Gerrit Master
    git gc
    Replication
    Plugin
    Push
    Git Repositories
    read
    write
    gc
    read
    ssh: create repos

    View Slide

  10. Add Slaves
    MySQL
    Master
    Apache-Git-
    backend
    MySQL
    Slave
    https: git push
    fetch transaction log
    Git Repositories
    write
    Fetch
    Gerrit Master
    git gc
    Replication
    Plugin
    Push
    Git Repositories
    read
    write
    gc
    read
    ssh: create repos

    View Slide

  11. Add Slaves
    MySQL
    Master
    Apache-Git-
    backend
    Gerrit Slave
    MySQL
    Slave
    https: git push
    fetch transaction log
    Git Repositories
    write
    Fetch
    Gerrit Master
    git gc
    Replication
    Plugin
    Push
    Git Repositories
    read
    write
    gc
    read
    ssh: create repos

    View Slide

  12. Add Slaves
    MySQL
    Master
    Apache-Git-
    backend
    Gerrit Slave
    MySQL
    Slave
    https: git push
    fetch transaction log
    Git Repositories
    write read
    Fetch
    Gerrit Master
    git gc
    Replication
    Plugin
    Push
    Git Repositories
    read
    write
    gc
    read
    ssh: create repos

    View Slide

  13. Add Slaves
    MySQL
    Master
    Apache-Git-
    backend
    Gerrit Slave
    MySQL
    Slave
    https: git push
    fetch transaction log
    Git Repositories
    write read
    Fetch
    Fetch
    Gerrit Master
    git gc
    Replication
    Plugin
    Push
    Git Repositories
    read
    write
    gc
    read
    ssh: create repos

    View Slide

  14. Add Slaves
    MySQL
    Master
    Apache-Git-
    backend
    git gc
    Gerrit Slave
    MySQL
    Slave
    https: git push
    fetch transaction log
    Git Repositories
    write gc read
    Fetch
    Fetch
    Gerrit Master
    git gc
    Replication
    Plugin
    Push
    Git Repositories
    read
    write
    gc
    read
    ssh: create repos

    View Slide

  15. Add Slaves
    MySQL
    Master
    Apache-Git-
    backend
    git gc
    Gerrit Slave
    MySQL
    Slave
    https: git push
    fetch transaction log
    Git Repositories
    write gc read
    Fetch
    Fetch
    Gerrit Master
    git gc
    Replication
    Plugin
    Push
    Git Repositories
    read
    write
    gc
    read
    ssh: create repos

    View Slide

  16. Add Slaves
    MySQL
    Master
    Apache-Git-
    backend
    git gc
    Gerrit Slave
    MySQL
    Slave
    https: git push
    fetch transaction log
    Git Repositories
    write gc read
    Fetch
    Fetch
    Gerrit Master
    git gc
    Replication
    Plugin
    Push
    Git Repositories
    read
    write
    gc
    read
    ssh: create repos

    View Slide

  17. Dockerize Slaves
    MySQL
    Master
    Apache-Git-
    backend
    git gc
    Gerrit Slave
    MySQL
    Slave
    ssh: create repos
    https: git push
    fetch transaction log
    Git Repositories
    write gc read
    Fetch
    Fetch
    Gerrit Master
    git gc
    Replication
    Plugin
    Push
    Git Repositories
    read
    write
    gc
    read

    View Slide

  18. From Containers to Kubernetes

    View Slide

  19. From Containers to Kubernetes
    VM
    Host OS
    Container Runtime
    Container

    View Slide

  20. From Containers to Kubernetes
    VM
    Host OS
    Container Runtime
    Container

    View Slide

  21. From Containers to Kubernetes
    VM
    Host OS
    Container Runtime
    Benefits
    Isolation
    Immutable infrastructure
    Portability
    Faster deployments
    Versioning
    Ease of sharing
    Container

    View Slide

  22. From Containers to Kubernetes
    VM
    Host OS
    Container Runtime
    Benefits
    Isolation
    Immutable infrastructure
    Portability
    Faster deployments
    Versioning
    Ease of sharing
    Challenges
    Networking
    Security
    Deployments
    Service Discovery
    Auto Scaling
    Persisting Data
    Logging, Monitoring
    Access Control
    Container

    View Slide

  23. From Containers to Kubernetes
    VM
    Host OS
    Container Runtime
    Benefits
    Isolation
    Immutable infrastructure
    Portability
    Faster deployments
    Versioning
    Ease of sharing
    Challenges
    Networking
    Security
    Deployments
    Service Discovery
    Auto Scaling
    Persisting Data
    Logging, Monitoring
    Access Control
    Container Scheduler
    Container

    View Slide

  24. From Containers to Kubernetes
    VM
    Host OS
    Container Runtime
    Benefits
    Isolation
    Immutable infrastructure
    Portability
    Faster deployments
    Versioning
    Ease of sharing
    Challenges
    Networking
    Security
    Deployments
    Service Discovery
    Auto Scaling
    Persisting Data
    Logging, Monitoring
    Access Control
    Container Scheduler
    Container

    View Slide

  25. From Containers to Kubernetes
    VM
    Host OS
    Container Runtime
    Benefits
    Isolation
    Immutable infrastructure
    Portability
    Faster deployments
    Versioning
    Ease of sharing
    Challenges
    Networking
    Security
    Deployments
    Service Discovery
    Auto Scaling
    Persisting Data
    Logging, Monitoring
    Access Control
    Container Scheduler
    Container

    View Slide

  26. From Containers to Kubernetes
    VM
    Host OS
    Container Runtime
    Benefits
    Isolation
    Immutable infrastructure
    Portability
    Faster deployments
    Versioning
    Ease of sharing
    Challenges
    Networking
    Security
    Deployments
    Service Discovery
    Auto Scaling
    Persisting Data
    Logging, Monitoring
    Access Control
    Container Scheduler
    Container

    View Slide

  27. From Containers to Kubernetes
    VM
    Host OS
    Container Runtime
    Benefits
    Isolation
    Immutable infrastructure
    Portability
    Faster deployments
    Versioning
    Ease of sharing
    Challenges
    Networking
    Security
    Deployments
    Service Discovery
    Auto Scaling
    Persisting Data
    Logging, Monitoring
    Access Control
    Kubernetes
    Orchestration of cluster of containers across
    multiple hosts
    • Automatic placements, networking,
    deployments, scaling, roll-out/-back, A/B testing
    Container Scheduler
    Container

    View Slide

  28. From Containers to Kubernetes
    VM
    Host OS
    Container Runtime
    Benefits
    Isolation
    Immutable infrastructure
    Portability
    Faster deployments
    Versioning
    Ease of sharing
    Challenges
    Networking
    Security
    Deployments
    Service Discovery
    Auto Scaling
    Persisting Data
    Logging, Monitoring
    Access Control
    Kubernetes
    Orchestration of cluster of containers across
    multiple hosts
    • Automatic placements, networking,
    deployments, scaling, roll-out/-back, A/B testing
    Declarative – not procedural
    • Declare target state, reconcile to desired state
    • Self-healing
    Container Scheduler
    Container

    View Slide

  29. From Containers to Kubernetes
    VM
    Host OS
    Container Runtime
    Benefits
    Isolation
    Immutable infrastructure
    Portability
    Faster deployments
    Versioning
    Ease of sharing
    Challenges
    Networking
    Security
    Deployments
    Service Discovery
    Auto Scaling
    Persisting Data
    Logging, Monitoring
    Access Control
    Kubernetes
    Orchestration of cluster of containers across
    multiple hosts
    • Automatic placements, networking,
    deployments, scaling, roll-out/-back, A/B testing
    Docker
    Workload Portability
    • Abstract from cloud provider specifics
    • Multiple container runtimes
    Declarative – not procedural
    • Declare target state, reconcile to desired state
    • Self-healing
    Container Scheduler
    Container

    View Slide

  30. Gerrit-Slave Helm Chart
    MySQL Helm Chart
    PV
    Slaves on Kubernetes
    MySQL
    Master
    Kubernetes
    deployment
    Apache-Git
    CronJob
    git gc
    Persistent Volume
    Git Repositories
    Fetch
    Fetch
    Gerrit Master
    git gc
    Replication
    Plugin
    Push
    Git Repositories
    read
    write
    gc
    read
    deployment
    replica set
    Gerrit
    Slave
    NFS Helm Chart
    NFS Server
    MySQL
    Slave
    Storage
    ingress
    ingress
    PVC PVC PVC
    PVC

    View Slide

  31. Gerrit-Slave Helm Chart
    MySQL Helm Chart
    PV
    Kubernetes Cluster
    deployment
    Apache-Git
    CronJob
    git gc
    Persistent Volume
    Git Repositories
    Fetch
    deployment
    replica set
    Gerrit
    Slave
    MySQL
    Slave
    NFS Volume
    ingress ingress
    PVC PVC PVC
    PVC
    Gerrit-Master Helm Chart
    MySQL Helm Chart
    PV
    job
    git gc
    Persistent Volume
    Git Repositories
    deployment
    replica set
    Gerrit
    Master
    MySQL
    Master
    NFS Volume
    PVC PVC
    PVC
    Fetch
    Push
    Master-Slave on Kubernetes

    View Slide

  32. Gardener - Kubernetes Clusters as a Service
    Gardener

    View Slide

  33. Gardener - Kubernetes Clusters as a Service
    Worker
    Master
    Master
    Master
    Worker
    Worker
    HA
    Gardener Cluster
    Gardener

    View Slide

  34. Gardener - Kubernetes Clusters as a Service
    Worker
    Master
    Master
    Master
    Worker
    Worker
    HA
    Gardener Cluster
    Gardener

    View Slide

  35. Worker
    Seed Cluster
    Master
    Master
    Master
    Worker
    Worker
    HA
    Gardener - Kubernetes Clusters as a Service
    Worker
    Master
    Master
    Master
    Worker
    Worker
    HA
    Gardener Cluster
    Gardener

    View Slide

  36. Worker
    Seed Cluster
    Master
    Master
    Master
    Worker
    Worker
    HA
    Gardener - Kubernetes Clusters as a Service
    Worker
    Master
    Master
    Master
    Worker
    Worker
    HA
    Gardener Cluster
    Gardener

    View Slide

  37. Worker
    Seed Cluster
    Master
    Master
    Master
    Worker
    Worker
    HA
    Shoot Clusters
    Gardener - Kubernetes Clusters as a Service
    Worker
    Master
    Master
    Master
    Worker
    Worker
    HA
    Gardener Cluster
    Gardener

    View Slide

  38. Worker
    Seed Cluster
    Master
    Master
    Master
    Worker
    Worker
    HA
    Shoot Clusters
    Worker
    Worker
    Worker
    Gardener - Kubernetes Clusters as a Service
    Worker
    Master
    Master
    Master
    Worker
    Worker
    HA
    Gardener Cluster
    Gardener
    apiVersion: garden.sapcloud.io/v1
    kind: Shoot
    metadata:
    name: my-cluster
    namespace: garden-project
    spec:
    dns:
    provider: aws-route53
    domain: cluster.ondemand.com
    cloud:
    aws:
    networks:
    vpc:
    cidr: 10.250.0.0/16
    workers:
    - name: cpu-worker
    machineType: m4.xlarge
    autoScalerMin: 5
    autoScalerMax: 20
    kubernetes:
    version: 1.11.2
    kubeAPIServer:
    featureGates: ...
    runtimeConfig: ...
    admissionPlugins: ...
    maintenance:
    timeWindow:
    begin: 220000+0000
    end: 230000+0000
    autoUpdate:
    kubernetesVersion: true
    status:
    ...
    cluster.yaml

    View Slide

  39. Worker
    Seed Cluster
    Master
    Master
    Master
    Worker
    Worker
    HA
    Shoot Clusters
    Worker
    Worker
    Worker
    Gardener - Kubernetes Clusters as a Service
    Worker
    Master
    Master
    Master
    Worker
    Worker
    HA
    Gardener Cluster
    Gardener

    View Slide

  40. Worker
    Seed Cluster
    Master
    Master
    Master
    Worker
    Worker
    HA
    Shoot Clusters
    Worker
    Worker
    Worker Worker
    Worker
    Gardener - Kubernetes Clusters as a Service
    Worker
    Master
    Master
    Master
    Worker
    Worker
    HA
    Gardener Cluster
    Gardener

    View Slide

  41. Worker
    Seed Cluster
    Master
    Master
    Master
    Worker
    Worker
    HA
    Shoot Clusters
    Worker
    Worker
    Worker Worker
    Worker
    Worker
    Gardener - Kubernetes Clusters as a Service
    Worker
    Master
    Master
    Master
    Worker
    Worker
    HA
    Gardener Cluster
    Gardener

    View Slide

  42. Worker
    Seed Cluster
    Master
    Master
    Master
    Worker
    Worker
    HA
    Shoot Clusters
    Worker
    Worker
    Worker Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Gardener - Kubernetes Clusters as a Service
    Worker
    Master
    Master
    Master
    Worker
    Worker
    HA
    Gardener Cluster
    Gardener

    View Slide

  43. Worker
    Seed Cluster
    Master
    Master
    Master
    Worker
    Worker
    HA
    Shoot Clusters
    Worker
    Worker
    Worker Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Gardener - Kubernetes Clusters as a Service
    Worker
    Master
    Master
    Master
    Worker
    Worker
    HA
    Gardener Cluster
    Inside a Seed
    Cluster Worker
    Gardener
    Zooming into the
    Seed Cluster reveals…

    View Slide

  44. Worker
    Seed Cluster
    Master
    Master
    Master
    Worker
    Worker
    HA
    Shoot Clusters
    Worker
    Worker
    Worker Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Gardener - Kubernetes Clusters as a Service
    Multiple Shoot Cluster
    Control Planes
    Worker
    Master
    Master
    Master
    Worker
    Worker
    HA
    Gardener Cluster
    Inside a Seed
    Cluster Worker
    manages
    API Server
    Scheduler
    Controller Mgr
    ETCD
    Gardener
    Zooming into the
    Seed Cluster reveals…

    View Slide

  45. Worker
    Seed Cluster
    Master
    Master
    Master
    Worker
    Worker
    HA
    Shoot Clusters
    Worker
    Worker
    Worker Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Gardener - Kubernetes Clusters as a Service
    Multiple Shoot Cluster
    Control Planes
    Worker
    Master
    Master
    Master
    Worker
    Worker
    HA
    Gardener Cluster
    Inside a Seed
    Cluster Worker
    manages
    API Server
    ETCD
    Scheduler
    Controller Mgr
    API Server
    Scheduler
    Controller Mgr
    ETCD
    Gardener
    Zooming into the
    Seed Cluster reveals…

    View Slide

  46. Worker
    Seed Cluster
    Master
    Master
    Master
    Worker
    Worker
    HA
    Shoot Clusters
    Worker
    Worker
    Worker Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Gardener - Kubernetes Clusters as a Service
    Multiple Shoot Cluster
    Control Planes
    Worker
    Master
    Master
    Master
    Worker
    Worker
    HA
    Gardener Cluster
    Inside a Seed
    Cluster Worker
    manages
    API Server
    ETCD
    Scheduler
    Controller Mgr
    API Server
    Scheduler
    Controller Mgr
    ETCD
    ETCD …
    API Server
    Gardener
    Zooming into the
    Seed Cluster reveals…

    View Slide

  47. Worker
    Seed Cluster
    Master
    Master
    Master
    Worker
    Worker
    HA
    Shoot Clusters
    Worker
    Worker
    Worker Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Gardener Machine Controller Manager
    Machine Provisioning
    Self-Healing
    Auto-Update
    Auto-Scaling
    Gardener - Kubernetes Clusters as a Service
    Multiple Shoot Cluster
    Control Planes
    Worker
    Master
    Master
    Master
    Worker
    Worker
    HA
    Gardener Cluster
    Inside a Seed
    Cluster Worker
    manages
    API Server
    ETCD
    Scheduler
    Controller Mgr
    API Server
    Scheduler
    Controller Mgr
    ETCD
    ETCD …
    API Server
    Gardener
    Zooming into the
    Seed Cluster reveals…

    View Slide

  48. Worker
    Seed Cluster
    Master
    Master
    Master
    Worker
    Worker
    HA
    Shoot Clusters
    Worker
    Worker
    Worker Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Worker
    Gardener Machine Controller Manager
    Machine Provisioning
    Self-Healing
    Auto-Update
    Auto-Scaling
    Gardener - Kubernetes Clusters as a Service
    Multiple Shoot Cluster
    Control Planes
    Worker
    Master
    Master
    Master
    Worker
    Worker
    HA
    Gardener Cluster
    Inside a Seed
    Cluster Worker
    manages
    API Server
    ETCD
    Scheduler
    Controller Mgr
    API Server
    Scheduler
    Controller Mgr
    ETCD
    ETCD …
    API Server
    Gardener
    Zooming into the
    Seed Cluster reveals…

    View Slide

  49. View Slide

  50. Demo
    Gerrit on Kubernetes
    managed by Gardener

    View Slide

  51. Next steps
    Storage
    • use Ceph / Rook to replace NFS
    Update chart to enable using master branch
    • get rid of database

    View Slide

  52. Index
    Replace Lucene by ElasticSearch

    View Slide

  53. Gerrit Operator
    based on operator framework
    to implement
    • Sharding
    • Multi-master
    • Auto-Scaling

    View Slide

  54. Logging
    EFK-Stack
    • Elastic search => Indexing
    • Fluentbit => Log-collection
    • Kibana => Visualization

    View Slide

  55. Monitoring
    Prometheus => data collection
    Grafana => visualization

    View Slide

  56. Contributions
    welcome !
    Picture Link

    View Slide

  57. Links
    Git Repository
    https://gerrit.googlesource.com/k8s-gerrit
    Docker Images
    https://hub.docker.com/u/k8sgerrit
    Gardener
    https://gardener.cloud

    View Slide