Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kubernetes manifests management and operation in Mercari

BABAROT
April 22, 2019

Kubernetes manifests management and operation in Mercari

BABAROT

April 22, 2019
Tweet

More Decks by BABAROT

Other Decks in Technology

Transcript

  1. @b4b4r07 (Apr 22, 2019) / Kubernete meetup tokyo #18
    Kubernetes manifests management &
    operation in Mercari

    View full-size slide

  2. BABAROT / @b4b4r07
    Mercari, Inc.

    SRE, Microservices Platform
    Blog / tellme.tokyo

    View full-size slide

  3. Monolith
    Our current status
    Microservices

    View full-size slide

  4. 100+ microservices
    Our current status

    View full-size slide

  5. 200+ contributors
    Our current status

    View full-size slide

  6. Our current status
    Service A namespace
    central GKE cluster
    Service A pods
    RBAC for Service A team
    Each namespace is managed
    by each microservices team
    GKE cluster itself is
    managed by platform team

    View full-size slide

  7. Agenda
    1. Kubernetes YAML
    1. GitHub Pull Requests
    2. GitOps
    3. Monorepo
    4. Directories
    2. Repository Ecosystem
    3. Recap

    View full-size slide

  8. Kubernetes YAML
    apiVersion: v1
    kind: Pod
    metadata:
    name: nginx-pod
    namespace: x-echo-jp-dev
    spec:
    containers:
    - name: nginx-container
    image: nginx
    ports:
    - containerPort: 80

    View full-size slide

  9. How do you manage YAML and operate it
    to your Kubernetes Clusters?

    View full-size slide

  10. Write Apply
    Some points…

    View full-size slide

  11. Write Apply
    How should we review it? How should we apply it?

    View full-size slide

  12. Write Apply
    How should we review it?
    How should we manage it?
    How should we apply it?
    How should we control it?

    View full-size slide

  13. Write Apply
    How should we review it?
    How should we manage it?
    How should we apply it?
    How should we control it?
    YAML Operations
    YAML Management

    View full-size slide

  14. Write Apply
    How should we review it?
    How should we manage it?
    How should we apply it?
    How should we control it?
    YAML Operations
    YAML Management
    Our practices are…

    View full-size slide

  15. Write Apply
    How should we review it?
    How should we manage it?
    How should we apply it?
    How should we control it?
    2. GitOps w/ kubectl
    1. Pull Requests
    3. Monorepo
    4. Directories

    View full-size slide

  16. Let’s see the each details

    View full-size slide

  17. 1. GitHub Pull Requests

    View full-size slide

  18. Kubernetes Object Management
    Imperative
    commands
    Imperative

    object

    configuration
    Declarative

    object

    configuration
    https://kubernetes.io/docs/concepts/overview/object-management-kubectl/overview/

    View full-size slide

  19. Kubernetes Object Management
    Imperative

    object

    configuration
    Declarative

    object

    configuration
    https://kubernetes.io/docs/concepts/overview/object-management-kubectl/overview/
    kubectl run nginx
    --image nginx

    View full-size slide

  20. Kubernetes Object Management
    Declarative

    object

    configuration
    https://kubernetes.io/docs/concepts/overview/object-management-kubectl/overview/
    kubectl run nginx
    --image nginx
    kubectl create
    -f nginx.yaml

    View full-size slide

  21. Kubernetes Object Management
    kubectl run nginx
    --image nginx
    https://kubernetes.io/docs/concepts/overview/object-management-kubectl/overview/
    kubectl create
    -f nginx.yaml
    kubectl apply
    -f manifests/

    View full-size slide

  22. •It can be stored in VCS such as Git.
    •It can integrate with processes such
    as reviewing changes.
    •It has better support for operating on
    directories and automatically detecting
    operation types per-object.
    Kubernetes Object Management
    Declarative

    object

    configuration
    https://kubernetes.io/docs/concepts/overview/object-management-kubectl/overview/

    View full-size slide

  23. Pull Requests
    •Easy to review
    •Easy to track operations and changes
    •Easy to recover from unexpected things
    • Just reverting

    View full-size slide

  24. GitOps - Operations by Pull Requests

    View full-size slide

  25. https://www.weave.works/blog/gitops-operations-by-pull-request

    View full-size slide

  26. Start 6 servers There are 6 servers

    View full-size slide

  27. Start 6 servers There are 6 servers
    <

    View full-size slide

  28. Let’s see Weaveworks case

    View full-size slide

  29. GitHub Repository
    Kubernetes Cluster

    View full-size slide

  30. Live Objects Source of Truth

    View full-size slide

  31. Live Objects Source of Truth

    View full-size slide

  32. Live Objects
    apply
    Source of Truth

    View full-size slide

  33. Live Objects
    apply
    kubectl create
    Source of Truth

    View full-size slide

  34. Live Objects
    apply
    kubectl create
    diff
    Source of Truth

    View full-size slide

  35. Live Objects
    apply
    kubectl create
    diff
    Source of Truth
    unknown
    objects

    View full-size slide

  36. Live Objects
    apply
    kubectl create
    diff
    Git is the source of truth
    Source of Truth
    unknown
    objects

    View full-size slide

  37. GitHub Repository
    Kubernetes Cluster
    diff & sync
    Pull strategy

    View full-size slide

  38. Let’s see our case

    View full-size slide

  39. GitHub Repository
    Kubernetes Cluster
    CI

    View full-size slide

  40. + -
    GitHub Repository
    Kubernetes Cluster
    CI
    diff

    View full-size slide

  41. + -
    GitHub Repository
    Kubernetes Cluster
    CI
    merge
    diff

    View full-size slide

  42. + -
    apply
    GitHub Repository
    Kubernetes Cluster
    CI
    merge
    diff

    View full-size slide

  43. + -
    apply
    GitHub Repository
    Kubernetes Cluster
    CI
    merge
    diff
    kubectl create

    View full-size slide

  44. + -
    apply
    GitHub Repository
    Kubernetes Cluster
    CI
    merge
    diff
    kubectl create
    not implemented yet

    (described later)

    View full-size slide

  45. GitHub Repository
    Kubernetes Cluster
    CI
    merge & apply
    Push strategy

    View full-size slide

  46. Let’s see the each differences

    View full-size slide

  47. Pull (Weaveworks case) Push (Mercari case)
    •Git can be the single
    source of truth
    •Bit difficult to implement
    the sync pipeline
    •Difficult to find
    divergence from the source
    •A common way of applying
    the changes

    View full-size slide

  48. Pull (Weaveworks case)
    We know they recommend "Pull" strategy
    https://www.weave.works/blog/kubernetes-anti-patterns-let-s-do-gitops-not-ciops
    •Git can be the single
    source of truth
    •Bit difficult to implement
    the sync pipeline

    View full-size slide

  49. Push (Mercari case)
    Why do we choose "Push" strategy?
    Simple
    Enough

    to start

    firstly
    Easy to
    implement
    •Difficult to find
    divergence from the source
    •A common way of applying
    the changes

    View full-size slide

  50. Why do we choose "Push" strategy?
    Simple
    Enough

    to start

    firstly
    Easy to
    implement
    Another most bigger reason is...
    Spinnaker

    View full-size slide

  51. Enough

    to start

    firstly
    Spinnaker
    Why using Spinnaker also?
    •Our CI pipeline which runs "kubectl
    apply" based on the changes is
    triggered by merging pull requests
    •However, in some resources (Job etc),
    we want to apply in our timing
    •Spinnaker "Provider V2" can handle
    Kubernetes manifests declaratively
    https://www.spinnaker.io/reference/providers/kubernetes-v2/

    View full-size slide

  52. + -
    apply
    GitHub Repository
    Kubernetes Cluster
    CI
    merge
    diff
    kubectl create
    not implemented yet

    (described later)
    Spinnaker
    kick

    View full-size slide

  53. Two type of repository styles

    View full-size slide

  54. One repository Multiple repositories

    View full-size slide

  55. Monorepo Polyrepo

    View full-size slide

  56. Monorepo Polyrepo
    Service A
    Service B
    Service A
    Service C
    Service B Service C

    View full-size slide

  57. https://medium.com/@adamhjk/monorepo-please-do-3657e08a4b70
    https://medium.com/@mattklein123/monorepos-please-dont-e9a279be011b

    View full-size slide

  58. Monorepo
    •Advantages
    • Easy to share YAML code
    • Easy to be reviewed by central team
    • Easy to be managed by central team
    • Easy to set up CI pipeline
    •Disadvantages
    • Take account into repo scale up
    • Take account into delegation of
    authority
    • CI/CD: No independence in each team

    View full-size slide

  59. •Advantages
    • Build CI/CD pipeline by yourself
    • No dependency of outside system
    • Easy to be scaled up themselves
    • Easy to change the pipeline cycle
    •Disadvantages
    • Not easy to share YAML code
    • Difficult to review by central team
    • Troublesome to build CI/CD pipeline
    by yourself
    Polyrepo

    View full-size slide

  60. •We chose "Monorepo" style.
    •It was a good option to start small.
    •The concrete reason will be shown in next section.

    View full-size slide

  61. 4. Directories

    View full-size slide

  62. microservice

    View full-size slide

  63. environment
    microservice

    View full-size slide

  64. environment
    kind
    microservice

    View full-size slide

  65. microservice
    environment
    kind
    resource

    View full-size slide

  66. Spinnaker case

    View full-size slide

  67. microservice

    View full-size slide

  68. microservice
    environment

    View full-size slide

  69. microservice
    environment
    pipeline

    View full-size slide

  70. microservice
    environment
    pipeline

    View full-size slide

  71. How do we apply the manifest changes?

    View full-size slide

  72. Just run "kubectl"
    https://groups.google.com/forum/#!msg/kubernetes-sig-cli/M6t40JP6n0g/U6Snz-bsFQAJ

    View full-size slide

  73. kubectl
    https://groups.google.com/forum/#!msg/kubernetes-sig-cli/M6t40JP6n0g/U6Snz-bsFQAJ
    •Microservices developers not only
    develop but also operate the service
    by themselves
    •So they are familiar with kubectl
    basically
    •It means less learning cost than
    introducing other tools

    View full-size slide

  74. kubectl
    https://groups.google.com/forum/#!msg/kubernetes-sig-cli/M6t40JP6n0g/U6Snz-bsFQAJ
    •Microservices developers not only
    develop but also operate the service
    by themselves
    •So they are familiar with kubectl
    basically
    •It means less learning cost than
    introducing other tools
    How do we apply the YAML
    files changed in the pull
    requests with kubectl?

    View full-size slide

  75. kubectl
    https://groups.google.com/forum/#!msg/kubernetes-sig-cli/M6t40JP6n0g/U6Snz-bsFQAJ
    •Microservices developers not only
    develop but also operate the service
    by themselves
    •So they are familiar with kubectl
    basically
    •It means less learning cost than
    introducing other tools
    We have some scripts to
    make apply pipeline easier.

    It can detect changed
    directories.

    View full-size slide

  76. Let’s say we’d add new manifest

    View full-size slide

  77. added
    Helper bash scripts detect changed
    directory from the PR
    manifests/microservices/mercari-echo-jp/development/PodDisruptionBudget

    View full-size slide

  78. changed_files() {
    declare basedir="${1}"
    declare current_branch="$(git rev-parse --abbrev-ref @)"
    if [[ ${current_branch} == "master" ]]; then
    # (apply)
    # In the master branch, when listing files edited
    # you need to compare with previous merge commit
    git diff --name-only "HEAD^" "HEAD" "${basedir}"
    else
    # (plan)
    # In the topic branch, when listing files edited in the branch,
    # you need to compare with the commit at the time
    # the branch was created
    # https://git-scm.com/docs/git-merge-base
    git diff --name-only $(git merge-base origin/HEAD HEAD) "${basedir}"
    fi
    }

    View full-size slide

  79. changed_dirs() {
    # Note:
    # If these files are changed
    # - manifests/microservices/x/development/Ingress
    # - manifests/microservices/x/development/PersistentVolumeClaim
    # - manifests/microservices/y/production/PersistentVolumeClaim
    # - manifests/microservices/y/production/Pod
    # - manifests/microservices/y/production/PodDisruptionBudget
    # The files we have to pass to script/apply are only two dirs
    # - manifests/microservices/x/development
    # - manifests/microservices/y/production
    declare basedir="${1}"
    for file in $(changed_files "${basedir}")
    do
    get_target_dir "${file}"
    done | sort | uniq
    }

    View full-size slide

  80. changed_dirs() {
    # Note:
    # If these files are changed
    # - manifests/microservices/x/development/Ingress
    # - manifests/microservices/x/development/PersistentVolumeClaim
    # - manifests/microservices/y/production/PersistentVolumeClaim
    # - manifests/microservices/y/production/Pod
    # - manifests/microservices/y/production/PodDisruptionBudget
    # The files we have to pass to script/apply are only two dirs
    # - manifests/microservices/x/development
    # - manifests/microservices/y/production
    declare basedir="${1}"
    for file in $(changed_files "${basedir}")
    do
    get_target_dir "${file}"
    done | sort | uniq
    }
    •We have to pay attention the order to
    apply multiple manifests
    • Namespace must be created before all
    other resources
    • ConfigMap must be created before
    Deployment
    How do we get the order of
    files to be applied?

    View full-size slide

  81. // InstallOrder is the order in which manifests should be installed (by Kind).
    // Those occurring earlier in the list get installed before those occurring later in the list.
    var InstallOrder SortOrder = []string{
    "Namespace",
    "ResourceQuota",
    "PodSecurityPolicy",
    "Secret",
    "ConfigMap",
    "PersistentVolume",
    "PersistentVolumeClaim",
    "CustomResourceDefinition",
    "Role",
    "RoleBinding",
    "Service",
    "DaemonSet",
    "Pod",
    "ReplicaSet",
    "Deployment",
    "StatefulSet",
    "Job",
    "CronJob",
    "Ingress",
    }
    https://github.com/helm/helm/blob/v2.10.0/pkg/tiller/kind_sorter.go

    View full-size slide

  82. // InstallOrder is the order in which manifests should be installed (by Kind).
    // Those occurring earlier in the list get installed before those occurring later in the list.
    var InstallOrder SortOrder = []string{
    "Namespace",
    "ResourceQuota",
    "PodSecurityPolicy",
    "Secret",
    "ConfigMap",
    "PersistentVolume",
    "PersistentVolumeClaim",
    "CustomResourceDefinition",
    "Role",
    "RoleBinding",
    "Service",
    "DaemonSet",
    "Pod",
    "ReplicaSet",
    "Deployment",
    "StatefulSet",
    "Job",
    "CronJob",
    "Ingress",
    }
    https://github.com/helm/helm/blob/v2.10.0/pkg/tiller/kind_sorter.go
    ResourceQuota
    Secret
    ConfigMap
    PersistentVolume
    PersistentVolumeClaim
    ServiceAccount
    Role
    RoleBinding
    Service
    DaemonSet
    Pod
    ReplicaSet
    Deployment
    StatefulSet
    Job
    CronJob
    Ingress
    HorizontalPodAutoscaler
    NetworkPolicy
    PodDisruptionBudget
    kind_install_order.txt

    View full-size slide

  83. sort_kinds_by_install_order() {
    kinds=( $(cat "kind_install_order.txt") )
    args=( "${@}" )
    for kind in "${kinds[@]}"
    do
    for arg in "${args[@]}"
    do
    if [[ $(get_kind "${arg}") == ${kind} ]]; then
    echo "${arg}"
    fi
    done
    done
    }
    .../development/Deployment
    .../development/ConfigMap
    .../development/PodDisruptionBudget
    ConfigMap
    Deployment
    PodDisruptionBudget
    kind_install_order.txt

    View full-size slide

  84. sort_kinds_by_install_order() {
    kinds=( $(cat "kind_install_order.txt") )
    args=( "${@}" )
    for kind in "${kinds[@]}"
    do
    for arg in "${args[@]}"
    do
    if [[ $(get_kind "${arg}") == ${kind} ]]; then
    echo "${arg}"
    fi
    done
    done
    }
    .../development/Deployment
    .../development/ConfigMap
    .../development/PodDisruptionBudget
    ConfigMap
    Deployment
    PodDisruptionBudget
    kind_install_order.txt
    How about using kustomize?

    View full-size slide

  85. It's easy to introduce our design but it has new concepts of "overlay" etc.

    View full-size slide

  86. •Our microservices is on the way
    •So developers are in the middle of being microservices developers
    •They have to learn a lot of things:
    • Kubernetes, Kubernetes YAML itself, Spinnaker, etc...
    • So introducing kustomize feature to our pipeline was not now.

    In the future.

    View full-size slide

  87. How do we apply
    PodDisruptionBudget?
    •Some resource kinds (e.g., PodDisruptionBudget)
    cannot update in-place
    •It means we cannot update existing
    PodDisruptionBudget or StatefulSet by kubectl
    apply.

    View full-size slide

  88. kubectl_apply() {
    # ...
    for manifest in $(sort_kinds_by_install_order "${manifests[@]}"); do
    case ${kind} in
    PodDisruptionBudget) # Need to be recreated if it already exists
    if kubectl get -n "${namespace}" "${kind}" "${resource}"; then
    kubectl delete -n "${namespace}" "${kind}" "${resource}"
    fi
    kubectl apply -n "${namespace}" -f "${manifes}" ;;
    Secret)
    ansible-vault view "${manifest}" \
    | kubectl apply -n "${namespace}" -f - ;;
    *)
    kubectl apply -n "${namespace}" -f "${manifest}" ;;
    esac
    done
    }
    In order to deal with those special kinds, we prepare for
    easy wrapper script for kubectl.

    View full-size slide

  89. The delegation of directory authority

    View full-size slide

  90. mercari-echo-jp team should not be able
    to change mercari-xxx-jp's team code,
    and vice versa

    View full-size slide

  91. GitHub CODEOWNERS feature
    https://blog.github.com/2017-07-06-introducing-code-owners/

    View full-size slide

  92. GitHub CODEOWNERS feature
    https://help.github.com/articles/about-codeowners/

    View full-size slide

  93. Repository Ecosystem

    View full-size slide

  94. Repository Ecosystem
    •Our microservices-kubernetes repository
    has some awesome tools to make it maintain
    easier and more handy like the ecosystem
    •One of those tools is a linter for
    Kubernetes YAML: Stein
    Stein Documentations

    View full-size slide

  95. Repository Ecosystem
    •For example, let's say you don't make
    the developers omit metadata.namespace
    field in their YAMLs to prevent from
    unexpected apply
    •However, do you have a way to do it
    in existing tools...?
    apiVersion: v1
    kind: Pod
    metadata:
    name: nginx-pod
    namespace: x-echo-jp-dev
    spec:
    containers:
    - name: nginx-container
    image: nginx
    ports:
    - containerPort: 80
    metadata:
    namespace: x-echo-jp-dev

    View full-size slide

  96. Stein can do that.

    View full-size slide

  97. rule "namespace_specification" {
    description = "Check namespace name is not empty"
    conditions = [
    "${jsonpath("metadata.namespace") != ""}",
    ]
    report {
    level = "ERROR"
    message = "Namespace is not specified"
    }
    }
    Stein configuration
    Stein allows you to enforce the rule defined by you based on
    your policy upon your YAML.

    View full-size slide

  98. rule "namespace_specification" {
    description = "Check namespace name is not empty"
    conditions = [
    "${jsonpath("metadata.namespace") != ""}",
    ]
    report {
    level = "ERROR"
    message = "Namespace is not specified"
    }
    }
    Stein configuration
    Stein allows you to enforce the rule defined by you based on
    your policy upon your YAML.
    rule definition
    the condition
    this rule
    fails or not
    if it fails it
    returns 1 with
    message according to
    this block

    View full-size slide

  99. rule "namespace_specification" {
    description = "Check namespace name is not empty"
    conditions = [
    "${jsonpath("metadata.namespace") != ""}",
    ]
    report {
    level = "ERROR"
    message = "Namespace is not specified"
    }
    }
    Stein configuration
    stein can interpret HCL like Terraform
    stein supports many built-in functions like Terraform

    View full-size slide

  100. $ stein apply
    x-echo-jp/development/Pod/test.yaml
    [ERROR] rule.namespace_specification Namespace is not specified
    =====================
    7 error(s), 2 warn(s)
    •Stein checks the policy files and applies them to your config
    files. If there are violation rules, Stein returns exit code 1.
    •Stein can work as a linter in CI step etc well.

    View full-size slide

  101. •Stein concepts and design comes from HashiCorp Sentinel one.
    •"Policy as Code" (PaC) is provided by HashiCorp and Sentinel.
    •PaC means the way to describe "ideal configuration files" and
    force it upon real configuration files.
    Policy as Code
    infrastructure code policy
    IaC PaC
    Policy as Code - Sentinel by HashiCorp
    Why Policy as Code? - HashiCorp Blog

    View full-size slide

  102. + -
    apply
    GitHub Repository
    Kubernetes Cluster
    CI
    merge
    diff
    kubectl create
    not implemented yet

    (described later)

    View full-size slide

  103. + -
    apply
    GitHub Repository
    Kubernetes Cluster
    CI
    merge
    diff
    kubectl create
    Stein: Adminmission Controller
    •TODO: Stein can work the admission controller also.
    • By doing so, it is possible to check whether YAML having
    violated rules is going to be applied.
    • It can be compatible with "Push" strategy.

    View full-size slide

  104. 1. Write manifests
    2. Send Pull Request
    •kubectl pipeline
    •stein lint step
    •Dir base delegation
    •apply when merged
    3. Run apply (dry-run)

    View full-size slide

  105. 1. Write manifests
    2. Send Pull Request
    •kubectl pipeline
    •stein lint step
    •Dir base delegation
    •apply when merged
    3. Run apply (dry-run)
    We can provide the common resources
    to all microservices

    View full-size slide

  106. •By using Monorepo style,
    • we can provide the common guard rail to start to develop &
    operate their own microservices
    • apply pipeline
    • review by central team
    • common lint step
    • Of course, it has also disadvantages
    • It's trade-offs for scaling up

    View full-size slide

  107. •Thank you

    View full-size slide