$30 off During Our Annual Pro Sale. View Details »

PIPE: CI/CD + Runtime as a Service for Better Developer Experience

PIPE: CI/CD + Runtime as a Service for Better Developer Experience


November 10, 2021

More Decks by LINE DEVDAY 2021

Other Decks in Technology


  1. None
  2. Agenda - About PIPE - PIPE Workflow - PIPE Runtime

    - Improve Service Level - Roadmap
  3. About PIPE

  4. What is PIPE? Workflow - Tekton Pipelines + Github Events

    - Fully automated CI/CD by declarative pipeline definition - Provides fast and reproducible build and test Runtime - Managed Kubernetes Platform - Designed to minimize external dependencies and in-house dialects
  5. Why PIPE? Open Platform - Start small and let them

    grow - Users don't have to be fully adapted from the beginning - Users can access PIPE Runtime directly via kubectl or 3rd party Kubernetes IDEs like LENS
  6. Why PIPE? Managed Container Runtime - Namespace as a Service

    tenancy model - Periodic update/upgrade PIPE services, Kubernetes and Kubernetes components without user intervention - Provide namespace administrator role to users, secured by Kyverno policy engine
  7. PIPE Workflow

  8. PIPE Workflow Tekton - Kubernetes Native CI/CD framework - Provides

    Tekton Pipelines, CLI, Catalog and more - Part of CD Foundation (https://cd.foundation) project - Without Jenkins dependency GitHub Events - Handle GitHub Events through GitHub Actions workflow syntax - Define event handler on Tekton Pipeline’s annotation field
  9. How Workflow works User User Action (e.g. Push,Release) Send event

    via webhook List tekton pipelines from repository Return pipelines Template PipelineRun when event triggers pipeline Create templated PipelineRun GitHub GitHub PIPE PIPE Container Runtime Container Runtime
  10. Workflow Status

  11. Workflow Log

  12. Workflow Scaffolder - 1

  13. Workflow Scaffolder - 2

  14. Improve Developer eXperience Reusable Tasks: Curated Pipelines from Tekton Hub

    - build-container-image - build-gradle - git-clone - github-create-deployment - github-create-deployment-status - helm-upgrade-from-source - send-slack-message - and more…
  15. Improve Developer eXperience Scaffolder: Support PIPE Users to adopt CI/CD

    best practices easily - ci.yaml on GitHub Push or Pull Request to non-default branch - build.yaml on GitHub Push or Pull Request to default branch - cd.yaml on GitHub Deployment
  16. Sample CI/CD Best Practices Changes on non default branch: Pull

    Request 1. ci.yaml triggered 2. Do lint and unit test apiVersion: tekton.dev/v1beta1 kind: Pipeline metadata: name: ci annotations: pipe.linecorp.com/event: | push: branches-ignore: - main
  17. Sample CI/CD Best Practices Changes on default branch: Push to

    main branch 1. build.yaml triggered 2. Do lint and unit test 3. Create GitHub deployment 4. Deploy to stage apiVersion: tekton.dev/v1beta1 kind: Pipeline metadata: name: build annotations: pipe.linecorp.com/event: | push: branches: - main
  18. Sample CI/CD Best Practices New tag created 1. cd.yaml triggered

    2. Deploy to production apiVersion: tekton.dev/v1beta1 kind: Pipeline metadata: name: cd annotations: pipe.linecorp.com/event: | push: tags: - v*
  19. PIPE Runtime

  20. About PIPE Runtime - Managed Kubernetes using Namespace as a

    Service tenancy model - Core Cluster x 1 - Serves PIPE API (including GitHub event handler) / Website - Handles OIDC Authentication - Runtime Cluster x N - Runs Workflow (Build) request - Serve user workloads - Deployed per region
  21. Kubernetes Base - Runs on Physical Machines - Deployment tool:

    kubeadm - Kubernetes version: v1.21.1 - CNI: Calico with Typha IPAM plugin - CRI: containerd - CSI: OpenEBS, VSFS (Verda Shared File System) - Node roles - control-plane, build, gateway (private / public), none(user workloads)
  22. Installed Components Service Mesh Istio Ingress Contour (default) / ingress-nginx

    Load Balancer Verda LoadBalancer Build Tekton / buildkitd Observability Fluent Bit / Prometheus AuthN/Z Dex Policy Kyverno Storage OpenEBS, VSFS (Verda Shared File System)
  23. Authentication flow K8SRuntime K8SRuntime K8SCore K8SCore GitHub GitHub Resource Request

    by expired ID Token (e.g. kubectl get pod) Need Authentication OAuth2 Flow Request ID Token by GitHub access token ID Token Resource Request by valid ID Token Response User User Discover OIDC endpoints Validate ID Token Dex Dex
  24. Authorization flow User User Administrator Administrator K8SCore K8SCore Apply Runtime

    Policy Request unapproved namespace resource Deny request Request invalid resource Deny request with explanation Request valid resource Check RBAC permission Check RBAC permission Check Kyverno policy Check RBAC permission Check approved policy Response Kyverno Kyverno
  25. Dashboard

  26. Improve Service Level

  27. Monitoring - Grafana - Prometheus - Alertmanager - kube-state-metrics -

    node-exporter - metrics-server - node-problem-detector
  28. Alerting Alertmanager sends following events to Slack channel - kube-state-metrics

    - KubernetesNodeReady - KubernetesClientCertificateExpiresNextWeek - KubernetesDeploymentReplicasMismatch - KubernetesDaemonsetRolloutStuck - KubernetesDaemonsetMisscheduled
  29. Alerting Alertmanager sends following events to Slack channel - kube-state-metrics

    - KubernetesPodNotHealthy - KubernetesPodCrashLooping - KubernetesEndpointAddressNotReady - Prometheus - PrometheusTargetMissing - Fluent Bit - NoOutputBytesProcessed
  30. Maintenance - Backup - etcd snapshot: Backed up for every

    30 minutes into Verda Object Storage - PIPE Runtime - Clean up user PipelineRuns older than 7 days to prevent etcd overload
  31. Maintenance - PIPE Control Plane - PIPE itself uses jenkins

    on VM for CI/CD and provisioning, to reduce circular dependency - Kubernetes cluster is managed by ansible playbook - Kubernetes components are managed by helmfile + kustomize
  32. Node Roles - Workloads are scheduled by Kubernetes NodeAffinity -

    User workloads (Application) runs on managed user nodes - Node without role (default) - PIPE workload runs on dedicate nodes - control-plane - build (+ builtkitd) - gateway (private / public)
  33. Roadmap

  34. High Availability Verda will support Multiple Availability Zone(AZ) on VM,

    LB, DNS and more PIPE is planning to utilize AZ for PIPE core and runtime for High Availability - Distribute & Replicate Control Plane nodes across Availability Zones - Distribute Worker nodes across Availability Zones
  35. Functions We released base infrastructure we needed, now is the

    time to aim higher. - Help developers to use Cloud Native events & boot workloads easily without deep knowledge on Kubernetes infrastructure - Like Heroku, Netlify or Google App Engine - Succeeds Verda Functions - With more events - With more languages - With improved DX
  36. CPaaS: Control planes as a Service Virtual Cluster: For users

    who need cluster-admin role Separated Namespace vCluster Separated Cluster Isolation Very weak Strong Very strong Access for Tenants Very restricted vCluster Admin Cluster Admin Cost Very cheap Cheap Expensive Resource Sharing Easy Easy Very hard Overhead Very low Very low Very high
  37. VM based CRI For users who need more secure runtime

    environment - Firecracker - Kata Containers - KubeVirt
  38. Thank you