PIPE: CI/CD + Runtime as a Service for Better Developer Experience

Seung Lee (Mark)
LINE Plus / Productivity Engineering / PE Team Developer / Manager

https://linedevday.linecorp.com/2021/ja/sessions/43
https://linedevday.linecorp.com/2021/en/sessions/43
https://linedevday.linecorp.com/2021/ko/sessions/43

Transcript

2. Agenda - About PIPE - PIPE Workflow - PIPE Runtime - Improve Service Level - Roadmap
3. About PIPE
4. What is PIPE? Workflow - Tekton Pipelines + Github Events - Fully automated CI/CD by declarative pipeline definition - Provides fast and reproducible build and test Runtime - Managed Kubernetes Platform - Designed to minimize external dependencies and in-house dialects
5. Why PIPE? Open Platform - Start small and let them grow - Users don't have to be fully adapted from the beginning - Users can access PIPE Runtime directly via kubectl or 3rd party Kubernetes IDEs like LENS
6. Why PIPE? Managed Container Runtime - Namespace as a Service tenancy model - Periodic update/upgrade PIPE services, Kubernetes and Kubernetes components without user intervention - Provide namespace administrator role to users, secured by Kyverno policy engine
7. PIPE Workflow
8. PIPE Workflow Tekton - Kubernetes Native CI/CD framework - Provides Tekton Pipelines, CLI, Catalog and more - Part of CD Foundation (https://cd.foundation) project - Without Jenkins dependency GitHub Events - Handle GitHub Events through GitHub Actions workflow syntax - Define event handler on Tekton Pipeline’s annotation field
9. How Workflow works User User Action (e.g. Push,Release) Send event via webhook List tekton pipelines from repository Return pipelines Template PipelineRun when event triggers pipeline Create templated PipelineRun GitHub GitHub PIPE PIPE Container Runtime Container Runtime
10. Workflow Status
11. Workflow Log
12. Workflow Scaffolder - 1
13. Workflow Scaffolder - 2
14. Improve Developer eXperience Reusable Tasks: Curated Pipelines from Tekton Hub - build-container-image - build-gradle - git-clone - github-create-deployment - github-create-deployment-status - helm-upgrade-from-source - send-slack-message - and more…
15. Improve Developer eXperience Scaffolder: Support PIPE Users to adopt CI/CD best practices easily - ci.yaml on GitHub Push or Pull Request to non-default branch - build.yaml on GitHub Push or Pull Request to default branch - cd.yaml on GitHub Deployment
16. Sample CI/CD Best Practices Changes on non default branch: Pull Request 1. ci.yaml triggered 2. Do lint and unit test apiVersion: tekton.dev/v1beta1 kind: Pipeline metadata: name: ci annotations: pipe.linecorp.com/event: | push: branches-ignore: - main
17. Sample CI/CD Best Practices Changes on default branch: Push to main branch 1. build.yaml triggered 2. Do lint and unit test 3. Create GitHub deployment 4. Deploy to stage apiVersion: tekton.dev/v1beta1 kind: Pipeline metadata: name: build annotations: pipe.linecorp.com/event: | push: branches: - main
18. Sample CI/CD Best Practices New tag created 1. cd.yaml triggered 2. Deploy to production apiVersion: tekton.dev/v1beta1 kind: Pipeline metadata: name: cd annotations: pipe.linecorp.com/event: | push: tags: - v*
19. PIPE Runtime
20. About PIPE Runtime - Managed Kubernetes using Namespace as a Service tenancy model - Core Cluster x 1 - Serves PIPE API (including GitHub event handler) / Website - Handles OIDC Authentication - Runtime Cluster x N - Runs Workflow (Build) request - Serve user workloads - Deployed per region
21. Kubernetes Base - Runs on Physical Machines - Deployment tool: kubeadm - Kubernetes version: v1.21.1 - CNI: Calico with Typha IPAM plugin - CRI: containerd - CSI: OpenEBS, VSFS (Verda Shared File System) - Node roles - control-plane, build, gateway (private / public), none(user workloads)
22. Installed Components Service Mesh Istio Ingress Contour (default) / ingress-nginx Load Balancer Verda LoadBalancer Build Tekton / buildkitd Observability Fluent Bit / Prometheus AuthN/Z Dex Policy Kyverno Storage OpenEBS, VSFS (Verda Shared File System)
23. Authentication flow K8SRuntime K8SRuntime K8SCore K8SCore GitHub GitHub Resource Request by expired ID Token (e.g. kubectl get pod) Need Authentication OAuth2 Flow Request ID Token by GitHub access token ID Token Resource Request by valid ID Token Response User User Discover OIDC endpoints Validate ID Token Dex Dex
24. Authorization flow User User Administrator Administrator K8SCore K8SCore Apply Runtime Policy Request unapproved namespace resource Deny request Request invalid resource Deny request with explanation Request valid resource Check RBAC permission Check RBAC permission Check Kyverno policy Check RBAC permission Check approved policy Response Kyverno Kyverno
25. Dashboard
26. Improve Service Level
27. Monitoring - Grafana - Prometheus - Alertmanager - kube-state-metrics - node-exporter - metrics-server - node-problem-detector
28. Alerting Alertmanager sends following events to Slack channel - kube-state-metrics - KubernetesNodeReady - KubernetesClientCertificateExpiresNextWeek - KubernetesDeploymentReplicasMismatch - KubernetesDaemonsetRolloutStuck - KubernetesDaemonsetMisscheduled
29. Alerting Alertmanager sends following events to Slack channel - kube-state-metrics - KubernetesPodNotHealthy - KubernetesPodCrashLooping - KubernetesEndpointAddressNotReady - Prometheus - PrometheusTargetMissing - Fluent Bit - NoOutputBytesProcessed
30. Maintenance - Backup - etcd snapshot: Backed up for every 30 minutes into Verda Object Storage - PIPE Runtime - Clean up user PipelineRuns older than 7 days to prevent etcd overload
31. Maintenance - PIPE Control Plane - PIPE itself uses jenkins on VM for CI/CD and provisioning, to reduce circular dependency - Kubernetes cluster is managed by ansible playbook - Kubernetes components are managed by helmfile + kustomize
32. Node Roles - Workloads are scheduled by Kubernetes NodeAffinity - User workloads (Application) runs on managed user nodes - Node without role (default) - PIPE workload runs on dedicate nodes - control-plane - build (+ builtkitd) - gateway (private / public)
33. Roadmap
34. High Availability Verda will support Multiple Availability Zone(AZ) on VM, LB, DNS and more PIPE is planning to utilize AZ for PIPE core and runtime for High Availability - Distribute & Replicate Control Plane nodes across Availability Zones - Distribute Worker nodes across Availability Zones
35. Functions We released base infrastructure we needed, now is the time to aim higher. - Help developers to use Cloud Native events & boot workloads easily without deep knowledge on Kubernetes infrastructure - Like Heroku, Netlify or Google App Engine - Succeeds Verda Functions - With more events - With more languages - With improved DX
36. CPaaS: Control planes as a Service Virtual Cluster: For users who need cluster-admin role Separated Namespace vCluster Separated Cluster Isolation Very weak Strong Very strong Access for Tenants Very restricted vCluster Admin Cluster Admin Cost Very cheap Cheap Expensive Resource Sharing Easy Easy Very hard Overhead Very low Very low Very high
37. VM based CRI For users who need more secure runtime environment - Firecracker - Kata Containers - KubeVirt
38. Thank you