Slide 1

Slide 1 text

No content

Slide 2

Slide 2 text

Agenda - About PIPE - PIPE Workflow - PIPE Runtime - Improve Service Level - Roadmap

Slide 3

Slide 3 text

About PIPE

Slide 4

Slide 4 text

What is PIPE? Workflow - Tekton Pipelines + Github Events - Fully automated CI/CD by declarative pipeline definition - Provides fast and reproducible build and test Runtime - Managed Kubernetes Platform - Designed to minimize external dependencies and in-house dialects

Slide 5

Slide 5 text

Why PIPE? Open Platform - Start small and let them grow - Users don't have to be fully adapted from the beginning - Users can access PIPE Runtime directly via kubectl or 3rd party Kubernetes IDEs like LENS

Slide 6

Slide 6 text

Why PIPE? Managed Container Runtime - Namespace as a Service tenancy model - Periodic update/upgrade PIPE services, Kubernetes and Kubernetes components without user intervention - Provide namespace administrator role to users, secured by Kyverno policy engine

Slide 7

Slide 7 text

PIPE Workflow

Slide 8

Slide 8 text

PIPE Workflow Tekton - Kubernetes Native CI/CD framework - Provides Tekton Pipelines, CLI, Catalog and more - Part of CD Foundation (https://cd.foundation) project - Without Jenkins dependency GitHub Events - Handle GitHub Events through GitHub Actions workflow syntax - Define event handler on Tekton Pipeline’s annotation field

Slide 9

Slide 9 text

How Workflow works User User Action (e.g. Push,Release) Send event via webhook List tekton pipelines from repository Return pipelines Template PipelineRun when event triggers pipeline Create templated PipelineRun GitHub GitHub PIPE PIPE Container Runtime Container Runtime

Slide 10

Slide 10 text

Workflow Status

Slide 11

Slide 11 text

Workflow Log

Slide 12

Slide 12 text

Workflow Scaffolder - 1

Slide 13

Slide 13 text

Workflow Scaffolder - 2

Slide 14

Slide 14 text

Improve Developer eXperience Reusable Tasks: Curated Pipelines from Tekton Hub - build-container-image - build-gradle - git-clone - github-create-deployment - github-create-deployment-status - helm-upgrade-from-source - send-slack-message - and more…

Slide 15

Slide 15 text

Improve Developer eXperience Scaffolder: Support PIPE Users to adopt CI/CD best practices easily - ci.yaml on GitHub Push or Pull Request to non-default branch - build.yaml on GitHub Push or Pull Request to default branch - cd.yaml on GitHub Deployment

Slide 16

Slide 16 text

Sample CI/CD Best Practices Changes on non default branch: Pull Request 1. ci.yaml triggered 2. Do lint and unit test apiVersion: tekton.dev/v1beta1 kind: Pipeline metadata: name: ci annotations: pipe.linecorp.com/event: | push: branches-ignore: - main

Slide 17

Slide 17 text

Sample CI/CD Best Practices Changes on default branch: Push to main branch 1. build.yaml triggered 2. Do lint and unit test 3. Create GitHub deployment 4. Deploy to stage apiVersion: tekton.dev/v1beta1 kind: Pipeline metadata: name: build annotations: pipe.linecorp.com/event: | push: branches: - main

Slide 18

Slide 18 text

Sample CI/CD Best Practices New tag created 1. cd.yaml triggered 2. Deploy to production apiVersion: tekton.dev/v1beta1 kind: Pipeline metadata: name: cd annotations: pipe.linecorp.com/event: | push: tags: - v*

Slide 19

Slide 19 text

PIPE Runtime

Slide 20

Slide 20 text

About PIPE Runtime - Managed Kubernetes using Namespace as a Service tenancy model - Core Cluster x 1 - Serves PIPE API (including GitHub event handler) / Website - Handles OIDC Authentication - Runtime Cluster x N - Runs Workflow (Build) request - Serve user workloads - Deployed per region

Slide 21

Slide 21 text

Kubernetes Base - Runs on Physical Machines - Deployment tool: kubeadm - Kubernetes version: v1.21.1 - CNI: Calico with Typha IPAM plugin - CRI: containerd - CSI: OpenEBS, VSFS (Verda Shared File System) - Node roles - control-plane, build, gateway (private / public), none(user workloads)

Slide 22

Slide 22 text

Installed Components Service Mesh Istio Ingress Contour (default) / ingress-nginx Load Balancer Verda LoadBalancer Build Tekton / buildkitd Observability Fluent Bit / Prometheus AuthN/Z Dex Policy Kyverno Storage OpenEBS, VSFS (Verda Shared File System)

Slide 23

Slide 23 text

Authentication flow K8SRuntime K8SRuntime K8SCore K8SCore GitHub GitHub Resource Request by expired ID Token (e.g. kubectl get pod) Need Authentication OAuth2 Flow Request ID Token by GitHub access token ID Token Resource Request by valid ID Token Response User User Discover OIDC endpoints Validate ID Token Dex Dex

Slide 24

Slide 24 text

Authorization flow User User Administrator Administrator K8SCore K8SCore Apply Runtime Policy Request unapproved namespace resource Deny request Request invalid resource Deny request with explanation Request valid resource Check RBAC permission Check RBAC permission Check Kyverno policy Check RBAC permission Check approved policy Response Kyverno Kyverno

Slide 25

Slide 25 text

Dashboard

Slide 26

Slide 26 text

Improve Service Level

Slide 27

Slide 27 text

Monitoring - Grafana - Prometheus - Alertmanager - kube-state-metrics - node-exporter - metrics-server - node-problem-detector

Slide 28

Slide 28 text

Alerting Alertmanager sends following events to Slack channel - kube-state-metrics - KubernetesNodeReady - KubernetesClientCertificateExpiresNextWeek - KubernetesDeploymentReplicasMismatch - KubernetesDaemonsetRolloutStuck - KubernetesDaemonsetMisscheduled

Slide 29

Slide 29 text

Alerting Alertmanager sends following events to Slack channel - kube-state-metrics - KubernetesPodNotHealthy - KubernetesPodCrashLooping - KubernetesEndpointAddressNotReady - Prometheus - PrometheusTargetMissing - Fluent Bit - NoOutputBytesProcessed

Slide 30

Slide 30 text

Maintenance - Backup - etcd snapshot: Backed up for every 30 minutes into Verda Object Storage - PIPE Runtime - Clean up user PipelineRuns older than 7 days to prevent etcd overload

Slide 31

Slide 31 text

Maintenance - PIPE Control Plane - PIPE itself uses jenkins on VM for CI/CD and provisioning, to reduce circular dependency - Kubernetes cluster is managed by ansible playbook - Kubernetes components are managed by helmfile + kustomize

Slide 32

Slide 32 text

Node Roles - Workloads are scheduled by Kubernetes NodeAffinity - User workloads (Application) runs on managed user nodes - Node without role (default) - PIPE workload runs on dedicate nodes - control-plane - build (+ builtkitd) - gateway (private / public)

Slide 33

Slide 33 text

Roadmap

Slide 34

Slide 34 text

High Availability Verda will support Multiple Availability Zone(AZ) on VM, LB, DNS and more PIPE is planning to utilize AZ for PIPE core and runtime for High Availability - Distribute & Replicate Control Plane nodes across Availability Zones - Distribute Worker nodes across Availability Zones

Slide 35

Slide 35 text

Functions We released base infrastructure we needed, now is the time to aim higher. - Help developers to use Cloud Native events & boot workloads easily without deep knowledge on Kubernetes infrastructure - Like Heroku, Netlify or Google App Engine - Succeeds Verda Functions - With more events - With more languages - With improved DX

Slide 36

Slide 36 text

CPaaS: Control planes as a Service Virtual Cluster: For users who need cluster-admin role Separated Namespace vCluster Separated Cluster Isolation Very weak Strong Very strong Access for Tenants Very restricted vCluster Admin Cluster Admin Cost Very cheap Cheap Expensive Resource Sharing Easy Easy Very hard Overhead Very low Very low Very high

Slide 37

Slide 37 text

VM based CRI For users who need more secure runtime environment - Firecracker - Kata Containers - KubeVirt

Slide 38

Slide 38 text

Thank you