Slide 1
Slide 1 text
No content
Slide 2
Slide 2 text
Agenda
- About PIPE
- PIPE Workflow
- PIPE Runtime
- Improve Service Level
- Roadmap
Slide 3
Slide 3 text
About PIPE
Slide 4
Slide 4 text
What is PIPE?
Workflow
- Tekton Pipelines + Github Events
- Fully automated CI/CD by declarative pipeline definition
- Provides fast and reproducible build and test
Runtime
- Managed Kubernetes Platform
- Designed to minimize external dependencies and in-house dialects
Slide 5
Slide 5 text
Why PIPE?
Open Platform
- Start small and let them grow
- Users don't have to be fully adapted from the beginning
- Users can access PIPE Runtime directly via kubectl or 3rd party
Kubernetes IDEs like LENS
Slide 6
Slide 6 text
Why PIPE?
Managed Container Runtime
- Namespace as a Service tenancy model
- Periodic update/upgrade PIPE services, Kubernetes and Kubernetes
components without user intervention
- Provide namespace administrator role to users, secured by Kyverno
policy engine
Slide 7
Slide 7 text
PIPE Workflow
Slide 8
Slide 8 text
PIPE Workflow
Tekton
- Kubernetes Native CI/CD framework
- Provides Tekton Pipelines, CLI, Catalog and more
- Part of CD Foundation (https://cd.foundation) project
- Without Jenkins dependency
GitHub Events
- Handle GitHub Events through GitHub Actions workflow syntax
- Define event handler on Tekton Pipeline’s annotation field
Slide 9
Slide 9 text
How Workflow works
User
User
Action (e.g. Push,Release)
Send event via webhook
List tekton pipelines from repository
Return pipelines
Template PipelineRun when event triggers pipeline
Create templated PipelineRun
GitHub
GitHub
PIPE
PIPE
Container Runtime
Container Runtime
Slide 10
Slide 10 text
Workflow Status
Slide 11
Slide 11 text
Workflow Log
Slide 12
Slide 12 text
Workflow Scaffolder - 1
Slide 13
Slide 13 text
Workflow Scaffolder - 2
Slide 14
Slide 14 text
Improve Developer eXperience
Reusable Tasks: Curated Pipelines from Tekton Hub
- build-container-image
- build-gradle
- git-clone
- github-create-deployment
- github-create-deployment-status
- helm-upgrade-from-source
- send-slack-message
- and more…
Slide 15
Slide 15 text
Improve Developer eXperience
Scaffolder: Support PIPE Users to adopt CI/CD best practices easily
- ci.yaml on GitHub Push or Pull Request to non-default branch
- build.yaml on GitHub Push or Pull Request to default branch
- cd.yaml on GitHub Deployment
Slide 16
Slide 16 text
Sample CI/CD Best Practices
Changes on non default branch:
Pull Request
1. ci.yaml triggered
2. Do lint and unit test
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
name: ci
annotations:
pipe.linecorp.com/event: |
push:
branches-ignore:
- main
Slide 17
Slide 17 text
Sample CI/CD Best Practices
Changes on default branch:
Push to main branch
1. build.yaml triggered
2. Do lint and unit test
3. Create GitHub deployment
4. Deploy to stage
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
name: build
annotations:
pipe.linecorp.com/event: |
push:
branches:
- main
Slide 18
Slide 18 text
Sample CI/CD Best Practices
New tag created
1. cd.yaml triggered
2. Deploy to production
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
name: cd
annotations:
pipe.linecorp.com/event: |
push:
tags:
- v*
Slide 19
Slide 19 text
PIPE Runtime
Slide 20
Slide 20 text
About PIPE Runtime
- Managed Kubernetes using Namespace as a Service tenancy model
- Core Cluster x 1
- Serves PIPE API (including GitHub event handler) / Website
- Handles OIDC Authentication
- Runtime Cluster x N
- Runs Workflow (Build) request
- Serve user workloads
- Deployed per region
Slide 21
Slide 21 text
Kubernetes Base
- Runs on Physical Machines
- Deployment tool: kubeadm
- Kubernetes version: v1.21.1
- CNI: Calico with Typha IPAM plugin
- CRI: containerd
- CSI: OpenEBS, VSFS (Verda Shared File System)
- Node roles
- control-plane, build, gateway (private / public), none(user workloads)
Slide 22
Slide 22 text
Installed Components
Service Mesh Istio
Ingress Contour (default) / ingress-nginx
Load Balancer Verda LoadBalancer
Build Tekton / buildkitd
Observability Fluent Bit / Prometheus
AuthN/Z Dex
Policy Kyverno
Storage OpenEBS, VSFS (Verda Shared File System)
Slide 23
Slide 23 text
Authentication flow
K8SRuntime
K8SRuntime
K8SCore
K8SCore
GitHub
GitHub
Resource Request by expired ID Token
(e.g. kubectl get pod)
Need Authentication
OAuth2 Flow
Request ID Token by GitHub access token
ID Token
Resource Request by valid ID Token
Response
User
User
Discover OIDC endpoints
Validate ID Token
Dex
Dex
Slide 24
Slide 24 text
Authorization flow
User
User
Administrator
Administrator
K8SCore
K8SCore
Apply Runtime Policy
Request unapproved namespace resource
Deny request
Request invalid resource
Deny request with explanation
Request valid resource
Check RBAC permission
Check RBAC permission
Check Kyverno policy
Check RBAC permission
Check approved policy
Response
Kyverno
Kyverno
Slide 25
Slide 25 text
Dashboard
Slide 26
Slide 26 text
Improve Service Level
Slide 27
Slide 27 text
Monitoring
- Grafana
- Prometheus
- Alertmanager
- kube-state-metrics
- node-exporter
- metrics-server
- node-problem-detector
Slide 28
Slide 28 text
Alerting
Alertmanager sends following events to Slack channel
- kube-state-metrics
- KubernetesNodeReady
- KubernetesClientCertificateExpiresNextWeek
- KubernetesDeploymentReplicasMismatch
- KubernetesDaemonsetRolloutStuck
- KubernetesDaemonsetMisscheduled
Slide 29
Slide 29 text
Alerting
Alertmanager sends following events to Slack channel
- kube-state-metrics
- KubernetesPodNotHealthy
- KubernetesPodCrashLooping
- KubernetesEndpointAddressNotReady
- Prometheus
- PrometheusTargetMissing
- Fluent Bit
- NoOutputBytesProcessed
Slide 30
Slide 30 text
Maintenance
- Backup
- etcd snapshot: Backed up for every 30 minutes into Verda Object
Storage
- PIPE Runtime
- Clean up user PipelineRuns older than 7 days to prevent etcd overload
Slide 31
Slide 31 text
Maintenance
- PIPE Control Plane
- PIPE itself uses jenkins on VM for CI/CD and provisioning, to reduce
circular dependency
- Kubernetes cluster is managed by ansible playbook
- Kubernetes components are managed by helmfile + kustomize
Slide 32
Slide 32 text
Node Roles
- Workloads are scheduled by Kubernetes NodeAffinity
- User workloads (Application) runs on managed user nodes
- Node without role (default)
- PIPE workload runs on dedicate nodes
- control-plane
- build (+ builtkitd)
- gateway (private / public)
Slide 33
Slide 33 text
Roadmap
Slide 34
Slide 34 text
High Availability
Verda will support Multiple Availability Zone(AZ) on VM, LB, DNS and more
PIPE is planning to utilize AZ for PIPE core and runtime for High Availability
- Distribute & Replicate Control Plane nodes across Availability Zones
- Distribute Worker nodes across Availability Zones
Slide 35
Slide 35 text
Functions
We released base infrastructure we needed, now is the time to aim higher.
- Help developers to use Cloud Native events & boot workloads easily
without deep knowledge on Kubernetes infrastructure
- Like Heroku, Netlify or Google App Engine
- Succeeds Verda Functions
- With more events
- With more languages
- With improved DX
Slide 36
Slide 36 text
CPaaS: Control planes as a Service
Virtual Cluster: For users who need cluster-admin role
Separated
Namespace
vCluster Separated Cluster
Isolation Very weak Strong Very strong
Access for Tenants Very restricted vCluster Admin Cluster Admin
Cost Very cheap Cheap Expensive
Resource Sharing Easy Easy Very hard
Overhead Very low Very low Very high
Slide 37
Slide 37 text
VM based CRI
For users who need more secure runtime environment
- Firecracker
- Kata Containers
- KubeVirt
Slide 38
Slide 38 text
Thank you