Upgrade to Pro — share decks privately, control downloads, hide ads and more …

PIPE: CI/CD + Runtime as a Service for Better Developer Experience

PIPE: CI/CD + Runtime as a Service for Better Developer Experience

LINE DEVDAY 2021

November 10, 2021
Tweet

More Decks by LINE DEVDAY 2021

Other Decks in Technology

Transcript

  1. Agenda
    - About PIPE
    - PIPE Workflow
    - PIPE Runtime
    - Improve Service Level
    - Roadmap

    View full-size slide

  2. What is PIPE?
    Workflow
    - Tekton Pipelines + Github Events
    - Fully automated CI/CD by declarative pipeline definition
    - Provides fast and reproducible build and test
    Runtime
    - Managed Kubernetes Platform
    - Designed to minimize external dependencies and in-house dialects

    View full-size slide

  3. Why PIPE?
    Open Platform
    - Start small and let them grow
    - Users don't have to be fully adapted from the beginning
    - Users can access PIPE Runtime directly via kubectl or 3rd party
    Kubernetes IDEs like LENS

    View full-size slide

  4. Why PIPE?
    Managed Container Runtime
    - Namespace as a Service tenancy model
    - Periodic update/upgrade PIPE services, Kubernetes and Kubernetes
    components without user intervention
    - Provide namespace administrator role to users, secured by Kyverno
    policy engine

    View full-size slide

  5. PIPE Workflow

    View full-size slide

  6. PIPE Workflow
    Tekton
    - Kubernetes Native CI/CD framework
    - Provides Tekton Pipelines, CLI, Catalog and more
    - Part of CD Foundation (https://cd.foundation) project
    - Without Jenkins dependency
    GitHub Events
    - Handle GitHub Events through GitHub Actions workflow syntax
    - Define event handler on Tekton Pipeline’s annotation field

    View full-size slide

  7. How Workflow works
    User
    User
    Action (e.g. Push,Release)
    Send event via webhook
    List tekton pipelines from repository
    Return pipelines
    Template PipelineRun when event triggers pipeline
    Create templated PipelineRun
    GitHub
    GitHub
    PIPE
    PIPE
    Container Runtime
    Container Runtime

    View full-size slide

  8. Workflow Status

    View full-size slide

  9. Workflow Log

    View full-size slide

  10. Workflow Scaffolder - 1

    View full-size slide

  11. Workflow Scaffolder - 2

    View full-size slide

  12. Improve Developer eXperience
    Reusable Tasks: Curated Pipelines from Tekton Hub
    - build-container-image
    - build-gradle
    - git-clone
    - github-create-deployment
    - github-create-deployment-status
    - helm-upgrade-from-source
    - send-slack-message
    - and more…

    View full-size slide

  13. Improve Developer eXperience
    Scaffolder: Support PIPE Users to adopt CI/CD best practices easily
    - ci.yaml on GitHub Push or Pull Request to non-default branch
    - build.yaml on GitHub Push or Pull Request to default branch
    - cd.yaml on GitHub Deployment

    View full-size slide

  14. Sample CI/CD Best Practices
    Changes on non default branch:
    Pull Request
    1. ci.yaml triggered
    2. Do lint and unit test
    apiVersion: tekton.dev/v1beta1
    kind: Pipeline
    metadata:
    name: ci
    annotations:
    pipe.linecorp.com/event: |
    push:
    branches-ignore:
    - main

    View full-size slide

  15. Sample CI/CD Best Practices
    Changes on default branch:
    Push to main branch
    1. build.yaml triggered
    2. Do lint and unit test
    3. Create GitHub deployment
    4. Deploy to stage
    apiVersion: tekton.dev/v1beta1
    kind: Pipeline
    metadata:
    name: build
    annotations:
    pipe.linecorp.com/event: |
    push:
    branches:
    - main

    View full-size slide

  16. Sample CI/CD Best Practices
    New tag created
    1. cd.yaml triggered
    2. Deploy to production
    apiVersion: tekton.dev/v1beta1
    kind: Pipeline
    metadata:
    name: cd
    annotations:
    pipe.linecorp.com/event: |
    push:
    tags:
    - v*

    View full-size slide

  17. PIPE Runtime

    View full-size slide

  18. About PIPE Runtime
    - Managed Kubernetes using Namespace as a Service tenancy model
    - Core Cluster x 1
    - Serves PIPE API (including GitHub event handler) / Website
    - Handles OIDC Authentication
    - Runtime Cluster x N
    - Runs Workflow (Build) request
    - Serve user workloads
    - Deployed per region

    View full-size slide

  19. Kubernetes Base
    - Runs on Physical Machines
    - Deployment tool: kubeadm
    - Kubernetes version: v1.21.1
    - CNI: Calico with Typha IPAM plugin
    - CRI: containerd
    - CSI: OpenEBS, VSFS (Verda Shared File System)
    - Node roles
    - control-plane, build, gateway (private / public), none(user workloads)

    View full-size slide

  20. Installed Components
    Service Mesh Istio
    Ingress Contour (default) / ingress-nginx
    Load Balancer Verda LoadBalancer
    Build Tekton / buildkitd
    Observability Fluent Bit / Prometheus
    AuthN/Z Dex
    Policy Kyverno
    Storage OpenEBS, VSFS (Verda Shared File System)

    View full-size slide

  21. Authentication flow
    K8SRuntime
    K8SRuntime
    K8SCore
    K8SCore
    GitHub
    GitHub
    Resource Request by expired ID Token
    (e.g. kubectl get pod)
    Need Authentication
    OAuth2 Flow
    Request ID Token by GitHub access token
    ID Token
    Resource Request by valid ID Token
    Response
    User
    User
    Discover OIDC endpoints
    Validate ID Token
    Dex
    Dex

    View full-size slide

  22. Authorization flow
    User
    User
    Administrator
    Administrator
    K8SCore
    K8SCore
    Apply Runtime Policy
    Request unapproved namespace resource
    Deny request
    Request invalid resource
    Deny request with explanation
    Request valid resource
    Check RBAC permission
    Check RBAC permission
    Check Kyverno policy
    Check RBAC permission
    Check approved policy
    Response
    Kyverno
    Kyverno

    View full-size slide

  23. Improve Service Level

    View full-size slide

  24. Monitoring
    - Grafana
    - Prometheus
    - Alertmanager
    - kube-state-metrics
    - node-exporter
    - metrics-server
    - node-problem-detector

    View full-size slide

  25. Alerting
    Alertmanager sends following events to Slack channel
    - kube-state-metrics
    - KubernetesNodeReady
    - KubernetesClientCertificateExpiresNextWeek
    - KubernetesDeploymentReplicasMismatch
    - KubernetesDaemonsetRolloutStuck
    - KubernetesDaemonsetMisscheduled

    View full-size slide

  26. Alerting
    Alertmanager sends following events to Slack channel
    - kube-state-metrics
    - KubernetesPodNotHealthy
    - KubernetesPodCrashLooping
    - KubernetesEndpointAddressNotReady
    - Prometheus
    - PrometheusTargetMissing
    - Fluent Bit
    - NoOutputBytesProcessed

    View full-size slide

  27. Maintenance
    - Backup
    - etcd snapshot: Backed up for every 30 minutes into Verda Object
    Storage
    - PIPE Runtime
    - Clean up user PipelineRuns older than 7 days to prevent etcd overload

    View full-size slide

  28. Maintenance
    - PIPE Control Plane
    - PIPE itself uses jenkins on VM for CI/CD and provisioning, to reduce
    circular dependency
    - Kubernetes cluster is managed by ansible playbook
    - Kubernetes components are managed by helmfile + kustomize

    View full-size slide

  29. Node Roles
    - Workloads are scheduled by Kubernetes NodeAffinity
    - User workloads (Application) runs on managed user nodes
    - Node without role (default)
    - PIPE workload runs on dedicate nodes
    - control-plane
    - build (+ builtkitd)
    - gateway (private / public)

    View full-size slide

  30. High Availability
    Verda will support Multiple Availability Zone(AZ) on VM, LB, DNS and more
    PIPE is planning to utilize AZ for PIPE core and runtime for High Availability
    - Distribute & Replicate Control Plane nodes across Availability Zones
    - Distribute Worker nodes across Availability Zones

    View full-size slide

  31. Functions
    We released base infrastructure we needed, now is the time to aim higher.
    - Help developers to use Cloud Native events & boot workloads easily
    without deep knowledge on Kubernetes infrastructure
    - Like Heroku, Netlify or Google App Engine
    - Succeeds Verda Functions
    - With more events
    - With more languages
    - With improved DX

    View full-size slide

  32. CPaaS: Control planes as a Service
    Virtual Cluster: For users who need cluster-admin role
    Separated
    Namespace
    vCluster Separated Cluster
    Isolation Very weak Strong Very strong
    Access for Tenants Very restricted vCluster Admin Cluster Admin
    Cost Very cheap Cheap Expensive
    Resource Sharing Easy Easy Very hard
    Overhead Very low Very low Very high

    View full-size slide

  33. VM based CRI
    For users who need more secure runtime environment
    - Firecracker
    - Kata Containers
    - KubeVirt

    View full-size slide