Building Build as a Service on Kubernetes

Kubernetes 위에서 개발자들이 코드를 빌드할 수 있는 서비스를 제공하기 위해 고민하고, 구현한 과정을 공유합니다.

Transcript

1. Sunghoon Kang / LINE Productivity Engineering Building Build as a Service on Kubernetes
2. Verda Productivity Engineering Team About Us • Improving CI / CD / CR with code for: • Reduce development cost • Reduce operation cost • Improve service reliability
3. Verda Productivity Engineering Team About Us • Developing PIPE • Gateway • Type LoadBalancer support for Kubernetes • Builder • Code to artifact service
4. Today’s topic - Why are we developing Build as a Service? - Journey to build artifacts on Kubernetes - Design Build as a Service for developers
5. Today’s topic - Why are we developing Build as a Service? - Journey to build artifacts on Kubernetes - Design Build as a Service for developers This session doesn’t contain: ! - Kubernetes cluster bootstrapping / operation
6. Services @ LINE - Most services are developed with Java - There are no strict rules for language selection: - Python, Typescript, Golang, Scala, Rust…
7. Build tools @ LINE - CI Services for Private Cloud (On-Premise) - Jenkins - Circle CI - Drone CI - Coming Soon - GitHub Actions (https://github.com/github/roadmap/issues/89)
8. Build tools @ LINE - Issues for current build tools - Speed - Scalability - Stability - Management Cost
9. “Reinventing the wheel”
10. Requirements for Build as a Service - Reproducibility (Build Environment) - Scalability - Speed - Stability
11. VM vs Container - VM - Snowflake problem - Slow if provisioning instance for each build request - MicroVM (e.g. Firecracker, Kata container…) can solve this issue ⚡
12. VM vs Container - VM - Snowflake problem - Slow if provisioning instance for each build request - MicroVM (e.g. Firecracker, Kata container…) can solve this issue ⚡ - Container - Can reproduce build environment faster - Easy to scale
13. Using Kubernetes as a platform - Approaches - Compose core resources (through custom controller) - Implement custom workflow engine (Custom Resource) - Open source workflow engine (e.g. Tekton, Argo…)
14. Using Kubernetes as a platform - Workflow Engines - Argo - Mature platform (v2.10.0) - Richer functions (Ansible like: loop, condition) → Imperative - Tekton - Declarative
15. Using Kubernetes as a platform - Argo - Workflow step = Pod - Tekton - Tasks → Pipeline - Task step = Container / Task = Pod Argo | Workflow Step (Pod) Container Step (Pod) Container Tekton | Pipeline Task (Pod) Step (Container) Step (Container)
16. Using Kubernetes as a platform - Tekton - Declarative - Task unit - Can define task step at container-level - Scheduling advantage - Sharing contexts between tasks - Managing PVC is expensive & slow (for multiple pods)
17. Constructing build environment - Multiple languages for build task (e.g. monorepo, node-gyp…) - Build is stateful task - Cannot be done with separate containers - Base image - Slim Image vs Fat Image - We want to reduce management cost - (For both) Low image layer cache hit ratio - Installing language is expensive operation
18. Constructing build environment - asdf (https://asdf-vm.com) for the rescue # - No need to manage language/version manually - Using ReadWriteManyPV as cache instead of image layer cache - Faster than installing language on-demand - Layer cache can be broken when installation ordering is broken
19. How Builder works User GitHub Builder API Tekton Triggers Pipeline Status Monitor Build Pipeline Parse DSL Install Languages Download Cache Build Artifacts Upload Cache Upload Artifacts Verda Object Storage Container Image Registry Webhook Create Pipeline Watch Status Update status to Check Run
20. Design Considerations - DSL - LINT : LINE for Next Ten years - Most projects are using monorepo - How do we represent multiple tasks by spec file? - How to reuse task - Overriding task variables
21. Design Considerations - DSL scenarios: cms: when: paths: - cms/** build: steps: - paramSetup: - buildCMS: server: when: paths: - server/** - protocol/** - storage/** - storage-protocol/** build: steps: - paramSetup: - buildServer: tasks: paramSetup: name: Parameter Setup run: | GIT_REV_SHORT=${GIT_COMMIT_SHA:0:7} DOCKER_IMAGE_BUILD_DATE=$(date +%Y%m%d%H%M%S) DOCKER_IMAGE_TAG=$(echo -n "${DOCKER_IMAGE_BUILD_DATE}-${GIT_REV_SHORT}") results: DOCKER_IMAGE_TAG: ${vars.DOCKER_IMAGE_TAG} buildCMS: name: Build cms run: ./gradlew clean cms:assemble buildServer: name: Build server run: ./gradlew clean server:assemble
22. Self questions after alpha release - Is container the right choice for build tasks? - Build tasks are stateful - Is our DSL the right choice for LINE developers? - Too hard to adopt
23. Next step - De-compose builder components as module for CI tools - (e.g. Jenkins plugin, GitHub Action) - Implement deployment related features - Hire more developers
24. End of Document
25. We are hiring $% - Job Openings & - Kubernetes ӝ߈ CI/CD ೒ۖಬ ѐߊ - Cloud Service ѐߊ ߂ ਍৔ - https://recruit.linepluscorp.com - ଻ਊ ந઱঴ ޷౴ न୒ೞӝ ⚡ - https://lin.ee/WqQCXcm/hrsu