Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Ingress Controller for LINE Private Cloud Platform

Ingress Controller for LINE Private Cloud Platform

Presentation materials at CloudNative Days Tokyo 2022

LINE Developers

November 22, 2022
Tweet

More Decks by LINE Developers

Other Decks in Technology

Transcript

  1. 2 ๺ᖒ ঵ଠ @kanatakita ShotaKitazawa ॴଐ: Verda ωοτϫʔΫ։ൃνʔϜ (2022/04~) ࠷ۙͷۀ຿Ingress

    controller ͷ։ൃ, ৴པੑͷՄࢹԽ ࠷ۙΩϟϯϓΛ࢝Ίͨ ීஈͷΞΠίϯ Speaker
  2. A Private Cloud Service for LINE Developers Verda 3 •

    Verda is the private cloud platform for LINE • This platform hosts many of LINE's services • With a high level of abstraction of compute and network resources, 
 we are continuously improving to enable LINE developers to use 
 a reliable service infrastructure at a low cost 99,000+ Virtual Machines 45,000+ Baremetals 7,600+ Hypervisors
  3. 4 NAT Load Balancer VM / Baremetal MySQL Elasticsearch Object

    Storage Shared Filesystem DNS App engine 
 (like Heroku) K8s And More… The services of Verda
  4. Verda Load Balancer As A Service (LBaaS) 5 Two types

    of Load Balancing ref. Software engineering that supports LINE-original LBaaS https://speakerdeck.com/line_devday2019/software-engineering-that-supports-line-original-lbaas Server Server Server Req Req Req Req Server Server Server Req Req Req Req Layer4 Load Balancer (L4LB) Layer7 Load Balancer (L7LB) a.k.a. Reverse Proxy
  5. L7LB supports TLS termination 6 Flow of activating a certificate

    Certs Management System L7LB API Developer Operator Security team 2. purchase 1. request to purchase Certi fi cation Authority audit 3. sync certificates 4. create LB 4. create • Developers can request to 
 purchase certificates & create 
 HTTPS L7LB after that • Security team audits certificates on Certs Management System 👎 There is only this flow even if DV certificates were sufficient on developers.
  6. L7LB supports TLS termination 7 Flow of upgrading certificates •

    👎 LB Admin has to do the following procedures manually to update existing L7 LBs. • LB Admin edits configs of each LB • LB Admin notifies to users (developers) • Users deploy by the above config • or LB Admin deploys forced by one L7LB API Developers LB for Developer A LB for Developer B LB for Developer C LB Admin ʜ 2. notify 1. create 
 config 3. deploy config 3. force deploy
  7. 8 • Dedicated user cluster • Worker Nodes are created

    on the user Project as VM Instance. • Based on Rancher • And some custom features (Etcd-backup, cluster upgrade, …) • Supported cloud-controller-manager for Verda • Users can use “type LoadBalancer Service” and so on • Possible to pre-install some applications as “add-on” Verda Kubernetes Service (VKS)
  8. Listen HTTPS using LBaaS & VKS 9 Case1: create L7LB

    manually 👎 unsupported scale in/out on VKS Nodes 👎 existing L7LB cannot L7-aware routing 
 (host-based, path-based routing) Node 
 Port Worker Worker Worker create manually L7LB App Pod App Pod
  9. Listen HTTPS using LBaaS & VKS 10 Case2: using cloud-controller-manager

    Node 
 Port Worker Worker Worker Verda ccm Pod L4LB Pod for Ingress App Pod 👎 Users need to prepare L7LB in the user cluster 👎 Users need to manage certificates for TLS create
  10. What’s the problems 11 • Ingress Controller which supported Verda

    L7LB does not exist. • Certificates activation/renewal flow is painful. • Verda L7LB is not L7-aware.
  11. What’s needed to resolve 12 Ingress Controller which supported Verda

    L7LB does not exist. • Develop Ingress Controller Certificates activation/renewal flow is painful. • Develop a mechanism to activate/renew certificates automatically Verda L7LB is not L7-aware. • Develop to extend L7LB API & DB schema
  12. Existing ways for managing certificates 13 github.com/cert-manager/cert-manager Control Plane Svc

    cert-manager Pod 3. Request on ACME 5. Store Secret CR 2. Watch 1. Create Auth DNS 4. HTTP-01 or DNS-01 
 Challenge Worker Worker Worker
  13. Concerns for managing certificates 14 • Accessing to Certs Management

    System & 
 L7LB certificates API by users role must be prohibited • Even if we have to register certificates to ones • We want not to make users manage certificates • Because unnecessary downloads can cause leaks github.com/cert-manager/cert-manager
  14. Existing ways for managing certificates 15 github.com/cert-manager/cert-manager cert-manager Pod Secret

    4. HTTP-01 or DNS-01 
 Challenge ❌ in user cluster ❌ in user cluster 1. Create 3. Request on ACME
  15. Ingress Controller for Verda 16 Design policies • Separate controllers

    (on each user cluster) and server (on shared environment) • Controllers have the responsibility for reconciling external services to desired state • Server has the responsibility for • requesting to issue certificate (DNS-01 challenge on ACME) • registering to Certs Management System • registering to L7LB certificate API (Admin API) • Don’t place certificates on the user clusters
  16. Ingress Controller for Verda 17 Overview Control Plane Node Port

    User Project Worker Worker Worker Controller App Verda DNS Certificate Server Certs Management System End users Developer C-plane side
  17. Ingress Controller for Verda 18 Control Plane Node Port User

    Project Worker Worker Worker Controller App Verda DNS Developer Ingress 1. Developer applies Ingress manifests Sequence for creating resources apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: app spec: tls: - hosts: - example.com rules: - host: example.com http: paths: - path: / pathType: Prefix backend: service: name: app port: number: 30020
  18. Ingress Controller for Verda 19 Control Plane Node Port User

    Project Worker Worker Worker Controller App Verda DNS Developer 2. Ingress controller detects Ingress object & 
 creates objects of ManagedCertificate and 
 RecordSet resources Ingress ManagedCertificate RecordSet Sequence for creating resources watch create
  19. Ingress Controller for Verda 20 Control Plane Node Port User

    Project Worker Worker Worker Controller App Verda DNS Developer 3. RecordSet controller creates 
 A Record with a dummy address 
 to Verda DNS in the Developer’s Project Ingress ManagedCertificate RecordSet Sequence for creating resources example.com. IN A 192.0.2.1 watch create A record
  20. Ingress Controller for Verda 21 Control Plane Node Port User

    Project Worker Worker Worker Controller App Verda DNS Developer 4. ManagedCertificate controller requests 
 CertificateServer to get a certificate Ingress ManagedCertificate RecordSet Sequence for creating resources example.com. IN A 192.0.2.1 Certificate Server C-plane side watch get example.com
  21. Ingress Controller for Verda 22 Control Plane Node Port User

    Project Worker Worker Worker Controller App Verda DNS Developer Ingress ManagedCertificate RecordSet Sequence for creating resources example.com. IN A 192.0.2.1 Certificate Server C-plane side 5. Authentication & Authorization of requests • authenticate: request to Identity API • authorize: check the existence of A records 
 corresponding to the requested FQDN get A record get example.com
  22. Ingress Controller for Verda 23 Control Plane Node Port User

    Project Worker Worker Worker Controller App Verda DNS Developer Ingress ManagedCertificate RecordSet Sequence for creating resources example.com. IN A 192.0.2.1 Certificate Server C-plane side 6. Check existence of the certificate (go to step 11 if a certificate exists 
 and hasn’t revoked) Certs Management System get get example.com get 
 (User API)
  23. Ingress Controller for Verda 24 Control Plane Node Port User

    Project Worker Worker Worker Controller App Verda DNS Developer Ingress ManagedCertificate RecordSet Sequence for creating resources example.com. IN A 192.0.2.1 Certificate Server C-plane side 7. ManagedCertificate controller requests 
 CertificateServer to issue a certificate (CertificateServer does authn/authz too) get A record request challenge 
 (async API) watch
  24. Ingress Controller for Verda 25 Control Plane Node Port User

    Project Worker Worker Worker Controller App Verda DNS Developer Ingress ManagedCertificate RecordSet Sequence for creating resources example.com. IN A 192.0.2.1 _acme-challenge.example.com. IN TXT <snip> Certificate Server C-plane side 8. ACME DNS-01 Challenge • Get token from Let’s Encrypt • POST TXT record to Verda DNS • Get the certificate issued by Let’s Encrypt create 
 TXT record verify
  25. Ingress Controller for Verda 26 Control Plane Node Port User

    Project Worker Worker Worker Controller App Verda DNS Developer Ingress ManagedCertificate RecordSet Sequence for creating resources example.com. IN A 192.0.2.1 Certificate Server C-plane side 9. Register to some services • Certs Management System • L7LB certificates API register 
 (Admin API) register Certs Management System
  26. Ingress Controller for Verda 27 Control Plane Node Port User

    Project Worker Worker Worker Controller App Verda DNS Developer Ingress ManagedCertificate RecordSet Sequence for creating resources example.com. IN A 192.0.2.1 Certificate Server C-plane side 10. ManagedCertificate controller checks 
 existence of a certificate periodically. 
 When it detects to complete challenge, it 
 update status of ManagedCertificate object get Certs Management System update status
  27. Ingress Controller for Verda 28 Control Plane Node Port User

    Project Worker Worker Worker Controller App Verda DNS Developer 11. Ingress controller detects the completion 
 of certificate challenge from the status 
 of ManagedCertificate object. 
 Then Ingress controller creates L7LB. 
 Ingress ManagedCertificate RecordSet Sequence for creating resources watch watch create L7LB (User API) example.com. IN A 192.0.2.1
  28. Ingress Controller for Verda 29 Control Plane Node Port User

    Project Worker Worker Worker Controller App Verda DNS Developer 12. Ingress controller creates LbDeployment 
 object to manage deployment on L7LB. 
 LbDeployment controller deploys L7LB, 
 and LbDeployment object is removed 
 after that. Ingress ManagedCertificate RecordSet Sequence for creating resources watch create & 
 remove LbDeployment example.com. IN A 192.0.2.1 deploy L7LB (User API) Address: 198.51.100.1
  29. Ingress Controller for Verda 30 Control Plane Node Port User

    Project Worker Worker Worker Controller App Verda DNS Developer 13. RecordSet controller detects to update 
 status of Ingress object and updates 
 A Record on Verda DNS. Then End users are able to access to App. Ingress ManagedCertificate RecordSet Sequence for creating resources example.com. IN A 198.51.100.1 update A record watch Address: 198.51.100.1 End users To https://example.com/
  30. Ingress Controller for Verda 31 Good points • Separate Certificate

    Server as shared API from controllers • Grant administrator role to only Certificate Server • Possible to manage rate-limit for Let’s Encrypt in one place • Don’t place certificates on the user clusters • Users never have to manage certificates manually • Separate controllers by each responsibility • For testability: since integration tests using envtest are performed for each controller, separating the controllers by the responsibility will simplify testing.
  31. Implementation Journey 33 System 
 Design Jun. Jul. Aug. Sep.

    Oct. Nov. Apr. ʙʙ ʙʙ I joined Shimizu-san 
 joined Development Release Preparation QA task force task force Release to 
 Verda Dev Build VT extend L7LB remaining implementation Offline Meeting To focus on development tasks during 1 sprint Build an environment where a set of Verda runs as a VM on one machine * deployment preparation * monitoring preparation *Etc... 2022 Start Project Achieved by 2 Project members
  32. Implementation Journey 35 Task force • For focusing on development

    tasks • Decrease interruptions to minimize • Task force members should attend only weekly meeting • How to proceed the task force • Define the MVP to be completed in 1 sprint (2 weeks) in advance • Report the progress on the half-of-sprint weekly meeting • Make a course correction based on comments • Demonstrate the MVP on the end-of-sprint weekly meeting
  33. Implementation Journey 36 Task force • 1st task force •

    Implement “Ingress controller” to create HTTP LB by requesting L7LB API • Implement “Certificate Server” to retrieve certificates and register one to 
 Certs Management System • Implement “ManagedCertificate controller” for UI on Certificate Server • 2nd task force • Implement all controllers that do everything 
 from retrieving certificates to creating HTTPS LBs 👍$POUJOVPVTJODSFNFOUTBMMPXFEBEFUBJMFESFWJFX 
 PGUIFTZTUFNBSDIJUFDUVSFFBDIUJNF
  34. Implementation Journey 37 Virtual Testbed (VT) • It is hard

    to prepare the Verda Component which 
 a development target depends on. • Mock isn’t enough on confirmation of behavior. • If using a shared staging environment, It can only be worked by 1 person. • Virtual Testbed: Build all Verda services by VMs on 1 physical machine • Possible to use as private development environment. • Possible to build by executing scripts (around 1 hour) • Anyone can build at all times
  35. Implementation Journey 38 Virtual Testbed (VT) • In this time,

    we needed to build VT by the following reasons • Ingress Controller requests L7LB API • We must make L7LB allow L7-aware routing 👍 Ready for development in about two weeks by using the VT
  36. Implementation Journey 39 Deployment • Certificate Server is deployed to

    shared environment. • Where to communicate with User VM and Admin APIs • How to deploy Ingress controller to user cluster? » VKS supports an add-on feature 
 realized by “vks-addon-controller”.
  37. • addon-controller applies manifests to all user clusters • Run

    based on addon-controller’s Reconciliation Loop • Possible to update addons by VKS Team • addon-controller generates manifests per environment (eg. K8s version) • Base-manifests are written by jsonnet vks-addon-controller 40 C-plane side addon-controller User Projects addon A addon B addonC manifests … base-manifests manifests
  38. Future works 42 • Collect metrics for User Cluster •

    Support Validation for applied manifests • CEL became Beta on K8s v1.25 • Support to use Secret on 
 User Cluster as certificates • Support GatewayAPI • Support Pod-native Load Balancing
  39. Future works 43 • Collect metrics for User Cluster •

    Support Validation for applied manifests • CEL became Beta on K8s v1.25 • Support to use Secret on 
 User Cluster as certificates • Support GatewayAPI • Support Pod-native Load Balancing For user experience For monitoring For specific service 
 requirements
  40. • The communication via NodePort is more latency • Because

    packets are DNAT-ed by proxy (eg. iptables) on Worker and forwarded to Pod on some Worker. • Especially, ad-technology is strict on latency • Need to be balanced 
 to Pods directly by L7LB • cf. GKE - Container-native load balancing Future works: Pod-native Load Balancing 44 App to Worker address User Projects Requirements on specific services Node Port Worker Worker Worker to Pod address by DNAT
  41. • Advertise Pod address to DC network on BGP •

    Develop CNI Plugin if needed • Add feature on Ingress Controller • Watch EndpointSlice for register Pod address to L7LB • Update Pod status for ReadinessGate • Prepare the approval workflow • Because Pod addresses is limited Future works: Pod-native Load Balancing 45 App Worker Worker Worker to Pod address directly BGP Speaker advertise to 
 DC network User Projects What’s needed to resolve
  42. Conclusion 46 • We developed Ingress Controller for Verda •

    Automated to activate/renew certificates • Supported L7-aware load balancing • Released in about 5 months • Some future works • Let us know if you’re interested!