Kubernetesを拡張して日々のオペレーションを自動化する

Cloud Native Days Tokyo 2019

Transcript

1. 

   ೔ʑͷΦϖϨʔγϣϯΛ
   ࣗಈԽ͢Δ
   Cloud Native Days Tokyo 2019
   Aya Igarashi @Ladicle
   Λ֦ுͯ͠
   Kubernetes
   

   View Slide

2. @Ladicle
   Software Engineer
   

   View Slide

3. Z Lab
   Automate
   Kubernetes
   Cluster
   Management
   KubernetesΛར༻ͯ͠Kubernetes Clusterͷ
   ࡞੒ɺ࡟আɺΞοϓάϨʔυɺো֐෮چͳͲͷ
   ΦϖϨʔγϣϯΛࣗಈԽ͍ͯ͠Δ
   

   View Slide

4. Why We Should
   Automate?
   

   View Slide

5. Why We Should
   Automate?
   If a human operator needs to touch
   your system during normal operations,
   you have a bug. The definition of
   normal changes as your systems grow.
   Carla Geisser, Google SRE
   Chapter 5 - Eliminating Toil
   “
   ”
   

   View Slide

6. 50
   clusters
   2018
   Do Not Work O(N)
   Manual Operations
   with Service Growth
   Kubernetes Clusterͷ؅ཧ͸ো֐࣌ͷ෮
   چ΍ΞοϓάϨʔυ͕͋Δ͕ɺαʔϏεͷ
   ੒௕ʹ߹Θͤͯ࡞ۀ΋O(N)Ͱ૿΍ͯ͠͠
   ·͏ͱ؅ཧ͖͠Εͳ͍
   Automation is Required
   ϠϑʔࣾͰͷར༻࣮੷˞
   ※ Cloud Native Days Tokyo 2019 ʮϠϑʔͷΫϥ΢υωΠςΟϒ΁ͷऔΓ
   ૊ΈͱͦΕΛࢧ͑ΔγεςϜ։ൃʯΑΓ
   400
   clusters
   2019
   x8
   

   View Slide

7. Faster
   ਓ͕ؒखॱॻ௨ΓʹίϐϖͰ
   ࡞ۀΛਐΊΔΑΓ΋ϓϩάϥ
   Ϝ࣮ߦͷํ͕ૣ͘ɺਂ໷࡞
   ۀͳͲ࣌ؒత੍໿΋ͳ͍
   Scalability
   ਓΑΓૉૣ͘܁Γฦ͠ॲཧ
   Λಘҙͱ͢ΔͨΊɺख࡞ۀ
   ΑΓ΋γεςϜͷൃలʹ߹Θ
   ͤͯεέʔϧ͠΍͍͢
   Reliability
   ਓؒ͸ϛεΛ͢Δ΋ͷͳͷ
   Ͱɺ৴པੑΛอͭΑ͏ʹɺಉ
   ͡࡞ۀΛਖ਼֬ʹ܁Γฦ͢͜
   ͱ͸ෆՄೳʹ͍ۙ
   The Profit of
   Automation
   

   View Slide

8. Why kubernetes?
   

   View Slide

9. Lv.01
   No
   Automation
   02
   Externally
   Maintained
   System-specific
   Automation
   03
   Externally
   Maintained
   Generic
   Automation
   04
   Internally
   Maintained
   System-specific
   Automation
   05
   Systems That
   Don’t Need
   Any
   Automation
   Automation
   Evolution
   Site Reliability Engineering
   Chapter 7 - A Hierarchy of Automation Classes
   

   View Slide

10. Platform Style
    Automation Makes
    Your System
    Robust
    Observability
    εΫϦϓτͷϝτϦΫεΛऔΔͷ͸େม͕ͩɺϓ
    ϥοτϑΥʔϜࣜͰ͋Ε͹ɺैདྷͷϞχλϦϯάγ
    εςϜ͕࢖͑ɺγεςϜͷ಺෦ঢ়گͷ೺Ѳʹ໾ཱͭ
    Testability
    σϓϩΠεΫϦϓτ͸සൟʹςετ͞Εͳ͍܏޲ʹ
    ͋Δ͕ɺϓϥοτϑΥʔϜͱͯ͠ߏங͢Ε͹ैདྷͷ
    ιϑτ΢ΣΞͱಉ͡Α͏ʹςετͰ͖Δ
    Maintainability
    γεςϜͷதʹॲཧؚ͕·Ε͍ͯΔͷͰɺࣗಈԽε
    ΫϦϓτΛແࢹͨ͠ઃܭʹͳΔ͜ͱ΋ͳ͘ɺϝϯς
    φϯε͠ଓ͚Δ͜ͱ͕Ͱ͖Δ
    

    View Slide

11. CRDΛ͸͡Ίͱͨ͠๛෋ͳ֦ுػೳΛඋ͓͑ͯΓɺ
    ج൫Λ࡞Δج൫ͱͯ͠࢖͍উख͕Α͍ͨΊ
    #02
    ௐ੔ϞσϧʹΑͬͯࣗ཯ͨ͠γεςϜͷ
    ߏஙՄೳͰ͋Γɺ·ͨɺखଓ͖తͰ͸ͳ
    ͘એݴతͳAPIΛఏڙͰ͖ΔͨΊ
    #01
    2 Reasons
    Why You Should Customize
    Kubernetes
    For Automation
    Reconciliation Model Custom Resource Definition
    

    View Slide

12. What’s Difference between
    Procedural Model and
    Reconciliation Model?
    Advantages of k8s #01
    Script: +Sc
    Server v1
    Desire:
    Current:
    Server v1
    Reconciler
    Observe
    Procedural Model Reconciliation Model
    Run Script Update Desire State
    

    View Slide

13. Advantages of k8s #01
    Script: +Sc
    Server v2
    Desire:
    Current:
    Server v2
    Reconciler
    Observe
    Reconcile
    Deploy
    Procedural Model Reconciliation Model
    What’s Difference between
    Procedural Model and
    Reconciliation Model?
    Run Script Update Desire State
    

    View Slide

14. Script: +Sc
    Server v2
    Desire:
    Current:
    Server v2
    Reconciler
    Observe
    Script: Fix B
    Failure Failure
    Advantages of k8s #01
    Procedural Model Reconciliation Model
    What’s Difference between
    Procedural Model and
    Reconciliation Model?
    Run Script
    

    View Slide

15. Reconciler Provides
    Declarative API to Keep a
    System Desired State
    Advantages of k8s #01
    Script: +Sc
    Server v3
    Desire:
    Current:
    Server v2
    Reconciler
    Recovery
    Procedural Model Reconciliation Model
    Script: Fix B
    Reconcile
    Observe
    Run Script
    

    View Slide

16. Advantages of k8s #02
    Kubernetes APIʹಠࣗͷϦιʔεΛఆٛ͢ΔͨΊͷػೳ
    CRDͰఆٛͨ͠Ϧιʔε͸ΧελϜϦιʔε(CR)ͱݺ͹ΕΔ
    Custom Resource Definition(CRD)
    /apis/apps/v1/namespaces/default/deployments
    /apis///namespaces/default/
    Platform
    Platform
    Platform
    API
    API
    CRD Defines Custom API
    The Automation Platform Can Be
    Further Extended
    Call
    Call
    Extended
    ɹɹɹ ed
    API Server Endpoints
    API
    

    View Slide

17. CRD
    Kubernetes
    Cluster
    CRD
    Kubernetes
    Cluster
    Reconciler
    CRD
    Kubernetes
    Cluster
    Reconciler
    Reconciler
    CR
    Kubernetes
    Cluster
    Z Lab Automate
    Kubernetes Cluster
    Management Using k8s
    Reconciler
    CRD
    Kubernetes
    Cluster
    Call
    API API Server
    Observe
    Register
    Reconcile
    

    View Slide

18. How Do We Customize
    Kubernetes?
    

    View Slide

19. Development
    Flow
    #01
    Design
    #03
    Testing
    #02
    Implementation
    #04
    Maintenance
    Kubernetes͕ԿΛͲ͜·Ͱอোͯ͠
    ͍Δͷ͔Λ೺Ѳ͠ɺReconciliation
    Loopͷಛ௃Λཧղ͢Δ
    ։ൃ؀ڥͷ੔උ͕ਐΜͰ͖͍ͯΔͷͰ
    Framework͸ੵۃతʹ׆༻͢Δ
    Kubernetesຊମ͸ྑ͍αϯϓϧίʔυ
    ௨ৗͷιϑτ΢ΣΞ։ൃͱಉ͡Α͏ʹ
    ςετͷ༻్ʹԠ࣮ͯ͡૷ํ๏΍ςε
    τ؀ڥͷߏஙํ๏Λݕ౼͢Δ
    όʔδϣϯߋ৽ͳͲɺ·ͩൃల్্ͷ
    ػೳ͕ଟ͍ͨΊɺKubernetes΍पลͷ
    ϓϩμΫτͷΞοϓσʔτΛ௥͏
    

    View Slide

20. #01
    Design
    Kubernetes͕ԿΛͲ͜·Ͱอোͯ͠
    ͍Δͷ͔Λ೺Ѳ͠ɺReconciliation
    Loopͷಛ௃Λཧղ͢Δ
    

    View Slide

21. Design Automation Platform
    Which Resource to Observe and
    How to Reconcile
    CR
    Reconciler
    CRD
    Call
    API API Server
    Observe
    Register
    Reconcile
    KubernetesͷػೳΛར༻͢Ε͹APIαʔό΍ΫϥΠ
    Ξϯτͷ։ൃ͸ෆཁͷͨΊɺϏδωεϩδοΫ෦෼
    ͷReconcilerͱॲཧର৅ͷCRDઃܭʹूதͰ͖Δ
    

    View Slide

22. Design CRD and Reconciler
    Example: Scaling the Number of RS replicas
    from 1 to 2
    ReplicaSet
    Controller
    API Server
    Observe
    ReplicaSet
    & Pod
    Reconcile
    apiVersion: apps/v1
    kind: ReplicaSet
    metadata:
    name: sample
    spec:
    replicas: 2
    …
    status:
    replicas: 1
    …
    apiVersion: v1
    kind: Pod
    metadata:
    name: sample-6f477f….
    …
    spec:
    containers:
    1. Analyse Resource
    2. Create Pod
    3. Update .Status
    ௐ੔ର৅ͱͳΔϦιʔεͷܾఆ
    ௐ੔ܖػͱͳΔ؂ࢹϦιʔεͷܾఆ
    ϦιʔεΛͲ͏ௐ੔͍͔ͯ͘͠
    ͭ·Γ… ҎԼͷ఺Λઃܭ͢Ε͹Α͍
    

    View Slide

23. CRD
    Design Tips
    CRD͸ࣗ༝ʹεΩʔϚΛઃఆͰ͖Δ
    ͕ɺ.Spec/.StatusΛఆٛ͠ͳ͍ͱk8s
    ͷ֤छԸܙ͕ड͚ΒΕͳ͍ɻ·ͨɺඪ
    ४ͷϦιʔεߏ଄͸ࢀߟʹͳΔͷͰɺε
    ΩʔϚͱͦͷಛ௃Λ೺Ѳ͓ͯ͘͠ͱ˓
    Values for Analysis and Action
    Changeability Condition
    ௐ੔ʹඞཁͳσʔλΛ੔ཧ͢Δ
    status:
    conditions:
    - lastProbeTime: null
    lastTransitionTime: "2019-05-01T15:05:05Z"
    status: "True"
    type: Initialized
    - lastProbeTime: null
    lastTransitionTime: "2019-05-01T15:32:41Z"
    status: "True"
    type: Ready
    - lastProbeTime: null
    Ϧιʔεͷ஋͸େ·͔ʹ .Spec ͱ .Status ͷࠩ෼͔
    ΒఆٛͱͷζϨݕग़ʹ࢖͏஋ͱɺζϨमਖ਼ͷΞΫ
    γϣϯ࣌ʹͷΈ࢖͏Φϓγϣϯʹ෼͚ΒΕΔɻ
    Φϓγϣϯ͸ߏ଄తʹอ࣋͢Δͱมߋ͠΍͍͢ɻ
    Condition͸มߋ͢Δ͜ͱ͕ଟ͍ͷͰ഑ྻͰ·ͱΊΔ
    

    View Slide

24. Update
    .Status
    Reconciler
    Design Tips
    .spec/.statusΛॲཧ͢ΔReconcilerΛ
    ੍ݶ͠ɺௐ੔ର৅͸ݪଇ1Ϧιʔεʹࢭ
    Ίґଘؔ܎ΛݮΒ͢ɻؔ࿈Ϧιʔεͷࢀ
    র͕ඞཁͳ৔߹͸ɺLabel΍.specͷε
    ΩʔϚΛ޻෉ͯ͠ܭࢉྔΛ࡟ݮ͢Δɻ
    ԿΛ͍ͭͲ͏ௐ੔͢Δ͔
    ChaosCron
    Reconciler
    ChaosSet
    Reconciler
    Chaos
    Reconciler
    Chaos
    C
    ChaosCron
    =A
    ChaosSet
    =B
    A
    ChaosCron
    B
    ChaosSet
    ChaosCron
    =A
    Create
    .Spec
    

    View Slide

25. Deletion Propagation Policy
    How to Delete Resources
    Correctly?
    ऴྃॲཧ͸.SpecͰ͸ͳ͘
    MetadataʹԠͯ͡ॲཧ͢Δ
    GC
    default
    Background
    Parent
    Child
    Parent
    Child
    Foreground
    Finalizer
    1. Delete Object 1. Delete Object
    Owner
    Reference
    2. Delete Child
    2. Added Finalize by
    API Server
    3. Delete Child and
    Remove Finalizer
    4.Deleted
    k8s GCΛར༻͢ΔͨΊʹΦϒδΣΫτʹ͸
    OwnerReferenceΛઃఆ͢Δ
    K8s GCର৅֎ͷ֎෦ϦιʔεΛࢠʹ࣋ͭΑ
    ͏ͳΦϒδΣΫτʹ͸FinalizerΛར༻͢Δ
    k8sඪ४Ϧιʔεʹର͢ΔFinalizer͸
    Validation͕͔͔͍ͬͯΔͷͰ໋໊ʹ஫ҙ
    / (ex. ladicle.com/gopher-cleanup)
    

    View Slide

26. #02
    Implementation
    ։ൃ؀ڥͷ੔උ͕ਐΜͰ͖͍ͯΔͷͰ
    Framework͸ੵۃతʹ׆༻͢Δ
    Kubernetesຊମ͸ྑ͍αϯϓϧίʔυ
    

    View Slide

27. controller-
    runtime
    client-go
    controller-
    tools
    Kubebuilder
    client-go
    k8sͷϕʔεͱͳΔclientϥΠϒϥϦͰɺcontroller-
    runtime͔Β͸ΩϟογϡपΓͷॲཧ͕࢖ΘΕ͍ͯΔ
    controller-runtime
    KubebuilderͷαϒϓϩδΣΫτͰɺReconcilerΛ։
    ൃ͢ΔͨΊʹඞཁͳఆܕॲཧͷϥΠϒϥϦ
    controller-tools
    ಉ͘͡αϒϓϩδΣΫτͰɺReconcilerͰར༻͢ΔϚ
    χϑΣετ΍ίʔυΛੜ੒͢ΔδΣωϨʔλ
    Kubebuilder
    ίʔυͷ਽ܗ/ϏϧυϑΝΠϧ/ςετ؀ڥߏஙεΫϦϓ
    τͳͲ։ൃʹඞཁͳҰࣜΛੜ੒ͯ͘͠ΕΔSDK
    Libraries and Tools for
    Developing
    Reconciler and CR
    

    View Slide

28. controller-runtime
    Overview
    API Server
    Operator
    Cache
    Manager
    Controller
    Reconciler
    Controller
    Reconciler
    Evt
    KV
    Evt
    

    View Slide

29. Reliability
    Scalability
    Operator
    Development Tips
    

    View Slide

30. Filter Reconciliation Target
    Controller
    Watcher
    Evt EvtEvt Source Event
    Handler
    Obj
    Evt
    Scalability
    Watcher
    Obj
    Evt
    …
    ctrl.NewControllerManagedBy(mgr).
    # Set Default Watcher for Reconcile Resource
    For(&event.CNDT{}).
    # Set Default Watcher for Own Resource
    Owns(&event.Session{}).
    # Set Custom Watcher
    Watches(
    &source.Channel{Source: events},
    &handler.EnqueueRequestsFromMapFunc{
    ToRequests: handler.ToRequestsFunc(
    func{a handler.MapObject} []reconcile.Request {
    if a.Meta.GetLabels()["year"] != "2019" {
    return nil
    }
    return []reconcile.Request{
    {
    Name: a.Meta.GetLabels()[“cndt”],
    Namespace: a.Meta.GetNamespace(),
    },
    },
    }),
    },
    &predicate.Funcs{
    Predicate Req
    Evt
    Source Event
    Handler
    Predicate
    Evt Req
    For( )
    Owns( )
    

    View Slide

31. Set the appropriate number of worker,
    and choose requeue method
    Scalability
    Controller
    Workqueue
    Worker
    …
    Req Req Req
    Reconciler
    Handler
    Req
    Worker
    Reconciler
    Handler
    Req
    Watcher
    Watcher
    vent
    andler
    vent
    andler
    Req
    Req
    Requeue
    Deduplication & RateLimited
    

    View Slide

32. Error and Retry
    Scalability & Reliability
    Worker
    …
    Reconciler
    Handler
    Req
    Worker
    Reconciler
    Handler
    Req
    Requeue
    mited
    Requeue Method
    RateLimited
    Error != nil
    Error == nil && Requeue = True
    After X duration (Reset Counter)
    Error == nil && RequeueAfter = X
    Delete from WorkQueue
    Else
    

    View Slide

33. API Server
    Operator
    Operator
    Operator
    Leader Election
    Reliability
    Leader
    Config
    Map
    Success to
    update CM
    ResourceVersio
    n is too Old..
    ctrlOpts := ctrl.Options{
    LeaderElection: leConfig,
    LeaderElectionNamespace: leNamespace,
    LeaderElectionID: leName,
    }
    Optimistic
    Resource Lock
    LeaderͷΈ͕Controller͕ىಈ͠ɺ࢒Γ
    ͷFollower͸ϗοτελϯόΠʹͳΔ
    CMͷݖݶʹ஫ҙ
    Just set LE configuration to Manager
    

    View Slide

34. Syntax and Semantic
    Validation
    Reliability
    KubeCon EU 2018 – Sig API Machinery Deep
    
    

    View Slide

35. #03
    Testing
    ௨ৗͷιϑτ΢ΣΞ։ൃͱಉ͡Α͏ʹς
    ετͷ༻్ʹԠ࣮ͯ͡૷ํ๏΍ςετ؀
    ڥͷߏஙํ๏Λݕ౼͢Δ
    

    View Slide

36. End-to-End Test
    Ginkgo & Gomega
    Unit-Test
    FakeClient
    ΤϯυπʔΤϯυςετ͸ɺΞϓϦέʔγϣϯ
    ͷϑϩʔ͕࠷ॳ͔Β࠷ޙ·ͰઃܭͲ͓Γʹ࣮ߦ
    ͞Ε͍ͯΔ͔Ͳ͏͔Λςετ͢Δख๏ɻ
    Ginkgo͸BDDελΠϧͷGoςετϑϨʔϜ
    ϫʔΫɻجຊతʹ͸GomegaͷmatcherϥΠ
    ϒϥϦͱར༻͢Δɻ
    controller-runtimeͰ͸ɺe2e༻ͷςετ؀ڥ
    ߏங(όΠφϦํࣜ)ͱkubeconfigಡΈࠐΈ෦
    ෼ͷϥΠϒϥϦΛఏڙ͍ͯ͠Δɻ
    ϊʔϚϧͳGoͷUnitςετΛॻ͘ํࣜɻ
    Reconcilerʹ౉͢KubernetesͷAPI ClientΛ
    fakeClientʹࠩ͠ସ͑ɺظ଴͢ΔAPIΞΫγϣ
    ϯΛઃఆͨ͠ΓɺΞΫγϣϯޙͷϦιʔεঢ়ଶ
    ͷ֬ೝʹར༻͢Δɻ(Finalizer౳͸ະରԠ)
    Kubebuilder͸ϝϯςͷ͠ਏ͔͞Β
    FakeClientํࣜΛඇਪ঑ͱ͍ͯ͠Δ
    OperatorSDK͸UT & e2eͷ2ஈߏ੒
    ̎ Testing Methods
    

    View Slide

37. Kubernetesͷςετ༻ʹ։ൃ͞
    ΕͨπʔϧͰ1ϊʔυΛ1ίϯςφ
    ͱͯ͠ىಈ͢ΔɻखܰʹϚϧν
    ϊʔυ΍ϚϧνΫϥελߏ੒ͷࢼ
    ݧ͕Ͱ͖Δ
    ࣮؀ڥͰςετ͸ߴίετ͕ͩɺ
    Node͕ίϯςφͰ͸ಈ࡞֬ೝͰ
    ͖ͳ͍ػೳ΍࣮؀ڥͰͷ֬ೝͷͨ
    Ίલೋͭͱ૊Έ߹Θ࣮ͤͯࢪ͢Δ
    EtcdͱAPI Serverͷ2όΠφϦͷ
    ΈΛىಈͤ͞ΔͨΊɺܰྔͩ
    ͕ɺ”APIϨϕϧ”ͷಈ࡞֬ೝ͔͠Ͱ
    ͖ͳ͍ɻControllerManager΋
    Kubelet΋͍ͳ͍ͷͰUTతʹͳΔ
    Container
    Kind
    Full Cluster
    Your Real Platform
    API server
    Testing Framework
    How to setup
    Kubernetes Cluster for Test?
    

    View Slide

38. #04
    Maintenance
    όʔδϣϯߋ৽ͳͲɺ·ͩൃల్্ͷػ
    ೳ͕ଟ͍ͨΊɺKubernetes΍पลͷϓ
    ϩμΫτͷΞοϓσʔτΛ௥͓͏
    

    View Slide

39. Event
    Recorder
    To Understand
    Internal Behavior
    kubectl get event΍ɺdescribeͨ࣌͠ʹऩूͯ͠Α͠ͳʹ֘౰͢ΔϦ
    ιʔεͷEventsϑΟʔϧυʹදࣔͯ͘͠ΕΔEvent
    k8sຊՈͷe2eͷΑ͏ʹςετ࣌ͷEventΛऩूͨ͠Γɺো֐͕ى͖ͨ࣌
    ʹ࠷ॳͷ͋ͨΓΛ͚ͭΔͨΊʹ֬ೝͨ͠ΓɺΫϥελΛ࡞੒͢Δͱ͖ͷ
    Ϧιʔεͷಈ͖Λ؍࡯ͨ͠Γͱɺͬ͘͟Γͱͨ͠ڍಈͷ೺Ѳʹศར
    $ kubectl get event
    LAST SEEN TYPE REASON OBJECT MESSAGE
    5s Normal Scheduled pod/cndt-cbb75cdc5-mws7l Successfully assigned default/cndt-cbb75cdc5-mws7l to worker3
    4s Normal Pulling pod/cndt-cbb75cdc5-mws7l Pulling image "gcr.io/hello-minikube-zero-install/hello-node"
    5s Normal SuccessfulCreate replicaset/cndt-cbb75cdc5 Created pod: cndt-cbb75cdc5-mws7l
    5s Normal ScalingReplicaSet deployment/cndt Scaled up replica set cndt-cbb75cdc5 to 1
    

    View Slide

40. Operator
    Metrics
    To Understand
    Internal Behavior
    ControllerΛӡ༻্͍ͯ͘͠ͰඞཁͳWorkqueueͷΩϡʔΠϯά਺΍
    Τϥʔ਺ͳͲɺϝτϦΫεެ։ʹඞཁͳ΋ͷ͸controller-runtimeͰ༻
    ҙ͞Ε͍ͯΔͨΊɺManagerͷઃఆ஋Λ௥Ճ͢Δ͚ͩɻ
    ϝτϦΫεσʔλ͸ɺPrometheus Metric Format
    $ curl http://localhost:8080/metrics
    # HELP controller_runtime_reconcile_errors_total Total number of reconcile errors per controller
    # TYPE controller_runtime_reconcile_errors_total counter
    controller_runtime_reconcile_errors_total{controller="mysql-controller"} 10
    # HELP controller_runtime_reconcile_queue_length Length of reconcile queue per controller
    # TYPE controller_runtime_reconcile_queue_length gauge
    

    View Slide

41. KubeCon EU 2018 – Sig API Machinery Deep
    
    Operator
    Upgrade
    

    View Slide

42. How to Deploy
    The Automated Platform?
    Operator
    Operator
    Operator
    Operator
    Bundle
    Deploy Automatied Platform
    

    View Slide

43. Conclusion
    KubernetesΛ֦ுͯ͠೔ʑͷΦϖϨʔγϣϯΛࣗಈԽ͢Δʹ͸?
    

    View Slide

44. Customize
    Kubernetes
    For Automation
    γεςϜͷ੒௕ʹରԠ͢ΔͨΊʹࣗಈԽ͕ඞཁ
    Platform StyleͷࣗಈԽΛ͢ΔͱγεςϜ͕ΑΓݎ࿚ʹͰ͖
    ͨΓɺ͞ΒͳΔࣗಈԽ͕ՄೳͱͳΔ
    ͜ͷΑ͏ͳࣗಈԽΛਐΊΔʹ͋ͨͬͯɺReconciliation
    ModelΛͱΓɺCRDͳͲͷ֦ுػೳ͕๛෋ͳKubernetes͸
    ج൫Λ࡞ΔͨΊͷج൫ͱͯ͠༏ल
    

    View Slide

45. Scalability
    ✓ Reconcilerͷ؂ࢹର৅ͱλ
    ΠϛϯάΛߜΔ
    ✓ Cache׆༻ͷͨΊ؂ࢹϦ
    ιʔεʹԠͯ͡Manager
    Λ۠੾Δ
    ✓ ద੾ͳWorker਺Λ
    Controllerʹઃఆ͢Δ
    ✓ ௐ੔ϧʔϓΛγϯϓϧʹ
    Maintenancebility
    ✓ SDKΛར༻͢Δ
    ✓ EventRecorder΍Metrics
    Λઃఆͯ͠ঢ়ଶ೺Ѳ͢Δ
    ✓ ConversionWebhookͰ
    ޙํޓ׵ੑΛอͭ
    ✓ ࣗ਎ͷϦϦʔε/σϓϩΠ
    ΋ࣗಈԽʹ૊ΈࠐΉ
    Reliability
    ✓ Self-HealingͰ͖ΔΑ͏
    ʹద੾ͳ؂ࢹͱௐ੔Λߦ͏
    ✓ ద੾ͳRetryΛઃఆ͢Δ
    ✓ FinalizerͰϦιʔεΛਖ਼
    ৗऴྃͤ͞Δ
    ✓ ςετΛ࣮ࢪ͢Δ
    ✓ LEΛઃఆ͠ো֐ʹඋ͑Δ
    Designing
    Automation Platform for RSM
    

    View Slide

46. WE ARE HIRING!
    ͝ڵຯͷ͋Δํ͸ɺZ Lab ࣾһʹ௚઀͝࿈བྷ͍ͩ͘͞
    

    View Slide

47. Thank You!
    For you time & we’ll see you soon
    @ladicle
    

    View Slide