Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Recap: KubeCon+CloudNativeCon 2021 NA / Kubernetes Meetup Tokyo #47

Recap: KubeCon+CloudNativeCon 2021 NA / Kubernetes Meetup Tokyo #47

Kubernetes Meetup Tokyo #47のRecapです
YouTube: https://youtu.be/fvxp8bUOgtI?t=1893
Event: https://k8sjp.connpass.com/event/229659/

71d7f6cdf5b1934a1b69f0624f5a7523?s=128

Aya (Igarashi) Ozawa

November 26, 2021
Tweet

Transcript

  1. Recap: Automated, Distributed Systems Testing for Kubernetes Controllers Aya Igarashi

    (@Ladicle), SWE at Z Lab Kubernetes Meetup Tokyo #47
  2. https://kccncna2021.sched.com/event/lV0S

  3. • Kuberenetesͷ(Custom)ControllerΛҟৗܥͰਖ਼ ͘͠ಈ͘͜ͱΛอূ͢Δͷ͸೉͍͠ • Fault Injectionͯ͠ControllerͷόάΛൃݟ͢Δͨ Ίͷsieveͱ͍͏OSSΛ։ൃͨ͠ • ಛ௃͸ιʔείʔυͷมߋ͕ෆཁͰɺൃݟͨ͠ό άͷ࠶ݱ͕Մೳͳ͜ͱ

    • cassandra-operator΍percona-xtradb-cluster- operatorͳͲͷOSSͰόάΛ࣮ࡍʹൃݟͨ͠ Recap: distributed-system pitfall \ /
  4. Interesting Points Event-oriented Common Bug Patterns • ͸͡Ίͯίϯτϩʔϥ࣮૷ͯ͠ ͍ͨ࣌ʹ೰ΜͩΑ͏ͳɺϋϚΓ ͕ͪͳϙΠϯτ͕ύλʔϯͱ͠

    ͯݴޠԽ͞Ε͍ͯΔ • πʔϧΛ࢖Θͳͯ͘΋͓֮͑ͯ ͘ͱ࣮૷ͷ࣌ʹࢀߟʹͳΔ͸ͣ (Chaos tool) Inject Fault = Bug!! Find Diff
  5. Bug Pattern 1 ίϯτϩʔϥ͕Πϕϯτͷड͚औ Γʹࣦഊ͢ΔͱɺϦιʔε͕ਖ਼ৗ ʹॲཧ͞Εͳ͍ύλʔϯ Observability-gaps Bug https://github.com/instaclustr/cassandra-operator/issues/398 ACTION౳ʹґଘͤͣಛఆͷΠϕϯ

    τΛऔΓ͜΅ͯ͠΋࣍ͷΠϕϯτ ౳ͰॲཧͰ͖ΔΑ͏ʹ࣮૷͢Δ όάͷྫ: Orphan PVC 1. Scale Down 2. Delete Pod
  6. Bug Pattern 2 Atomicity Violation https://github.com/rabbitmq/cluster-operator/issues/782 όάͷྫ: Reconcileͷ్தͰ༧ظͤ͵Τϥʔ ΍ఀࢭ͕ىͬͯ͜΋࠶։Ͱ͖ΔΑ ͏ʹ࣮૷͢Δ

    Reconcile͕׬ྃ͢Δલʹίϯτ ϩʔϥ͕ఀࢭͯ͠ɺϦιʔε͕த ్൒୺ͳঢ়ଶʹͳΔύλʔϯ 1. Extend PVC volume to 2Gi 2. Delete STS 3. Crash controller 4. restart controller 5. Create New STS reference 1Gi PVC
  7. Bug Pattern 3 Time Traveling https://jira.percona.com/browse/K8SPXC-716 HAߏ੒ͷΫϥελʹ͓͍ͯɺίϯ τϩʔϥͷ࠶ىಈ࣌ʹݹ͍Ϧιʔ εΛऔಘͯ͠όάʹͳΔύλʔϯ Ϧιʔεͷόʔδϣϯ΍ґଘؔ܎

    (OwnerReference)͕ਖ਼͍͔͠ͳͲ ॲཧલͷόϦσʔγϣϯʹ஫ҙ όάͷྫ: 1. Create Cluster w/ same ID 2. Receive old event Reference invalid STS...
  8. References Event Schedule: https://kccncna2021.sched.com/event/lV0S YouTube: https://youtu.be/6JnhjgOaZVk GitHub: https://github.com/sieve-project/sieve Research Paper:

    https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s11-sun.pdf
  9. Thank you! Aya Igarashi, Z Lab Twitter & GitHub: @Ladicle