Recap: Automated, Distributed Systems Testing for Kubernetes Controllers Aya Igarashi (@Ladicle), SWE at Z Lab Kubernetes Meetup Tokyo #47

https://kccncna2021.sched.com/event/lV0S

• Kuberenetesͷ(Custom)ControllerΛҟৗܥͰਖ਼ ͘͠ಈ͘͜ͱΛอূ͢Δͷ͸೉͍͠ • Fault Injectionͯ͠ControllerͷόάΛൃݟ͢Δͨ Ίͷsieveͱ͍͏OSSΛ։ൃͨ͠ • ಛ௃͸ιʔείʔυͷมߋ͕ෆཁͰɺൃݟͨ͠ό άͷ࠶ݱ͕Մೳͳ͜ͱ • cassandra-operator΍percona-xtradb-cluster- operatorͳͲͷOSSͰόάΛ࣮ࡍʹൃݟͨ͠ Recap: distributed-system pitfall \ /

Interesting Points Event-oriented Common Bug Patterns • ͸͡Ίͯίϯτϩʔϥ࣮૷ͯ͠ ͍ͨ࣌ʹ೰ΜͩΑ͏ͳɺϋϚΓ ͕ͪͳϙΠϯτ͕ύλʔϯͱ͠ ͯݴޠԽ͞Ε͍ͯΔ • πʔϧΛ࢖Θͳͯ͘΋͓֮͑ͯ ͘ͱ࣮૷ͷ࣌ʹࢀߟʹͳΔ͸ͣ (Chaos tool) Inject Fault = Bug!! Find Diff

Bug Pattern 1 ίϯτϩʔϥ͕Πϕϯτͷड͚औ Γʹࣦഊ͢ΔͱɺϦιʔε͕ਖ਼ৗ ʹॲཧ͞Εͳ͍ύλʔϯ Observability-gaps Bug https://github.com/instaclustr/cassandra-operator/issues/398 ACTION౳ʹґଘͤͣಛఆͷΠϕϯ τΛऔΓ͜΅ͯ͠΋࣍ͷΠϕϯτ ౳ͰॲཧͰ͖ΔΑ͏ʹ࣮૷͢Δ όάͷྫ: Orphan PVC 1. Scale Down 2. Delete Pod

Bug Pattern 2 Atomicity Violation https://github.com/rabbitmq/cluster-operator/issues/782 όάͷྫ: Reconcileͷ్தͰ༧ظͤ͵Τϥʔ ΍ఀࢭ͕ىͬͯ͜΋࠶։Ͱ͖ΔΑ ͏ʹ࣮૷͢Δ Reconcile͕׬ྃ͢Δલʹίϯτ ϩʔϥ͕ఀࢭͯ͠ɺϦιʔε͕த ్൒୺ͳঢ়ଶʹͳΔύλʔϯ 1. Extend PVC volume to 2Gi 2. Delete STS 3. Crash controller 4. restart controller 5. Create New STS reference 1Gi PVC

Bug Pattern 3 Time Traveling https://jira.percona.com/browse/K8SPXC-716 HAߏ੒ͷΫϥελʹ͓͍ͯɺίϯ τϩʔϥͷ࠶ىಈ࣌ʹݹ͍Ϧιʔ εΛऔಘͯ͠όάʹͳΔύλʔϯ Ϧιʔεͷόʔδϣϯ΍ґଘؔ܎ (OwnerReference)͕ਖ਼͍͔͠ͳͲ ॲཧલͷόϦσʔγϣϯʹ஫ҙ όάͷྫ: 1. Create Cluster w/ same ID 2. Receive old event Reference invalid STS...

References Event Schedule: https://kccncna2021.sched.com/event/lV0S YouTube: https://youtu.be/6JnhjgOaZVk GitHub: https://github.com/sieve-project/sieve Research Paper: https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s11-sun.pdf

Thank you! Aya Igarashi, Z Lab Twitter & GitHub: @Ladicle