Kubernetesで分散処理をやってみる-Kafkaを添えて-

Transcript

1. 2025-09-13 / ෼ࢄٕज़ษڧձ~Ϋϥ΢υͱ෼ࢄٕज़ͷΑ΋΍·࿩~ KubernetesͰ෼ࢄॲཧΛͯ͠ΈΔ Katsuya Ogata
2. ࣗݾ঺հ • ໊લ: ॹํ ࠀ࠸ (͓͕ͨ ͔ͭ΍) • ॴଐ: େࡕେֶ৘ใֶݚڀՊ • ࣗ୐αʔόʔ࢝Ί·ͨ͠ • Ӄࢼظؒதʹ5kgଠΓ·ͨ͠ • X: @ogata_katsuya • ݸਓ: www.ogatakatsuya.com
3. ෼ࢄॲཧλεΫͷ঺հ
4. ϨΠτϨʔγϯά ޫͷ൓ࣹɾ۶ંΛܭࢉͯ͠ɺߴ඼࣭ͳը૾Λੜ੒͢Δ • ޫઢͷ௥੻ʹΑΔը૾ੜ੒ख๏ • Ծ૝Χϝϥ͔ΒͷޫઢΛγϛϡϨʔγϣϯ • ޫͷ൓ࣹɾ۶ંɾӨΛ࠶ݱ͢Δ • ܭࢉίετ͕ߴ͍ • ө૾΍CGɺݐஙՄࢹԽʹ͓͍ͯ࢖ΘΕΔ https://www.scratchapixel.com/lessons/3d-basic-rendering/introduction-to-ray-tracing/implementing-the-raytracing-algorithm.htm ޫઢͷ௥੻ʹΑΔը૾ੜ੒ख๏
5. ϨΠτϨʔγϯά ޫͷ൓ࣹɾ۶ંΛܭࢉͯ͠ɺߴ඼࣭ͳը૾Λੜ੒͢Δ • ܭࢉྔ O ( P * R * S * D) • P: ϐΫηϧ਺ • R: ϐΫηϧ͋ͨΓͷޫઢ਺ • S: ΦϒδΣΫτ਺ • D: ൓ࣹ/۶ંਂ౓ • ࠓճͷ৔߹ɺO ( 800*800 * 50 * 8 * 8 ) = O ( 2,048,000,000 ) • ͜ΕΛ100pixel͝ͱʹ෼ׂͯ͠ɺॲཧΛฒྻ࣮ߦ͢Δ
6. Kubernetesͷ෼ࢄॲཧ Node/PodʹॲཧΛ෼ࢄ͢Δ͜ͱʹΑ࣮ͬͯݱ͢Δ https://kubernetes.io/ja/docs/concepts/overview/components/
7. ߟ͑ΒΕΔΞʔΩςΫνϟ ͋ΔPod͕WorkerʹԿΒ͔ͷ௨৴खஈͰσʔλΛ౉ͯ͠ฦ͢
8. ߟ͑ΒΕΔΞʔΩςΫνϟ ͋ΔPod͕WorkerʹԿΒ͔ͷ௨৴खஈͰσʔλΛ౉ͯ͠ฦ͢ • σϝϦοτ • Worker PodͷΞυϨε͸ʁʁ • Producer - Workerؒͷ௨৴खஈ͸ʁʁ • ࠶ૹɾϦτϥΠॲཧ͕ඞཁʹͳΔʁʁ • ॲཧதʹαʔόʔ͕མͪͨΒϝοηʔδ͸ফ͑ͯ͠·͏ʁʁ • ॲཧ͕ॏ͘ͳͬͨ࣌ʹͲ͏΍ͬͯରԠ͢Δʁʁ
9. ͦ͏ͩɺKafkaΛ࢖͓͏
10. KafkaΛ༻͍ͨ෼ࢄॲཧ Pub/Sub messaging model https://aws.amazon.com/jp/what-is/pub-sub-messaging/
11. KafkaΛ༻͍ͨ෼ࢄॲཧ Pub/Sub messaging model • ϝϦοτ • ௨৴ͱϏδωεϩδοΫ͕ૄ݁߹ʹͳΔ • PublisherͱSubscriberΛ༰қʹεέʔϧͰ͖Δ • ඇಉظͰॲཧΛ࣮ߦͰ͖Δ • ΠϕϯτۦಈΞʔΩςΫνϟ΁ͷԠ༻ • ଱ো֐ੑ
12. ͜ΕΛKubernetes্Ͱ࣮ݱ͍ͨ͠
13. KubernetesͷΞʔΩςΫνϟ Pub/Sub messaging model
14. KubernetesͷΞʔΩςΫνϟ Pub/Sub messaging model
15. KubernetesͷΞʔΩςΫνϟ Pub/Sub messaging model
16. KubernetesͷΞʔΩςΫνϟ Pub/Sub messaging model
17. ࣮ݧ
18. ࣮ݧ಺༰ GKEΛ༻͍ͯɺSubscriberͷ਺ΛมԽͤ͞Δ • 3~6 Node • n2-standard-4 (4vCPU / 16GB) • ϦιʔεͷϦΫΤετʹԠͯ͡εέʔϧ • Subscriber • 1 / 10 / 20 / 30 / 40 / 50 / 60 / 70 / 80 Pod • ֤PodͷϦιʔε • CPU: 250m ~ 500m • ϝϞϦ: 256Mi ~ 512Mi
19. ݁Ռ
20. ࣮ݧ݁Ռ Pod਺ ॲ ཧ ࣌ ؒ 0 550 1100 1650 2200 1 10 20 30 40 50 60 70 80 1 pod 2105 seconds 20 pods 689 seconds 80 pods 368 seconds
21. ·ͱΊ
22. • KubernetesΛ༻͍Δ͜ͱͰɺPub/SubϞσϧͰͷ෼ࢄॲཧΛखܰʹߦ͏͜ͱ ͕Ͱ͖Δ • KafkaΛ༻͍Δ͜ͱͰɺSubscriberΛεέʔϧͤͯ͞ॲཧ࣌ؒΛ୹ॖ͢Δ͜ͱ ͕Ͱ͖Δ • https://github.com/ogatakatsuya/distributed-tasks • ڵຯͷ͋Δํ͸೷͍ͯΈ͍ͯͩ͘͞ • ٞ࿦΍վળ఺౳͋Ε͹ڭ͍͑ͯͩ͘͞ʂ ·ͱΊ
23. ࢀߟจݙ • Kubernetes • Kubernetes׬શΨΠυ ୈ2൛ • Ray-tracing • https://www.scratchapixel.com/lessons/3d-basic-rendering • Kafka • https://speakerdeck.com/oracle4engineer/shin-kafka • https://speakerdeck.com/hashitokyo/apache-kafka-and-the-world-of- streams