Kubernetesで分散処理をやってみる-Kafkaを添えて-

by Ogata Katsuya

Embed

Start on current slide

Slide 1

Slide 1 text

2025-09-13 / ෼ࢄٕज़ษڧձ~Ϋϥ΢υͱ෼ࢄٕज़ͷΑ΋΍·࿩~ KubernetesͰ෼ࢄॲཧΛͯ͠ΈΔ Katsuya Ogata

Slide 2

Slide 2 text

ࣗݾ঺հ • ໊લ: ॹํ ࠀ࠸ (͓͕ͨ ͔ͭ΍) • ॴଐ: େࡕେֶ৘ใֶݚڀՊ • ࣗ୐αʔόʔ࢝Ί·ͨ͠ • Ӄࢼظؒதʹ5kgଠΓ·ͨ͠ • X: @ogata_katsuya • ݸਓ: www.ogatakatsuya.com

Slide 3

Slide 3 text

෼ࢄॲཧλεΫͷ঺հ

Slide 4

Slide 4 text

ϨΠτϨʔγϯά ޫͷ൓ࣹɾ۶ંΛܭࢉͯ͠ɺߴ඼࣭ͳը૾Λੜ੒͢Δ • ޫઢͷ௥੻ʹΑΔը૾ੜ੒ख๏ • Ծ૝Χϝϥ͔ΒͷޫઢΛγϛϡϨʔγϣϯ • ޫͷ൓ࣹɾ۶ંɾӨΛ࠶ݱ͢Δ • ܭࢉίετ͕ߴ͍ • ө૾΍CGɺݐஙՄࢹԽʹ͓͍ͯ࢖ΘΕΔ https://www.scratchapixel.com/lessons/3d-basic-rendering/introduction-to-ray-tracing/implementing-the-raytracing-algorithm.htm ޫઢͷ௥੻ʹΑΔը૾ੜ੒ख๏

Slide 5

Slide 5 text

ϨΠτϨʔγϯά ޫͷ൓ࣹɾ۶ંΛܭࢉͯ͠ɺߴ඼࣭ͳը૾Λੜ੒͢Δ • ܭࢉྔ O ( P * R * S * D) • P: ϐΫηϧ਺ • R: ϐΫηϧ͋ͨΓͷޫઢ਺ • S: ΦϒδΣΫτ਺ • D: ൓ࣹ/۶ંਂ౓ • ࠓճͷ৔߹ɺO ( 800*800 * 50 * 8 * 8 ) = O ( 2,048,000,000 ) • ͜ΕΛ100pixel͝ͱʹ෼ׂͯ͠ɺॲཧΛฒྻ࣮ߦ͢Δ

Slide 6

Slide 6 text

Kubernetesͷ෼ࢄॲཧ Node/PodʹॲཧΛ෼ࢄ͢Δ͜ͱʹΑ࣮ͬͯݱ͢Δ https://kubernetes.io/ja/docs/concepts/overview/components/

Slide 7

Slide 7 text

ߟ͑ΒΕΔΞʔΩςΫνϟ ͋ΔPod͕WorkerʹԿΒ͔ͷ௨৴खஈͰσʔλΛ౉ͯ͠ฦ͢

Slide 8

Slide 8 text

ߟ͑ΒΕΔΞʔΩςΫνϟ ͋ΔPod͕WorkerʹԿΒ͔ͷ௨৴खஈͰσʔλΛ౉ͯ͠ฦ͢ • σϝϦοτ • Worker PodͷΞυϨε͸ʁʁ • Producer - Workerؒͷ௨৴खஈ͸ʁʁ • ࠶ૹɾϦτϥΠॲཧ͕ඞཁʹͳΔʁʁ • ॲཧதʹαʔόʔ͕མͪͨΒϝοηʔδ͸ফ͑ͯ͠·͏ʁʁ • ॲཧ͕ॏ͘ͳͬͨ࣌ʹͲ͏΍ͬͯରԠ͢Δʁʁ

Slide 9

Slide 9 text

ͦ͏ͩɺKafkaΛ࢖͓͏

Slide 10

Slide 10 text

KafkaΛ༻͍ͨ෼ࢄॲཧ Pub/Sub messaging model https://aws.amazon.com/jp/what-is/pub-sub-messaging/

Slide 11

Slide 11 text

KafkaΛ༻͍ͨ෼ࢄॲཧ Pub/Sub messaging model • ϝϦοτ • ௨৴ͱϏδωεϩδοΫ͕ૄ݁߹ʹͳΔ • PublisherͱSubscriberΛ༰қʹεέʔϧͰ͖Δ • ඇಉظͰॲཧΛ࣮ߦͰ͖Δ • ΠϕϯτۦಈΞʔΩςΫνϟ΁ͷԠ༻ • ଱ো֐ੑ

Slide 12

Slide 12 text

͜ΕΛKubernetes্Ͱ࣮ݱ͍ͨ͠

Slide 13

Slide 13 text

KubernetesͷΞʔΩςΫνϟ Pub/Sub messaging model

Slide 14

Slide 14 text

KubernetesͷΞʔΩςΫνϟ Pub/Sub messaging model

Slide 15

Slide 15 text

KubernetesͷΞʔΩςΫνϟ Pub/Sub messaging model

Slide 16

Slide 16 text

KubernetesͷΞʔΩςΫνϟ Pub/Sub messaging model

Slide 17

Slide 17 text

࣮ݧ

Slide 18

Slide 18 text

࣮ݧ಺༰ GKEΛ༻͍ͯɺSubscriberͷ਺ΛมԽͤ͞Δ • 3~6 Node • n2-standard-4 (4vCPU / 16GB) • ϦιʔεͷϦΫΤετʹԠͯ͡εέʔϧ • Subscriber • 1 / 10 / 20 / 30 / 40 / 50 / 60 / 70 / 80 Pod • ֤PodͷϦιʔε • CPU: 250m ~ 500m • ϝϞϦ: 256Mi ~ 512Mi

Slide 19

Slide 19 text

݁Ռ

Slide 20

Slide 20 text

࣮ݧ݁Ռ Pod਺ ॲ ཧ ࣌ ؒ 0 550 1100 1650 2200 1 10 20 30 40 50 60 70 80 1 pod 2105 seconds 20 pods 689 seconds 80 pods 368 seconds

Slide 21

Slide 21 text

·ͱΊ

Slide 22

Slide 22 text

• KubernetesΛ༻͍Δ͜ͱͰɺPub/SubϞσϧͰͷ෼ࢄॲཧΛखܰʹߦ͏͜ͱ ͕Ͱ͖Δ • KafkaΛ༻͍Δ͜ͱͰɺSubscriberΛεέʔϧͤͯ͞ॲཧ࣌ؒΛ୹ॖ͢Δ͜ͱ ͕Ͱ͖Δ • https://github.com/ogatakatsuya/distributed-tasks • ڵຯͷ͋Δํ͸೷͍ͯΈ͍ͯͩ͘͞ • ٞ࿦΍վળ఺౳͋Ε͹ڭ͍͑ͯͩ͘͞ʂ ·ͱΊ

Slide 23

Slide 23 text

ࢀߟจݙ • Kubernetes • Kubernetes׬શΨΠυ ୈ2൛ • Ray-tracing • https://www.scratchapixel.com/lessons/3d-basic-rendering • Kafka • https://speakerdeck.com/oracle4engineer/shin-kafka • https://speakerdeck.com/hashitokyo/apache-kafka-and-the-world-of- streams