Kubernetes (EKS) で構築する スケーラブルなジョブ実行基盤

Slide 1

Slide 1 text

Kubernetes (EKS) Ͱߏங͢Δ  εέʔϥϒϧͳδϣϒ࣮ߦج൫ Kenta Suzuki (twitter: @kekekenta) CTO, primeNumber inc 2019/8/28 Kubernetes Meetup Tokyo #22

Slide 2

Slide 2 text

ࣗݾ঺հ • Kenta Suzuki • twitter: @kekekenta • CTO, primeNumber inc.  εϥΠυͷ࠷ޙʹձࣾ঺հࡌͤ·ͨ͠ͷͰɺ͝ڵຯ͋Γ·ͨ͠Βޚཡ͍ͩ͘͞ •෼ੳج൫ߏஙɾӡ༻ࢧԉSaaSʮtrocco®ʯ

Slide 3

Slide 3 text

.Z42- 4 4BMFTGPSDF #JH2VFSZ ෼ੳج൫ߏஙɾӡ༻ σʔλసૹ αϚϦʔςʔϒϧ࡞੒ σʔλ؂ࢹ ֎෦࿈ܞ

Slide 4

Slide 4 text

.Z42- 4 4BMFTGPSDF #JH2VFSZ σʔλసૹ αϚϦʔςʔϒϧ࡞੒ σʔλ؂ࢹ ֎෦࿈ܞ ෼ੳج൫ͷߏஙɾӡ༻Λ·ͱΊͯΧόʔ

Slide 5

Slide 5 text

troccoͷσʔλసૹδϣϒ • σʔλιʔε • MySQLɺPostgreSQLɺSQLServerͳͲ • εέδϡʔϧ࣮ߦ • ࠷୹1෼ִؒ ೚ҙͷσʔλιʔεͷςʔϒϧ౳ͷ಺༰ΛDWHʹసૹ troccoͷσʔλసૹ෦෼͸OSSͷembulkΛଟ༻͍ͯ͠·͢ɻ  ͜ͷ৔ΛआΓͯɺίϛολʔͷํʑɺembulkίϛϡχςΟͷํʑʹײँਃ্͛͠·͢ɻ ࠓޙ͸embulkͷpluginެ։౳Ͱߩݙ͍ͯ͘͠༧ఆͰ͢ʂ

Slide 6

Slide 6 text

troccoͷసૹδϣϒ • δϣϒ࣮ߦ਺ • ʙ10,000/೔ • δϣϒ͝ͱͷ࣮ߦ࣌ؒ • min: 5s ʙ max: 5h • ࣮ߦ࣌ؒଳ • ਺࣌ؒʹ1౓ͷεύΠΫ

Slide 7

Slide 7 text

Fan out on ECS چΞʔΩςΫνϟ

Slide 8

Slide 8 text

Fan out on ECSαʔϏε SQS εέδϡʔϥʔ 8PSLFS &$4UBTL 8PSLFS &$4UBTL 8PSLFS &$4UBTL ECS ECS NODE δϣϒ polling

Slide 9

Slide 9 text

໰୊఺

Slide 10

Slide 10 text

λεΫεέʔϧΞ΢τͷλΠϛϯά೉͍͠ • SQSͷΩϡʔͷαΠζΛ؂ࢹ • CloudwatchϝτϦΫε͕࠷୹1෼͝ͱ • εέʔϧ·Ͱɺ1෼଴ͨͳ͍ͱߦ͚ͳ͍

Slide 11

Slide 11 text

ϗετεέʔϧΞ΢τͷλΠϛϯά೉͍͠ • λεΫ͕΋͏௥ՃͰ্ཱ͕ͪΔϦιʔε͕ ͳ͘ͳͬͨΒεέʔϧ • ىಈ଴ͪͷλεΫ͕ग़དྷͨλΠϛϯάͰϗ ετεέʔϧͯ͠΄͍͠ɻɻ

Slide 12

Slide 12 text

λεΫεέʔϧΠϯ೉͍͠ • ECSλεΫΛ҆શʹऴྃͤ͞Δඞཁ͕͋Δ • సૹδϣϒ͸5hʹٴͿ৔߹΋͋Δ • TERMγάφϧͷτϥοϓͯ͠graceful shutdownΛ ࣮૷ • ͨͩ͠ɺECSͷίϯιʔϧ্ͷ࣮ߦதλεΫҰཡ͔ΒɺTERMΛૹͬͯ͠͹Β͘͢Δ ͱফ͑ͯ͠·͏

Slide 13

Slide 13 text

ϗετͷεέʔϧΠϯ΋೉͍͠ • Auto Scaling Group ͷ Life cycle hook Λ࢖ͬͯɺΠϯ ελϯεఀࢭલʹDRAININGΛ࣮ࢪ • ىಈதͷλεΫ͕ͳΔ·Ͱ؂ࢹ͠ɺͳ͘ͳͬͨΒΠϯε λϯεఀࢭΛ͢͢ΊΔ • લड़ͷ௨ΓɺىಈதͷλεΫҰཡ͔Βফ͑ΔͨΊ͝ʹΐ ͝ʹΐ΍ͬͯͨ

Slide 14

Slide 14 text

Ϧιʔεׂ౰ • సૹߴ଎ԽΦϓγϣϯͱͯ͠ɺCPU/ϝϞϦ্ݶΛઃఆ Ͱ͖ΔΑ͏ʹ͍ͨ͠ • ECSͰ΋λεΫ͝ͱʹϦιʔε্ݶΛઃఆͰ͖Δ͕ɺى ಈ࣌ͷΈ • Ϧιʔε্ݶ͝ͱʹαʔϏεΛ෼͚Δඞཁ͕͋Γ

Slide 15

Slide 15 text

೉͍͠

Slide 16

Slide 16 text

ͦ͜Ͱɺk8s

Slide 17

Slide 17 text

k8sͷpodεέδϡʔϥʔʹͭ ͍ͯ

Slide 18

Slide 18 text

k8sͷpodεέδϡʔϥʔʹͭ ͍ͯ

Slide 19

Slide 19 text

k8sͷpodεέδϡʔϥʔʹͭ ͍ͯ statusͷભҠ "Pending" -> "Running" -> "Succeeded"

Slide 20

Slide 20 text

k8sͷpodεέδϡʔϥʔ • nodeϦιʔεͷۭ͖ঢ়گʹԠͯ͡ɺεέδϡʔϥʔ͕ॱ࣍podΛ nodeʹׂΓ౰ͯͯ͘ΕΔ • podΛ࡞੒͓͚ͯ͠͹ɺ͋ͱ͸࣮ߦ͞ΕΔͷΛ଴͚ͭͩ ,VCFSOFUFT QPE /0%& QPE QPE pod্ཱͪ͛ ࣮ߦ଴ͪpod create pod

Slide 21

Slide 21 text

ͱΓ͋͑ͣࡶʹpodΛcreate͢Ε ͹ɺ͋ͱ͸k8s͕࣮ߦ·Ͱ໘౗ݟ ͯ͘ΕΔ

Slide 22

Slide 22 text

ϊʔυͷεέʔϧΞ΢τ/Πϯ • EKSͷ৔߹ɺcluster autoscalerΛ࢖͏ͷ͕σϑΝΫτʢͱ ࢥΘΕΔʣ  https://github.com/kubernetes/autoscaler/tree/master/ cluster-autoscaler/cloudprovider/aws • podεέδϡʔϦϯάͷঢ়گʹԠͯ͡ɺEC2ϊʔυͷASG ܦ༝ͰεέʔϧΠϯɾΞ΢τΛ΍ͬͯ͘ΕΔ

Slide 23

Slide 23 text

ϊʔυͷεέʔϧΞ΢τ QPE &$/0%& QPE QPE 1FOEJOH ࠓͷnodeϦιʔεͰ͸ɺ  podΛ্ཱͪ͛Δ͜ͱ͕ग़དྷͳ͍

Slide 24

Slide 24 text

ϊʔυͷεέʔϧΞ΢τ QPE &$/0%& QPE &$/0%& QPE ClusterAutoscaler͕ASGܦ༝ͰnodeΛεέʔϧΞ΢τ

Slide 25

Slide 25 text

ϊʔυεέʔϧΠϯ QPE &$/0%& QPE &$/0%& QPE

Slide 26

Slide 26 text

ϊʔυεέʔϧΠϯ QPE &$/0%& QPE &$/0%& Ϧιʔεʹ༨༟͕͋Δ

Slide 27

Slide 27 text

ϊʔυεέʔϧΠϯ QPE &$4/0%& QPE εέʔϧΠϯ

Slide 28

Slide 28 text

1. create pod ࣮ͯ͠ߦΛ଴ͭ 2. ඞཁʹԠͯ͡ɺnode͕εέʔϧΞ΢ τ͢ΔͷΛ଴ͭ εέʔϥϒϧδϣϒج൫ര஀

Slide 29

Slide 29 text

k8sʹΑΔδϣϒج൫

Slide 30

Slide 30 text

ݱࡏͷΞʔΩςΫνϟ SQS εέδϡʔϥʔ 8PSLFS 1PE +PC %JTQBUDIFS Kubernetes δϣϒ create pod schuedule pod &,4/0%& 8PSLFS 1PE $MVTUFS "VUPTDBMFS Scale in/out

Slide 31

Slide 31 text

Job Dispatcher SQS εέδϡʔϥʔ 8PSLFS 1PE Kubernetes δϣϒ create pod schuedule pod &,4/0%& 8PSLFS 1PE $MVTUFS "VUPTDBMFS Scale in/out +PC %JTQBUDIFS

Slide 32

Slide 32 text

Job Dispatcher ˞ίʔυ͸ΠϝʔδͰ͢ • SQS Λ polling • k8s API ʹϦΫΤετΛ౤͛ɺ pod Λ࡞੒

Slide 33

Slide 33 text

Job Dispatcher ˞ίʔυ͸ΠϝʔδͰ͢ LT"1*ʹϦΫΤετ • SQS Λ polling • k8s API ʹϦΫΤετΛ౤͛ɺ pod Λ࡞੒

Slide 34

Slide 34 text

Job Dispatcher ˞ίʔυ͸ΠϝʔδͰ͢ EPDLFSΠϝʔ δͷλάࢦఆ • SQS Λ polling • k8s API ʹϦΫΤετΛ౤͛ɺ pod Λ࡞੒

Slide 35

Slide 35 text

Job Dispatcher ˞ίʔυ͸ΠϝʔδͰ͢ QPEͰ࣮ߦ͢Δ ίϚϯυΛࢦఆ • SQS Λ polling • k8s API ʹϦΫΤετΛ౤͛ɺ pod Λ࡞੒

Slide 36

Slide 36 text

Job Dispatcher • SQS Λ polling • k8s API ʹϦΫΤετΛ౤͛ɺ pod Λ࡞੒ ˞ίʔυ͸ΠϝʔδͰ͢ $16ϝϞϦࢦఆ

Slide 37

Slide 37 text

Worker pod SQS εέδϡʔϥʔ +PC %JTQBUDIFS Kubernetes δϣϒ create pod schuedule pod &,4/0%& $MVTUFS "VUPTDBMFS Scale in/out 8PSLFS 1PE 8PSLFS 1PE

Slide 38

Slide 38 text

Worker pod • ࣮ࡍͷసૹΛ࣮ߦ • సૹ։࢝Ͱ্ཱ͕ͪΓɺసૹ͕ऴྃͨ͠Βpod΋ऴྃ ˞ίʔυ͸ΠϝʔδͰ͢

Slide 39

Slide 39 text

Cluster autoscaler SQS εέδϡʔϥʔ +PC %JTQBUDIFS Kubernetes δϣϒ create pod schuedule pod &,4/0%& Scale in/out 8PSLFS 1PE 8PSLFS 1PE $MVTUFS "VUPTDBMFS

Slide 40

Slide 40 text

Cluster autoscaler ˞ίʔυ͸ΠϝʔδͰ͢

Slide 41

Slide 41 text

k8sδϣϒج൫ಋೖͷ੒Ռ • ѹ౗తίετ࡟ݮʹ੒ޭ • εέʔϧͷλΠϛϯά͕೉͍͠໰୊͕͍͍ ײ͡ʹղܾͰ͖ͨ • cluster autoscaler༷༷ • ։ൃ؀ڥͱຊ൪؀ڥͷಉҰԽ

Slide 42

Slide 42 text

؂ࢹʹ͍ͭͯ • create podͰ͖ͳ͍ • pod͕pendingͷ··্ཱ͕ͪΒͳ͍ • pod͕runningதʹಥવࢮ ຊ൪ӡ༻࣮੷͕ແ͍ͷͰɺຊ౰ʹظ଴௨Γʹ ಈ͔͘ͱ͍͏ҙຯͰ؂ࢹΛ࣮ࢪ

Slide 43

Slide 43 text

troccoͷδϣϒεςʔλεભҠ • DB্ʹδϣϒ͝ͱͷεςʔλεΛอ࣋ • dispatcher΍podͳͲ͕ͦΕͧΕߋ৽͢Δ RVFVFE TFUUJOH@VQ FYFDVUJOH TVDDFFEFE SQSʹΩϡʔΠϯά create pod ׬ྃ pod্ཱ͕ͪΓ·Γ pod͕࣮ߦத సૹ׬ྃ

Slide 44

Slide 44 text

redashͰͷ؂ࢹ RVFVFE TFUUJOH@VQ FYFDVUJOH TVDDFFEFE ֤εςʔλεͰ଺ཹ͍ͯͨ͠ΒΞϥʔτ

Slide 45

Slide 45 text

·ͱΊ • γϯϓϧ • k8sͷݞʹ৐Ε͹ɺεέʔϥϒϧͳδϣϒج൫ߏங΋γϯϓ ϧʹͳΔ • ؂ࢹ • ӡ༻࣮੷͕ແ͍ͷͰɺͱΓ͋͑ͣऔΕΔσʔλ͸ͱ͓ͬͯ͘ • k8s΁ͷҠߦΛৼΓฦͬͯ • όον͔Βk8sಋೖ͢Δͷ͸ׂͱ΍Γ΍͍͢ͷͰ͓͢͢Ί • কདྷతʹ • কདྷతʹ͸ɺϗετͷ؅ཧͱ͔΍Βͳͯ͘ྑ͘ͳΔͱخ͍͠

Slide 46

Slide 46 text

͝ਗ਼ௌ ͋Γ͕ͱ͏͍͟͝·ͨ͠ :bow Α͚Ε͹ձࣾ঺հ΋ޚཡ͍ͩ͘͞

Slide 47

Slide 47 text

גࣜձࣾ primeNumber • 2015೥11݄ ઃཱ • ࣾһ9໊ʢ͏ͪΤϯδχΞ8໊ʣ  ύʔτφʔΤϯδχΞ30໊Ҏ্ • 2018೥11݄ trocco®ϦϦʔε • 2019೥7݄ γϦʔζAࢿۚௐୡʢ໿1.3ԯԁʣ࣮ࢪ https://primenumber.co.jp

Slide 48

Slide 48 text

ಇ͘؀ڥ • @த໨ࠇͷ઒Ԋ͍ • ίΞλΠϜ੍ • 11ʙ15࣌ • िத೔ɺ2೔·ͰϦϞʔτՄ ※ࢼ༻த • δϜશֹิॿ

Slide 49

Slide 49 text

We are hiring • ιϑτ΢ΣΞΤϯδχΞ • ΠϯϑϥΤϯδχΞ • ͳͲͳͲɺଟํ໘ઈࢍืूத