CNDF2023前夜祭 - 玄界灘のクラウドネイティブなデータ基盤運用の実践

by Kazuhiko Yamashita

Slide 1

Slide 1 text

ݰքṗͷΫϥ΢υωΠςΟϒͳ σʔλج൫ӡ༻ͷ࣮ફ ʙॵ͍ʂʂՆͩʂʂʂૣ͘ϏΞΨʔσϯߦ͖͍ͨʂʂ̍ฤʙ

Slide 2

Slide 2 text

ࢁԼ࿨඙!QZBNB (.0ϖύϘٕज़ج൫νʔϜ γχΞɾϓϦϯγύϧ ɹΩϟϯϓɺཱྀߦɺώϧτϯ८ΓɺιϫχΤ८Γ ($1ɺ"JS fl PXɺ1VC4VCɺ%BUB fl PX

Slide 3

Slide 3 text

ࠓ೔࿩͢͜ͱ γεςϜ֓ཁ ͦ͜ʹ͋ͬͨ՝୊ ͲͷΑ͏ʹղܾͨ͠ͷ͔

Slide 4

Slide 4 text

ϗεςΟϯάࣄۀ &$ࢧԉࣄۀ ϋϯυϝΠυɾͦͷଞࣄۀ

Slide 5

Slide 5 text

γεςϜߏ੒ Ingest Pipeline DataFlow CloudComposer Extract Analytics ML BigQuery Vertex AI source monitor Cloud  Logging Cloud  Monitoring DBσʔλɺϩάΛBigQueryʹू໿ Pub/Sub

Slide 6

Slide 6 text

σʔλج൫ͷར׆༻ ྨࣅը૾ݕࡧ ঎඼ਪન ੜ࢈ੑࢦඪͷଌఆ

Slide 7

Slide 7 text

ͳͥ๻͕ʁʁʁ

Slide 8

Slide 8 text

https://note.com/udzura/n/n5c8647d38 ff f ΑΓҾ༻

Slide 9

Slide 9 text

ͦ͜ʹ͋ͬͨ՝୊ མͪଓ͚ΔDAGͱຫੑతͳτΠϧ ΤϯυϢʔβʔ͔ΒͷϑΟʔυόοΫͰݦࡏԽ͢Δো֐ ޮ཰ͷѱ͍։ൃ

Slide 10

Slide 10 text

མͪଓ͚ΔDAGͱ ຫੑతͳτΠϧ

Slide 11

Slide 11 text

γεςϜߏ੒ Ingest Pipeline DataFlow CloudComposer Extract Analytics ML BigQuery Vertex AI source monitor Cloud  Logging Cloud  Monitoring DBσʔλɺϩάΛBigQueryʹू໿ Pub/Sub

Slide 12

Slide 12 text

མͪଓ͚ΔDAGͱຫੑతͳτΠϧ DAG = Directed Acyclic Graph = ༗޲ඇ८ճάϥϑ https://air fl ow.apache.org/docs/apache-air fl ow/stable/core-concepts/dags.html ΑΓҾ༻

Slide 13

Slide 13 text

DAGͷࣦഊཁҼ 1.༷ʑͳλΠϜΞ΢τ 2.࿈ٳʹΑΔσʔλͷܽଛ 3.ιʔεͷσʔλߏ଄ͷมߋɺଐੑ 4.มߋʹΑΔΤϯόά

Slide 14

Slide 14 text

τΠϧͨΔॴҎ 1.༷ʑͳλΠϜΞ΢τ → ϦτϥΠɺλΠϜΞ΢τᮢ஋มߋ 2.࿈ٳʹΑΔσʔλͷܽଛ → ϦτϥΠɺεΩοϓ 3.ιʔεͷσʔλߏ଄ɺଐੑͷมߋ → มߋ͓ͯ͠͠·͍ 4.มߋʹΑΔΤϯόά → मਖ਼͓ͯ͠͠·͍

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

ॳख SLI/SLOͷࡦఆ

Slide 17

Slide 17 text

SLO/SLOͷࡦఆ ʮͳʹ͔Β΍Δ͔ʁʯ - վળͷ༏ઌॱҐ ʮͲΕ͘Β͍΍Δ͔ʯ - ਺஋Ͱվળͷ౓߹͍ΛܾΊΔ ·ͣࢦඪΛ࡞Δ͜ͱͰɺ਺஋ۦಈͰҙࢥܾఆͰ͖Δ

Slide 18

Slide 18 text

SLO/SLOͷࡦఆ SREϫʔΫϒοΫ - Ch 13. Data Processing Pipelines ͲͷΑ͏ʹܾΊΔ͔ʁ https://sre.google/workbook/data-processing/ https://cloud.google.com/stackdriver/docs/solutions/slo-monitoring/sli-metrics/data-proc-metrics?hl=ja Google Cloud ΦϖϨʔγϣϯεΠʔτ σʔλॲཧαʔϏε

Slide 19

Slide 19 text

SLO/SLOͷࡦఆ GrafanaͰ؆୯ʹμογϡϘʔυԽͰ͖Δ

Slide 20

Slide 20 text

SLO/SLOͷࡦఆ 1.DAGͷ੒ޭ཰ 2.DataFlowδϣϒʹ͓͚ΔΤϥʔͷൃੜ཰ 3.DataFlowδϣϒͷεςοϓʹ͓͚Δσʔλͷ઱౓

Slide 21

Slide 21 text

վળͷଧͪख λΠϜΞ΢τͷཁҼͱͳΔɺ஗ԆΛऔΓআ͘ɺՁ஋؍Λৢ੒͢Δ λΠϜΞ΢τ஋ʹ͍ͭͯɺϝτϦΫε͔Βଥ౰ͳ஋͔Λ֬ೝ͢Δ 1. ༷ʑͳλΠϜΞ΢τ

Slide 22

Slide 22 text

վળͷଧͪख holiday-jpΛ׆༻ͯ͠ɺ࿈ٳ࣌ͷॲཧΛ࣮૷ BigQuery͔Β΋UDFͰ࿈ٳͷ৚݅ΛఆٛͰ͖ΔΑ͏ʹͨ͠ 2.࿈ٳʹΑΔσʔλͷܽଛ

Slide 23

Slide 23 text

վળͷଧͪख DebeziumͰDDLΛݕ஌ͨ͠ΒSlack௨஌͢Δ σʔλͷNULLൺ཰͕૿͑ͨΒ௨஌͢Δ 3.ιʔεͷσʔλߏ଄ɺଐੑͷมߋ https://github.com/pyama86/debezium-ddl-noti fi er/

Slide 24

Slide 24 text

վળͷଧͪख ຖ೔શDAGΛޕલɺޕޙʹ࣮ߦͯ͠ɺຊ൪࣮ߦΑΓ  σʔλͷෆ੔߹ͳͲΛૣ͘ݕ஌ Ϣχοτςετͷ֦ॆ 4. มߋʹΑΔΤϯόά

Slide 25

Slide 25 text

࠷ۙ͸͓ئ͍ͯ͠ɺTABΛԡ͚ͩ͢ͷ࢓ࣄͰ͢ ## ೖྗ࢓༷ - Air fl owͷDAGͷϑΝΠϧΛೖྗͱ͠·͢ ## αϯϓϧ࣮૷1 ```python {sample1} ``` ## αϯϓϧ࣮૷2 ```python {sample2} ``` ## ࢦࣔ - ೖྗͱͯ͠౉͞ΕͨDAGͷςετίʔυΛpytestΛ༻͍࣮ͯ૷͍ͯͩ͘͠͞ɻ - Ϋϥεʹ͍ͭͯ͸spy΍mockΛ༻͍ͯϞοΫԽ͍ͯͩ͘͠͞ɻ - 2ͭͷαϯϓϧ࣮૷Λࣔ͠·͢ɻͦΕΛࢀߟʹ͍ͯͩ͘͠͞ɻ - ͋ͳ͕ͨੜ੒ͨ͠ςετ͸ͦͷ··ϑΝΠϧʹอଘͯ͠ɺ࣮ߦ͠·͢ɻग़ྗ಺༰ͷઆ໌ͳͲ͸ෆཁͰ͢ɻ ## ೖྗ ϑΝΠϧ໊:{dag_ fi le_path} ```python {dag_ fi le_content} ``` ... $ poetry run python bin/test_generator.py dags/example.py

Slide 26

Slide 26 text

ΤϯυϢʔβʔ͔Βͷ   ϑΟʔυόοΫͰݦࡏԽ͢Δো֐

Slide 27

Slide 27 text

ʁʁʁʮࠓ೔ͷσʔλ͕ೖͬͯͳ͍Έ͍ͨ   ͳΜͰ͚͢ͲɺͳΜ͔͋Γ·ͨ͠ʁʯ

Slide 28

Slide 28 text

ͳΜͱɺ؂ࢹ͕   ΄ͱΜͲͳ͍ͷͰ͋Δ (؆қతͳσʔλ੔߹ੑͷςετ͕͋Δ͚ͩ)

Slide 29

Slide 29 text

γεςϜߏ੒ Ingest Pipeline DataFlow CloudComposer Extract Analytics ML BigQuery Vertex AI source monitor Cloud  Logging Cloud  Monitoring DBσʔλɺϩάΛBigQueryʹू໿ Pub/Sub

Slide 30

Slide 30 text

؂ࢹࣾձ΁ AWSͰECS࢖͍ͬͯΔͱ͜Ζ͸Mackerel Container Agentಋೖ Cloud MonitoringͷΞϥʔτఆٛΛSLIʹ߹ΘͤͯTerraformͰ࣮૷ GCPͱൺ΂ͯར༻ن໛͕খ͍ͨ͞ΊɺશࣾͰར༻͞Ε͍ͯΔMackerelΛར༻͢Δ͜ͱͰ ֶशɺಋೖίετͷ࡟ݮ

Slide 31

Slide 31 text

σʔλύΠϓϥΠϯͷ؂ࢹ ΰʔϧσϯσʔλͷ౤ೖ @type dummy @label @INPUT tag example dummy { "accessed_at": "2022-01-01T00:00:00Z", "account_id": 1, "client_id": "12345abcde", "event": "example_event", } https://docs. fl uentd.org/v/0.12/input/dummy ຖඵμϛʔσʔλΛ ύΠϓϥΠϯʹྲྀ͠ɺ ΤϯυϙΠϯτͰ؂ࢹ

Slide 32

Slide 32 text

ޮ཰ͷѱ͍։ൃ

Slide 33

Slide 33 text

γεςϜߏ੒ Ingest Pipeline DataFlow CloudComposer Extract Analytics ML BigQuery Vertex AI source monitor Cloud  Logging Cloud  Monitoring DBσʔλɺϩάΛBigQueryʹू໿ Pub/Sub

Slide 34

Slide 34 text

։ൃ؀ڥͱۚͱݖྗ Cloud Composerͷ؀ڥͰ݁߹ςετΛ࣮ߦ͢ΔͨΊʹɺ  ςετ༻ͷ؀ڥΛຖேࣗಈ࡞੒ɺຖ൩ࣗಈ࡟আ ίετͱνʔϜߏ੒ΛؑΈͯɺຖே̎؀ڥ࡞੒ͯ͠ɺڞ༻

Slide 35

Slide 35 text

։ൃ؀ڥͱۚͱݖྗ Cloud Composerͷ؀ڥͰ݁߹ςετΛ࣮ߦ͢ΔͨΊʹɺ  ςετ༻ͷ؀ڥΛຖேࣗಈ࡞੒ɺຖ൩ࣗಈ࡟আ → ো֐࣌ɺ࣌ؒ֎ʹ࢖͑ͳ͍ ίετͱνʔϜߏ੒͔Β̎؀ڥ࡞੒ͯ͠ɺڞ༻ → ςετ଴ͪͷൃੜɺग़ྗ݁Ռͷࠞ߹

Slide 36

Slide 36 text

I'm Kubernetes Rock Star ΦϯϓϨϛεͷKubernetes্ͰPR͝ͱʹ࣮ߦ؀ڥΛ࡞੒ https://tech.pepabo.com/2023/07/07/data-platform-ci/

Slide 37

Slide 37 text

ਓͱνΣοΫϦετ DAG A DAG B ࣮૷ͷ֬ೝΛਓ͕΍͍ͬͯͨ EmptyOperator( taks_id="external_task_sensor_target )

Slide 38

Slide 38 text

ྲྀΕͷRuby Rock Star ʮͦΕASTݟͨΒ͍͍͡ΌΜʯ

Slide 39

Slide 39 text

Abstract Syntax Tree from air fl ow.operators.empty import EmptyOperator EmptyOperator(task_id="child_task3") % python -m ast example.py Module( body=[ ImportFrom( module='air fl ow.operators.empty', names=[ alias(name='EmptyOperator')], level=0), Expr( value=Call( func=Name(id='EmptyOperator', ctx=Load()), args=[], keywords=[ keyword( arg='task_id', value=Constant(value='child_task3'))]))], type_ignores=[]) grepͰͰ͖ͳ͍͜ͱ΋ɺ੩తղੳͳΒղܾͰ͖Δ

Slide 40

Slide 40 text

͜͜·Ͱ࿩ͨ͜͠ͱ σʔλج൫ͷ։ൃɺӡ༻ʹ͓͚Δ՝୊Λ  SREͷεΩϧͰվળ

Slide 41

Slide 41 text

Ҿୀࣦഊʂʂ1

Slide 42

Slide 42 text

શํҐ࠾༻͍ͯ͠·͢ɻ   .-΋ͬͱ΍Γ͍ͨ͠ɺ43&΋ͬͱ͓΋͠Ζ͍ͨ͘͠ ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ !QC@SFDSVJU

Slide 43

Slide 43 text

ಥવͰ͕͢ɺ ൧ͷ࿩Λ͠·͢

Slide 44

Slide 44 text

झຯιϫχΤΊ͙Γ https://www.web-soigner.jp/magazine_hakata_gourmet_guide/ Α͏ͦ͜෱Ԭ΁ʂʂ̍

Slide 45

Slide 45 text

ډञ԰

Slide 46

Slide 46 text

໨ར͖ͷ͔ͨࢤ ٬୯Ձ :4000ʙ6000ԁ ৔ॴ: ᷫԂ ਪ঑ਓ਺: 2 - 4ਓ ਪ͠ɿڕ͔Βͳʹ·Ͱɺ͢΂͕ͯඒຯ͍ɺ  ೔ຊञ΋छྨ͕͋Γ๛෋ɻ  ͦͯ͠ɺWEB༧໿Ͱ͖Δɻ  ࢲతNo.1

Slide 47

Slide 47 text

͍ͤ΋Μ෷͍ ٬୯Ձ :6000ʙ8000ԁ ৔ॴ: தऱ ਪ঑ਓ਺: 2 - 8ਓ ਪ͠ɿ҆͘͸ͳ͍͕ɺԿ৯΂ͯ΋ඒຯ͍ɻ  ΤϏϑϥΠ͕৴͡ΒΕͳ͍σΧ͞ͳͷͱ  ࢗ͠਎΋ඒຯ͍ɻ

Slide 48

Slide 48 text

ḉ࿠ ٬୯Ձ :4000ʙ8000ԁ ৔ॴ: தऱ ਪ঑ਓ਺: 2 - 8ਓ ਪ͠ɿͱʹ͔͘ḉͱञɻ  ͋Μ·ΓຊभͰ৯΂Εͳ͍ḉͩͱࢥ͏

Slide 49

Slide 49 text

ڕ๪ࡾಙ ٬୯Ձ :3000ʙ5000ԁ ৔ॴ: ᷫԂ ਪ঑ਓ਺: 2 - 8ਓ ਪ͠ɿڕͱञ͕͍҆ɻͱʹ͔҆ͯ͘͘ɺඒຯ͍

Slide 50

Slide 50 text

ୋউ ٬୯Ձ :4000ʙ6000ԁ ৔ॴ: தऱ ਪ঑ਓ਺: 2 - 8ਓ ਪ͠ɿ෦԰ͷ໊લ͕ϗʔΫεͷબखɻ  ΊͪΌඒຯ͍Θ͚͡Όͳ͍͚Ͳɺ  ԿͰ΋͋Δɻศརɻ

Slide 51

Slide 51 text

ϥʔϝϯ

Slide 52

Slide 52 text

Ұ૒ ٬୯Ձ :1000ԁ ৔ॴ: ᷫԂɺതଟ ਪ͠ɿಲࠎofಲࠎ ͍͍ͩͨ৯͏ͱ͖ ᬦᬡͯ͠Δ͔Β ࣸਅͳ͔ͬͨ

Slide 53

Slide 53 text

໊ౡ௾ ٬୯Ձ :1000ԁ ৔ॴ: തଟɺ໊ౡ ਪ͠ɿͦΜͳʹ೏͍΋ڧ͘ͳͯ͘ɺ  ɹɹɹόϥϯε͕͍͍ɻ޷͖

Slide 54

Slide 54 text

݉ދ ٬୯Ձ :1000ԁ ৔ॴ: ఱਆɺതଟ ਪ͠ɿ෱ԬͰྲྀߦͬͯΔ͚ͭ໙ɻ  ɹɹɹύϧί͕Ұ൪ฒ͹ͳ͍͔΋

Slide 55

Slide 55 text

௕඿φϯόʔϫϯ ٬୯Ձ :1000ԁ ৔ॴ: ఱਆɺതଟɺᷫԂ ਪ͠ɿ͜ͷลͰ৯΂ΕΔ௕඿ܥͰҰ൪޷͖ɻ  ී௨ͷϥʔϝϯʹމຑͱߚੜᇙ͍Εͯ  ৯΂Δͷ͕͓͢͢Ί

Slide 56

Slide 56 text

ͦͷଞ

Slide 57

Slide 57 text

΢Τετ ٬୯Ձ :1000ԁ ৔ॴ: Ͳ͜Ͱ΋͋Δ ਪ͠ɿϩʔΧϧ͏ͲΜνΣʔϯɻ  ೑͏ͲΜࡉ໙ʹωΪଟΊ͕޷͖ɻ  ໷͸ډञ԰Ͱɺ৴͡ΒΕͳ͍ίεύ

Slide 58

Slide 58 text

໷ؒඈߦ ٬୯Ձ :3000-5000ԁ ৔ॴ: തଟ ਪ͠ɿ೔ޫʹೖͬͯΔϗςϧόʔɻ  Ҝࢠ͕͍͍ͷͱΏͬ͘Γ࿩ͤΔɻ  ɹɹɹτϫΠϥΠτͱ͍͏ΧΫςϧ͕޷͖

Slide 59

Slide 59 text

ϐΤτϩ ٬୯Ձ :1000-3000ԁ ৔ॴ: Ͳ͜Ͱ΋͋Δ ਪ͠ɿϥϯνͰ΋໷Ͱ΋ɺεύʔΫϦϯάͱ  ɹɹɹύελorϐβͰܹ҆ඒຯ͍

Slide 60

Slide 60 text

࠷ߴͷ໷CNDF2023ʹ   ͠·͠ΐ͏ʂʂʂ