CNDF2023前夜祭 - 玄界灘のクラウドネイティブなデータ基盤運用の実践

CNDF2023前夜祭で話したデータ基盤のSREに関連する話です。

Transcript

1. ݰքṗͷΫϥ΢υωΠςΟϒͳ σʔλج൫ӡ༻ͷ࣮ફ ʙॵ͍ʂʂՆͩʂʂʂૣ͘ϏΞΨʔσϯߦ͖͍ͨʂʂ̍ฤʙ
2. ࢁԼ
   ࿨඙
   
   !QZBNB
   (.0ϖύϘ
   ٕज़ج൫νʔϜ
   
   
   γχΞɾϓϦϯγύϧ
   
   
   ɹΩϟϯϓɺཱྀߦɺώϧτϯ८ΓɺιϫχΤ८Γ
   
   
   
   
   
   ($1ɺ"JS fl PXɺ1VC4VCɺ%BUB fl PX
3. ࠓ೔࿩͢͜ͱ γεςϜ֓ཁ ͦ͜ʹ͋ͬͨ՝୊ ͲͷΑ͏ʹղܾͨ͠ͷ͔
4. ϗεςΟϯάࣄۀ &$ࢧԉࣄۀ ϋϯυϝΠυɾͦͷଞࣄۀ
5. γεςϜߏ੒ Ingest Pipeline DataFlow CloudComposer Extract Analytics ML BigQuery Vertex AI source monitor Cloud
 Logging Cloud
 Monitoring DBσʔλɺϩάΛBigQueryʹू໿ Pub/Sub
6. σʔλج൫ͷར׆༻ ྨࣅը૾ݕࡧ ঎඼ਪન ੜ࢈ੑࢦඪͷଌఆ
7. ͳͥ๻͕ʁʁʁ
8. https://note.com/udzura/n/n5c8647d38 ff f ΑΓҾ༻
9. ͦ͜ʹ͋ͬͨ՝୊ མͪଓ͚ΔDAGͱຫੑతͳτΠϧ ΤϯυϢʔβʔ͔ΒͷϑΟʔυόοΫͰݦࡏԽ͢Δো֐ ޮ཰ͷѱ͍։ൃ
10. མͪଓ͚ΔDAGͱ ຫੑతͳτΠϧ
11. γεςϜߏ੒ Ingest Pipeline DataFlow CloudComposer Extract Analytics ML BigQuery Vertex AI source monitor Cloud
 Logging Cloud
 Monitoring DBσʔλɺϩάΛBigQueryʹू໿ Pub/Sub
12. མͪଓ͚ΔDAGͱຫੑతͳτΠϧ DAG = Directed Acyclic Graph = ༗޲ඇ८ճάϥϑ https://air fl ow.apache.org/docs/apache-air fl ow/stable/core-concepts/dags.html ΑΓҾ༻
13. DAGͷࣦഊཁҼ 1.༷ʑͳλΠϜΞ΢τ 2.࿈ٳʹΑΔσʔλͷܽଛ 3.ιʔεͷσʔλߏ଄ͷมߋɺଐੑ 4.มߋʹΑΔΤϯόά
14. τΠϧͨΔॴҎ 1.༷ʑͳλΠϜΞ΢τ → ϦτϥΠɺλΠϜΞ΢τᮢ஋มߋ 2.࿈ٳʹΑΔσʔλͷܽଛ → ϦτϥΠɺεΩοϓ 3.ιʔεͷσʔλߏ଄ɺଐੑͷมߋ → มߋ͓ͯ͠͠·͍ 4.มߋʹΑΔΤϯόά → मਖ਼͓ͯ͠͠·͍
16. ॳख SLI/SLOͷࡦఆ
17. SLO/SLOͷࡦఆ ʮͳʹ͔Β΍Δ͔ʁʯ - վળͷ༏ઌॱҐ ʮͲΕ͘Β͍΍Δ͔ʯ - ਺஋Ͱվળͷ౓߹͍ΛܾΊΔ ·ͣࢦඪΛ࡞Δ͜ͱͰɺ਺஋ۦಈͰҙࢥܾఆͰ͖Δ
18. SLO/SLOͷࡦఆ SREϫʔΫϒοΫ - Ch 13. Data Processing Pipelines ͲͷΑ͏ʹܾΊΔ͔ʁ https://sre.google/workbook/data-processing/ https://cloud.google.com/stackdriver/docs/solutions/slo-monitoring/sli-metrics/data-proc-metrics?hl=ja Google Cloud ΦϖϨʔγϣϯεΠʔτ σʔλॲཧαʔϏε
19. SLO/SLOͷࡦఆ GrafanaͰ؆୯ʹμογϡϘʔυԽͰ͖Δ
20. SLO/SLOͷࡦఆ 1.DAGͷ੒ޭ཰ 2.DataFlowδϣϒʹ͓͚ΔΤϥʔͷൃੜ཰ 3.DataFlowδϣϒͷεςοϓʹ͓͚Δσʔλͷ઱౓
21. վળͷଧͪख λΠϜΞ΢τͷཁҼͱͳΔɺ஗ԆΛऔΓআ͘ɺՁ஋؍Λৢ੒͢Δ λΠϜΞ΢τ஋ʹ͍ͭͯɺϝτϦΫε͔Βଥ౰ͳ஋͔Λ֬ೝ͢Δ 1. ༷ʑͳλΠϜΞ΢τ
22. վળͷଧͪख holiday-jpΛ׆༻ͯ͠ɺ࿈ٳ࣌ͷॲཧΛ࣮૷ BigQuery͔Β΋UDFͰ࿈ٳͷ৚݅ΛఆٛͰ͖ΔΑ͏ʹͨ͠ 2.࿈ٳʹΑΔσʔλͷܽଛ
23. վળͷଧͪख DebeziumͰDDLΛݕ஌ͨ͠ΒSlack௨஌͢Δ σʔλͷNULLൺ཰͕૿͑ͨΒ௨஌͢Δ 3.ιʔεͷσʔλߏ଄ɺଐੑͷมߋ https://github.com/pyama86/debezium-ddl-noti fi er/
24. վળͷଧͪख ຖ೔શDAGΛޕલɺޕޙʹ࣮ߦͯ͠ɺຊ൪࣮ߦΑΓ
 σʔλͷෆ੔߹ͳͲΛૣ͘ݕ஌ Ϣχοτςετͷ֦ॆ 4. มߋʹΑΔΤϯόά
25. ࠷ۙ͸͓ئ͍ͯ͠ɺTABΛԡ͚ͩ͢ͷ࢓ࣄͰ͢ ## ೖྗ࢓༷ - Air fl owͷDAGͷϑΝΠϧΛೖྗͱ͠·͢ ## αϯϓϧ࣮૷1 ```python {sample1} ``` ## αϯϓϧ࣮૷2 ```python {sample2} ``` ## ࢦࣔ - ೖྗͱͯ͠౉͞ΕͨDAGͷςετίʔυΛpytestΛ༻͍࣮ͯ૷͍ͯͩ͘͠͞ɻ - Ϋϥεʹ͍ͭͯ͸spy΍mockΛ༻͍ͯϞοΫԽ͍ͯͩ͘͠͞ɻ - 2ͭͷαϯϓϧ࣮૷Λࣔ͠·͢ɻͦΕΛࢀߟʹ͍ͯͩ͘͠͞ɻ - ͋ͳ͕ͨੜ੒ͨ͠ςετ͸ͦͷ··ϑΝΠϧʹอଘͯ͠ɺ࣮ߦ͠·͢ɻग़ྗ಺༰ͷઆ໌ͳͲ͸ෆཁͰ͢ɻ ## ೖྗ ϑΝΠϧ໊:{dag_ fi le_path} ```python {dag_ fi le_content} ``` ... $ poetry run python bin/test_generator.py dags/example.py
26. ΤϯυϢʔβʔ͔Βͷ 
 ϑΟʔυόοΫͰݦࡏԽ͢Δো֐
27. ʁʁʁʮࠓ೔ͷσʔλ͕ೖͬͯͳ͍Έ͍ͨ 
 ͳΜͰ͚͢ͲɺͳΜ͔͋Γ·ͨ͠ʁʯ
28. ͳΜͱɺ؂ࢹ͕ 
 ΄ͱΜͲͳ͍ͷͰ͋Δ (؆қతͳσʔλ੔߹ੑͷςετ͕͋Δ͚ͩ)
29. γεςϜߏ੒ Ingest Pipeline DataFlow CloudComposer Extract Analytics ML BigQuery Vertex AI source monitor Cloud
 Logging Cloud
 Monitoring DBσʔλɺϩάΛBigQueryʹू໿ Pub/Sub
30. ؂ࢹࣾձ΁ AWSͰECS࢖͍ͬͯΔͱ͜Ζ͸Mackerel Container Agentಋೖ Cloud MonitoringͷΞϥʔτఆٛΛSLIʹ߹ΘͤͯTerraformͰ࣮૷ GCPͱൺ΂ͯར༻ن໛͕খ͍ͨ͞ΊɺશࣾͰར༻͞Ε͍ͯΔMackerelΛར༻͢Δ͜ͱͰ ֶशɺಋೖίετͷ࡟ݮ
31. σʔλύΠϓϥΠϯͷ؂ࢹ ΰʔϧσϯσʔλͷ౤ೖ <source> @type dummy @label @INPUT tag example dummy { "accessed_at": "2022-01-01T00:00:00Z", "account_id": 1, "client_id": "12345abcde", "event": "example_event", } </source> https://docs. fl uentd.org/v/0.12/input/dummy ຖඵμϛʔσʔλΛ ύΠϓϥΠϯʹྲྀ͠ɺ ΤϯυϙΠϯτͰ؂ࢹ
32. ޮ཰ͷѱ͍։ൃ
33. γεςϜߏ੒ Ingest Pipeline DataFlow CloudComposer Extract Analytics ML BigQuery Vertex AI source monitor Cloud
 Logging Cloud
 Monitoring DBσʔλɺϩάΛBigQueryʹू໿ Pub/Sub
34. ։ൃ؀ڥͱۚͱݖྗ Cloud Composerͷ؀ڥͰ݁߹ςετΛ࣮ߦ͢ΔͨΊʹɺ
 ςετ༻ͷ؀ڥΛຖேࣗಈ࡞੒ɺຖ൩ࣗಈ࡟আ ίετͱνʔϜߏ੒ΛؑΈͯɺຖே̎؀ڥ࡞੒ͯ͠ɺڞ༻
35. ։ൃ؀ڥͱۚͱݖྗ Cloud Composerͷ؀ڥͰ݁߹ςετΛ࣮ߦ͢ΔͨΊʹɺ
 ςετ༻ͷ؀ڥΛຖேࣗಈ࡞੒ɺຖ൩ࣗಈ࡟আ → ো֐࣌ɺ࣌ؒ֎ʹ࢖͑ͳ͍ ίετͱνʔϜߏ੒͔Β̎؀ڥ࡞੒ͯ͠ɺڞ༻ → ςετ଴ͪͷൃੜɺग़ྗ݁Ռͷࠞ߹
36. I'm Kubernetes Rock Star ΦϯϓϨϛεͷKubernetes্ͰPR͝ͱʹ࣮ߦ؀ڥΛ࡞੒ https://tech.pepabo.com/2023/07/07/data-platform-ci/
37. ਓͱνΣοΫϦετ DAG A DAG B ࣮૷ͷ֬ೝΛਓ͕΍͍ͬͯͨ EmptyOperator( taks_id="external_task_sensor_target )
38. ྲྀΕͷRuby Rock Star ʮͦΕASTݟͨΒ͍͍͡ΌΜʯ
39. Abstract Syntax Tree from air fl ow.operators.empty import EmptyOperator EmptyOperator(task_id="child_task3") % python -m ast example.py Module( body=[ ImportFrom( module='air fl ow.operators.empty', names=[ alias(name='EmptyOperator')], level=0), Expr( value=Call( func=Name(id='EmptyOperator', ctx=Load()), args=[], keywords=[ keyword( arg='task_id', value=Constant(value='child_task3'))]))], type_ignores=[]) grepͰͰ͖ͳ͍͜ͱ΋ɺ੩తղੳͳΒղܾͰ͖Δ
40. ͜͜·Ͱ࿩ͨ͜͠ͱ σʔλج൫ͷ։ൃɺӡ༻ʹ͓͚Δ՝୊Λ
 SREͷεΩϧͰվળ
41. Ҿୀࣦഊʂʂ1
42. શํҐ࠾༻͍ͯ͠·͢ɻ 
 .-΋ͬͱ΍Γ͍ͨ͠ɺ43&΋ͬͱ͓΋͠Ζ͍ͨ͘͠ ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ !QC@SFDSVJU
43. ಥવͰ͕͢ɺ ൧ͷ࿩Λ͠·͢
44. झຯιϫχΤΊ͙Γ https://www.web-soigner.jp/magazine_hakata_gourmet_guide/ Α͏ͦ͜෱Ԭ΁ʂʂ̍
45. ډञ԰
46. ໨ར͖ͷ͔ͨࢤ ٬୯Ձ :4000ʙ6000ԁ ৔ॴ: ᷫԂ ਪ঑ਓ਺: 2 - 4ਓ ਪ͠ɿڕ͔Βͳʹ·Ͱɺ͢΂͕ͯඒຯ͍ɺ
 ೔ຊञ΋छྨ͕͋Γ๛෋ɻ
 ͦͯ͠ɺWEB༧໿Ͱ͖Δɻ
 ࢲతNo.1
47. ͍ͤ΋Μ෷͍ ٬୯Ձ :6000ʙ8000ԁ ৔ॴ: தऱ ਪ঑ਓ਺: 2 - 8ਓ ਪ͠ɿ҆͘͸ͳ͍͕ɺԿ৯΂ͯ΋ඒຯ͍ɻ
 ΤϏϑϥΠ͕৴͡ΒΕͳ͍σΧ͞ͳͷͱ
 ࢗ͠਎΋ඒຯ͍ɻ
48. ḉ࿠ ٬୯Ձ :4000ʙ8000ԁ ৔ॴ: தऱ ਪ঑ਓ਺: 2 - 8ਓ ਪ͠ɿͱʹ͔͘ḉͱञɻ
 ͋Μ·ΓຊभͰ৯΂Εͳ͍ḉͩͱࢥ͏
49. ڕ๪ࡾಙ ٬୯Ձ :3000ʙ5000ԁ ৔ॴ: ᷫԂ ਪ঑ਓ਺: 2 - 8ਓ ਪ͠ɿڕͱञ͕͍҆ɻͱʹ͔҆ͯ͘͘ɺඒຯ͍
50. ୋউ ٬୯Ձ :4000ʙ6000ԁ ৔ॴ: தऱ ਪ঑ਓ਺: 2 - 8ਓ ਪ͠ɿ෦԰ͷ໊લ͕ϗʔΫεͷબखɻ
 ΊͪΌඒຯ͍Θ͚͡Όͳ͍͚Ͳɺ
 ԿͰ΋͋Δɻศརɻ
51. ϥʔϝϯ
52. Ұ૒ ٬୯Ձ :1000ԁ ৔ॴ: ᷫԂɺതଟ ਪ͠ɿಲࠎofಲࠎ ͍͍ͩͨ৯͏ͱ͖ ᬦᬡͯ͠Δ͔Β ࣸਅͳ͔ͬͨ
53. ໊ౡ௾ ٬୯Ձ :1000ԁ ৔ॴ: തଟɺ໊ౡ ਪ͠ɿͦΜͳʹ೏͍΋ڧ͘ͳͯ͘ɺ
 ɹɹɹόϥϯε͕͍͍ɻ޷͖
54. ݉ދ ٬୯Ձ :1000ԁ ৔ॴ: ఱਆɺതଟ ਪ͠ɿ෱ԬͰྲྀߦͬͯΔ͚ͭ໙ɻ
 ɹɹɹύϧί͕Ұ൪ฒ͹ͳ͍͔΋
55. ௕඿φϯόʔϫϯ ٬୯Ձ :1000ԁ ৔ॴ: ఱਆɺതଟɺᷫԂ ਪ͠ɿ͜ͷลͰ৯΂ΕΔ௕඿ܥͰҰ൪޷͖ɻ
 ී௨ͷϥʔϝϯʹމຑͱߚੜᇙ͍Εͯ
 ৯΂Δͷ͕͓͢͢Ί
56. ͦͷଞ
57. ΢Τετ ٬୯Ձ :1000ԁ ৔ॴ: Ͳ͜Ͱ΋͋Δ ਪ͠ɿϩʔΧϧ͏ͲΜνΣʔϯɻ
 ೑͏ͲΜࡉ໙ʹωΪଟΊ͕޷͖ɻ
 ໷͸ډञ԰Ͱɺ৴͡ΒΕͳ͍ίεύ
58. ໷ؒඈߦ ٬୯Ձ :3000-5000ԁ ৔ॴ: തଟ ਪ͠ɿ೔ޫʹೖͬͯΔϗςϧόʔɻ
 Ҝࢠ͕͍͍ͷͱΏͬ͘Γ࿩ͤΔɻ
 ɹɹɹτϫΠϥΠτͱ͍͏ΧΫςϧ͕޷͖
59. ϐΤτϩ ٬୯Ձ :1000-3000ԁ ৔ॴ: Ͳ͜Ͱ΋͋Δ ਪ͠ɿϥϯνͰ΋໷Ͱ΋ɺεύʔΫϦϯάͱ
 ɹɹɹύελorϐβͰܹ҆ඒຯ͍
60. ࠷ߴͷ໷CNDF2023ʹ 
 ͠·͠ΐ͏ʂʂʂ