CNDF2023前夜祭 - 玄界灘のクラウドネイティブなデータ基盤運用の実践

CNDF2023前夜祭で話したデータ基盤のSREに関連する話です。

Transcript

1. ݰքṗͷΫϥ΢υωΠςΟϒͳ
   σʔλج൫ӡ༻ͷ࣮ફ
   ʙॵ͍ʂʂՆͩʂʂʂૣ͘ϏΞΨʔσϯߦ͖͍ͨʂʂ̍ฤʙ
   

   View Slide

2. ࢁԼ࿨඙!QZBNB
   (.0ϖύϘٕज़ج൫νʔϜ
   γχΞɾϓϦϯγύϧ
   ɹΩϟϯϓɺཱྀߦɺώϧτϯ८ΓɺιϫχΤ८Γ
   ($1ɺ"JS
   fl
   PXɺ1VC4VCɺ%BUB
   fl
   PX
   

   View Slide

3. ࠓ೔࿩͢͜ͱ
   γεςϜ֓ཁ
   
   ͦ͜ʹ͋ͬͨ՝୊
   
   ͲͷΑ͏ʹղܾͨ͠ͷ͔
   

   View Slide

4. ϗεςΟϯάࣄۀ &$ࢧԉࣄۀ ϋϯυϝΠυɾͦͷଞࣄۀ
   

   View Slide

5. γεςϜߏ੒
   Ingest Pipeline
   DataFlow
   CloudComposer
   Extract
   Analytics
   
   
   ML
   BigQuery
   Vertex AI
   source
   monitor
   Cloud

   Logging
   Cloud

   Monitoring
   DBσʔλɺϩάΛBigQueryʹू໿
   Pub/Sub
   

   View Slide

6. σʔλج൫ͷར׆༻
   ྨࣅը૾ݕࡧ
   
   ঎඼ਪન
   
   ੜ࢈ੑࢦඪͷଌఆ
   

   View Slide

7. ͳͥ๻͕ʁʁʁ
   

   View Slide

8. https://note.com/udzura/n/n5c8647d38
   ff
   f ΑΓҾ༻
   

   View Slide

9. ͦ͜ʹ͋ͬͨ՝୊
   མͪଓ͚ΔDAGͱຫੑతͳτΠϧ
   
   ΤϯυϢʔβʔ͔ΒͷϑΟʔυόοΫͰݦࡏԽ͢Δো֐
   
   ޮ཰ͷѱ͍։ൃ
   
   

   View Slide

10. མͪଓ͚ΔDAGͱ
    
    
    ຫੑతͳτΠϧ
    

    View Slide

11. γεςϜߏ੒
    Ingest Pipeline
    DataFlow
    CloudComposer
    Extract
    Analytics
    
    
    ML
    BigQuery
    Vertex AI
    source
    monitor
    Cloud

    Logging
    Cloud

    Monitoring
    DBσʔλɺϩάΛBigQueryʹू໿
    Pub/Sub
    

    View Slide

12. མͪଓ͚ΔDAGͱຫੑతͳτΠϧ
    DAG = Directed Acyclic Graph = ༗޲ඇ८ճάϥϑ
    https://air
    fl
    ow.apache.org/docs/apache-air
    fl
    ow/stable/core-concepts/dags.html ΑΓҾ༻
    

    View Slide

13. DAGͷࣦഊཁҼ
    1.༷ʑͳλΠϜΞ΢τ
    
    2.࿈ٳʹΑΔσʔλͷܽଛ
    
    3.ιʔεͷσʔλߏ଄ͷมߋɺଐੑ
    
    4.มߋʹΑΔΤϯόά
    

    View Slide

14. τΠϧͨΔॴҎ
    1.༷ʑͳλΠϜΞ΢τ → ϦτϥΠɺλΠϜΞ΢τᮢ஋มߋ
    
    2.࿈ٳʹΑΔσʔλͷܽଛ → ϦτϥΠɺεΩοϓ
    
    3.ιʔεͷσʔλߏ଄ɺଐੑͷมߋ → มߋ͓ͯ͠͠·͍
    4.มߋʹΑΔΤϯόά → मਖ਼͓ͯ͠͠·͍
    

    View Slide

15. View Slide

16. ॳख
    
    
    SLI/SLOͷࡦఆ
    

    View Slide

17. SLO/SLOͷࡦఆ
    ʮͳʹ͔Β΍Δ͔ʁʯ - վળͷ༏ઌॱҐ
    
    ʮͲΕ͘Β͍΍Δ͔ʯ - ਺஋Ͱվળͷ౓߹͍ΛܾΊΔ
    
    ·ͣࢦඪΛ࡞Δ͜ͱͰɺ਺஋ۦಈͰҙࢥܾఆͰ͖Δ
    

    View Slide

18. SLO/SLOͷࡦఆ
    SREϫʔΫϒοΫ - Ch 13. Data Processing Pipelines
    
    ͲͷΑ͏ʹܾΊΔ͔ʁ
    https://sre.google/workbook/data-processing/
    https://cloud.google.com/stackdriver/docs/solutions/slo-monitoring/sli-metrics/data-proc-metrics?hl=ja
    Google Cloud ΦϖϨʔγϣϯεΠʔτ σʔλॲཧαʔϏε
    
    

    View Slide

19. SLO/SLOͷࡦఆ
    GrafanaͰ؆୯ʹμογϡϘʔυԽͰ͖Δ
    

    View Slide

20. SLO/SLOͷࡦఆ
    1.DAGͷ੒ޭ཰
    
    2.DataFlowδϣϒʹ͓͚ΔΤϥʔͷൃੜ཰
    
    3.DataFlowδϣϒͷεςοϓʹ͓͚Δσʔλͷ઱౓
    

    View Slide

21. վળͷଧͪख
    λΠϜΞ΢τͷཁҼͱͳΔɺ஗ԆΛऔΓআ͘ɺՁ஋؍Λৢ੒͢Δ
    
    λΠϜΞ΢τ஋ʹ͍ͭͯɺϝτϦΫε͔Βଥ౰ͳ஋͔Λ֬ೝ͢Δ
    
    1. ༷ʑͳλΠϜΞ΢τ
    

    View Slide

22. վળͷଧͪख
    holiday-jpΛ׆༻ͯ͠ɺ࿈ٳ࣌ͷॲཧΛ࣮૷
    
    BigQuery͔Β΋UDFͰ࿈ٳͷ৚݅ΛఆٛͰ͖ΔΑ͏ʹͨ͠
    
    2.࿈ٳʹΑΔσʔλͷܽଛ
    

    View Slide

23. վળͷଧͪख
    DebeziumͰDDLΛݕ஌ͨ͠ΒSlack௨஌͢Δ
    
    σʔλͷNULLൺ཰͕૿͑ͨΒ௨஌͢Δ
    
    3.ιʔεͷσʔλߏ଄ɺଐੑͷมߋ
    https://github.com/pyama86/debezium-ddl-noti
    fi
    er/
    

    View Slide

24. վળͷଧͪख
    ຖ೔શDAGΛޕલɺޕޙʹ࣮ߦͯ͠ɺຊ൪࣮ߦΑΓ

    σʔλͷෆ੔߹ͳͲΛૣ͘ݕ஌
    
    Ϣχοτςετͷ֦ॆ
    
    4. มߋʹΑΔΤϯόά
    

    View Slide

25. ࠷ۙ͸͓ئ͍ͯ͠ɺTABΛԡ͚ͩ͢ͷ࢓ࣄͰ͢
    ## ೖྗ࢓༷
    
    
    - Air
    fl
    owͷDAGͷϑΝΠϧΛೖྗͱ͠·͢
    
    
    ## αϯϓϧ࣮૷1
    
    
    ```python
    
    
    {sample1}
    
    
    ```
    
    
    ## αϯϓϧ࣮૷2
    
    
    ```python
    
    
    {sample2}
    
    
    ```
    
    
    ## ࢦࣔ
    
    
    - ೖྗͱͯ͠౉͞ΕͨDAGͷςετίʔυΛpytestΛ༻͍࣮ͯ૷͍ͯͩ͘͠͞ɻ
    
    
    - Ϋϥεʹ͍ͭͯ͸spy΍mockΛ༻͍ͯϞοΫԽ͍ͯͩ͘͠͞ɻ
    
    
    - 2ͭͷαϯϓϧ࣮૷Λࣔ͠·͢ɻͦΕΛࢀߟʹ͍ͯͩ͘͠͞ɻ
    
    
    - ͋ͳ͕ͨੜ੒ͨ͠ςετ͸ͦͷ··ϑΝΠϧʹอଘͯ͠ɺ࣮ߦ͠·͢ɻग़ྗ಺༰ͷઆ໌ͳͲ͸ෆཁͰ͢ɻ
    
    
    ## ೖྗ ϑΝΠϧ໊:{dag_
    fi
    le_path}
    
    
    ```python
    
    
    {dag_
    fi
    le_content}
    
    
    ```
    
    
    ...
    
    
    $ poetry run python bin/test_generator.py dags/example.py
    

    View Slide

26. ΤϯυϢʔβʔ͔Βͷ
    

    ϑΟʔυόοΫͰݦࡏԽ͢Δো֐
    

    View Slide

27. ʁʁʁʮࠓ೔ͷσʔλ͕ೖͬͯͳ͍Έ͍ͨ
    

    ͳΜͰ͚͢ͲɺͳΜ͔͋Γ·ͨ͠ʁʯ
    

    View Slide

28. ͳΜͱɺ؂ࢹ͕
    

    ΄ͱΜͲͳ͍ͷͰ͋Δ
    (؆қతͳσʔλ੔߹ੑͷςετ͕͋Δ͚ͩ)
    

    View Slide

29. γεςϜߏ੒
    Ingest Pipeline
    DataFlow
    CloudComposer
    Extract
    Analytics
    
    
    ML
    BigQuery
    Vertex AI
    source
    monitor
    Cloud

    Logging
    Cloud

    Monitoring
    DBσʔλɺϩάΛBigQueryʹू໿
    Pub/Sub
    

    View Slide

30. ؂ࢹࣾձ΁
    AWSͰECS࢖͍ͬͯΔͱ͜Ζ͸Mackerel Container Agentಋೖ
    
    Cloud MonitoringͷΞϥʔτఆٛΛSLIʹ߹ΘͤͯTerraformͰ࣮૷
    GCPͱൺ΂ͯར༻ن໛͕খ͍ͨ͞ΊɺશࣾͰར༻͞Ε͍ͯΔMackerelΛར༻͢Δ͜ͱͰ
    
    ֶशɺಋೖίετͷ࡟ݮ
    

    View Slide

31. σʔλύΠϓϥΠϯͷ؂ࢹ
    ΰʔϧσϯσʔλͷ౤ೖ
    
    
    
    @type dummy
    
    
    @label @INPUT
    
    
    tag example
    
    
    dummy {
    
    
    "accessed_at": "2022-01-01T00:00:00Z",
    
    
    "account_id": 1,
    
    
    "client_id": "12345abcde",
    
    
    "event": "example_event",
    
    
    }
    
    
    
    https://docs.
    fl
    uentd.org/v/0.12/input/dummy
    ຖඵμϛʔσʔλΛ
    
    
    ύΠϓϥΠϯʹྲྀ͠ɺ
    
    
    ΤϯυϙΠϯτͰ؂ࢹ
    

    View Slide

32. ޮ཰ͷѱ͍։ൃ
    

    View Slide

33. γεςϜߏ੒
    Ingest Pipeline
    DataFlow
    CloudComposer
    Extract
    Analytics
    
    
    ML
    BigQuery
    Vertex AI
    source
    monitor
    Cloud

    Logging
    Cloud

    Monitoring
    DBσʔλɺϩάΛBigQueryʹू໿
    Pub/Sub
    

    View Slide

34. ։ൃ؀ڥͱۚͱݖྗ
    Cloud Composerͷ؀ڥͰ݁߹ςετΛ࣮ߦ͢ΔͨΊʹɺ

    ςετ༻ͷ؀ڥΛຖேࣗಈ࡞੒ɺຖ൩ࣗಈ࡟আ
    
    ίετͱνʔϜߏ੒ΛؑΈͯɺຖே̎؀ڥ࡞੒ͯ͠ɺڞ༻
    

    View Slide

35. ։ൃ؀ڥͱۚͱݖྗ
    Cloud Composerͷ؀ڥͰ݁߹ςετΛ࣮ߦ͢ΔͨΊʹɺ

    ςετ༻ͷ؀ڥΛຖேࣗಈ࡞੒ɺຖ൩ࣗಈ࡟আ
    
    → ো֐࣌ɺ࣌ؒ֎ʹ࢖͑ͳ͍
    ίετͱνʔϜߏ੒͔Β̎؀ڥ࡞੒ͯ͠ɺڞ༻
    
    → ςετ଴ͪͷൃੜɺग़ྗ݁Ռͷࠞ߹
    

    View Slide

36. I'm Kubernetes Rock Star
    ΦϯϓϨϛεͷKubernetes্ͰPR͝ͱʹ࣮ߦ؀ڥΛ࡞੒
    https://tech.pepabo.com/2023/07/07/data-platform-ci/
    

    View Slide

37. ਓͱνΣοΫϦετ
    DAG A DAG B
    ࣮૷ͷ֬ೝΛਓ͕΍͍ͬͯͨ
    EmptyOperator(
    
    
    taks_id="external_task_sensor_target
    
    
    )
    

    View Slide

38. ྲྀΕͷRuby Rock Star
    
    
    ʮͦΕASTݟͨΒ͍͍͡ΌΜʯ
    

    View Slide

39. Abstract Syntax Tree
    from air
    fl
    ow.operators.empty import
    EmptyOperator
    
    
    EmptyOperator(task_id="child_task3")
    
    
    % python -m ast example.py
    
    
    Module(
    
    
    body=[
    
    
    ImportFrom(
    
    
    module='air
    fl
    ow.operators.empty',
    
    
    names=[
    
    
    alias(name='EmptyOperator')],
    
    
    level=0),
    
    
    Expr(
    
    
    value=Call(
    
    
    func=Name(id='EmptyOperator', ctx=Load()),
    
    
    args=[],
    
    
    keywords=[
    
    
    keyword(
    
    
    arg='task_id',
    
    
    value=Constant(value='child_task3'))]))],
    
    
    type_ignores=[])
    
    
    grepͰͰ͖ͳ͍͜ͱ΋ɺ੩తղੳͳΒղܾͰ͖Δ
    

    View Slide

40. ͜͜·Ͱ࿩ͨ͜͠ͱ
    σʔλج൫ͷ։ൃɺӡ༻ʹ͓͚Δ՝୊Λ

    SREͷεΩϧͰվળ
    

    View Slide

41. Ҿୀࣦഊʂʂ1
    

    View Slide

42. શํҐ࠾༻͍ͯ͠·͢ɻ
    

    .-΋ͬͱ΍Γ͍ͨ͠ɺ43&΋ͬͱ͓΋͠Ζ͍ͨ͘͠
    ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ !QC@SFDSVJU
    

    View Slide

43. ಥવͰ͕͢ɺ
    
    
    ൧ͷ࿩Λ͠·͢
    

    View Slide

44. झຯιϫχΤΊ͙Γ
    https://www.web-soigner.jp/magazine_hakata_gourmet_guide/
    Α͏ͦ͜෱Ԭ΁ʂʂ̍
    

    View Slide

45. ډञ԰
    

    View Slide

46. ໨ར͖ͷ͔ͨࢤ
    ٬୯Ձ :4000ʙ6000ԁ
    
    ৔ॴ: ᷫԂ
    
    ਪ঑ਓ਺: 2 - 4ਓ
    
    ਪ͠ɿڕ͔Βͳʹ·Ͱɺ͢΂͕ͯඒຯ͍ɺ

    ೔ຊञ΋छྨ͕͋Γ๛෋ɻ

    ͦͯ͠ɺWEB༧໿Ͱ͖Δɻ

    ࢲతNo.1
    

    View Slide

47. ͍ͤ΋Μ෷͍
    ٬୯Ձ :6000ʙ8000ԁ
    
    ৔ॴ: தऱ
    
    ਪ঑ਓ਺: 2 - 8ਓ
    
    ਪ͠ɿ҆͘͸ͳ͍͕ɺԿ৯΂ͯ΋ඒຯ͍ɻ

    ΤϏϑϥΠ͕৴͡ΒΕͳ͍σΧ͞ͳͷͱ

    ࢗ͠਎΋ඒຯ͍ɻ
    

    View Slide

48. ḉ࿠
    ٬୯Ձ :4000ʙ8000ԁ
    
    ৔ॴ: தऱ
    
    ਪ঑ਓ਺: 2 - 8ਓ
    
    ਪ͠ɿͱʹ͔͘ḉͱञɻ

    ͋Μ·ΓຊभͰ৯΂Εͳ͍ḉͩͱࢥ͏
    

    View Slide

49. ڕ๪ࡾಙ
    ٬୯Ձ :3000ʙ5000ԁ
    
    ৔ॴ: ᷫԂ
    
    ਪ঑ਓ਺: 2 - 8ਓ
    
    ਪ͠ɿڕͱञ͕͍҆ɻͱʹ͔҆ͯ͘͘ɺඒຯ͍
    

    View Slide

50. ୋউ
    ٬୯Ձ :4000ʙ6000ԁ
    
    ৔ॴ: தऱ
    
    ਪ঑ਓ਺: 2 - 8ਓ
    
    ਪ͠ɿ෦԰ͷ໊લ͕ϗʔΫεͷબखɻ

    ΊͪΌඒຯ͍Θ͚͡Όͳ͍͚Ͳɺ

    ԿͰ΋͋Δɻศརɻ
    

    View Slide

51. ϥʔϝϯ
    

    View Slide

52. Ұ૒
    ٬୯Ձ :1000ԁ
    
    ৔ॴ: ᷫԂɺതଟ
    
    ਪ͠ɿಲࠎofಲࠎ
    ͍͍ͩͨ৯͏ͱ͖
    
    
    ᬦᬡͯ͠Δ͔Β
    
    
    ࣸਅͳ͔ͬͨ
    

    View Slide

53. ໊ౡ௾
    ٬୯Ձ :1000ԁ
    
    ৔ॴ: തଟɺ໊ౡ
    
    ਪ͠ɿͦΜͳʹ೏͍΋ڧ͘ͳͯ͘ɺ

    ɹɹɹόϥϯε͕͍͍ɻ޷͖
    

    View Slide

54. ݉ދ
    ٬୯Ձ :1000ԁ
    
    ৔ॴ: ఱਆɺതଟ
    
    ਪ͠ɿ෱ԬͰྲྀߦͬͯΔ͚ͭ໙ɻ

    ɹɹɹύϧί͕Ұ൪ฒ͹ͳ͍͔΋
    

    View Slide

55. ௕඿φϯόʔϫϯ
    ٬୯Ձ :1000ԁ
    
    ৔ॴ: ఱਆɺതଟɺᷫԂ
    
    ਪ͠ɿ͜ͷลͰ৯΂ΕΔ௕඿ܥͰҰ൪޷͖ɻ

    ී௨ͷϥʔϝϯʹމຑͱߚੜᇙ͍Εͯ

    ৯΂Δͷ͕͓͢͢Ί
    

    View Slide

56. ͦͷଞ
    

    View Slide

57. ΢Τετ
    ٬୯Ձ :1000ԁ
    
    ৔ॴ: Ͳ͜Ͱ΋͋Δ
    
    ਪ͠ɿϩʔΧϧ͏ͲΜνΣʔϯɻ

    ೑͏ͲΜࡉ໙ʹωΪଟΊ͕޷͖ɻ

    ໷͸ډञ԰Ͱɺ৴͡ΒΕͳ͍ίεύ
    

    View Slide

58. ໷ؒඈߦ
    ٬୯Ձ :3000-5000ԁ
    
    ৔ॴ: തଟ
    
    ਪ͠ɿ೔ޫʹೖͬͯΔϗςϧόʔɻ

    Ҝࢠ͕͍͍ͷͱΏͬ͘Γ࿩ͤΔɻ

    ɹɹɹτϫΠϥΠτͱ͍͏ΧΫςϧ͕޷͖
    

    View Slide

59. ϐΤτϩ
    ٬୯Ձ :1000-3000ԁ
    
    ৔ॴ: Ͳ͜Ͱ΋͋Δ
    
    ਪ͠ɿϥϯνͰ΋໷Ͱ΋ɺεύʔΫϦϯάͱ

    ɹɹɹύελorϐβͰܹ҆ඒຯ͍
    

    View Slide

60. ࠷ߴͷ໷CNDF2023ʹ
    

    ͠·͠ΐ͏ʂʂʂ
    

    View Slide