$30 off During Our Annual Pro Sale. View Details »

CNDF2023前夜祭 - 玄界灘のクラウドネイティブなデータ基盤運用の実践

CNDF2023前夜祭 - 玄界灘のクラウドネイティブなデータ基盤運用の実践

CNDF2023前夜祭で話したデータ基盤のSREに関連する話です。

Kazuhiko Yamashita

August 02, 2023
Tweet

More Decks by Kazuhiko Yamashita

Other Decks in Technology

Transcript

  1. ݰքṗͷΫϥ΢υωΠςΟϒͳ
    σʔλج൫ӡ༻ͷ࣮ફ
    ʙॵ͍ʂʂՆͩʂʂʂૣ͘ϏΞΨʔσϯߦ͖͍ͨʂʂ̍ฤʙ

    View Slide

  2. ࢁԼ࿨඙!QZBNB
    (.0ϖύϘٕज़ج൫νʔϜ
    γχΞɾϓϦϯγύϧ
    ɹΩϟϯϓɺཱྀߦɺώϧτϯ८ΓɺιϫχΤ८Γ
    ($1ɺ"JS
    fl
    PXɺ1VC4VCɺ%BUB
    fl
    PX

    View Slide

  3. ࠓ೔࿩͢͜ͱ
    γεςϜ֓ཁ

    ͦ͜ʹ͋ͬͨ՝୊

    ͲͷΑ͏ʹղܾͨ͠ͷ͔

    View Slide

  4. ϗεςΟϯάࣄۀ &$ࢧԉࣄۀ ϋϯυϝΠυɾͦͷଞࣄۀ

    View Slide

  5. γεςϜߏ੒
    Ingest Pipeline
    DataFlow
    CloudComposer
    Extract
    Analytics


    ML
    BigQuery
    Vertex AI
    source
    monitor
    Cloud

    Logging
    Cloud

    Monitoring
    DBσʔλɺϩάΛBigQueryʹू໿
    Pub/Sub

    View Slide

  6. σʔλج൫ͷར׆༻
    ྨࣅը૾ݕࡧ

    ঎඼ਪન

    ੜ࢈ੑࢦඪͷଌఆ

    View Slide

  7. ͳͥ๻͕ʁʁʁ

    View Slide

  8. https://note.com/udzura/n/n5c8647d38
    ff
    f ΑΓҾ༻

    View Slide

  9. ͦ͜ʹ͋ͬͨ՝୊
    མͪଓ͚ΔDAGͱຫੑతͳτΠϧ

    ΤϯυϢʔβʔ͔ΒͷϑΟʔυόοΫͰݦࡏԽ͢Δো֐

    ޮ཰ͷѱ͍։ൃ

    View Slide

  10. མͪଓ͚ΔDAGͱ


    ຫੑతͳτΠϧ

    View Slide

  11. γεςϜߏ੒
    Ingest Pipeline
    DataFlow
    CloudComposer
    Extract
    Analytics


    ML
    BigQuery
    Vertex AI
    source
    monitor
    Cloud

    Logging
    Cloud

    Monitoring
    DBσʔλɺϩάΛBigQueryʹू໿
    Pub/Sub

    View Slide

  12. མͪଓ͚ΔDAGͱຫੑతͳτΠϧ
    DAG = Directed Acyclic Graph = ༗޲ඇ८ճάϥϑ
    https://air
    fl
    ow.apache.org/docs/apache-air
    fl
    ow/stable/core-concepts/dags.html ΑΓҾ༻

    View Slide

  13. DAGͷࣦഊཁҼ
    1.༷ʑͳλΠϜΞ΢τ

    2.࿈ٳʹΑΔσʔλͷܽଛ

    3.ιʔεͷσʔλߏ଄ͷมߋɺଐੑ

    4.มߋʹΑΔΤϯόά

    View Slide

  14. τΠϧͨΔॴҎ
    1.༷ʑͳλΠϜΞ΢τ → ϦτϥΠɺλΠϜΞ΢τᮢ஋มߋ

    2.࿈ٳʹΑΔσʔλͷܽଛ → ϦτϥΠɺεΩοϓ

    3.ιʔεͷσʔλߏ଄ɺଐੑͷมߋ → มߋ͓ͯ͠͠·͍
    4.มߋʹΑΔΤϯόά → मਖ਼͓ͯ͠͠·͍

    View Slide

  15. View Slide

  16. ॳख


    SLI/SLOͷࡦఆ

    View Slide

  17. SLO/SLOͷࡦఆ
    ʮͳʹ͔Β΍Δ͔ʁʯ - վળͷ༏ઌॱҐ

    ʮͲΕ͘Β͍΍Δ͔ʯ - ਺஋Ͱվળͷ౓߹͍ΛܾΊΔ

    ·ͣࢦඪΛ࡞Δ͜ͱͰɺ਺஋ۦಈͰҙࢥܾఆͰ͖Δ

    View Slide

  18. SLO/SLOͷࡦఆ
    SREϫʔΫϒοΫ - Ch 13. Data Processing Pipelines

    ͲͷΑ͏ʹܾΊΔ͔ʁ
    https://sre.google/workbook/data-processing/
    https://cloud.google.com/stackdriver/docs/solutions/slo-monitoring/sli-metrics/data-proc-metrics?hl=ja
    Google Cloud ΦϖϨʔγϣϯεΠʔτ σʔλॲཧαʔϏε

    View Slide

  19. SLO/SLOͷࡦఆ
    GrafanaͰ؆୯ʹμογϡϘʔυԽͰ͖Δ

    View Slide

  20. SLO/SLOͷࡦఆ
    1.DAGͷ੒ޭ཰

    2.DataFlowδϣϒʹ͓͚ΔΤϥʔͷൃੜ཰

    3.DataFlowδϣϒͷεςοϓʹ͓͚Δσʔλͷ઱౓

    View Slide

  21. վળͷଧͪख
    λΠϜΞ΢τͷཁҼͱͳΔɺ஗ԆΛऔΓআ͘ɺՁ஋؍Λৢ੒͢Δ

    λΠϜΞ΢τ஋ʹ͍ͭͯɺϝτϦΫε͔Βଥ౰ͳ஋͔Λ֬ೝ͢Δ

    1. ༷ʑͳλΠϜΞ΢τ

    View Slide

  22. վળͷଧͪख
    holiday-jpΛ׆༻ͯ͠ɺ࿈ٳ࣌ͷॲཧΛ࣮૷

    BigQuery͔Β΋UDFͰ࿈ٳͷ৚݅ΛఆٛͰ͖ΔΑ͏ʹͨ͠

    2.࿈ٳʹΑΔσʔλͷܽଛ

    View Slide

  23. վળͷଧͪख
    DebeziumͰDDLΛݕ஌ͨ͠ΒSlack௨஌͢Δ

    σʔλͷNULLൺ཰͕૿͑ͨΒ௨஌͢Δ

    3.ιʔεͷσʔλߏ଄ɺଐੑͷมߋ
    https://github.com/pyama86/debezium-ddl-noti
    fi
    er/

    View Slide

  24. վળͷଧͪख
    ຖ೔શDAGΛޕલɺޕޙʹ࣮ߦͯ͠ɺຊ൪࣮ߦΑΓ

    σʔλͷෆ੔߹ͳͲΛૣ͘ݕ஌

    Ϣχοτςετͷ֦ॆ

    4. มߋʹΑΔΤϯόά

    View Slide

  25. ࠷ۙ͸͓ئ͍ͯ͠ɺTABΛԡ͚ͩ͢ͷ࢓ࣄͰ͢
    ## ೖྗ࢓༷


    - Air
    fl
    owͷDAGͷϑΝΠϧΛೖྗͱ͠·͢


    ## αϯϓϧ࣮૷1


    ```python


    {sample1}


    ```


    ## αϯϓϧ࣮૷2


    ```python


    {sample2}


    ```


    ## ࢦࣔ


    - ೖྗͱͯ͠౉͞ΕͨDAGͷςετίʔυΛpytestΛ༻͍࣮ͯ૷͍ͯͩ͘͠͞ɻ


    - Ϋϥεʹ͍ͭͯ͸spy΍mockΛ༻͍ͯϞοΫԽ͍ͯͩ͘͠͞ɻ


    - 2ͭͷαϯϓϧ࣮૷Λࣔ͠·͢ɻͦΕΛࢀߟʹ͍ͯͩ͘͠͞ɻ


    - ͋ͳ͕ͨੜ੒ͨ͠ςετ͸ͦͷ··ϑΝΠϧʹอଘͯ͠ɺ࣮ߦ͠·͢ɻग़ྗ಺༰ͷઆ໌ͳͲ͸ෆཁͰ͢ɻ


    ## ೖྗ ϑΝΠϧ໊:{dag_
    fi
    le_path}


    ```python


    {dag_
    fi
    le_content}


    ```


    ...


    $ poetry run python bin/test_generator.py dags/example.py

    View Slide

  26. ΤϯυϢʔβʔ͔Βͷ

    ϑΟʔυόοΫͰݦࡏԽ͢Δো֐

    View Slide

  27. ʁʁʁʮࠓ೔ͷσʔλ͕ೖͬͯͳ͍Έ͍ͨ

    ͳΜͰ͚͢ͲɺͳΜ͔͋Γ·ͨ͠ʁʯ

    View Slide

  28. ͳΜͱɺ؂ࢹ͕

    ΄ͱΜͲͳ͍ͷͰ͋Δ
    (؆қతͳσʔλ੔߹ੑͷςετ͕͋Δ͚ͩ)

    View Slide

  29. γεςϜߏ੒
    Ingest Pipeline
    DataFlow
    CloudComposer
    Extract
    Analytics


    ML
    BigQuery
    Vertex AI
    source
    monitor
    Cloud

    Logging
    Cloud

    Monitoring
    DBσʔλɺϩάΛBigQueryʹू໿
    Pub/Sub

    View Slide

  30. ؂ࢹࣾձ΁
    AWSͰECS࢖͍ͬͯΔͱ͜Ζ͸Mackerel Container Agentಋೖ

    Cloud MonitoringͷΞϥʔτఆٛΛSLIʹ߹ΘͤͯTerraformͰ࣮૷
    GCPͱൺ΂ͯར༻ن໛͕খ͍ͨ͞ΊɺશࣾͰར༻͞Ε͍ͯΔMackerelΛར༻͢Δ͜ͱͰ

    ֶशɺಋೖίετͷ࡟ݮ

    View Slide

  31. σʔλύΠϓϥΠϯͷ؂ࢹ
    ΰʔϧσϯσʔλͷ౤ೖ



    @type dummy


    @label @INPUT


    tag example


    dummy {


    "accessed_at": "2022-01-01T00:00:00Z",


    "account_id": 1,


    "client_id": "12345abcde",


    "event": "example_event",


    }



    https://docs.
    fl
    uentd.org/v/0.12/input/dummy
    ຖඵμϛʔσʔλΛ


    ύΠϓϥΠϯʹྲྀ͠ɺ


    ΤϯυϙΠϯτͰ؂ࢹ

    View Slide

  32. ޮ཰ͷѱ͍։ൃ

    View Slide

  33. γεςϜߏ੒
    Ingest Pipeline
    DataFlow
    CloudComposer
    Extract
    Analytics


    ML
    BigQuery
    Vertex AI
    source
    monitor
    Cloud

    Logging
    Cloud

    Monitoring
    DBσʔλɺϩάΛBigQueryʹू໿
    Pub/Sub

    View Slide

  34. ։ൃ؀ڥͱۚͱݖྗ
    Cloud Composerͷ؀ڥͰ݁߹ςετΛ࣮ߦ͢ΔͨΊʹɺ

    ςετ༻ͷ؀ڥΛຖேࣗಈ࡞੒ɺຖ൩ࣗಈ࡟আ

    ίετͱνʔϜߏ੒ΛؑΈͯɺຖே̎؀ڥ࡞੒ͯ͠ɺڞ༻

    View Slide

  35. ։ൃ؀ڥͱۚͱݖྗ
    Cloud Composerͷ؀ڥͰ݁߹ςετΛ࣮ߦ͢ΔͨΊʹɺ

    ςετ༻ͷ؀ڥΛຖேࣗಈ࡞੒ɺຖ൩ࣗಈ࡟আ

    → ো֐࣌ɺ࣌ؒ֎ʹ࢖͑ͳ͍
    ίετͱνʔϜߏ੒͔Β̎؀ڥ࡞੒ͯ͠ɺڞ༻

    → ςετ଴ͪͷൃੜɺग़ྗ݁Ռͷࠞ߹

    View Slide

  36. I'm Kubernetes Rock Star
    ΦϯϓϨϛεͷKubernetes্ͰPR͝ͱʹ࣮ߦ؀ڥΛ࡞੒
    https://tech.pepabo.com/2023/07/07/data-platform-ci/

    View Slide

  37. ਓͱνΣοΫϦετ
    DAG A DAG B
    ࣮૷ͷ֬ೝΛਓ͕΍͍ͬͯͨ
    EmptyOperator(


    taks_id="external_task_sensor_target


    )

    View Slide

  38. ྲྀΕͷRuby Rock Star


    ʮͦΕASTݟͨΒ͍͍͡ΌΜʯ

    View Slide

  39. Abstract Syntax Tree
    from air
    fl
    ow.operators.empty import
    EmptyOperator


    EmptyOperator(task_id="child_task3")


    % python -m ast example.py


    Module(


    body=[


    ImportFrom(


    module='air
    fl
    ow.operators.empty',


    names=[


    alias(name='EmptyOperator')],


    level=0),


    Expr(


    value=Call(


    func=Name(id='EmptyOperator', ctx=Load()),


    args=[],


    keywords=[


    keyword(


    arg='task_id',


    value=Constant(value='child_task3'))]))],


    type_ignores=[])


    grepͰͰ͖ͳ͍͜ͱ΋ɺ੩తղੳͳΒղܾͰ͖Δ

    View Slide

  40. ͜͜·Ͱ࿩ͨ͜͠ͱ
    σʔλج൫ͷ։ൃɺӡ༻ʹ͓͚Δ՝୊Λ

    SREͷεΩϧͰվળ

    View Slide

  41. Ҿୀࣦഊʂʂ1

    View Slide

  42. શํҐ࠾༻͍ͯ͠·͢ɻ

    .-΋ͬͱ΍Γ͍ͨ͠ɺ43&΋ͬͱ͓΋͠Ζ͍ͨ͘͠
    ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ !QC@SFDSVJU

    View Slide

  43. ಥવͰ͕͢ɺ


    ൧ͷ࿩Λ͠·͢

    View Slide

  44. झຯιϫχΤΊ͙Γ
    https://www.web-soigner.jp/magazine_hakata_gourmet_guide/
    Α͏ͦ͜෱Ԭ΁ʂʂ̍

    View Slide

  45. ډञ԰

    View Slide

  46. ໨ར͖ͷ͔ͨࢤ
    ٬୯Ձ :4000ʙ6000ԁ

    ৔ॴ: ᷫԂ

    ਪ঑ਓ਺: 2 - 4ਓ

    ਪ͠ɿڕ͔Βͳʹ·Ͱɺ͢΂͕ͯඒຯ͍ɺ

    ೔ຊञ΋छྨ͕͋Γ๛෋ɻ

    ͦͯ͠ɺWEB༧໿Ͱ͖Δɻ

    ࢲతNo.1

    View Slide

  47. ͍ͤ΋Μ෷͍
    ٬୯Ձ :6000ʙ8000ԁ

    ৔ॴ: தऱ

    ਪ঑ਓ਺: 2 - 8ਓ

    ਪ͠ɿ҆͘͸ͳ͍͕ɺԿ৯΂ͯ΋ඒຯ͍ɻ

    ΤϏϑϥΠ͕৴͡ΒΕͳ͍σΧ͞ͳͷͱ

    ࢗ͠਎΋ඒຯ͍ɻ

    View Slide

  48. ḉ࿠
    ٬୯Ձ :4000ʙ8000ԁ

    ৔ॴ: தऱ

    ਪ঑ਓ਺: 2 - 8ਓ

    ਪ͠ɿͱʹ͔͘ḉͱञɻ

    ͋Μ·ΓຊभͰ৯΂Εͳ͍ḉͩͱࢥ͏

    View Slide

  49. ڕ๪ࡾಙ
    ٬୯Ձ :3000ʙ5000ԁ

    ৔ॴ: ᷫԂ

    ਪ঑ਓ਺: 2 - 8ਓ

    ਪ͠ɿڕͱञ͕͍҆ɻͱʹ͔҆ͯ͘͘ɺඒຯ͍

    View Slide

  50. ୋউ
    ٬୯Ձ :4000ʙ6000ԁ

    ৔ॴ: தऱ

    ਪ঑ਓ਺: 2 - 8ਓ

    ਪ͠ɿ෦԰ͷ໊લ͕ϗʔΫεͷબखɻ

    ΊͪΌඒຯ͍Θ͚͡Όͳ͍͚Ͳɺ

    ԿͰ΋͋Δɻศརɻ

    View Slide

  51. ϥʔϝϯ

    View Slide

  52. Ұ૒
    ٬୯Ձ :1000ԁ

    ৔ॴ: ᷫԂɺതଟ

    ਪ͠ɿಲࠎofಲࠎ
    ͍͍ͩͨ৯͏ͱ͖


    ᬦᬡͯ͠Δ͔Β


    ࣸਅͳ͔ͬͨ

    View Slide

  53. ໊ౡ௾
    ٬୯Ձ :1000ԁ

    ৔ॴ: തଟɺ໊ౡ

    ਪ͠ɿͦΜͳʹ೏͍΋ڧ͘ͳͯ͘ɺ

    ɹɹɹόϥϯε͕͍͍ɻ޷͖

    View Slide

  54. ݉ދ
    ٬୯Ձ :1000ԁ

    ৔ॴ: ఱਆɺതଟ

    ਪ͠ɿ෱ԬͰྲྀߦͬͯΔ͚ͭ໙ɻ

    ɹɹɹύϧί͕Ұ൪ฒ͹ͳ͍͔΋

    View Slide

  55. ௕඿φϯόʔϫϯ
    ٬୯Ձ :1000ԁ

    ৔ॴ: ఱਆɺതଟɺᷫԂ

    ਪ͠ɿ͜ͷลͰ৯΂ΕΔ௕඿ܥͰҰ൪޷͖ɻ

    ී௨ͷϥʔϝϯʹމຑͱߚੜᇙ͍Εͯ

    ৯΂Δͷ͕͓͢͢Ί

    View Slide

  56. ͦͷଞ

    View Slide

  57. ΢Τετ
    ٬୯Ձ :1000ԁ

    ৔ॴ: Ͳ͜Ͱ΋͋Δ

    ਪ͠ɿϩʔΧϧ͏ͲΜνΣʔϯɻ

    ೑͏ͲΜࡉ໙ʹωΪଟΊ͕޷͖ɻ

    ໷͸ډञ԰Ͱɺ৴͡ΒΕͳ͍ίεύ

    View Slide

  58. ໷ؒඈߦ
    ٬୯Ձ :3000-5000ԁ

    ৔ॴ: തଟ

    ਪ͠ɿ೔ޫʹೖͬͯΔϗςϧόʔɻ

    Ҝࢠ͕͍͍ͷͱΏͬ͘Γ࿩ͤΔɻ

    ɹɹɹτϫΠϥΠτͱ͍͏ΧΫςϧ͕޷͖

    View Slide

  59. ϐΤτϩ
    ٬୯Ձ :1000-3000ԁ

    ৔ॴ: Ͳ͜Ͱ΋͋Δ

    ਪ͠ɿϥϯνͰ΋໷Ͱ΋ɺεύʔΫϦϯάͱ

    ɹɹɹύελorϐβͰܹ҆ඒຯ͍

    View Slide

  60. ࠷ߴͷ໷CNDF2023ʹ

    ͠·͠ΐ͏ʂʂʂ

    View Slide