$30 off During Our Annual Pro Sale. View Details »

Observability: Mackerel による観測と Mackerel の観測 / NoOps Meetup Tokyo #8

astj
PRO
September 17, 2019

Observability: Mackerel による観測と Mackerel の観測 / NoOps Meetup Tokyo #8

astj
PRO

September 17, 2019
Tweet

More Decks by astj

Other Decks in Technology

Transcript

  1. Observability:
    Mackerel ʹΑΔ؍ଌ
    ͱ Mackerel ͷ؍ଌ
    2019/09/17 NoOps Meetup Tokyo #8
    Asato WAKISAKA (Hatena)

    View Slide

  2. Asato WAKISAKA
    GitHub: @astj / Twitter @ast_j
    ͸ͯͳ ΞϓϦέʔγϣϯΤϯδχΞ
    Mackerel ։ൃνʔϜ (2016/08~)
    Mackerel ςοΫϦʔυ (2018/05~)

    View Slide

  3. कඋൣғ:
    ϑϩϯτΤϯυ͔ΒSREing·Ͱʢࣗশʣ
    Scala, Golang, TypeScript, …

    View Slide

  4. Mackerel

    View Slide

  5. View Slide

  6. • Mackerel ஌ͬͯΔਓ
    • ͋Γ͕ͱ͏͍͟͝·͢ʂ
    • Mackerel ৮ͬͨ͜ͱ͕͋Δਓ
    • ͋Γ͕ͱ͏͍͟͝·͢ʂʂ
    • Mackerel ࢖ͬͯΔਓ
    • ͋Γ͕ͱ͏͍͟͝·͢ʂʂʂʂ
    Ξϯέʔτ

    View Slide

  7. View Slide

  8. ؂ࢹ

    View Slide

  9. Monitoring

    View Slide

  10. Observability

    View Slide

  11. ?

    View Slide

  12. Observability ͱ Monitoring

    View Slide

  13. • Observability: ʮՄ؍ଌੑʯ
    • γεςϜͷ಺෦ঢ়ଶ͕औಘͰ͖Δঢ়ଶ
    • 3ͭͷओཁͳߏ੒ཁૉ
    • Metrics, Logging, Tracing
    • Mackerel ͸͜ͷ͏ͪ Metrics த৺

    View Slide

  14. • ʮՄ؍ଌੑʯ≠ʮ؍ଌʯ
    • ʮՄ؍ଌੑʯ͸γεςϜ͕࣋ͭੑ࣭
    • ʮ؍ଌʯ͸γεςϜ֎෦ͷߦಈ

    View Slide

  15. Metrics Observability
    Log Observability
    Trace Observability
    Metrics Observation
    Log Observation
    Trace Observation

    View Slide

  16. Metrics Observability
    Log Observability
    Trace Observability
    Metrics Observation
    Log Observation
    Trace Observation
    • Server Metrics
    • Middleware Metrics
    • Application Metrics
    • Business KPI
    • Etc…
    • Alerting
    • Dashboard Visualization
    • Analysis
    • Etc…

    View Slide

  17. Metrics Observability
    Log Observability
    Trace Observability
    Metrics Observation
    Log Observation
    Trace Observation
    ʮϝτϦοΫ؂ࢹʯ͸͜ͷลʁ
    • Server Metrics
    • Middleware Metrics
    • Application Metrics
    • Business KPI
    • Etc…
    • Alerting
    • Dashboard Visualization
    • Analysis
    • Etc…

    View Slide

  18. • ͱ͸͍͑ɺͦͷઌͰ͸ਓ͕ؒ؍ଌ͍ͯ͠Δ͸ͣ
    • ݹయతʹ͸ΞϥʔτରԠͳͲ
    • ʮάϥϑΛݟΔʯ΋؍ଌ

    View Slide

  19. Metrics Observability
    Log Observability
    Trace Observability
    Metrics Observation
    Log Observation
    Trace Observation
    wਓؒ
    • ͋Ε΍
    • ͜Ε΍
    ˞͜Ε͸ۃ୺

    View Slide

  20. • લஈʹ͸ʮՄ؍ଌੑʯͦͷ΋ͷ
    • ޙஈͷʮ؍ଌʯΛ;ͨͭʹ෼͚ͯଊ͑Δ
    • ʮ؍ଌͷ࣮૷ʯ
    • ʮʢਓؒͷʣ؍ଌʯ

    View Slide

  21. Observability
    Observability
    Observability
    Observation
    Observation
    Observation
    Observe impl.
    Observe impl.
    Observe Impl.

    View Slide

  22. Observability
    Observability
    Observability
    Observation
    Observation
    Observation
    Observe impl.
    Observe impl.
    Observe Impl.
    ٕज़ͱͯ͠ͷ Observability ͕

    ͜ͷลΓΛࢦ͢͜ͱ΋͋Ε͹

    View Slide

  23. Observability
    Observability
    Observability
    Observation
    Observation
    Observation
    Observe impl.
    Observe impl.
    Observe Impl.
    ιϦϡʔγϣϯͷ࿩୊Ͱ
    Observability ͕ݴٴ͞ΕΔ࣌͸

    ͜͜·Ͱؚ·ΕΔ͜ͱ΋ଟ͍

    View Slide

  24. ……Ͱɺ͍͍Ͱ͔͢Ͷʁ

    View Slide

  25. • ٞ࿦ͷ༨஍͸͍Ζ͍Ζ͋Γͦ͏
    • ͔͠͠ࠓ೔͸͜ͷ࿩ͷਂ௥͍͸͠·ͤΜ
    • ౎߹্ࠨ୺ΛՄ؍ଌੑͱͯ͠࿩͠·͢

    View Slide

  26. Observability
    Observability
    Observability
    Observation
    Observation
    Observation
    Observe impl.
    Observe impl.
    Observe Impl.
    • ϝτϦοΫ؂ࢹ͸:
    • ϝτϦοΫΛऩू͢Δ
    • ऩूͨ͠ϝτϦοΫΛूੵ͢Δ
    • άϥϑ΍μογϡϘʔυͰՄࢹԽ͢Δ
    • ΞϥʔτΛൃใ͢Δ

    View Slide

  27. Observability
    Observability
    Observability
    Observation
    Observation
    Observation
    Observe impl.
    Observe impl.
    Observe Impl.
    ϝτϦοΫऩू
    ϝτϦοΫूੵ • άϥϑ
    • Ξϥʔτ
    • μογϡϘʔυ
    ˞Πϝʔδ
    • ਓؒ
    • ࣗಈԽରԠ

    View Slide

  28. Observability
    Observability
    Observability
    Observation
    Observation
    Observation
    Observe impl.
    Observe impl.
    Observe Impl.
    • ؍ଌ͢ΔͨΊʹ͸؍ଌՄೳʹ͢Δඞཁ͕͋Δ
    • Մ؍ଌੑ͸؂ࢹͷલఏͱͳΔ
    • ʢͷͰɺ؂ࢹͷจ຺ͰՄ؍ଌੑʹݴٴ͞ΕΔʣ

    View Slide

  29. View Slide

  30. Mackerel ͷ֓೦

    View Slide

  31. ϗετ

    View Slide

  32. • ؂ࢹɾ؅ཧͷى఺ʹͳΔ֓೦
    • యܕతʹ͸αʔόʔ
    • ʮαʔόʔͷΑ͏ͳ΋ͷʯ΋ϗετ
    • ίϯςφͷू߹ (e.g. ECS Task / k8s pod)
    • Ϋϥ΢υϦιʔε (e.g. Lambda Function)
    ϗετ

    View Slide

  33. Service:Blog
    Role:App Role:DB
    Host A
    Host B
    Host C
    Host D
    Service:Bookmark
    Role:App
    Host E
    Host F

    αʔϏεɾϩʔϧ

    View Slide

  34. • ϗετΛʮ໾ׂʯͰ෼ྨ
    • μογϡϘʔυ্Ͱάϧʔϐϯά͞ΕΔ
    • αʔϏε΍ϩʔϧ୯ҐͰ؂ࢹϧʔϧΛ؅ཧ
    αʔϏεɾϩʔϧ

    View Slide

  35. View Slide

  36. View Slide

  37. View Slide

  38. ಺෦ঢ়ଶͷऩू

    View Slide

  39. • ࣌ܥྻͰࢀরՄೳ
    ϝτϦοΫ

    View Slide

  40. • ୯७ͳ੒൱ͱϝοηʔδΛه࿥
    • ฏৗ࣌ͷཤྺ͸ه࿥ʹ࢒Βͳ͍
    νΣοΫ؂ࢹ

    View Slide

  41. • ϩάʹର͢Δ(ϝτϦοΫ|νΣοΫ)؂ࢹͰදݱ
    • ϩάΛ૸ࠪͯ͠ύλʔϯϚον͢Δ
    • LTSV ܗࣜͷΞΫηεϩάΛूܭ͢Δ
    ϩάΛ؂ࢹ͢Δ

    View Slide

  42. View Slide

  43. ऩूͷλʔήοτ
    • ΤʔδΣϯτܥ
    • αʔόʔ
    • ίϯςφ

    • αʔϏεϝτϦοΫ
    • URL ֎ܗ؂ࢹ
    • ΠϯςάϨʔγϣϯ

    View Slide

  44. • αʔόʔ্Ͱಈ͘ΤʔδΣϯτ
    • ΤʔδΣϯτ͔Β Mackerel ʹ POST ͢Δ
    • γεςϜϝτϦοΫ͸ࣗಈͰऩू
    • CPU, memory, interface…
    • ϓϥάΠϯͰऔಘ৘ใΛ֦ுͰ͖Δ
    αʔόʔ͔Βͷऩू

    View Slide

  45. Server
    Mackerel
    mackerel-agent
    Fluentd
    Golang app
    OS stats
    monitor-agent
    Plugins
    HTTP gostats
    POST

    View Slide

  46. • ECS 1 Task ΍ k8s 1 pod Λ1ϗετ
    • αΠυΧʔίϯςφͱͯ͠ΤʔδΣϯτΛಋೖ
    • γεςϜϝτϦοΫΛࣗಈͰऩू
    • ϓϥάΠϯͰ֦ுՄೳ
    ίϯςφ͔Βͷऩू

    View Slide

  47. Task
    Task
    Task
    Mackerel
    Agent
    Container
    Fluentd
    container
    Golang app
    container
    monitor-agent
    Plugins
    HTTP gostats
    POST
    Orchestration
    Metadata

    View Slide

  48. • ಛఆͷϗετʹඥ෇͔ͳ͍ϝτϦοΫ
    • Ϗδωε KPI ɺαʔϏεશମͷϨΠςϯγ
    • API Ͱࣗ༝ʹ౤ߘ
    • e.g. fluentd
    αʔϏεϝτϦοΫ
    Service:Bookmark
    Role:App
    Host

    View Slide

  49. • Mackerel ଆ͔Β URL ʹ઀ଓ
    • ઀ଓՄ൱΍ϨεϙϯελΠϜͳͲΛऩू
    URL ֎ܗ؂ࢹ

    View Slide

  50. • Ϋϥ΢υࣄۀऀͷϞχλϦϯά API ͔Βऔಘ
    • ΤʔδΣϯτͳ͠ͰऩूͰ͖Δ
    • ऩूϝτϦοΫ͸Ϋϥ΢υࣄۀऀґଘ
    AWS / Azure ΠϯςάϨʔγϣϯ
    Mackerel
    User
    Cloud
    Services
    Crawl

    View Slide

  51. ऩू͢Δ಺༰

    View Slide

  52. • γεςϜϝτϦοΫ͸ࣗಈͰऩू
    • ϛυϧ΢ΣΞ͸ެࣜϓϥάΠϯͰऩू
    • ΞϓϦέʔγϣϯͷϝτϦοΫ͸ʁʁ

    View Slide

  53. • γεςϜϝτϦοΫ͸ࣗಈͰऩू
    • ϛυϧ΢ΣΞ͸ެࣜϓϥάΠϯͰऩू
    • ΞϓϦέʔγϣϯͷϝτϦοΫ͸ʁʁ

    View Slide

  54. • ݴޠ΍ϑϨʔϜϫʔΫ޲͚ͷϓϥάΠϯ͸͋Δ
    • JVM ΍ Go ͸ GC ౷ܭ౳΋؍ଌͰ͖Δ
    • ଍Γͳ͍ϝτϦοΫͷ؍ଌखஈ͕ඞཁ

    View Slide

  55. Observability
    Observability
    Observability
    Observation
    Observation
    Observation
    Observe impl.
    Observe impl.
    Observe Impl.
    • ؍ଌ͢ΔͨΊʹ͸؍ଌՄೳʹ͢Δඞཁ͕͋Δ
    • Մ؍ଌੑ͸؂ࢹͷલఏͱͳΔ
    • ʢͷͰɺ؂ࢹͷจ຺ͰՄ؍ଌੑʹݴٴ͢Δʣ
    ࠶ܝ
    • ؍ଌ͢ΔͨΊʹ͸؍ଌՄೳʹ͢Δඞཁ͕͋Δ

    View Slide

  56. ؍ଌ͍ͨ͠ϝτϦοΫ͸ԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ
    ؍ଌՄೳʹ͢Δ
    ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ
    NO
    YES

    View Slide

  57. ؍ଌ͍ͨ͠ϝτϦοΫ͸ԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ
    ؍ଌՄೳʹ͢Δ
    ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ
    NO
    YES

    View Slide

  58. • ͲͷΑ͏ʹ؍ଌ͞ΕΔ͔Ͱ࣮૷͕ܾ·Δ
    ؍ଌՄೳʹ͢Δ

    View Slide

  59. ؍ଌ͍ͨ͠ϝτϦοΫ͸ԿΒ͔ͷखஈͰ؍ଌՄೳͰ͔͢ʁ
    ؍ଌՄೳʹ͢Δ
    ΤʔδΣϯτʢͷϓϥάΠϯʣͰ؍ଌɾऩू͢Δ
    NO
    YES

    View Slide

  60. • ϓϥάΠϯΛࣗ࡞ͯ͠೚ҙͷखஈͰ؍ଌ
    • HTTP Ͱ JSON ܗࣜΛಡΈࠐΜͰ؍ଌ
    ؍ଌखஈ

    View Slide

  61. • ϓϥάΠϯ = ࣮ߦϑΝΠϧ
    • ॴఆͷग़ྗͰϝτϦοΫ໊ͱ஋Λग़ྗ͢Δ͚ͩ
    • ࣮ߦ؀ڥ಺Ͱ“ݟ͑Δ”஋ͳΒԿͰ΋औΕΔ
    ࣗ࡞ϓϥάΠϯͰͷϝτϦοΫऩू
    $ mackerel-plugin-uptime
    uptime.seconds 7749392 1568662368

    View Slide

  62. • ࣮૷ࣗମ͸ Bash Ͱ΋ Go Ͱ΋ԿͰ΋
    • ग़ྗϑΥʔϚοτ͕ૉ๿ͳͷͰ࣮૷Λબ͹ͳ͍
    • ؍ଌର৅ʹ͋Θ࣮ͤͯ૷͢Δඞཁ͕͋Δ
    • ؍ଌର৅ଆͷࣗ༝౓͸ߴ͘ͳΔ
    ࣗ࡞ϓϥάΠϯͰͷϝτϦοΫऩू

    View Slide

  63. HTTP ܦ༝Ͱͷऩू
    • mackerel-plugin-json
    • ؍ଌ͍ͨ͠஋Λ JSON Ͱ HTTP ഑৴͢Δ
    • ϓϥάΠϯ͔ΒͦͷΤϯυϙΠϯτΛಡΈࠐΉ

    View Slide

  64. ૝ఆ໰౴
    • Q: Prometheus Exporter Έ͍ͨͳʁ
    • => ํ޲ੑ͸ yesʢ΋ͬͱγϯϓϧʣ
    • Q: Prometheus Exporter ࢖͑ͳ͍ͷʁ
    • => ……!

    View Slide

  65. HTTP ܦ༝Ͱͷऩू
    $ mackerel-plugin-json \
    > -url=http://localhost/stats
    active3 1568666460
    waiting 2 1568666460
    $ curl -fsSL http://localhost/stats
    {“active”:3,”waiting”:2}

    View Slide

  66. HTTP ܦ༝Ͱͷऩू

    View Slide

  67. • ͲͪΒ͕ྑ͍͔͸৔߹ʹΑΔ
    • طʹHTTPΛ஻ΔͳΒHTTP͕ศར
    • طଘͷ؍ଌखஈ͕͋Ε͹࠶ར༻͍ͨ͠

    View Slide

  68. ऩूͨ͠ϝτϦοΫͷར༻

    View Slide

  69. • Mackerel Ͱͷओͳ࢖͍ಓ
    • ؂ࢹʢΞϥʔτʣ
    • ՄࢹԽʢάϥϑɾμογϡϘʔυʣ

    View Slide

  70. ؂ࢹʢΞϥʔτʣ

    View Slide

  71. ;ͨͭͷΞϥʔτ
    • ଈ࣌ରԠΛଅ͢Ξϥʔτ
    • ӡ༻ऀͱͯ͠͸θϩʹ͍ͨ͠ʂ
    • Information ͱͯ͠ͷΞϥʔτ

    View Slide

  72. ଈ࣌ରԠͷΞϥʔτ (on-call)
    • γεςϜͷՔಇ͕ଛͳΘΕ͍ͯΔ
    • ରԠΛඞཁͱ͢Δ
    • “௚Δ·Ͱ์ஔ”͕ڐ༰ͳΒଈ࣌ରԠͰ͸ͳ͍

    View Slide

  73. Information ͱͯ͠ͷΞϥʔτ
    • ٸ͕ͳ͍͕ԿΒ͔ͷΞΫγϣϯΛٻΊΔ
    • ྫ: ৑௕Խϗετ͕1୆μ΢ϯ
    • γεςϜ͕ଓߦՄೳͳൣғͷʮҟৗʯ
    • ಉ͡ࢦඪ͕͞ΒʹѱԽ͢Ε͹ଈ࣌ରԠ

    View Slide

  74. • ࣮ࡍʹγεςϜ͕Քಇ͍ͯ͠Δ͔

    Ͳ͏͔Λ؂ࢹ͢Δ
    • e.g. ֎ܗ؂ࢹ, ΤϥʔϨʔτ
    • ˱ CPU 100%Ͱ΋ಈ͍ͯΕ͹ྑ͍
    ଈ࣌ରԠͷΞϥʔτ

    View Slide

  75. • ͱ͸͍͑৔߹ʹΑΓ͚Γ
    • ௚઀ͷՔಇͷ؂ࢹͰͳͯ͘΋ɺ

    ͔ͦ͜ΒՔಇঢ়گ͕ԋ៷ՄೳͳΒ༗༻

    View Slide

  76. Information ͱͯ͠ͷΞϥʔτ
    • γεςϜՔಇҎ֎ͷࢦඪ΋औΓಘΔ
    • ଈ࣌ରԠͱಉ͡௨஌ܦ࿏ʹࡌͤͳ͍

    View Slide

  77. Ξϥʔτશൠ
    • ͳΜͰ΋͔ΜͰ΋Ξϥʔτʹ͠ͳ͍
    • ຊ౰ʹඞཁͳΞϥʔτ͕ຒ΋ΕΔ
    • Information ͸ͩΒͩΒ૿͕͑ͪ

    View Slide

  78. ՄࢹԽ

    View Slide

  79. • ಺෦ͷঢ়ଶΛ࣌ܥྻ͖ͭͰه࿥͢Δ
    • खಈͰऩू͠ͳͯ͘΋؍ଌͰ͖Δ
    • ݱࡏͷঢ়ଶΛ೺Ѳ͢Δखஈ
    • ఆৗ࣌ɾো֐࣌໰Θͣ
    • աڈͷঢ়ଶΛৼΓฦΔखஈ
    άϥϑɾμογϡϘʔυ

    View Slide

  80. ؍ଌΛҭͯΔ

    View Slide

  81. ඞཁͳ؍ଌྔ͕࠷ॳ͔Βἧ͏ͱ͸ݶΒͳ͍

    View Slide

  82. • ։ൃɾݕূதʹؾͮ͘
    • ؂ࢹ༻ҙ͠Α͏ͱͯ͠ෆ଍ʹؾͮ͘
    • ো֐ى͖ͯΈͯෆ଍ʹؾͮ͘
    ͨͱ͑͹…

    View Slide

  83. ඞཁͳ؍ଌ΋࠷ॳ͔Βἧ͏ͱ͸ݶΒͳ͍

    View Slide

  84. • ։ൃɾݕূதʹؾͮ͘
    • ෛՙࢼݧͰؾͮ͘ͱඒ͍͠ʂ
    • ো֐ى͖ͯΈͯෆ଍ʹؾͮ͘
    ͨͱ͑͹…

    View Slide

  85. αΠΫϧΛճ͍ͯ͘͠

    View Slide

  86. • ໪࿦࠷ॳ͔Β੔උͰ͖Ε͹ྑ͍
    • ಛʹඞਢͷ߲໨͸཈͍͑ͯ͘
    • γεςϜͷՔಇͷ؂ࢹʂʂ
    • ܧଓతʹมߋɾվળ͍ͯ͘͠

    View Slide

  87. View Slide

  88. Mackerel Ͱ
    Mackerel Λ؍ଌ͢Δ

    View Slide

  89. Mackerel ͷٕज़ελοΫ
    • AWS ্ʹߏங
    • EC2 ӡ༻ͱ ECS Fargate ͕ࠞࡏ
    • ϝΠϯͷ JVM ͳͲ͕ EC2
    • গͣͭ͠ίϯςφϫʔΫϩʔυʹҠߦத

    View Slide

  90. ୤ઢ

    View Slide

  91. ίϯςφศར

    View Slide

  92. • Πϯελϯεͷੈ࿩͕ཁΒͳ͍
    • Πϯϑϥ·ͰؔΘΔߏ੒มߋ͕खܰ
    • “chef ߋ৽ͯ͠Πϯελϯε࡞Γ௚ͯ͠ೖΕ
    ׵͑ͯ…”
    • ʮخ͘͠ͳ͍͜ͱʯ͡ΌΜ……

    View Slide

  93. ୤ઢ͜͜·Ͱ

    View Slide

  94. Mackerel ͷٕज़ελοΫ
    ࣌ܥྻσʔλϕʔε: ϚωʔδυαʔϏεϕʔε

    View Slide

  95. ୤ઢ

    View Slide

  96. αʔόϨεΞʔΩςΫνϟศར

    View Slide

  97. • ૄ݁߹ɾϦτϥΠՄೳͳઃܭ
    • ϝοηʔδΩϡʔΛڬΉ
    • Lambda ͷॲཧ಺༰Λႈ౳ʹઃܭ
    • ύϥϝʔλมߋͰεέʔϧ͢Δ
    • ʢ໪࿦ FaaS ಛ༗ͷӡ༻՝୊͸θϩͰ͸ͳ͍ʣ

    View Slide

  98. ୤ઢ͜͜·Ͱ

    View Slide

  99. Mackerel ͷٕज़ελοΫ
    ʢڵຯ͕͋Ε͹աڈͷొஃࢿྉΛ͝ཡԼ͍͞ʂʣ

    View Slide

  100. JVM ΞϓϦέʔγϣϯͷ؍ଌ
    • ϝτϦΫεɾ؂ࢹ: mackerel
    • jstat ͱ jolokia ʹΑΔJVM಺෦ߏ଄ͷ؍ଌ
    • αʔόʔ͝ͱͷϨΠςϯγ؂ࢹ
    • ίωΫγϣϯϓʔϧ؂ࢹ
    • etc…

    View Slide

  101. JVM ΞϓϦέʔγϣϯͷ؍ଌ
    • APM: NewRelic APM
    • ΑΓਫ਼៛ͳ಺෦෼ੳ͕ඞཁͳͱ͖ʹ
    • ϩάղੳ: fluentd => Kibana
    • ΞΫηεϩάɺΞϓϦέʔγϣϯϩά

    View Slide

  102. ։ൃ؀ڥͷ؍ଌ
    • ։ൃ؀ڥʢ݉εςʔδϯά؀ڥʣ
    • ຊ൪ͱಉ༷ͷߏ੒ʢن໛͸খ͍͞ʣ
    • ಉ༷ͷ؍ଌπʔϧߏ੒
    • => ϦϦʔεલ͔Β։ൃ؀ڥͰ؍ଌ

    View Slide

  103. ૝ఆ໰౴
    • Q: ։ൃ؀ڥͷ؂ࢹͷίετ͸ʁ
    • => զʑ͸MackerelΛແྉͰ࢖͑Δʢ൓ଇʣ

    View Slide

  104. • த֩ʹ Mackerel
    • ීஈͷʮ؂ࢹʯʮ؅ཧʯͷى఺͸͜͜
    • ଞͷπʔϧͱ૊Έ߹Θͤͯ؍ଌΛิڧ

    View Slide

  105. View Slide

  106. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠

    View Slide