Upgrade to Pro — share decks privately, control downloads, hide ads and more …

seesaa meets Mackerel

yseto
February 05, 2018

seesaa meets Mackerel

Mackerel Meetup #11 Tokyo #mackerelio

https://mackerelio.connpass.com/event/76678/

yseto

February 05, 2018
Tweet

Other Decks in Technology

Transcript

  1. seesaa meets Mackerel
    γʔαʔגࣜձࣾ
    ੉ށ༞थ

    View full-size slide

  2. ࣗݾ঺հ
    • ੉ށ༞थ
    • γʔαʔגࣜձࣾΤϯδχΞ
    • த్ೖࣾ4೥໨
    • Mackerel͸2015೥͔Β
    • ϑϩϯτɺόοΫΤϯυɺӡ༻

    View full-size slide

  3. ձࣾͷ঺հ
    • γʔαʔגࣜձࣾ
    • 2003೥10݄૑ۀ
    • γʔαʔϒϩάɺͨ͠Β͹ܝࣔ൘ɺγʔαʔ
    wikiͳͲͷϝσΟΞӡӦ
    • ϒϩάγεςϜͷOEMఏڙ

    View full-size slide

  4. ໨࣍
    • Πϯϑϥӡ༻ͷ࢓ࣄʹ͍ͭͯ
    • MackerelҎલɾҎޙ
    • งғؾͰ؂ࢹ͍ͯͨ͠ɾ͍ͯ͠Δ
    • ؂ࢹ͢ΔྖҬ͸૿͍͑ͯͬͨ
    • ͜Μͳ؂ࢹΛ͍ͯ͠·͢

    View full-size slide

  5. Πϯϑϥɾӡ༻ʹ͍ͭͯ
    • ΦϯϓϨϛεɺAWSΛओʹར༻͍ͯ͠Δ
    • ओʹӡ༻Λ΍͍ͬͯΔͷ͸3໊ఔ౓
    • ख़࿅ऀ+एऀͷ૊Έ߹Θͤ
    • ࡉ͔͍ͱ͜Ζʹख͕ճΒͳ͍
    • ࣌ʑൃੜ͢ΔϏοάόϯతΠϕϯτͰࠞཚ

    View full-size slide

  6. ӡ༻ϝϯόʔͷओͳ࢓ࣄ
    • ೔ࠒͷΞϥʔτରԠ
    • Mackerel͔ΒͷΞϥʔτΛݟͯҟৗ͔Ͳ͏͔ͷݟۃΊ
    • ӡ༻໘͕ڧ͍։ൃ
    • طଘΞϓϦέʔγϣϯͷDockerԽͷςετ
    • ࣗࣾϝσΟΞαʔϏεʢϒϩάɾܝࣔ൘ɾWikiʣ͸
    DockerԽࡁΈ

    View full-size slide

  7. ΦϯϓϨϛε
    • DBͳͲͷෛՙ͕༧૝͞ΕΔαʔόʔ͸CentOSΛಋೖͯ͠ӡ

    • ΞϓϦέʔγϣϯαʔόʔͳͲ͸ɺXen ServerΛ༻͍ͯɺVM
    ʢCentOSʣʹͯӡ༻
    • ݹ͔͘ΒͷαʔϏε͕ଟ͘ɺػثϦϓϨʔεͳͲʹ߹Θͤͯ
    VMɺmysqlͷઃఆͳͲฏ४Խ͍ͤͯ͞Δ
    • ҆ఆՔಇͯ͠΍ͬͱฏۉ఺

    View full-size slide

  8. AWS
    • ৽نαʔϏεͳͲ͸ੵۃతʹ࠾༻͢Δ܏޲
    • mackerel-agent-pluginsʹAWSؔ࿈͕ॆ࣮͍ͯ͠
    ΔͷͰ҆৺
    • ΠϯςάϨʔγϣϯػೳ͸ঢ়گʹԠͯ͡࢖͍෼͚
    • ࢖Θͳ͍৔߹͸EC2ͷϗετϝτϦοΫʹ಺แ

    View full-size slide

  9. MackerelҎલɾҎޙ

    View full-size slide

  10. MackerelҎલ
    • ࣾ಺ΦϯϓϨϛε؀ڥ؂ࢹ
    • savacan ʢಡΈ:͞͹͔ΜʣͰ؂ࢹ͍ͯͨ͠
    • cacti Ͱάϥϑ
    • 1ਓӡ༻୲౰ऀ͕ுΓ෇͖

    View full-size slide

  11. savacanͷ؆୯ͳ঺հ
    • αʔόʔ୆ா
    • ϥοΫɺVMͷऩ༰ϗετ୆ா
    • ϩʔυόϥϯαͷίϯτϩʔϥత໾ׂ
    • ϑϩϯτͷwebαʔόʔ͕ઃఆΛಡΈࠐΉ
    • Ξϥʔτൃใʢϝʔϧʣ
    • ϩάϩʔςʔγϣϯิॿ

    View full-size slide

  12. savacanͷΞʔΩςΫνϟ
    • ؂ࢹ͢ΔϗετʹεΫϦϓτΛΠϯετʔϧɺperl੡
    • CartonͳͲΛߟྀͯ͠ͳ͍࡞ΓɺΠϯετʔϧ͕ෳࡶ
    • ϝτϦοΫ͸ू໿͢ΔmysqlʹmysqlͰ௨৴
    • ؂ࢹ͸γϯάϧεϨου
    • ࣌ʑ٧·ͬͯ؂ࢹ͕ࢮʹ͕ͪ…
    • εέʔϧ͠ͳ͍ɺಠࣗCLI͸ॳ৺ऀʹͱ͔ͬͯͳΓϋʔυϧ

    View full-size slide

  13. savacanͷCLIʢྫʣ
    sdev@xxxx$ svcn stat bl-app21
    bl-app21 bl app active 1 20.1 N/A N/A
    sdev@xxxx$ svcn scan stat bl-app21
    bl-app21 df 0 95 on
    bl-app21 loads 0 35 on 35
    bl-app21 nfs_mount 0 3 on
    bl-app21 ping 0 0 on
    bl-app21 port 0 3 on 5000
    sdev@xxxx$ svcn update bl-app21 status active
    sdev@xxxx$ svcn update bl-app21 status ignored

    View full-size slide

  14. savacanͷCLIߏจ
    • svcn update HOSTNAME KEY1 VALUE1
    • σʔλΛೖΕΔmysqlͷߏ଄͕ͦͷ··࢖ΘΕ͍ͯΔ
    • ςʔϒϧߏ଄͕Θ͔Βͳ͍ͱ࢖͑ͳ͍
    • ӡ༻͢Δਓ͔͠Θ͔ͬͯͳ͍
    • ӡ༻ϝϯόʔҎ֎ͷࢀೖোนʹͳ͍ͬͯΔ

    View full-size slide

  15. savacanͷcliͰݟΕΔ৘ใ
    • load averageɺdfͷঢ়ଶ
    • pingɺϙʔτ؂ࢹɺNFSϚ΢ϯτͷ؂ࢹঢ়ଶ
    • ϩʔυόϥϯεର৅Ͱ͋Δ͔ͷ༗ແ
    • load averageɺdfͷঢ়ଶͳͲʹΑͬͯ͸঎༻
    αʔϏε͔Βࣗಈతʹ཭୤͠ɺΞϥʔτൃใ

    View full-size slide

  16. savacanͰݟΕΔաڈ
    • աڈͷ஋͸ࣺ͍ͯͯͨͷͰɺܧଓͨ͠ϝτϦοΫ؂ࢹ͸
    Ͱ͖͍ͯͳ͔ͬͨ
    • ॏཁੑΛೝࣝͰ͖͍ͯͳ͔ͬͨͷͩͱࢥ͏
    • MackerelʹΑͬͯաڈ͕ܧଓͯ͠ݟΕΔΑ͏ʹͳΓɺӡ
    ༻ෛՙΛ၆ᛌతʹݟ΍͘͢ͳͬͨ
    • cactiͰ͸ɺload averageఔ౓͔͠औಘ͍ͯ͠ͳ͔ͬͨ
    ͨΊ

    View full-size slide

  17. ͋Δౙɺӡ༻୲౰ऀ͕ࣙΊͨ

    View full-size slide

  18. cactiΛҾ͖ܧ͝͏ͱͨ͠
    • snmpͷmibΛ֦ு͠ɺएׯΦϨΦϨͳ؂ࢹΛ
    ͍ͯͨ͠ɻ
    • ଐਓత͗͢Δ…
    • cactiͷੵۃతͳܧଓӡ༻Λஅ೦
    • ژ౎ͷํ޲͔Βޫ͕ࠩ͠ࠐΜͰ͖ͨ

    View full-size slide

  19. MackerelΛࢼ༻

    View full-size slide

  20. ΤʔδΣϯτͷηοτΞοϓ
    • ଟछଟ༷ͳ؀ڥʹࡶʹΤʔδΣϯτΛೖΕͯ
    άϥϑͷงғؾʹײಈ
    • ଈܾͰ໿200ϗετͷ؂ࢹΛ࢝ΊΔ
    • ઃఆͷࣽ٧Ί͕؁͘ɺΤʔδΣϯτΛೖΕͨ
    ͚ͩɻͱ͍͏ࡶ͗͢Δϗετ΋͋ͬͨ

    View full-size slide

  21. MackerelΛೖΕͯؾ͍ͮͨ͜ͱ
    • VMͷੑೳઃܭ͕ద੾Ͱͳ͘ɺΧπΧπؾຯ
    • ֎ܗ؂ࢹʹΑͬͯࡉ͔͍εϩʔμ΢ϯͳͲݟ
    ͔ͭΔ
    • ࠓ·Ͱͷπέ͕શ෦ݟ͑ΔԽ͞ΕΔ

    View full-size slide

  22. MackerelʹΑͬͯಘΒΕͨ؂ࢹ
    • ֎ܗ؂ࢹ
    • Ҏલ͔Βଘࡏ͸͍ͯͨ͠΋ͷͷࡉ͔͍࣌ؒ
    ࣠ͷ৘ใ͕ܽམ͕ͪ͠
    • طଘ؂ࢹͰ͸ɺଟ͘ͷ৘ใΛऔಘ͍ͯ͠ͳ͔ͬ
    ͨͷͰɺࠓࢥ͑͹ɺงғؾͰରࡦͳͲ͍ͯ͠
    ͨͷ͔΋͠Εͳ͍

    View full-size slide

  23. Mackerelͷ͍͍ͱ͜Ζ
    • ·ͣ͸ΤʔδΣϯτΛೖΕͯঢ়گΛݟΔɻͱ͍͏ͱ͜Ζ
    ͔Β࢝ΊΒΕΔɻ
    • ΤʔδΣϯτ͸yumͳͲͰΠϯετʔϧͰ͖Δɻߋ৽΋
    ׆ൃ
    • ෆ۩߹ɺෆ໌఺͸͙͢ʹղܾ͞ΕΔ
    • TwitterͰࠔͬͨɻͱݴΘͣʹʮαϙʔτνʔϜ΁࿈བྷʯ

    View full-size slide

  24. զʑ͸งғؾͰ؂ࢹ͍ͯͨ͠

    View full-size slide

  25. ؾ͍͖͔͚ͮͨͬ
    • ಥવ͸ͯͳ͞Μ͔Βɺ࢖͍উखͳͲΠϯλϏϡʔ
    ͍ͨ͠ͷͰ๚໰ͯ͠ྑ͍͔?ͱͷ͓੠͕͔͔Δ
    • ओʹɺ͜Ε·ͰʹϦϦʔεͨ͠ػೳͷར༻ͷ༗
    ແɺ࢖͍উखͷώΞϦϯά
    • a-know͞ΜʮདྷिɺΠϕϯτ͕͋ΔͷͰͥͻ
    དྷͯ΄͍͠ʯ

    View full-size slide

  26. Driving Mercari with 50+ custom plugins /
    Mackerel DAY by kazeburo

    View full-size slide

  27. Mackerel DAY
    • ͦΕͳΓʹ؂ࢹϧʔϧͷݟ௚͠ͳͲ͸͍ͯ͠
    ͨͭ΋Γͩͬͨ
    • ͔͠͠ɺ΄ͱΜͲ͕৔౰ͨΓతͳݟ௚͠
    • ࡉ͔͍ͱ͜Ζ·Ͱࢹ໺͕ಧ͍ͯͳ͔ͬͨ
    • ඦฉ͸Ұݟʹ೗͔ͣɻงғؾͰ؂ࢹ͍ͯͨ͠

    View full-size slide

  28. PDCA͕ճͬͯͳ͔ͬͨ
    • ઃఆϑΝΠϧ͸αʔϏε͝ͱʹ؅ཧ
    • ࠷ॳʹ࡞ͬͨαʔϏεͷઃఆϑΝΠϧΛίϐʔͯ͠ɺઃ
    ఆΛࣽ٧ΊΔ
    • ޙͷ΄͏ʹͳΔͱɺઃఆ߲໨͕ॆ࣮
    • ઃఆͷݟ௚͕͠ͳ͍ͷͰɺ࠷ॳʹೖΕͨͷ͸๨ΕΒΕΔ
    • ʮ͋Εɺ͜Εʹ͸͜ͷϝτϦοΫ͕ͳ͍Ͷʯ

    View full-size slide

  29. Mackerelͷઃఆݟ௚͠
    • งғؾա͗ͨઃఆͩͬͨͱ൓ল
    • mackerel-agentΛೖΕ͍ͯͨ໿300ϗετͷઃఆ
    ݟ௚͠Λܾஅ
    • mkrίϚϯυͰαʔόʔҰཡΛ࡞ΓɺઃఆϑΝΠϧ
    Λμ΢ϯϩʔυ
    • શ਺ݕࠪΛߦ͏͜ͱʹ

    View full-size slide

  30. ઃఆΛ௚ͨ͠ͱ͜Ζ
    • ϓϩηεͷ؂ࢹΛ͍͕ͨ͠ɺΑ͘Θ͔Βͳ͍ͷͰɺ
    શ෦άϥϑʹग़ͤ͹͍͍Μ͡ΌͶʁ
    • ͦΕɺcheck-procsͰͰ͖ΔΑ
    • ͦ΋ͦ΋γεςϜʹඞཁෆՄܽͳϓϩηεΛશ෦
    ؂ࢹͰ͖ͯͳ͔ͬͨ
    • ྫ͑͹ɺϩʔΧϧͷDNSΩϟογϡͳͲ

    View full-size slide

  31. աڈ΍͍ࣦͬͯͨഊ
    [plugin.checks.xxxx]
    command = "TIME=`date +%s`; for x in xxx1 xxx2; do printf 'XXXXX.%s
    %d %d\n' $x `command $x` $TIME ; done”

    View full-size slide

  32. ΍ͬͯ͠·͍͕ͪ
    • ϫϯϥΠφʔ
    • ΍͚ͬͭͰॻ͍ͨΓ͢Δ
    • ޙ͔ΒɺಡΈͮΒ͍
    • ͜͏͍ͬͨ΋ͷ͸શͯγΣϧεΫϦϓτԽ
    • ϓϥάΠϯ͕͋ͬͨΒͦͬͪʹࡌͤସ͑

    View full-size slide

  33. Mackerel DAYͰͷऩ֭
    • ϩʔϧ͝ͱʹ෼͚Δ֓೦Λਖ਼͘͠஌Δ
    • ϩʔϧͷઃఆ=include͢ΔઃఆϑΝΠϧΛ
    ಉҰͱ͢ΔελΠϧ
    • ΍͹͍ɺ͔͍͍ͬ͜

    View full-size slide

  34. MackerelͰݟ͍ͨ৘ใ
    • ྫ͑͹AWSͳͲͩͱɺAZͳͲ͕ݟΕΔɻ
    • XenαʔόʔͷVM͸Ͳ͜ʹϗετ͍ͯ͠Δͷ
    ͔Θ͔Βͳ͍
    • savacanͱͷೋॏ؅ཧɺMackerelݟ͚ͨͩͰ
    ೺Ѳ͍ͨ͠

    View full-size slide

  35. Whitebait
    • ͠Β͢ - VM͕Ͳ͜ʹډΔ͔஌Β͢
    • Xen ServerͷRPCΛୟ͍ͯVMͷҐஔ৘ใͱIPΞυϨε
    ҰཡΛΩϟογϡ
    • mackerel-agentͷchecksϓϥάΠϯͷ݁ՌͰऩ༰ϗε
    τΛࣔͤΕΔΑ͏ʹ
    • https://github.com/yseto/whitebait

    View full-size slide

  36. ऩ༰ϗετ͕Θ͔Δ

    View full-size slide

  37. MackerelͷઃఆϨϙδτϦ
    • Mackerelͷઃఆ͸ࣾ಺GitͰ؅ཧ͞Ε͍ͯΔ
    • ͦ΋ͦ΋ɺϗετ͝ͱʹ֎෦ͷαʔόʔͷϨϙδτϦ
    ʹऔΓ͜Έʹߦ͘ͷ͸credential؅ཧͳͲ໰୊͕͋Δ
    • ސ٬ͷػີ৘ใ͕ೖͬͯͳ͚Ε͹໰୊ͳ͍ͱ൑அ͠ɺ
    WhitebaitʹGitϨϙδτϦͷϗεςΟϯάػೳΛ࣋ͨ
    ͤΔʢpull͚ͩͰ͖Δʣ

    View full-size slide

  38. ૿͑Δ؂ࢹྖҬ

    View full-size slide

  39. ΦϯϓϨϛεεΠον
    • ΦϯϓϨϛε؀ڥͷεΠονɺࠓ·Ͱৄ͘͠τϥ
    ϑΟοΫͷ֬ೝͳͲ͍ͯ͠ͳ͔ͬͨ
    • τϥϒϧ͕͋ͬͨͷͰɺMackerelͰ؂ࢹ͢Δ͜ͱ
    ʹ…
    • ͦ΋ͦ΋ɺMackerelͰεΠονͷάϥϑग़ͨ͠
    ͍͚ͩͳͷ͕ͩ…

    View full-size slide

  40. OSSΛௐ΂Δ
    • https://github.com/sorah/snmp2mkr
    • CookpadͷͦΒ͸͞Μ͕࡞͍ͬͯΔOSS
    • SNMPΛू໿ͯ͠ɺϗετͱͯ͠POSTͯ͘͠ΕΔ
    • ݩ͸ɺΠϕϯτͷWi-Fi؂ࢹ༻ͷΑ͏͕ͩɺͱΓ͋͑ͣઃఆ
    • ର৅εΠον͸ϙʔτ਺͕ଟ͘ɺऔΓ͜΅͕͋͠Δ΋ͷͷɺ࣌
    ؒ࣠Λେ͖͘ͱΕ͹܏޲͸͔ͭΊΔ
    • ͪΌΜͱه࿥͍ͨ͠ͳΒɺ͕Μ͹Δ΂͖

    View full-size slide

  41. ʮϓϦϯλಈ͔ͳ͍Μ͚ͩͲʁʯ
    ϑϦʔૉࡐͺͨͦ͘ ΑΓ

    View full-size slide

  42. ΦϑΟεͰ͋Δ͋Δ
    • ؾܰʹ੠͕͔͚ΒΕΔΤϯδχΞʹॿ͚Λٻ
    Ίͯ͘Δύλʔϯ
    • ΦϑΟεͷॄث΋؂ࢹͰ͖Δ΍ͭ͸ͯ͠͠·
    ͓͏
    • ΞϥʔτൃใͰɺτϥϒϧ͕ൃੜ͢Δલʹ
    ରॲ͢Δ

    View full-size slide

  43. ؂ࢹ͞ΕΔΦϑΟεػث

    View full-size slide

  44. ΦϑΟε಺ͷ؂ࢹ
    • pingͷԠ౴ΛݟΔ͜ͱͰ؂ࢹ
    • ΋ͪΖΜɺϧʔλʔͳͲ΋؂ࢹ͍ͯ͠Δ
    • ԕִ஍ΦϑΟεͷϧʔλʔ΋τϥϑΟοΫ؂
    ࢹ͠ɺτϥϒϧରԠ͕Ͱ͖ΔΑ͏ʹ͍ͯ͠Δ

    View full-size slide

  45. ͜Μͳ؂ࢹ΋͍ͯ͠·͢
    ͜Μͳ͜ͱ΋͍ͯ͠·͢

    View full-size slide

  46. mysqlεϨʔϒ
    • Slave_IO_RunningͳͲΛ؂ࢹ
    • ఆ࣌όοΫΞοϓͳͲͷؒ͸εςʔλεΛૢ

    • Ξϥʔτ͕ෆඞཁʹ໐Βͳ͍Α͏ʹ͢Δ
    • ॲཧʹࣦഊ͔ͨ͠Λcheck-file-ageͳͲͰ؂ࢹ

    View full-size slide

  47. keepalivedͷ؂ࢹ
    • ϓϩηε؂ࢹ
    • ঢ়ଶ؂ࢹʢΞΫςΟϒɾελϯόΠʣ
    • γΣϧεΫϦϓτͰIPΞυϨεͷଘࡏΛνΣοΫ
    ͠ɺঢ়ଶ͕੾ΓସΘͬͨ࣌ʹΞϥʔτΛൃใ
    • ো֐ɺఆܕ࡞ۀ࣌ͷ੾ΓସΘΓ͕ɺӡ༻ϝϯόʔ
    શମʹ఻ΘΔ

    View full-size slide

  48. ϋʔυ΢ΣΞRAIDͷ؂ࢹ
    • DELL OpenManageͳͲͷCLI݁ՌΛύʔε
    • σΟεΫނোΛݕ஌
    • ·ͨɺOpenManage͕ಥવࢮ͍ͯͨ͜͠ͱ΋
    ͋ΔͷͰɺϓϩηεͷࢮ׆؂ࢹ
    • ஌ݟ͕͋·Γཷ·͍ͬͯͳ͍

    View full-size slide

  49. ͘͞ΒͷΫϥ΢υͷ՝ۚ؂ࢹ
    • ࡞ۀͰΠϯελϯεΛ࡞ͬͯͦͷ··ɻͷΑ
    ͏ͳ͜ͱ͕ͳ͍Α͏ʹɺ՝ۚͷάϥϑΛ
    MackerelʹPOST
    • ΠϯελϯεͷλάͰαʔϏεΛ͚ͭͯɺαʔ
    Ϗε͝ͱͷ՝ֹۚΛ͓͓ΑͦͰݟ͑ΔԽ͢Δ

    View full-size slide

  50. lambdaɺCloudFrontͷ؂ࢹ
    • lambda΍CloudFrontɺAmazon Elasticsearch
    ServiceͳͲ࢖༻͍ͯ͠ΔͷͰɺmackerel-
    agent-pluginsͰ؂ࢹ
    • Elasticsearch Service͸ɺϗετͱͯ͠ొ࿥͠
    ͍ͯΔ

    View full-size slide

  51. Dockerͷ؂ࢹ
    • ίϯςφͷىಈ͍ͯ͠Δ͔Λdocker psΛύʔ
    εͯ͠؂ࢹ
    • ·ͨ͸ɺϙʔτΛ࢖͍ͬͯΔͷͰͦΕΛ؂ࢹ
    ͢ΔͳͲ
    • ෳ਺ίϯςφΛ1ϗετͰ͋͛ΔΫϥελͷΑ
    ͏ͳӡ༻͸·͍ͩͯ͠ͳ͍

    View full-size slide

  52. ঎༻؀ڥͷdockerͷঢ়ଶ

    View full-size slide

  53. ฐࣾɺChatWork࢖ͬͯ·͢
    • ChatWorkɺը૾ͷURLΛషͬͯ΋ల։ͯ͘͠Ε
    ͳ͍…
    • ChatWorkͷAPIʹը૾Λड͚෇͚ΔAPI͕ͳ͍
    • MackerelͷάϥϑΛChatWorkͰݟ͍ͨ
    • https://github.com/yseto/slack_to_chatwork

    View full-size slide

  54. ࢓૊Έ
    • Mackerel͔Βม׵༻ͷΞϓϦΛSlackͷΤϯυ
    ϙΠϯτͱͯ͠ొ࿥
    • chatworkͷwebͷૢ࡞Λௐ΂ͯɺΞοϓϩʔ
    υૢ࡞Λ૊Έཱͯͯૹ৴

    View full-size slide

  55. chatworkͰMackerelͷάϥϑΛோΊΔ
    άϥϑ͕ݟΕΔ͜ͱͰٞ࿦͕νϟοτʹͪΌΜͱ࢒Δ

    View full-size slide

  56. ·ͱΊ
    • ΦϯϓϨϛεͰ͋ͬͯ΋AWSͰ͋ͬͯ΋ɺΠ
    ϯετʔϧ͢Δ͚ͩͰ࢖͑ΔMackerel͸ӡ༻
    ͷຯํ
    • ΦϑΟεͷτϥϒϧ΋MackerelͰ؂ࢹ͢Δͱ
    ḿΔ

    View full-size slide

  57. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠

    View full-size slide