AWSの「隙間」を埋める隙間家具 OSS 開発 / AWS DevDay Tokyo 2019

Transcript

1. AWSͷʮ伱ؒʯΛຒΊΔ 伱ؒՈ۩ OSS ։ൃ ౻ݪढ़Ұ࿠ @fujiwara

2. @fujiwara SREνʔϜ github.com/fujiwara sfujiwara.hatenablog.com

3. Game & Community

4. Agenda AWSͷʮ伱ؒʯͱ͸ ʮ伱ؒՈ۩ OSSʯͷ࣮ྫͱઃܭࢥ૝ Rin / s32cs / ssmwrap ͳͥ

   OSS ͳͷ͔

5. AWSͷʮ伱ؒʯͱ͸

6. AWSͷʮ伱ؒʯͱ͸ ϚωʔδυαʔϏε͸࠷ॳ͸ίΞػೳͰϦϦʔε͞Ε (ཁ๬ΛऔΓೖΕͳ͕Β) ঃʑʹػೳ͕૿͍͑ͯ͘

7. ྫ: RDS for MySQL ͷྺ࢙ 2009-10 ϩʔϯν 2010-05 Multi-AZػೳ 2015-12

   ೚ҙλΠϜκʔϯઃఆ 2018-01 CloudWatch Logsʹϩάग़ྗػೳ αʔϏεͷࠜװʹؔΘΔίΞػೳ (ྫ:Multi-AZ) ͸ ͘͝ॳظʹ࣮૷͞ΕΔ͕… ޻෉ͰԿͱ͔ͳΔػೳ (ྫ:λΠϜκʔϯ) ଞαʔϏεͱͷ࿈ܞศརػೳ (ྫ:Logsग़ྗ) ͸ޙճ͠ʹ͞Ε͕ͪ ← 伱ؒ

8. ޻෉ͰԿͱ͔ͳΔػೳ͸ޙճ͠ʹͳΓ͕ͪ ྫ͑͹λΠϜκʔϯΛ JST ʹઃఆ͍ͨ͠৔߹ ύϥϝʔλάϧʔϓͷ init_connect ͰҎԼͷ SQL Λࢦఆ1 SET

   SESSION time_zone = CASE WHEN POSITION('rds' IN CURRENT_USER()) = 1 THEN 'UTC' ELSE 'Asia/Tokyo' END; rds ͔Β࢝·ΔϢʔβʔ(RDS಺෦Ͱར༻͢Δ΋ͷ)Ͱ͸ UTC ͦΕҎ֎ͷϢʔβʔʹ͸ Asia/Tokyo (JST) ʹઃఆ͢Δ Bad Know-how... 1 https://qiita.com/j3tm0t0/items/089ef96ba131df079ca4

9. ଞͷαʔϏεͱͷ࿈ܞศརػೳ΋… RDS ͷϩάΛ֎෦ʹྲྀ͔ͨͬͨ͠ github.com/kenjiskywalker/fluent-plugin-rds-slowlog github.com/shinsaka/fluent-plugin-rds-log github.com/acidlemon/rds-throwlog ΈΜͳؤு͍ͬͯͨ CloudWatch Logs ࿈ܞϦϦʔεͰઃఆϙνͰ׬ྃ͢ΔΑ͏ʹ

10. 伱͕ؒଟ͍ϚωʔδυαʔϏεΛ࢖͏͔Ͳ͏͔ ϚωʔδυαʔϏε͸উखʹڧ͘ͳΔ ࣗલӡ༻͸উखʹ͸ڧ͘ͳΒͳ͍ ύονద༻ɺόʔδϣϯΞοϓͳͲͷਓతίετΛे෼ʹ෷͑ͳ͍ͳΒ ଟগෆศͰ΋(க໋తͰͳ͍ͳΒ)ϚωʔδυαʔϏεΛબ୒͠ ޙ೔ͷػೳ֦ுΛظ଴ͭͭ͠ӡ༻ͷखؒΛݮΒͨ͠΄͏͕Α͍

11. 伱ؒՈ۩Λࣗ࡞͢Δ খ͘͞ɺͦͷϢʔεέʔε಺Ͱద੾ͳ൚༻౓Λ࣋ͬͨ΋ͷΛ࡞Δ ຊՈ͕伱ؒΛຒΊͨΒࣺͯΒΕΔཻ౓Ͱ࡞Δ

12. ʮ伱ؒՈ۩ OSSʯͷ࣮ྫͱઃܭࢥ૝

13. 伱ؒͷ࣮ྫ - S3 ͱ Redshift ͷؒ Redshift ʹσʔλΛऔΓ͜Ήʹ͸ S3 ͔Βͷίϐʔ͕ඞਢ

    Redshift ʹ઀ଓ͠ɺऔΓࠐΉΦϒδΣΫτΛࢦఆͯ͠ COPY ΫΤϦΛൃߦ 2012-11 ʹ Redshift ͕ϩʔϯνͯ͠3೥ S3͔ΒܧଓతʹऔΓ͜ΉϚωʔδυͳํ๏͸ଘࡏ͠ͳ͔ͬͨ

14. 伱ؒՈ۩ OSS ͷ࣮ྫ - Rin github.com/fujiwara/Rin S3 Πϕϯτ௨஌Ͱ SQS ʹૹ৴

    SQS ͷϝοηʔδΛݩʹ Redshiftʹ COPY Λൃߦ ͯ͠औΓࠐΈΛߦ͏ Go ੡πʔϧ 2015.05 ʹ։ൃ

15. ݩʑผͷํ๏Λ࢖༻͍ͯͨ͠ fluent-plugin-redshift Fluentd ͷ output plugin ϝοηʔδͷoutputॲཧͱͯ͠ S3 ΁Ξοϓϩʔυ Redshift

    ΁ͷCOPYൃߦ Λಉظతʹ࣮ߦ Fluentd ʹϩάΛૹΔ͚ͩͰ Redshift ·ͰऔΓ͜ ·ΕΔͷͰศརͱࢥͬͯಋೖ

16. ӡ༻͍ͯͯ͠ͷ໰୊ S3΁ͷΞοϓϩʔυͱRedshift΁ͷऔΓࠐΈ͕ෆՄ෼ Redshift ͸ఆظతʹϝϯςφϯεͰఀࢭ ΫϥελϦαΠζͰ΋ఀࢭ ఀࢭ࣌͸ COPY ͕ࣦഊ͢Δ S3 ΁ͷΞοϓϩʔυͱ

    COPY ͕ҰମͷͨΊ ϦτϥΠ͸Ξοϓϩʔυ͔Β΍Γͳ͓͠ → S3 ʹΰϛ͕ཷ·Δ

17. Rin ͷઃܭํ਑ S3 ΁ͷΞοϓϩʔυ͸ଞʹ೚ͤΔ (fluentd) S3, SQS ͷՄ༻ੑ͸େมߴ͍ Redshift ͷμ΢ϯλΠϜ͸ൺֱతେ͖͍

    ϛεϚον෦෼ΛҰ౓ʹॲཧ͠ͳ͍͜ͱͰ ϦτϥΠΛ༰қʹ Redshift ΁ͷ COPY ൃߦʹಛԽ͢Δπʔϧ

18. Rin ͷઃఆϑΝΠϧ queue_name: my_queue_name # SQS queue name targets: -

    s3: region: ap-northeast-1 bucket: my.test.bucket key_prefix: test/foo/ redshift: host: redshift.example.com port: 5439 dbname: test user: test password: xxxxxxxx schema: public table: foo sql_option: "JSON 'auto' GZIP"

19. ॊೈͳऔΓࠐΈઃఆ targets: - s3: bucket: my.test.bucket key_regexp: test/schema-([a-z0-9]+)/table-([a-z0-9]+)/ redshift: schema:

    $1 # ^ͷਖ਼نදݱͰcaptureͨ͠஋Λల։ table: $2 େ఍ͷΞϓϦέʔγϣϯͰ͸औΓࠐΈςʔϒϧ͸ෳ਺ʹͳΔ ॊೈʹऔΓ͜ΊΔΑ͏ʹಈతઃఆΛՄೳʹ

20. ։ൃͯ͠5ϲ݄ޙ 伱͕ؒຒ·ͬͨ 2015.10 Kinesis Firehose ൃද

21. Before / After Fluentd ΁ϩάΛૹ৴ 㱺 Redshift ʹॱ࣍औΓ͜·ΕΔ ߏ଄͸ͦͷ··Ϛωʔδυʹ !

    ౦ژϦʔδϣϯʹདྷͨͷ͸ 2017.07 (2೥ޙ)

22. Rin ͷण໋͸௵͑ͨͷ͔? ࣮͸·ͩੜ͖͍ͯ·͢ Fluentd͔ΒͷऔΓࠐΈ͸Firehoseܦ༝ͰΑ͘ͳ͕ͬͨ ผͷϢʔεέʔεͰ࢖͑Δ ELB / ALB ͷϩά͕ S3

    ʹ഑ஔ͞ΕΔ΋ͷΛஞ࣍औΓࠐΈ͢Δͷʹ׆༂த ϢʔεέʔεΛߜͬͨద౓ͳ൚༻ੑ͕૗ޭ͍ͯ͠Δ

23. Rin ͷڭ܇ Ͳ͏ߟ͑ͯ΋Ϛωʔδυʹͳͬͯ΄͍͠΋ͷ͸ͦͷ͏ͪͳΔ (धཁ͕͋Ε͹…) ෳ਺ͷαʔϏεؒ࿈ܞΛҰ౓ʹॲཧ͢ΔͱϦτϥΠ͕ෳࡶʹͳΔ ୯ػೳʹಛԽͭͭ͠ɺͦͷυϝΠϯͰͷ൚༻ੑΛ࣋ͨͤΔͱ࢖͍ճͤΔ Ϛωʔδυʹͳͬͨͱ͖ʹ͖Ε͍ʹऔΓ֎ͤΔઃܭ͕େࣄ

24. 伱ؒՈ۩ OSS ͷྫ - s32cs Amazon CloudSearch 2012-04 ϦϦʔε 2019-10

    ݱࡏ ϚωʔδυͰσʔλΛܧଓతʹऔΓ͜Ήํ๏͸ͳ͍ (console/ aws-cli / API / HTTP POST ͷΈ)

25. github.com/fujiwara/s32cs (S3 to CloudSearch) S3 ͷΠϕϯττϦΨͰىಈ͢Δ Lambda ؔ਺ͱ࣮ͯ͠૷ S3 ʹ഑ஔ͞Εͨ

    ndjson (Newline Delimited JSON) ΛՃ޻ HTTP POST Ͱ CloudSearch ʹ౤ೖ͢Δ

26. CloudSearch ΁ͷ౤ೖॲཧ CloudSearch ΁͸ 5MB ҎԼͷ JSON ഑ྻΛ౤ೖ ࡉ͔͍୯ҐͰ౤ೖ͢ΔͱύϑΥʔϚϯε͕ྼԽ͢Δ s32cs

    = ഑ྻܗࣜ΁ͷՃ޻ɺ෼ׂͱ HTTP POST Λ΍ͬͯ͘ΕΔ͚ͩͷπʔϧ {"id": "123", "type": "add", "fields": {"title": "hoge", "message": "Θ͍Θ͍"]}} {"id": "345", "type": "delete"} ↓ [ {"id": "123", "type": "add", "fields": {"title": "hoge", "message": "Θ͍Θ͍"]}}, {"id": "345", "type": "delete"} ]

27. Firehose → S3 ͷΠϕϯττϦΨ͔Βͳʹ͔͢Δύλʔϯ n෼͝ͱʹσʔλॲཧ͢ΔόονΛΠϕϯτυϦϒϯʹม׵͍ͯ͠Δ s32cs Ҏલ cronͰաڈ5෼ͷσʔλΛDB͔ΒऔಘɺՃ޻ɺHTTP POST Ͱ౤ೖ͢Δόον

    σʔλऔಘɺՃ޻ɺ౤ೖ͕ҰମͰෆՄ෼ s32cs Ҏޙ ੜ੒͸ΞϓϦέʔγϣϯͰͷϩάૹ৴ 5෼͝ͱʹσʔλΛ·ͱΊΔͷ͸ Firehose ·ͱ·ͬͨσʔλΛՃ޻ͯ͠౤ೖ͢Δ͚͕ͩ࢓ࣄ

28. ຊ౰ʹ΍Γ͍ͨ͜ͱ = ॱ࣍ൃੜ͢ΔσʔλΛద੾ͳ୯ҐͰॲཧ͍ͨ͠ ͜ΕΛn෼͝ͱͷcronͰ࣮૷ͯ͠͠·͏ͱ… 1ճॲཧ͕ൈ͚Δͱ࣍ճ࣮ߦͰ͸2ճ෼Λॲཧ͢Δඞཁ͕ → ࠷ޙͷॲཧ஍఺Λه࿥͓ͯ͘͠ඞཁ͕͋Δ → γϦΞϧॲཧʹͳΔ →

    Մ༻ੑ΍ύϑΥʔϚϯεͷͨΊͷ෼ࢄॲཧ͕ͮ͠Β͍ → ΋͠n೔෼Λॲཧ͠Α͏ͱ͢Δͱ…? εέʔϧ͠ͳ͍ + ࣦഊ͢Δͱۓு͕૸Δ

29. Firehose ͰͷετϦʔϜ → S3ΠϕϯττϦΨॲཧ n෼͝ͱͷcronΛ୅ସ ετϦʔϜΛn෼͝ͱͷchunkͰS3ʹు͖ग़͢ॲཧ = Firehose S3 ʹ͸

    ࢦఆ࣌ؒ (60ʙ900ඵ) ΋͘͠͸ࢦఆαΠζ (1ʙ128MB)ͷ ઌʹୡͨ͠৚݅ͰΦϒδΣΫτ͕ੜ੒͞ΕΔ ෼ࢄॲཧ͕ՄೳʹͳΔ όονॲཧଆʹঢ়ଶΛ࣋ͭඞཁ͕ͳ͍ ҆৺ͯ͠ΦϯϝϞϦॲཧ͕Ͱ͖Δʂ

30. ϚωʔδυαʔϏεʹ΋͍Ζ͍Ζ͋Δ Ϋϥ΢υωΠςΟϒ = ΠϯελϯεΛҙࣝ͠ͳͯ͘Α͍(Ͱ͖ͳ͍) SQS, S3, CloudWatch, Labmda, DynamoDB... Ϋϥ΢υωΠςΟϒʹ͍ۙ

    = Πϯελϯε͸ᐓؾʹݟ͑Δ͕εέʔϧ͕ࣗಈ ELB, CloudSearch... Ϋϥ΢υωΠςΟϒʹͳΓ͖Εͳ͍=Πϯελϯε͕ݟ͑εέʔϧ͕ࣗಈͰͳ͍ RDS, ElastiCache... Ϋϥ΢υωΠςΟϒͳ΋ͷΛ׆༻͍ͯ͘͠ͱεέʔϧ͠΍͍͢ Ϋϥ΢υωΠςΟϒͳࢥߟ๏͸େࣄ

31. s32cs ͷڭ܇ n෼͝ͱͷcronॲཧɺຊ౰ʹ΍Γ͍ͨ͜ͱ͸࣌ؒґଘͰ͸ͳ͍͔΋ʁ ద੾ͳ୯ҐͰ੍໿͞ΕͨσʔλͷΠϕϯτυϦϒϯॲཧ͸࣮૷ָ͕ ؾ݉Ͷͳ͘ΦϯϝϞϦॲཧ͕Ͱ͖Δ ঢ়ଶ͸ϚωʔδυαʔϏεʹɺঢ়ଶΛ࣋ͨͳ͍ॲཧ͚ͩॻ͘ͱεέʔϧ͕༰қ

32. 伱ؒՈ۩ OSS ͷྫ - ssmwrap github.com/handlename/ssmwrap SSM Parameter Store ͷ஋Λ؀ڥม਺ʹઃఆͯ͠ίϚϯυΛ

    exec ͢Δ wrapper ੈͷதʹྨࣅ඼͸͍ͬͺ͍͋Δ͕͋͑ͯ… github.com/remind101/ssm-env github.com/okzk/env-injector github.com/jamietsao/aws-ssm-env

33. ssmwrap $ ssmwrap -path /prod/ -- mycommand ྫ͑͹ SSM ύϥϝʔλετΞʹ

    /prod/DB_PASSWORD, /prod/API_TOKEN ͕ೖ͍ͬͯΔ৔߹ ؀ڥม਺ DB_PASSWORD, API_TOKEN ʹ஋Λઃఆͯ͠ mycommand Λ࣮ߦ 2018.07 ECS task ʹ SSM ͷ஋Λ౉ͨ͢Ίʹ։ൃ

34. ssmwrap ͷ͍͍ͱ͜Ζ exec ͢Δ(ࢠϓϩηεͱͯ͠ىಈ͠ͳ͍) wrapper ͳͷͰ ίϯςφͷ entrypoint ʹ௚઀ࢦఆͰ͖Δ #

    ssmwrap -path /prod/ -- command ͱಉ౳ͷىಈॲཧ ENTRYPOINT ["/usr/bin/ssmwrap"] ENV SSMWRAP_PATHS=/prod/ CMD ["--", "mycommand"] ϦτϥΠ͕ࢦఆͰ͖Δ (ྫ -retries=3 or SSMWRAP_RETRIES=3) ύϥϝʔλετΞ͕ΤϥʔΛฦͨ͠৔߹ɺϦτϥΠͰ͖ͳ͍ͱ

35. ͔͋Β͞·ͳ伱ؒ͸͙͢ຒ·Δ 2018-11 ECS task ʹ SSM ύϥϝʔλετΞͷ஋Λ؀ڥม਺ͱͯ͠ઃఆͰ͖Δ secrets ػೳϦϦʔε !

36. ssmwrap ͷར༻Ձ஋͸ͳ͘ͳͬͨͷ͔? ECS Ҏ֎Ͱ΋࢖͑Δ EC2 Ͱಈ࡞͢Δ shell script ͷ shebang

    ʹࢦఆͨ͠Γ #!/usr/bin/ssmwrap -path /prod/ -- /bin/sh echo $DB_PASS ... AWS֎ (ྫ͑͹ CircleCI) Ͱͷ࣮ߦʹ΋ಉ༷ʹ࢓ࠐΊΔ γεςϜ͢΂͕ͯ ECS Ͱͳ͍ঢ়ଶͰ͸·ͩ·ͩར༻Ձ஋͸͋Δ

37. ࢦఆͨ͠஋ΛϑΝΠϧͱͯ͠ॻ͖ग़͔ͯ͠Β exec ssmwrap -paths="/prod/" \ -file "Name=/prod/ID_RSA_KEY,Path=/root/.ssh/id_rsa,Mode=400" \ -file "Name=/prod/ID_RSA_PUB,Path=/root/.ssh/id_rsa.pub,Mode=644"

    \ -- mycommand 伴ͳͲɺ؀ڥม਺Ͱ͸ͳ͘ϑΝΠϧ഑ஔ͕લఏͷΞϓϦέʔγϣϯ΋ଟ͍

38. GoͷϥΠϒϥϦͱͯ͠΋࢖͑Δ ͜Ε͕໌ه͞Ε͍ͯΔྨࣅπʔϧ͸গͳ͍(͸ͣ) godoc.org/github.com/handlename/ssmwrap#Export ϥΠϒϥϦͱͯ͠࢖͑Δ → Lambda Ͱ࢖͑Δ import "github.com/handlename/ssmwrap" err

    := ssmwrap.Export(ssmwrap.ExportOptions{ Paths: []string{"/prod/"}, Retries: 3, }) // ී௨ʹ؀ڥม਺Λࢀর͢Δ͚ͩ dbpass := os.Getenv("DB_PASS")

39. Lambda Ͱ΋ύϥϝʔλετΞͷ஋Λ؀ڥม਺ʹ͍ͨ͠(Ͱ͢ΑͶ) ϚωʔδυͰઃఆ͢Δํ๏͸(·ͩ)ͳ͍͕Ͱ͖ͯવΔ΂͖ ϚωʔδυͰͰ͖ΔΑ͏ʹͳͬͨ৔߹ͷίʔυमਖ਼͕ۃগ ssmwrap.Export() Λআڈ͢Δ͚ͩ Lambda ͰύϥϝʔλετΞΛࢀর͢ΔίʔυΛॻ͘ΑΓݟ௨͕͠Α͍

40. ssmwrap ͷࢥ૝ Ϛωʔδυʹͳͬͨͱ͖ʹऔΓ֎͠΍͍͢Α͏ͳઃܭ ؀ڥม਺/ϑΝΠϧʹSSMύϥϝʔλετΞͷ஋Λઃఆ͢Δ͜ͱʹूத͢Δ ECSɿsecrets ͰαϙʔτࡁΈ Lambda: ͦͷ͏ͪͰ͖Δ(͖ͬͱ) ϑΝΠϧʹॻ͖ग़͔ͯͯ͠͠Βίϯςφىಈ →

    k8sͰ͸Ͱ͖Δɺͭ·ΓECS΋… কདྷϚωʔδυʹͳΓͦ͏ͳ෦෼ΛΞϓϦέʔγϣϯͷ΂ͨॻ͖Ͱղܾ͠ͳ͍ খ͍͞πʔϧ/ϥΠϒϥϦʹ੾Γग़͓ͯ͘͠ औΓ֎͕͠༰қʹͳΔ

41. ͦͷ͏ͪऔΓ֎͢΋ͷΛͳͥΘ͟Θ͟࡞Δͷ͔ ΞϓϦέʔγϣϯͰ௚઀ॻ͘ͱີ݁߹͢Δ औΓ֎͍ͨ࣌͠ʹີ݁߹͍ͯ͠Δͱվमͮ͠Β͍ ϥΠϒϥϦͰ͸ͳ͍ίʔυ͸ԟʑʹͯ͠ผϓϩδΣΫτʹίϐϖ͞ΕΔ ͍͟मਖ਼ΛೖΕΑ͏ͱͯ͠΋ίϐϖͳͷͰ… վमͮ͠Β͍ΞϓϦέʔγϣϯ → ͦͷ͏ͪ୭΋৮Γͨ͘ͳ͘ͳΔ վम͠΍͍͢ΞϓϦέʔγϣϯ →

    ௿ίετͰۙ୅ԽΛਐΊ΍͍͢

42. OSS ͱͯ͠࡞Δ

43. OSS ͱͯ͠࡞Δ ࣗ෼Β͔͠࢖Θͳͯ͘΋ OSS ʹͯ͠͠·͏ ઃܭࢦ਑ͱͯ͠ ͜ΕΛ OSS ʹ͢ΔͳΒͲ͏͢Δ͔ Λߟ͑Δͷ͸༗༻

    (ࣄ৘Ͱ OSS ʹͰ͖ͳͯ͘΋)

44. OSS ͱͯ͠࡞Δཧ༝(1) υΩϡϝϯτΛॻ͘ؾʹͳΔ README ͙Β͍͸ؤுͬͯॻ͘ϞνϕʔγϣϯʹͳΔ ۭͩͱஏ͔͍ͣ͠ͷͰ……

45. OSS ͱͯ͠࡞Δཧ༝(2) ա౓ͳࣾ಺ࣄ৘ͷࠞೖΛ๷͙ ʮࣾ಺/ͦͷϓϩδΣΫτҎ֎ʹҙຯ͕͋Δػೳͳͷ͔?ʯ ੹೚෼ք఺ΛݟۃΊΔ͜ͱͰকདྷ֎͠΍͘͢ͳΔ ઃܭ͕͖Ε͍ʹͳΔ

46. OSS ͱͯ͠࡞Δཧ༝(3) ຐվ଄൛͕૿৩͢ΔͷΛ๷͙ ʮGitHubͰόΠφϦఏڙͯ͠ΔͷͰͦΕΛ࢖͍ͬͯͩ͘͞ʯ ʮRubyGems / CPAN / npm /

    etc. ͔ΒΠϯετʔϧ͍ͯͩ͘͠͞ʯ ಠཱͨ͠ύοέʔδʹͳͬͯͳ͍ͱίϐϖ͞ΕΔ ίϐϖޙʹվ଄͞ΕΔͱ৽ػೳ΍όάϑΟοΫεʹ௥ैͰ͖ͳ͍

47. OSS ͱͯ͠࡞Δཧ༝(4) ಉ͡Α͏ͳ伱ؒՈ۩ΛΈΜͳ͕ຖճखॻ͖͢Δͷ͸ແବʂʂ ͱ͸͍͑ϢʔεέʔεʹΑͬͯ͸࠶࣮૷͢Δ͜ͱ΋ (ssmwrap)

48. ·ͱΊ AWSͷϚωʔδυαʔϏεʹ͸ʮ伱ؒʯ͕͋Δ 伱ؒՈ۩Λ࡞ͬͯΑΓΑ͍ӡ༻Λ ඞཁʹԠͯ͡औΓ֎ͤΔΑ͏ʹઃܭ͢Δ OSS ͱͯ͠࡞Δ͜ͱΛߟ͑Δ͜ͱͰɺΑΓΑ͍ઃܭͱ࣮૷ʹ ཁ๬͸ͪΌΜͱ఻͑·͠ΐ͏ʂ

49. Thank You!