Link
Embed
Share
Beginning
This slide
Copy link URL
Copy link URL
Copy iframe embed code
Copy iframe embed code
Copy javascript embed code
Copy javascript embed code
Share
Tweet
Share
Tweet
Slide 1
Slide 1 text
։ൃͱӡ༻ͰαʔϏεͷ৴པੑΛߴΊΔ ʮSREʯͷ࣮ફ Enterprise Development Conference 2017/09/01 גࣜձࣾϝϧΧϦ Masahiro Nagano @kazeburo
Slide 2
Slide 2 text
ࣗݾհ • Masahiro Nagano / խ • @kazeburo (twitter/github) • גࣜձࣾϝϧΧϦ ϓϦϯγύϧΤϯδχΞ Site Reliability Engineering (SRE) νʔϜ • BASE, Inc ٕज़ΞυόΠβʔ
Slide 3
Slide 3 text
ࣗݾհ(ܦྺɾ׆ಈ) • ܦྺ • 2006 mixi - ΞϓϦӡ༻νʔϜ • 2010 livedoor (LINE) - ։ൃࢧԉνʔϜ • 2015 ݱ৬ - SRE • 15Ҏ্ WebαʔϏεΛΠϯϑϥ͔Βࢧ͑Δۀ • ొஃʗࣥච • AWS Dev Day Tokyo 2017 ొஃ • WEB+DB PRESS Vol. 100 هࣄࣥච
Slide 4
Slide 4 text
ΞδΣϯμ • ࣗݾհ • ϝϧΧϦʹ͍ͭͯ • SREͱ • ϝϧΧϦͷSREͷࣄྫհ
Slide 5
Slide 5 text
ϝϧΧϦʹ͍ͭͯ
Slide 6
Slide 6 text
ϝϧΧϦ • ࠃ࠷େڃͷϑϦϚΞϓϦ • 3Ͱ؆୯ʹग़ 1) ࣸਅΛࡱΔ 2) ใΛهೖ 3) ग़ϘλϯΛԡ͢ • ҆৺҆શͳܾࡁɾऔҾ • ΤεΫϩʔ • ͓ۚͷΓͱΓ͕ࣾؒʹհࡏ • ಗ໊ૹ
Slide 7
Slide 7 text
ถࠃ/ӳࠃ ͷల։ JP US UK
Slide 8
Slide 8 text
KPI μϯϩʔυ GMV(૯औҾֹ) 7500ສDL(JP+US) ݄ؒ100ԯԁҎ্ ग़ 1100ສҎ্ (ϐʔΫ࣌ؒଳ1ؒʹ1000ग़Ҏ্)
Slide 9
Slide 9 text
ϝϧΧϦγεςϜ֓ཁ ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. Client Multimedia Corporate data center Traditional server Mobile Client IAM Add-on Example: IAM Add-on ence ) Assignment/ Task Requester Workers ग़! DB Search 5-දࣔ ݕࡧө ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporate data center Traditional server Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Amazon Mechanical Turk On-Demand Workforce Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific େྔͷϦΫΤετ ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corp data c Mobile Client Internet AWS Management Console IAM Add-on Example: IAM Add-on Human Intelligence Tasks (HIT) Assignment/ Task Requester Workers Amazon Mechanical Turk Non-Service Specific ϦΫΤετԠ DB Search ߪೖ! ඵʙ30ඵ ඵʙ ߴʹฒߦͯ͠େྔͷτϥϯβΫγϣϯΛѻ͏ ը૾ ܾࡁ AI ϑΟʔυόοΫ
Slide 10
Slide 10 text
ΠϯϑϥετϥΫνϟ JP US UK DNS: Amazon Route53 CDN: Akamai, Fastly, ImageFlux Storage: Amazon S3 Analysis: Google BigQuery ܾࡁ/ྲྀαʔϏε ܾࡁ/ྲྀαʔϏε ܾࡁ/ྲྀαʔϏε
Slide 11
Slide 11 text
SREͱ
Slide 12
Slide 12 text
SREͱ • Site Reliability Engineering/Engineer ͷུ • Reliability = ৴པੑ • γεςϜཧͱαʔϏεӡ༻ͷํ๏ͱͯ͠Googleͷӡ༻νʔϜΛ͍͍ͯͨ Ben Treynor͕ఏএ • USΛத৺ʹେنͳITΠϯϑϥΛӡ༻͢Δ֤ࣾʹ·Δ • ໌֬ͳఆٛͳ͍͕ʮιϑτΣΞΤϯδχΞϦϯάʹΑͬͯɺΠϯϑϥετϥΫ νϟɾαʔϏεશମͷՄ༻ੑɺੑೳɺηΩϡϦςΟΛվળ͢ΔʯΤϯδχΞ/νʔϜ
Slide 13
Slide 13 text
Google SRE • ιϑτΣΞΤϯδχΞϦϯάʹՃ͑ɺγεςϜɾӡ༻ͷೳྗ͕ٻΊΒΕ Δɻ·ͨ৫ɾνʔϜͷ͋ΓํؚΜͰ͍Δ • ιϑτΣΞΤϯδχΞϦϯάʮࣗಈԽʯʹϑΥʔΧε • SREͷਓαʔϏεͷنʹൺྫͤ͞ͳ͍(ݱ࣮తʹͰ͖ͳ͍) • ʮτΠϧʯͷ໓ • ख࡞ۀͰߦΘΕɺࣗಈԽՄೳͰ܁Γฦ͢͜ͱʹՁΛ࣋ͨͳ͍
Slide 14
Slide 14 text
Google SRE • ۀ࣌ؒͷ50%ιϑτΣΞΤϯδχΞϦϯάΛߦ͏ • ࣗಈԽ(ࣗԽ)ɺ৴པੑ্ʹ͋ͯΔ • 50%Λ͑Δ͜ͱ͕͋Εۀͷେ෯ͳݟ͠ΛഭΒΕΔ • SLAɺΤϥʔόδΣοτ(༧ࢉ)ʹΑΔ։ൃऀͷརௐ • ։ൃऀνʔϜͱՄ༻ੑͷඪΛαʔϏε͝ͱʹઃఆ • ΤϥʔόδΣοτʹ͋Δͱ͖։ൃऀੵۃతͳϦϦʔεΛߦ͍ɺ༧ࢉΛ ͑Δ߹৴པੑճ෮ͷͨΊͷ։ൃʹઐ೦͢Δ͜ͱ͕ٻΊΒΕΔ
Slide 15
Slide 15 text
ຊࠃͰͷSRE • 201511݄ ϝϧΧϦٕज़blogͰSREΛհ • RettyɺαΠϘζɺCookPadɺMixiɺͯͳͳͲWebܥاۀΛத৺ʹSREͷ࠾༻͕ਐΜͰ͍Δ • ΠϯϑϥνʔϜ/ΠϯϑϥΤϯδχΞ͔ΒͷҠߦ͕ଟ͍ • ΫϥυԽΛഎܠʹΠϯϑϥΤϯδχΞͱ͍͏ݺͼํ͕దͰͳ͘ͳ͖ͬͯͨ • SRE Tech Talk։࠵ • ୈҰճ: 20166݄ɻୈೋճ: 20171݄ • 100໊Ҏ্ͷࢀՃऀ
Slide 16
Slide 16 text
SREͷظͷߴ·Γ • ॻ੶/ࡶࢽ • ΦϥΠϦʔʮSRE αΠτϦϥΠΞϏϦςΟΤϯδχΞϦϯάʯ • ܦBPʮܦSYSTEM 2017/7ʯ • Πϯλʔωοτ্ͷಛूهࣄ • ITPro - άʔάϧൃͷ৽ख๏ʮSREʯɺຊͰ֦େ • http://itpro.nikkeibp.co.jp/atcl/column/14/346926/030600869/ • @IT - ಛूɿγεʹٻΊΒΕΔʮSREʯͱ͍͏৽ͨͳׂ • http://www.atmarkit.co.jp/ait/series/4503/
Slide 17
Slide 17 text
ϝϧΧϦ SRE
Slide 18
Slide 18 text
ϝϧΧϦ SRE • ͍ͭͰշద͔ͭ҆શʹར༻Ͱ͖Δʮ৴པੑͷߴ͍ʯαʔϏεͷ࣮ݱ • ʮ৽نαʔϏεͷ։ൃҎ֎ͷιϑτΣΞΤϯδχΞϦϯάશ෦Δʯ • ݱࡏϝϯόʔʮ10ਓʯ • શϝϯόʔ౦ژۈɻग़ுͰถࠃʹ͍͘͜ͱ • େنͳWebαʔϏεͰͷܦݧ͕͋Δத్͕ଟ͍͕ɺ৽ଔϝϯόʔࡏ੶ • ࣾπʔϧͷӡ༻վળɺAIܥͷۀʹܞΘΔϝϯόʔ͍Δ
Slide 19
Slide 19 text
ϝϧΧϦͰSREΛ࠾༻ͨ͠ཧ༝ • ϝϧΧϦΛ͓٬͞·ʹͬͯ͘Β͏ʹʮ͍ͭͰշదʹ҆શʹ ͑Δʯ৴པੑ͕ॏཁ • ΠϯϑϥͰͳ͘ɺιϑτΣΞؚΊ৴པੑʹΛ࣋ͭ͜ͱΛ໌Β ͔ʹ͢Δ • ւ֎Ͱ௨͡Δ໊শ • JP/US/UKͰͷల։ɻάϩʔόϧͰͷ࠾༻ৗʹҙࣝ • ઌਐతͳऔΓΈͱͯ͠ͷૂ͍
Slide 20
Slide 20 text
Mercari SRE ͷۀൣғ Operations Software Eng. ج൫ߏங OnCall (োରԠ) Automation εέʔϥϏϦςΟɾՄ༻ੑվળ DBAɺϛυϧΣΞߏங ΞϓϦέʔγϣϯͷઃܭϨϏϡʔ ϩάऩूɾੳج൫ͷߏஙɺӡ༻ αʔόϓϩϏδϣχϯάɾσϓϩΠɺϚΠΫϩαʔϏεج൫ͷඋ ηΩϡϦςΟʗෆਖ਼ར༻ݕग़ γεςϜӡ༻ΛʮΈʯͱͯ͠ ࡞Γ্͛Δ͜ͱ͕ٻΊΒΕ͍ͯΔ
Slide 21
Slide 21 text
ϝϧΧϦ SREͷ࣮ફࣄྫ
Slide 22
Slide 22 text
ϝϧΧϦSRE࣮ફࣄྫ • OnCall/൪ରԠ • ηΩϡϦςΟͷऔΓΈ • ύεϫʔυϦετ߈ܸͷࣄྫ
Slide 23
Slide 23 text
OnCall/൪ରԠ
Slide 24
Slide 24 text
SRE൪/OnCall • ΞϥʔτରԠ • ΞϥʔτΛड͚औΔεϚϗΛৗʹOnʹ͠ɺҰ࣍ରԠΛߦ͏ • ి൪ • SlackΛ௨ͨ͡ۓٸిͷड৴ • ʮোͷݕ͔ΒରԠ։࢝·Ͱͷ࣌ؒͷॖʯʮଞϦʔδϣϯ ͔ΒͷۓٸґཔͷରԠʯ͕త
Slide 25
Slide 25 text
SRE൪/OnCall • ༵0͔࣌Β༵24࣌·Ͱ1िؒͰަ • ฏνʔϜϝϯόʔ͕ग़ࣾ͢Δ·Ͱࣗػ • 9͔࣌Βࣗػ͠ɺUS͔ΒͷґཔͳͲʹରԠ • ٳ15-20ҎʹରԠ։࢝Ͱ͖Δ͜ͱ͕·͘͠ɺߦಈʹ੍ ݶ͋Δ • ਂٳͷରԠͳͲɺՈͷڠྗඞཁ
Slide 26
Slide 26 text
൪/OnCall Λࢧ͑Δٕज़ • ࢹ • Mackerel • ௨/Scheduling • Slack, PagerDuty • BotʹΑΔిݺͼग़͠
Slide 27
Slide 27 text
mackerel: ΫϥυܕࢹαʔϏε • גࣜձࣾͯͳ ఏڙͷࢹαʔϏε • ͯͳࣾͰͷαʔόӡ༻ϊϋ • ֤छAPI͕༻ҙ͞ΕDevOpsͱͷ૬ੑྑ͍ • PluginͰࢹ߲ͷ֦ு͕Մೳ • 40ݸҎ্ͷSREνʔϜ։ൃͷPluginΛར༻ • αʔόͷঢ়ଶ͚ͩͰͳ͘ɺ֎ܗࢹɺαʔϏεʹؔΘΔͷՄࢹԽɺΞϥʔτઃఆՄೳ • Ξϥʔτͷ௨SlackΛ࢝Ί֤छαʔϏε࿈ܞ͕༻ҙ
Slide 28
Slide 28 text
slackͱΞϥʔτͷ • SlackϏδωε͚νϟοταʔϏε • OSSձࣾؒͷίϛϡχέʔγϣϯʹΘΕΔ • Chat botΛར༻ͨ͠ChatOpsͳͲߦ͏ • Ξϥʔτͷ௨ઌͱͯ͠దͰͳ͍߹ • ௨ԻͷΧελϚΠζͰ͖ͳ͍ • 1ճ͔͠௨Ͱ͖ͳ͍ɻྲྀΕ͕ͪ • ετϨε⤴ / ӡ༻ʹͳΓ͕ͪ
Slide 29
Slide 29 text
PagerDuty: Πϯγσϯτཧ • ΠϯγσϯτཧͷΫϥυαʔϏε • ༷ʑͳखஈʹΑΔ௨ • ௨ͷϩʔςʔγϣϯཧ • ΤεΧϨʔγϣϯϧʔϧͷઃఆ
Slide 30
Slide 30 text
PagerDutyʹΑΔ௨ • ༷ʑͳखஈͰ௨Λߦ͏͜ͱ͕Ͱ͖Δɻ Ԡ͢Δ·Ͱଓ͘ • mail • SMS • App (iOS, Android) • ి • ʮ10Λ͑ͨͱ͜ΖͰҰిΛೖ ΕΔʯϧʔϧͰӡ༻
Slide 31
Slide 31 text
BotʹΑΔۓٸి • SlackͰbotʹର໋ͯ͠ྩ͢Δ͜ͱͰɺ൪ʹి͕ ͔͔ΔΈ • SRE͕։ൃɾӡ༻ • PagerDutyͷAPIΛར༻͠ɺ൪ͷ࿈བྷઌΛऔಘ • TwilioΛͬͯࣗಈͰి • US/UKؚΊɺؾܰʹͬͯΒ͑ΔΑ͏ࣾࠂ ࠷ۙͰSREνʔϜͰΤεΧϨʔγϣϯతͰ ར༻ ☎
Slide 32
Slide 32 text
༷ʑͳ ChatOps • ൪ͷిͷଞʹbotΛར༻ͨ͠ӡ༻Λߦͳͬͯ ͍·͢ • ൪ใ • CDNͷΩϟογϡΫϦΞ • ຊ൪ڥͰͷSQLͷ࣮ߦܭը֬ೝ • σϓϩΠ
Slide 33
Slide 33 text
ηΩϡϦςΟͷऔΓΈ ύεϫʔυϦετ߈ܸͷࣄྫ
Slide 34
Slide 34 text
ύεϫʔυϦετ߈ܸ • ͓٬͞·ͷΞΧϯτʹରͯ͠ɺϥϯμϜͳύεϫʔυ͘͠ผͰ࿙Ӯ ͨ͠ύεϫʔυจࣈྻΛ͍ϩάΠϯΛࢼߦɺෆਖ਼ϩάΠϯΛߦ͏ • ͞·͟·ͳنͷ߈ܸ͕ߦΘΕɺଞͷαʔϏεͰͷࣄྫଟ͍
Slide 35
Slide 35 text
ύεϫʔυϦετ߈ܸͷରԠ • Ϧετ߈ܸͷݕ • ߈ܸ͕ߦΘΕ͍ͯΔ͜ͱΛͰ͖ΔݶΓૣ͘ݕ͠ɺΞϥʔτΛ͋͛Δ • ߈ܸ؇ࡦͷ࣮ࢪ • ΞΫηεύλʔϯ͔ΒࣗಈͰޚ • ߈ܸΛड͚ͨΞΧϯτͷύεϫʔυϦηοτ
Slide 36
Slide 36 text
ύεϫʔυϦετ߈ܸͷݕ • ϩάΠϯࣦഊΛAPIͷϩάͱͯ͠ه • ϩάΛϦΞϧλΠϜͰूܭɺ mackerelͰՄࢹԽͱࢹΛߦ͏ • ϝϧΧϦ͕TVʹऔΓ্͛ΒΕΔͱ Ξϥʔτ͕དྷΔ͜ͱ
Slide 37
Slide 37 text
ύεϫʔυϦετ߈ܸͷ؇ • ൺֱత୯७ͳ߈ܸΞϓϦέʔγϣϯͰࣗಈతʹޚ • ಉҰͷϝʔϧΞυϨεʹΑΔෳճͷϩάΠϯࢼߦ • ಉҰͷIPΞυϨεʹΑΔෳΞΧϯτͷϩάΠϯࢼߦ • ߈ܸͱஅ͞Εͨ߹֘IPΛҰఆظؒڋ൱ • ϩάΠϯ͞ΕͨͱࢥΘ͖͠ɺ͓٬͞·ʹύεϫʔυͷϦηοτଅ͢
Slide 38
Slide 38 text
େنͳύεϫʔυϦετ߈ܸࣄྫ • ւ֎͚αʔϏεࠃ͚αʔϏεΑΓ߈ܸΛड͚Δճ͕ଟ͍ • ߈ܸͷن͕େ͖͘ͳΓɺखޱෳࡶʹͳΔ
Slide 39
Slide 39 text
େنύεϫʔυϦετ߈ܸࣄྫ • 2016ʹ࣮ࡍʹى͖ͨ߈ܸͷΞΫ ηεݩͷࠃ • ࣍ʑʹIPΛมߋ͠ɺͦΕͧΕͷIPͰ ճ͔͠ϩάΠϯࢼߦͤͣɺࣗಈ Ͱ͙͜ͱ͕͍͠ ͦͷଞ 18% Armenia 2% Azerbaijan 2% Bahrain 2% Georgia 2% Japan 2% Russian 2% Indonesia 3% Nepal 3% Pakistan 5% Thailand 5% Taiwan 6% Viet Nam 6% Brazil 10% India 30%
Slide 40
Slide 40 text
େنύεϫʔυϦετ߈ܸͷඋ͑ • ߈ܸ͞Ε͍͢Web൛ͰreCAPTCHAΛಋೖ • Client Reputation ͷར༻ • GeoIP: IPΞυϨε͔Βࠃఆ • ಗ໊ϓϩΩγͷར༻ͷ༗ແ • IP Reputation
Slide 41
Slide 41 text
Client reputation/GeoIP • ֎෦ͷIPσʔλϕʔεΛར༻ • ࠃ/भ/ݝ/ࢢϨϕϧ·Ͱਪఆ • σʔληϯλʔ͕ར༻͍ͯ͠ΔIP • ಗ໊ԽϓϩΩγͷఆ • ࠃෆಠࣗʹऩू https://www.ip2location.com/demo
Slide 42
Slide 42 text
Client reputation/IP reputation • ओʹSPAMϑΟϧλͳͲʹΘΕΔٕज़ • աڈʹSPAMͷૹ৴Λ͓͜ͳͬͨIPɺBotnetͱ ͯ͠ΘΕͨIPΛݕग़ • ௐࠪͷ݁ՌɺύεϫʔυϦετ߈ܸʹΘΕ ͍ͯΔ͜ͱ͕໌ • ϦεΫͷߴ͍IP͔ΒͷΞΫηεڋ൱͢Δ http://www.cyren.com/security-center/ip-reputation-check
Slide 43
Slide 43 text
߈ܸݩIP Reputation • 2016ͷ߈ܸݩௐࠪͷ݁Ռ • 4ׂ͙͜ͱ͕Ͱ͖ͨՄೳੑ͕͋Δ* • ݱࡏͰɺ༷ʑͳใϦιʔεΛΈ߹Θͤͯ αʔϏεͷ҆શੑΛߴΊ͍ͯ·͢ -PX .JEEMF )JHI * ߈ܸͷ͋ͬͨλΠϛϯάͷreputationͰͳ͍ͷͰଟগζϨ͕͋Γ·͢
Slide 44
Slide 44 text
·ͱΊ • SREͱ • Googleʹ͓͚ΔγεςϜӡ༻ͷ࣮ફ • ιϑτΣΞΤϯδχΞϦϯάʹΑͬͯ৴པੑΛ্ͤ͞Δ • ϝϧΧϦͷSREͷࣄྫհ • OnCall/ӡ༻൪ • ύεϫʔυϦετ߈ܸ͔Βͷޚࣄྫ
Slide 45
Slide 45 text
͝੩ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ ຊεϥΠυspeakerdeckʹͯެ։͠·͢ https://speakerdeck.com/kazeburo/