Go Conference 2017 Springでの講演スライドです。
࣮ફ!Go/GAE+DDDͰͷΫϩʔϥʔߏங@__timakin__GoConference 2017 Spring
View Slide
ࣗݾհ
ࣗݾհ• twitter: @__timakin__• github: timakin• גࣜձࣾGunosy ৽نࣄۀ։ൃࣨ ← New!• ओͳGoϥΠϒϥϦ։ൃ• gopli (DBϨϓϦέʔγϣϯπʔϧ)• gonvert (จࣈίʔυมϥΠϒϥϦ)• octop (githubͷissue, PRϏϡʔ༻CLIπʔϧ)• ts (ٕज़ɾϏδωεܥχϡʔε८ճCLIπʔϧ)
Copyright© Gunosy Inc. All Rights Reserved 4Go / Python ΤϯδχΞืूத▶https://gunosy.co.jp/recruit/Gunosyɺ౦ژେֶʹ௨͏3ਓͷֶੜͷ ʮใΛੈքதͷਓʹ࠷దʹಧ͚͍ͨʯͱ͍͏͍͔Β࢝·Γ·ͨ͠ɻ౦ূϚβʔζ্ɺຊώϧζͷΦϑΟεҠసΛܦͯɺ େ͖͍ͯ͘͠ΔձࣾͰ׆༂͍ͨ͠ϝϯόʔΛืू͍ͯ͠·͢ɻ
ΞδΣϯμ
ΞδΣϯμ• Go/GAE + DDDͰ࡞ΔΫϩʔϥʔ• APIͷ֓ཁ• σΟϨΫτϦߏ• ڞ௨ॲཧɺݸผͷυϝΠϯͷৄࡉ• Go/GAEͰ٧·ͬͨϙΠϯτ• ·ͱΊ
ࠓճͷ։ൃϓϩηε
APIͷ֓ཁ
APIͷ֓ཁ• ओͳ༻ٕज़• GAE SDK: go version go1.6.3 (appengine-1.9.48) darwin/amd64• Web Framework: echo v3.0.3• Vendoring: dep• ػೳཁ݅• Facebook, TwitterͷAPI & Σϒϖʔδ͔ΒίϯςϯπΛऔಘ• ͋Β͔͡ΊऔಘઌͷީิΛDBʹొ͓͖ͯ͠ɺcronδϣϒͰదٓΫϩʔϧ• λΠτϧɺbodyɺαϜωΠϧͷڞ௨ϓϩύςΟʹ֨ೲ• ҎલΫϩʔϧͨ͠λΠϛϯά͔Βߋ৽͕ͳ͔ͬͨΒಡΈࠐ·ͳ͍• Ϋϩʔϧ݁ՌJSONΦϒδΣΫτʹ·ͱΊͯ࠷ޙʹS3ʹΞοϓϩʔυ͢Δ• ը૾ͷՃͱ͔ͳ͠ɻ͍͔ͭإೝࣝͱ͔γϡοͱΓ͍ͨɻ
ίϯςΩετϚοϓ(DDD)ఆظऩूΫϩʔϧδϣϒͷൃՐίϯςϯπऔಘऔಘઌͷཧίϯςϯπऔಘʢੜͷϨεϙϯεʣ༰ͷՃऔಘͨ͠ใΛՃͯ͠+40/༻ͷύϥϝʔλʔʹม͢ΔอଘετϨʔδ 4ʹΞοϓϩʔυA BcallsWorkerFetcherParserUploader
σΟϨΫτϦߏ
σΟϨΫτϦߏGAEͷίϯςΩετੜϧʔςΟϯάGAEͷδϣϒఆٛίΞػೳͷ࣮෦ґଘύοέʔδGAEϏϧυͷ߹্ىಈεΫϦϓτͱ࣮ɺvendorσΟϨΫτϦผʹ͚Δ(package໊ͷিಥɺunsafeͷඇਪػೳͱ͔ͰΞϥʔτ͕ग़Δ)
σΟϨΫτϦߏΞϓϦͷڞ௨ઃఆΛѻ͏Օॴʢޙड़ʣ֤υϝΠϯͷ࣮DBΞΫηεΛ୲͏ϦϙδτϦ܈Contextઃఆ
DDDߏʹͯ͠Α͔ͬͨͱ͜Ζ• ࣮ݱ͍ͨ͠ϏδωεϩδοΫ୯ҐͰ package໊͕౷Ұ͞ΕΔͷͰɺ package໊ͷিಥ͕ى͜Γʹ͘͘ͳͬͨɻ• Ϧιʔεͱ͍͏ʮϞϊʯͰͳ͘ɺ υϝΠϯͱ͍͏ʮߦҝʯʹͯ͠ ίʔυΛॻ͘͜ͱͰɺʮ͍ͭ͜ԿΛͯ͘͠ΕΔͭͳͷʁʯ ͱ͍͏ٙͷղ͕͙͢ʹಘΒΕͯɺ୯७ʹಡΈਐΊ͘͢ͳͬͨɻ• repositoryΛΓ͢ͱɺ࣮ͱσʔλΞΫηεͷ͕ؔ ૄʹͳΓɺϝϯςφϯε͔ͬͨ͢͠ɻ (CloudSQL͔ΒDataStoreͷҠߦͱ͔͔ͬͨ͢͠)
ڞ௨ॲཧݸผͷυϝΠϯͷৄࡉ
࣮ɿڞ௨ॲཧGAEϏϧυʹඞཁͳechoͷίʔυҎԼΛࢀߟʹ͍ͯͩ͘͠͞ɻhttps://echo.labstack.com/cookbook/google-app-engine
࣮ɿڞ௨ॲཧGAEϏϧυʹඞཁͳechoͷίʔυҎԼΛࢀߟʹ͍ͯͩ͘͠͞ɻhttps://echo.labstack.com/cookbook/google-app-engineʁ
࣮ɿڞ௨ॲཧGAEͷίϯςΩετΛɺɾtimeout limitΛઃఆɾίϯςΩετʹrepositoryͷΠϯελϯεΛ࣋ͨͤΔͱ͍͏͜ͱΛ্ͨ͠ͰɺechoͷΧελϜίϯςΩετͱͯ͠ઃఆ͢Δ
࣮ɿڞ௨ॲཧecho.Contextɺඪ४ύοέʔδͷcontext.Context͡Όͳ͍…echo.ContextΛར༻ͨ͠GrawlerCtxͱ͍͏ΧελϜίϯςΩετΛ࡞ͯ͠ɺContextͷ1มͱͯ͠AppEngineCtxΛ࣋ͭɻ
FetcherɿίϯςϯπऔಘઌొϦΫΤετύϥϝʔλʔΛͱʹɺ”Entity”Λ࡞ͬͯɺίϯςΩετ෦ͷrepository͔ΒɺίϯςϯπऔಘઌΛొ͢Δ
ิɿrepositoryͷར༻ํ๏ΧελϜίϯςΫετͷத͔ΒAppEngineͷContextΛऔಘ͠ɺͦͷContext͕࣋ͭFetcher(·ͨଞͷυϝΠϯͷ)DBͷΞΫηαϝιουΛݺͼग़͢
ิɿrepositoryͷ෦Entity(ࣝผࢠΛ࣋ͬͨσʔλߏ)Λࡐྉʹͯ͠σʔλΞΫηε͢ΔɻݸผͷϦϙδτϦΛೖΕࢠͰࢀরͯ͠ɺ୯ҰίϯςΩετ͔ΒશϦϙδτϦʹΞΫηεͰ͖ΔΑ͏ʹ͢Δɻ
ิɿrepositoryͷ෦Entity(ࣝผࢠΛ࣋ͬͨσʔλߏ)Λࡐྉʹͯ͠σʔλΞΫηε͢Δɻ
FetcherɿΠϯλʔϑΣʔε
FetcherɿΠϯλʔϑΣʔεϨεϙϯεΛͦͷ··interface{}ͱͯ͠ฦ͢֎෦ͷϦΫΤετappengine/urlfetchΛ͏
ParserɿΠϯλʔϑΣʔεHTMLύʔαʔͱ͔ɺαΠτʹΑͬͯେ෯ʹparseͷํ͕ࣜҟͳΔɻͦͷͨΊɺParserFactory.CreateΛܦ༝ͯ͠ɺParseServiceΛ࡞͢ΔfetcherInstanceϖʔδϯά࣌ͷϦΫΤετ༻
ParserɿHTMLύʔαͷఆٛFactoryͰͲͷύʔαʔΛ࡞͢Δ͔ಛఆ͢Δͱ͖ʹ͏ʢεϥΠυͰ߹্໊લΛมߋͯ͠·͢ʣappengineͷόʔδϣϯͷ߹্ɺcontext.Contextx/netͷͷΛ͏ɻ
ParserɿParserFactoryࢦఆ͞ΕͨparserKeyʹԠͯ͡ParseServiceΛ࡞ɻFactoryTypeߏମΛ࡞Δ্Ͱக͠ํͳ͘༻ҙ͍ͯͯ͠ɺ”html”ͱ͍͏จࣈྻΛೖΕͯΔ͚ͩɻ
ParserɿParse࣮ՕॴίϯςΫετΛड͚औͬͯFetch͠ɺͦͷ݁ՌΛparse͢Δɻ) ຊParserʹFetcherΛ͢ͷݏͩͬͨͷͰ͕͢ɺɹ ϖʔδϯάͰͲ͏ͯ͠ඞཁͩͬͨͷͰɺFetcherΛɹ ࣋ͨͤΔ͜ͱʹ͠·ͨ͠ɻͭΒ͍ʂ FacebookParserͱ͔ϖʔδϯά͍Βͳ͍ͷͰɺશʹFetcherͱParser͕͍ͯ͠·͢ɻ
ParserɿParse࣮Օॴyhat/scrapeΛͬͯཁૉΛऔΓग़͢ɻgoqueryΈ͍ͨʹίʔϧόοΫͷॻ͖ํ͠ͳ͍͠γϯϓϧͰ͢ɻྫ) aλά͕ΘΕͯͯɺ͕h1λάɺͦͯ֘͠NodeͷΫϥε͕“skin-entryTitle”Ͱ͋Δͷɺ”title”ͱͯ͠ύʔε͢Δ
ParserɿParse࣮ՕॴϒϩάهࣄҰཡ͔Βɺهࣄৄࡉϖʔδͷ༰Λ࠶FetchͦͷதͷTitleཁૉͱ͔Λऔಘ͢Δॲཧ
UploaderɿΠϯλʔϑΣʔεJSONΞοϓϩʔυ༻ͷڞ௨σʔλߏʹ parse݁ՌΛ٧Ίͨͷʢ[]feeditem.FeedItemʣΛͯ͠ɺಛఆετϨʔδʹΞοϓϩʔυ͢Δ
UploaderɿॳظԽॲཧ(ྫ: S3)S3ΞοϓϩʔμͷॳظԽॲཧɻγʔΫϨοτͳใΛͯ͠ɺS3ΞΫηε༻ͷΦϒδΣΫτΛ࡞
Uploaderɿ࣮Օॴ͜͜Ͱappengine/urlfetchͰੜͨ͠HTTPClientΛઃఆ͠ͳ͍ͱɺϦΫΤετ͕௨Βͳ͍ॗʑͱbodyΒcredentialsΛઃఆͯ͠ɺS3ClientΛ࡞͜ΕҎ߱requestΛsend()͢Δ͚ͩͳͷͰলུ
WorkerɿϋϯυϥGAEͷcronͰୟ͘ΤϯυϙΠϯτͷϋϯυϥΛ༻ҙ͢ΔworkerΛ࡞ͬͯɺCrawlϝιουΛݺͿ͚ͩɻ
Workerɿ࣮Օॴfetch -> parse·ͰɻΈΜͳେ͖goroutineɻsync/errgroupΛ͑ɺΤϥʔϋϯυϦϯά͓ͯ͘͢͢͢͠ΊͰ͢ɻ
Workerɿ࣮Օॴcontext.ContextΛड͚औͬͯΔͷͰɺδϣϒ࣮ߦதʹλΠϜΞτͨ͠ΒΤϥʔΛฦ͢ɻΞοϓϩʔυॲཧΞοϓϩʔυ͕ ྃͨ͠Βɺ࠷ऴऩू࣌ࠁΛߋ৽
croncron.yamlͱ͍͏ͷΛ༻ҙ͢ΔͱɺಛఆͷΤϯυϙΠϯτʹܾ·ͬͨසͰGETϦΫΤετΛૹͬͯ͘ΕΔͷͰɺ͜ΕΛͬͯworkerͷΤϯυϙΠϯτΛୟ͘
Go/GAEͰ٧·ͬͨϙΠϯτ
σΟϨΫτϦߏͷݟ͠• ͍ܰؾ࣋ͪͰGAEࢼͯ͠ΈΑ͏ͱɺ ॳظσϓϩΠΛ͠ͳ͍··ਐΊͯͨΒɺ ͋ͱ͋ͱมߋ͕͍ͬͺ͍ೖΔɻ• ಛʹvendoringπʔϧΛ͏ͱɺ package໊িಥͱ͍͏ΫϦςΟΧϧͳॴͰ Ϗϧυ͕௨Βͳ͍ͷͰɺGAE্ʹσϓϩΠ͢ΔͳΒ ࠷ॳ͔Β༷ࣜʹ߹ΘͤΔɻ
ൿີใͷཧ• access tokenͱ͔ΛͲ͜ʹஔ͔͘ɻ• ࠷ॳtomlͰཧͯͨ͠Μ͚ͩͲɺgoapp deployͨ͠Β srcҎԼͷtoml͕ফ໓͢ΔɻࠔΔɻ• app.yamlͷenvʹઃఆ͢Δͱ͍͏ํ๏͋Δ͕ɺ gitignoreͰ͖ͳ͍ͷͰौ͍ɻ• ConfigurationRepositoryΛ࡞ͬͯɺDataStore্ʹ อଘ͢Δͱ͍͏ํΛͱͬͨɻɹ
ൿີใͷཧ
CloudSQL or DataStore• ࠷ॳCloudSQL(gormܦ༝)Λར༻͍͕ͯͨ͠ɺ ͳʹΒ͕͔͔͍ͬͯΔɻ ͜ͷAPIࣗମͷϦΫΤετଟ͘ͳ͍ͣͳͷͰɺ খنͰ͔͔ۚΔͷौ͍ɻ• ConfigurationRepositoryΛ࡞ΔλΠϛϯάͰɺ શ෦ετϨʔδΛDataStoreҠߦͨ͠ɻ• RepositoryΛ͍ͯ͠ΕҠߦָͩ͠ɺ ίϯιʔϧ͔ΒΤϯςΟςΟใݟΕΔͷͰ DataStoreͷํ͕Αͦ͞͏ɻ
beforeafter
·ͱΊ• ΫϩʔϥʔΛ࡞Γ·ͨ͠ɻ• DDDݟ௨͕͠Α͘ɺpackage໊ͷিಥආ͚ΒΕͯΑ͍ɻ• ಛʹɺͪΌΜͱinterfaceΛఆٛ͢Εɺ υϝΠϯ͝ͱͷίʔυͷՄಡੑ͕ඈ༂తʹ্͕ΔͷͰɺ Goͱ૬ੑ͕͍͍ͱࢥ͏ɻ• GAEσΟϨΫτϦߏͷ໘ͰΫη͋Γ·͕͢ɺ cronΒDataStoreΒͰԸܙ͕͋ΔͷͰɺҰ୴ڥΛ ߏங͢Δͱେมศརɻඪ४ύοέʔδͷcontext͕ ར༻Ͱ͖ΔΑ͏ʹͳΔͱ͍͍ͳɻ
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂtwitter: @__timakin__github: timakin