実践!Go/GAE+DDDでのクローラー構築

Transcript

1. ࣮ફ!Go/GAE+DDD ͰͷΫϩʔϥʔߏங @__timakin__ GoConference 2017 Spring

2. ࣗݾ঺հ

3. ࣗݾ঺հ • twitter: @__timakin__ • github: timakin • גࣜձࣾGunosy ৽نࣄۀ։ൃࣨ

   ← New! • ओͳGoϥΠϒϥϦ։ൃ෺ • gopli (DBϨϓϦέʔγϣϯπʔϧ) • gonvert (จࣈίʔυม׵ϥΠϒϥϦ) • octop (githubͷissue, PRϏϡʔ༻CLIπʔϧ) • ts (ٕज़ɾϏδωεܥχϡʔε८ճCLIπʔϧ)

4. Copyright© Gunosy Inc. All Rights Reserved 4 Go / Python

   ΤϯδχΞืूத ▶https://gunosy.co.jp/recruit/ Gunosy͸ɺ౦ژେֶʹ௨͏3ਓͷֶੜͷ
 ʮ৘ใΛੈքதͷਓʹ࠷దʹಧ͚͍ͨʯͱ͍͏૝͍͔Β࢝·Γ·ͨ͠ɻ ౦ূϚβʔζ্৔ɺ࿡ຊ໦ώϧζ΁ͷΦϑΟεҠసΛܦͯɺ
 େ͖͘੒௕͍ͯ͠ΔձࣾͰ׆༂͍ͨ͠ϝϯόʔΛืू͍ͯ͠·͢ɻ

5. ΞδΣϯμ

6. ΞδΣϯμ • Go/GAE + DDDͰ࡞ΔΫϩʔϥʔ • APIͷ֓ཁ • σΟϨΫτϦߏ੒ •

   ڞ௨ॲཧɺݸผͷυϝΠϯͷৄࡉ • Go/GAEͰ٧·ͬͨϙΠϯτ • ·ͱΊ

7. ࠓճͷ։ൃϓϩηε

8. APIͷ֓ཁ

9. APIͷ֓ཁ • ओͳ࢖༻ٕज़ • GAE SDK: go version go1.6.3 (appengine-1.9.48)

   darwin/amd64 • Web Framework: echo v3.0.3 • Vendoring: dep • ػೳཁ݅ • Facebook, Twitter౳ͷAPI & ΢Σϒϖʔδ͔ΒίϯςϯπΛऔಘ • ͋Β͔͡ΊऔಘઌͷީิΛDBʹొ࿥͓͖ͯ͠ɺcronδϣϒͰదٓΫϩʔϧ • λΠτϧɺbodyɺαϜωΠϧ౳ͷڞ௨ϓϩύςΟʹ֨ೲ • ҎલΫϩʔϧͨ͠λΠϛϯά͔Βߋ৽͕ͳ͔ͬͨΒಡΈࠐ·ͳ͍ • Ϋϩʔϧ݁Ռ͸JSONΦϒδΣΫτʹ·ͱΊͯ࠷ޙʹS3ʹΞοϓϩʔυ͢Δ • ը૾ͷՃ޻ͱ͔͸ͳ͠ɻ͍͔ͭإೝࣝͱ͔γϡοͱ΍Γ͍ͨɻ

10. ίϯςΩετϚοϓ(DDD) ఆظऩू Ϋϩʔϧδϣϒͷ
    ൃՐ
    ίϯςϯπऔಘ औಘઌͷ؅ཧ ίϯςϯπऔಘ
    ʢੜͷϨεϙϯεʣ ಺༰ͷՃ޻ औಘͨ͠৘ใΛՃ޻ͯ͠+40/༻ͷ
    

    ύϥϝʔλʔʹม׵͢Δ อଘ ετϨʔδ 4 ʹΞοϓϩʔυ A B calls Worker Fetcher Parser Uploader

11. σΟϨΫτϦߏ੒

12. σΟϨΫτϦߏ੒ GAEͷίϯςΩετੜ੒ ϧʔςΟϯά GAEͷδϣϒఆٛ ίΞػೳͷ࣮૷෦෼ ґଘύοέʔδ GAEϏϧυͷ౎߹্ىಈεΫϦϓτͱ࣮૷ɺvendorσΟϨΫτϦ͸ผʹ෼͚Δ (package໊ͷিಥɺunsafe౳ͷඇਪ঑ػೳͱ͔ͰΞϥʔτ͕ग़Δ)

13. σΟϨΫτϦߏ੒ ΞϓϦ಺ͷڞ௨ઃఆ Λѻ͏Օॴʢޙड़ʣ ֤υϝΠϯͷ࣮૷ DBΞΫηεΛ୲͏ ϦϙδτϦ܈ Contextઃఆ

14. DDDߏ੒ʹͯ͠Α͔ͬͨͱ͜Ζ • ࣮ݱ͍ͨ͠ϏδωεϩδοΫ୯ҐͰ
 package໊͕౷Ұ͞ΕΔͷͰɺ
 package໊ͷিಥ͕ى͜Γʹ͘͘ͳͬͨɻ • Ϧιʔεͱ͍͏ʮϞϊʯͰ͸ͳ͘ɺ
 υϝΠϯͱ͍͏ʮߦҝʯʹ஫໨ͯ͠
 ίʔυΛॻ͘͜ͱͰɺʮ͍ͭ͜͸ԿΛͯ͘͠ΕΔ΍ͭͳͷʁʯ
 ͱ͍͏ٙ໰ͷղ͕͙͢ʹಘΒΕͯɺ୯७ʹಡΈਐΊ΍͘͢ͳͬͨɻ

    • repositoryΛ੾Γ཭͢ͱɺ࣮૷ͱσʔλΞΫηεͷؔ܎͕
 ૄʹͳΓɺϝϯςφϯε͠΍͔ͬͨ͢ɻ
 (CloudSQL͔ΒDataStore΁ͷҠߦͱ͔͠΍͔ͬͨ͢)

15. ڞ௨ॲཧ ݸผͷυϝΠϯͷৄࡉ

16. ࣮૷ɿڞ௨ॲཧ GAEϏϧυʹඞཁͳechoͷίʔυ͸ҎԼΛࢀߟʹ͍ͯͩ͘͠͞ɻ https://echo.labstack.com/cookbook/google-app-engine

17. ࣮૷ɿڞ௨ॲཧ GAEϏϧυʹඞཁͳechoͷίʔυ͸ҎԼΛࢀߟʹ͍ͯͩ͘͠͞ɻ https://echo.labstack.com/cookbook/google-app-engine ʁ

18. ࣮૷ɿڞ௨ॲཧ GAEͷίϯςΩετΛɺ ɾtimeout limitΛઃఆ ɾίϯςΩετʹrepositoryͷΠϯελϯεΛ࣋ͨͤΔ ͱ͍͏͜ͱΛ্ͨ͠ͰɺechoͷΧελϜίϯςΩετͱͯ͠ઃఆ͢Δ

19. ࣮૷ɿڞ௨ॲཧ echo.Contextɺඪ४ύοέʔδͷcontext.Context͡Όͳ͍… echo.ContextΛར༻ͨ͠GrawlerCtxͱ͍͏ΧελϜίϯςΩετΛ࡞੒ͯ͠ɺ Contextͷ1ม਺ͱͯ͠AppEngineCtxΛ࣋ͭɻ

20. Fetcherɿίϯςϯπऔಘઌొ࿥ ϦΫΤετύϥϝʔλʔΛ΋ͱʹɺ”Entity”Λ࡞ͬͯɺ ίϯςΩετ಺෦ͷrepository͔ΒɺίϯςϯπऔಘઌΛొ࿥͢Δ

21. ิ଍ɿrepositoryͷར༻ํ๏ ΧελϜίϯςΫετͷத͔ΒAppEngineͷContextΛऔಘ͠ɺ ͦͷContext͕࣋ͭFetcher(·ͨ͸ଞͷυϝΠϯͷ) DB΁ͷΞΫηαϝιουΛݺͼग़͢

22. ิ଍ɿrepositoryͷ಺෦ Entity(ࣝผࢠΛ࣋ͬͨσʔλߏ଄)Λࡐྉʹͯ͠σʔλΞΫηε͢Δɻ ݸผͷϦϙδτϦΛ ೖΕࢠͰࢀরͯ͠ɺ ୯ҰίϯςΩετ ͔ΒશϦϙδτϦʹ ΞΫηεͰ͖Δ Α͏ʹ͢Δɻ

23. ิ଍ɿrepositoryͷ಺෦ Entity(ࣝผࢠΛ࣋ͬͨσʔλߏ଄)Λࡐྉʹͯ͠σʔλΞΫηε͢Δɻ

24. FetcherɿΠϯλʔϑΣʔε

25. FetcherɿΠϯλʔϑΣʔε

26. FetcherɿΠϯλʔϑΣʔε ϨεϙϯεΛ ͦͷ··interface{} ͱͯ͠ฦ͢ ֎෦΁ͷϦΫΤετ͸ appengine/urlfetchΛ ࢖͏

27. ParserɿΠϯλʔϑΣʔε HTMLύʔαʔͱ͔͸ɺαΠτʹΑͬͯେ෯ʹparseͷํ͕ࣜҟͳΔɻ ͦͷͨΊɺParserFactory.CreateΛܦ༝ͯ͠ɺParseServiceΛ࡞੒͢Δ fetcherInstance͸ϖʔδϯά࣌ͷϦΫΤετ༻

28. ParserɿHTMLύʔαͷఆٛ FactoryͰͲͷ ύʔαʔΛ࡞੒͢Δ͔ ಛఆ͢Δͱ͖ʹ࢖͏ ʢεϥΠυͰ͸ ౎߹্໊લΛมߋͯ͠·͢ʣ appengineͷόʔδϣϯ ͷ౎߹্ɺcontext.Context͸ x/netͷ΋ͷΛ࢖͏ɻ

29. ParserɿParserFactory ࢦఆ͞ΕͨparserKeyʹԠͯ͡ParseServiceΛ࡞੒ɻ FactoryType͸ߏ଄ମΛ࡞Δ্Ͱக͠ํͳ͘༻ҙ͍ͯͯ͠ɺ ”html”ͱ͍͏จࣈྻΛೖΕͯΔ͚ͩɻ

30. ParserɿParse࣮૷Օॴ ίϯςΫετΛड͚औͬͯFetch͠ɺͦͷ݁ՌΛparse͢Δɻ ஫) ຊ౰͸ParserʹFetcherΛ౉͢ͷ͸ݏͩͬͨͷͰ͕͢ɺ ɹ ϖʔδϯά౳ͰͲ͏ͯ͠΋ඞཁͩͬͨͷͰɺFetcherΛ ɹ ࣋ͨͤΔ͜ͱʹ͠·ͨ͠ɻͭΒ͍ʂ
 FacebookParserͱ͔͸ϖʔδϯά͍Βͳ͍ͷͰɺ ׬શʹFetcherͱParser͕෼཭͍ͯ͠·͢ɻ

31. ParserɿParse࣮૷Օॴ yhat/scrapeΛ࢖ͬͯ ཁૉΛऔΓग़͢ɻ goqueryΈ͍ͨʹ ίʔϧόοΫͷॻ͖ํ ͠ͳ͍͠γϯϓϧͰ͢ɻ ྫ) aλά͕࢖ΘΕͯͯɺ ਌͕h1λάɺͦͯ͠ ֘౰NodeͷΫϥε͕

    “skin-entryTitle”Ͱ͋Δ ΋ͷ͸ɺ”title”ͱͯ͠ ύʔε͢Δ

32. ParserɿParse࣮૷Օॴ ϒϩάهࣄҰཡ͔Βɺ هࣄৄࡉϖʔδͷ಺༰Λ ࠶౓Fetch ͦͷதͷTitleཁૉͱ͔Λ औಘ͢Δॲཧ

33. UploaderɿΠϯλʔϑΣʔε JSONΞοϓϩʔυ༻ͷڞ௨σʔλߏ଄ʹ
 parse݁ՌΛ٧Ίͨ΋ͷʢ[]feeditem.FeedItemʣΛ౉ͯ͠ɺ ಛఆετϨʔδʹΞοϓϩʔυ͢Δ

34. UploaderɿॳظԽॲཧ(ྫ: S3) S3ΞοϓϩʔμͷॳظԽॲཧɻ γʔΫϨοτͳ৘ใΛ౉ͯ͠ɺS3ΞΫηε༻ͷΦϒδΣΫτΛ࡞੒

35. Uploaderɿ࣮૷Օॴ ͜͜Ͱ΋appengine/urlfetch Ͱੜ੒ͨ͠HTTPClientΛ ઃఆ͠ͳ͍ͱɺϦΫΤετ͕௨Βͳ͍ ॗʑͱbody΍Β credentialsΛઃఆͯ͠ɺ S3ClientΛ࡞੒ ͜ΕҎ߱͸requestΛ send()͢Δ͚ͩͳͷͰ লུ

36. Workerɿϋϯυϥ GAEͷcronͰୟ͘ΤϯυϙΠϯτͷϋϯυϥΛ༻ҙ͢Δ workerΛ࡞ͬͯɺCrawlϝιουΛݺͿ͚ͩɻ

37. Workerɿϋϯυϥ GAEͷcronͰୟ͘ΤϯυϙΠϯτͷϋϯυϥΛ༻ҙ͢Δ workerΛ࡞ͬͯɺCrawlϝιουΛݺͿ͚ͩɻ

38. Workerɿ࣮૷Օॴ fetch -> parse·Ͱɻ ΈΜͳେ޷͖goroutineɻ sync/errgroupΛ࢖͑͹ɺ ΤϥʔϋϯυϦϯά͠΍ͯ͘͢ ͓͢͢ΊͰ͢ɻ

39. Workerɿ࣮૷Օॴ context.ContextΛड͚औͬͯΔͷͰɺ δϣϒ࣮ߦதʹλΠϜΞ΢τͨ͠Β ΤϥʔΛฦ͢ɻ Ξοϓϩʔυॲཧ Ξοϓϩʔυ͕
 ׬ྃͨ͠Βɺ ࠷ऴऩू࣌ࠁΛߋ৽

40. cron cron.yamlͱ͍͏ͷΛ༻ҙ͢Δͱɺ ಛఆͷΤϯυϙΠϯτʹܾ·ͬͨස౓ͰGETϦΫΤετΛૹͬͯ͘ΕΔͷͰɺ ͜ΕΛ࢖ͬͯworkerͷΤϯυϙΠϯτΛୟ͘

41. Go/GAEͰ٧·ͬͨϙΠϯτ

42. σΟϨΫτϦߏ੒ͷݟ௚͠ • ͍ܰؾ࣋ͪͰGAEࢼͯ͠ΈΑ͏ͱɺ
 ॳظσϓϩΠΛ͠ͳ͍··ਐΊͯͨΒɺ
 ͋ͱ͋ͱมߋ͕͍ͬͺ͍ೖΔɻ • ಛʹvendoringπʔϧΛ࢖͏ͱɺ
 package໊িಥͱ͍͏ΫϦςΟΧϧͳॴͰ
 Ϗϧυ͕௨Βͳ͍ͷͰɺGAE্ʹσϓϩΠ͢ΔͳΒ
 ࠷ॳ͔Β༷ࣜʹ߹ΘͤΔɻ

43. None

44. ൿີ৘ใͷ؅ཧ • access tokenͱ͔ΛͲ͜ʹஔ͔͘ɻ • ࠷ॳtomlͰ؅ཧͯͨ͠Μ͚ͩͲɺgoapp deployͨ͠Β
 srcҎԼͷtoml͕ফ໓͢ΔɻࠔΔɻ • app.yamlͷenvʹઃఆ͢Δͱ͍͏ํ๏΋͋Δ͕ɺ


    gitignoreͰ͖ͳ͍ͷͰौ͍ɻ • ConfigurationRepositoryΛ࡞ͬͯɺDataStore্ʹ
 อଘ͢Δͱ͍͏ํ਑Λͱͬͨɻɹ

45. ൿີ৘ใͷ؅ཧ

46. CloudSQL or DataStore • ࠷ॳCloudSQL(gormܦ༝)Λར༻͍͕ͯͨ͠ɺ
 ͳʹ΍Β ͕͔͔͍ͬͯΔɻ
 ͜ͷAPIࣗମ΁ͷϦΫΤετ͸ଟ͘ͳ͍͸ͣͳͷͰɺ
 খن໛Ͱ͔͔ۚΔͷ͸ौ͍ɻ •

    ConfigurationRepositoryΛ࡞ΔλΠϛϯάͰɺ
 શ෦ετϨʔδΛDataStore΁Ҡߦͨ͠ɻ • RepositoryΛ෼཭͍ͯ͠Ε͹Ҡߦ΋ָͩ͠ɺ
 ίϯιʔϧ͔ΒΤϯςΟςΟ৘ใݟΕΔͷͰ
 DataStoreͷํ͕Αͦ͞͏ɻ

47. before after

48. ·ͱΊ • ΫϩʔϥʔΛ࡞Γ·ͨ͠ɻ • DDD͸ݟ௨͕͠Α͘ɺpackage໊ͷিಥ΋ආ͚ΒΕͯΑ͍ɻ • ಛʹɺͪΌΜͱinterfaceΛఆٛ͢Ε͹ɺ
 υϝΠϯ͝ͱͷίʔυͷՄಡੑ͕ඈ༂తʹ্͕ΔͷͰɺ
 Goͱ૬ੑ͕͍͍ͱࢥ͏ɻ •

    GAE͸σΟϨΫτϦߏ੒౳ͷ໘ͰΫη΋͋Γ·͕͢ɺ
 cron΍ΒDataStore΍ΒͰԸܙ͕͋ΔͷͰɺҰ୴؀ڥΛ
 ߏங͢Δͱେมศརɻඪ४ύοέʔδͷcontext͕
 ར༻Ͱ͖ΔΑ͏ʹͳΔͱ͍͍ͳɻ

49. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ twitter: @__timakin__ github: timakin