実践!Go/GAE+DDDでのクローラー構築

by Seiji Takahashi

Slide 1

Slide 1 text

࣮ફ!Go/GAE+DDD ͰͷΫϩʔϥʔߏங @__timakin__ GoConference 2017 Spring

Slide 2

Slide 2 text

ࣗݾ঺հ

Slide 3

Slide 3 text

ࣗݾ঺հ • twitter: @__timakin__ • github: timakin • גࣜձࣾGunosy ৽نࣄۀ։ൃࣨ ← New! • ओͳGoϥΠϒϥϦ։ൃ෺ • gopli (DBϨϓϦέʔγϣϯπʔϧ) • gonvert (จࣈίʔυม׵ϥΠϒϥϦ) • octop (githubͷissue, PRϏϡʔ༻CLIπʔϧ) • ts (ٕज़ɾϏδωεܥχϡʔε८ճCLIπʔϧ)

Slide 4

Slide 4 text

Copyright© Gunosy Inc. All Rights Reserved 4 Go / Python ΤϯδχΞืूத ▶https://gunosy.co.jp/recruit/ Gunosy͸ɺ౦ژେֶʹ௨͏3ਓͷֶੜͷ  ʮ৘ใΛੈքதͷਓʹ࠷దʹಧ͚͍ͨʯͱ͍͏૝͍͔Β࢝·Γ·ͨ͠ɻ ౦ূϚβʔζ্৔ɺ࿡ຊ໦ώϧζ΁ͷΦϑΟεҠసΛܦͯɺ  େ͖͘੒௕͍ͯ͠ΔձࣾͰ׆༂͍ͨ͠ϝϯόʔΛืू͍ͯ͠·͢ɻ

Slide 5

Slide 5 text

ΞδΣϯμ

Slide 6

Slide 6 text

ΞδΣϯμ • Go/GAE + DDDͰ࡞ΔΫϩʔϥʔ • APIͷ֓ཁ • σΟϨΫτϦߏ੒ • ڞ௨ॲཧɺݸผͷυϝΠϯͷৄࡉ • Go/GAEͰ٧·ͬͨϙΠϯτ • ·ͱΊ

Slide 7

Slide 7 text

ࠓճͷ։ൃϓϩηε

Slide 8

Slide 8 text

APIͷ֓ཁ

Slide 9

Slide 9 text

APIͷ֓ཁ • ओͳ࢖༻ٕज़ • GAE SDK: go version go1.6.3 (appengine-1.9.48) darwin/amd64 • Web Framework: echo v3.0.3 • Vendoring: dep • ػೳཁ݅ • Facebook, Twitter౳ͷAPI & ΢Σϒϖʔδ͔ΒίϯςϯπΛऔಘ • ͋Β͔͡ΊऔಘઌͷީิΛDBʹొ࿥͓͖ͯ͠ɺcronδϣϒͰదٓΫϩʔϧ • λΠτϧɺbodyɺαϜωΠϧ౳ͷڞ௨ϓϩύςΟʹ֨ೲ • ҎલΫϩʔϧͨ͠λΠϛϯά͔Βߋ৽͕ͳ͔ͬͨΒಡΈࠐ·ͳ͍ • Ϋϩʔϧ݁Ռ͸JSONΦϒδΣΫτʹ·ͱΊͯ࠷ޙʹS3ʹΞοϓϩʔυ͢Δ • ը૾ͷՃ޻ͱ͔͸ͳ͠ɻ͍͔ͭإೝࣝͱ͔γϡοͱ΍Γ͍ͨɻ

Slide 10

Slide 10 text

ίϯςΩετϚοϓ(DDD) ఆظऩू Ϋϩʔϧδϣϒͷ ൃՐ ίϯςϯπऔಘ औಘઌͷ؅ཧ ίϯςϯπऔಘ ʢੜͷϨεϙϯεʣ ಺༰ͷՃ޻ औಘͨ͠৘ใΛՃ޻ͯ͠+40/༻ͷ ύϥϝʔλʔʹม׵͢Δ อଘ ετϨʔδ 4 ʹΞοϓϩʔυ A B calls Worker Fetcher Parser Uploader

Slide 11

Slide 11 text

σΟϨΫτϦߏ੒

Slide 12

Slide 12 text

σΟϨΫτϦߏ੒ GAEͷίϯςΩετੜ੒ ϧʔςΟϯά GAEͷδϣϒఆٛ ίΞػೳͷ࣮૷෦෼ ґଘύοέʔδ GAEϏϧυͷ౎߹্ىಈεΫϦϓτͱ࣮૷ɺvendorσΟϨΫτϦ͸ผʹ෼͚Δ (package໊ͷিಥɺunsafe౳ͷඇਪ঑ػೳͱ͔ͰΞϥʔτ͕ग़Δ)

Slide 13

Slide 13 text

σΟϨΫτϦߏ੒ ΞϓϦ಺ͷڞ௨ઃఆ Λѻ͏Օॴʢޙड़ʣ ֤υϝΠϯͷ࣮૷ DBΞΫηεΛ୲͏ ϦϙδτϦ܈ Contextઃఆ

Slide 14

Slide 14 text

DDDߏ੒ʹͯ͠Α͔ͬͨͱ͜Ζ • ࣮ݱ͍ͨ͠ϏδωεϩδοΫ୯ҐͰ  package໊͕౷Ұ͞ΕΔͷͰɺ  package໊ͷিಥ͕ى͜Γʹ͘͘ͳͬͨɻ • Ϧιʔεͱ͍͏ʮϞϊʯͰ͸ͳ͘ɺ  υϝΠϯͱ͍͏ʮߦҝʯʹ஫໨ͯ͠  ίʔυΛॻ͘͜ͱͰɺʮ͍ͭ͜͸ԿΛͯ͘͠ΕΔ΍ͭͳͷʁʯ  ͱ͍͏ٙ໰ͷղ͕͙͢ʹಘΒΕͯɺ୯७ʹಡΈਐΊ΍͘͢ͳͬͨɻ • repositoryΛ੾Γ཭͢ͱɺ࣮૷ͱσʔλΞΫηεͷؔ܎͕  ૄʹͳΓɺϝϯςφϯε͠΍͔ͬͨ͢ɻ  (CloudSQL͔ΒDataStore΁ͷҠߦͱ͔͠΍͔ͬͨ͢)

Slide 15

Slide 15 text

ڞ௨ॲཧ ݸผͷυϝΠϯͷৄࡉ

Slide 16

Slide 16 text

࣮૷ɿڞ௨ॲཧ GAEϏϧυʹඞཁͳechoͷίʔυ͸ҎԼΛࢀߟʹ͍ͯͩ͘͠͞ɻ https://echo.labstack.com/cookbook/google-app-engine

Slide 17

Slide 17 text

࣮૷ɿڞ௨ॲཧ GAEϏϧυʹඞཁͳechoͷίʔυ͸ҎԼΛࢀߟʹ͍ͯͩ͘͠͞ɻ https://echo.labstack.com/cookbook/google-app-engine ʁ

Slide 18

Slide 18 text

࣮૷ɿڞ௨ॲཧ GAEͷίϯςΩετΛɺ ɾtimeout limitΛઃఆ ɾίϯςΩετʹrepositoryͷΠϯελϯεΛ࣋ͨͤΔ ͱ͍͏͜ͱΛ্ͨ͠ͰɺechoͷΧελϜίϯςΩετͱͯ͠ઃఆ͢Δ

Slide 19

Slide 19 text

࣮૷ɿڞ௨ॲཧ echo.Contextɺඪ४ύοέʔδͷcontext.Context͡Όͳ͍… echo.ContextΛར༻ͨ͠GrawlerCtxͱ͍͏ΧελϜίϯςΩετΛ࡞੒ͯ͠ɺ Contextͷ1ม਺ͱͯ͠AppEngineCtxΛ࣋ͭɻ

Slide 20

Slide 20 text

Fetcherɿίϯςϯπऔಘઌొ࿥ ϦΫΤετύϥϝʔλʔΛ΋ͱʹɺ”Entity”Λ࡞ͬͯɺ ίϯςΩετ಺෦ͷrepository͔ΒɺίϯςϯπऔಘઌΛొ࿥͢Δ

Slide 21

Slide 21 text

ิ଍ɿrepositoryͷར༻ํ๏ ΧελϜίϯςΫετͷத͔ΒAppEngineͷContextΛऔಘ͠ɺ ͦͷContext͕࣋ͭFetcher(·ͨ͸ଞͷυϝΠϯͷ) DB΁ͷΞΫηαϝιουΛݺͼग़͢

Slide 22

Slide 22 text

ิ଍ɿrepositoryͷ಺෦ Entity(ࣝผࢠΛ࣋ͬͨσʔλߏ଄)Λࡐྉʹͯ͠σʔλΞΫηε͢Δɻ ݸผͷϦϙδτϦΛ ೖΕࢠͰࢀরͯ͠ɺ ୯ҰίϯςΩετ ͔ΒશϦϙδτϦʹ ΞΫηεͰ͖Δ Α͏ʹ͢Δɻ

Slide 23

Slide 23 text

ิ଍ɿrepositoryͷ಺෦ Entity(ࣝผࢠΛ࣋ͬͨσʔλߏ଄)Λࡐྉʹͯ͠σʔλΞΫηε͢Δɻ

Slide 24

Slide 24 text

FetcherɿΠϯλʔϑΣʔε

Slide 25

Slide 25 text

FetcherɿΠϯλʔϑΣʔε

Slide 26

Slide 26 text

FetcherɿΠϯλʔϑΣʔε ϨεϙϯεΛ ͦͷ··interface{} ͱͯ͠ฦ͢ ֎෦΁ͷϦΫΤετ͸ appengine/urlfetchΛ ࢖͏

Slide 27

Slide 27 text

ParserɿΠϯλʔϑΣʔε HTMLύʔαʔͱ͔͸ɺαΠτʹΑͬͯେ෯ʹparseͷํ͕ࣜҟͳΔɻ ͦͷͨΊɺParserFactory.CreateΛܦ༝ͯ͠ɺParseServiceΛ࡞੒͢Δ fetcherInstance͸ϖʔδϯά࣌ͷϦΫΤετ༻

Slide 28

Slide 28 text

ParserɿHTMLύʔαͷఆٛ FactoryͰͲͷ ύʔαʔΛ࡞੒͢Δ͔ ಛఆ͢Δͱ͖ʹ࢖͏ ʢεϥΠυͰ͸ ౎߹্໊લΛมߋͯ͠·͢ʣ appengineͷόʔδϣϯ ͷ౎߹্ɺcontext.Context͸ x/netͷ΋ͷΛ࢖͏ɻ

Slide 29

Slide 29 text

ParserɿParserFactory ࢦఆ͞ΕͨparserKeyʹԠͯ͡ParseServiceΛ࡞੒ɻ FactoryType͸ߏ଄ମΛ࡞Δ্Ͱக͠ํͳ͘༻ҙ͍ͯͯ͠ɺ ”html”ͱ͍͏จࣈྻΛೖΕͯΔ͚ͩɻ

Slide 30

Slide 30 text

ParserɿParse࣮૷Օॴ ίϯςΫετΛड͚औͬͯFetch͠ɺͦͷ݁ՌΛparse͢Δɻ ஫) ຊ౰͸ParserʹFetcherΛ౉͢ͷ͸ݏͩͬͨͷͰ͕͢ɺ ɹ ϖʔδϯά౳ͰͲ͏ͯ͠΋ඞཁͩͬͨͷͰɺFetcherΛ ɹ ࣋ͨͤΔ͜ͱʹ͠·ͨ͠ɻͭΒ͍ʂ  FacebookParserͱ͔͸ϖʔδϯά͍Βͳ͍ͷͰɺ ׬શʹFetcherͱParser͕෼཭͍ͯ͠·͢ɻ

Slide 31

Slide 31 text

ParserɿParse࣮૷Օॴ yhat/scrapeΛ࢖ͬͯ ཁૉΛऔΓग़͢ɻ goqueryΈ͍ͨʹ ίʔϧόοΫͷॻ͖ํ ͠ͳ͍͠γϯϓϧͰ͢ɻ ྫ) aλά͕࢖ΘΕͯͯɺ ਌͕h1λάɺͦͯ͠ ֘౰NodeͷΫϥε͕ “skin-entryTitle”Ͱ͋Δ ΋ͷ͸ɺ”title”ͱͯ͠ ύʔε͢Δ

Slide 32

Slide 32 text

ParserɿParse࣮૷Օॴ ϒϩάهࣄҰཡ͔Βɺ هࣄৄࡉϖʔδͷ಺༰Λ ࠶౓Fetch ͦͷதͷTitleཁૉͱ͔Λ औಘ͢Δॲཧ

Slide 33

Slide 33 text

UploaderɿΠϯλʔϑΣʔε JSONΞοϓϩʔυ༻ͷڞ௨σʔλߏ଄ʹ  parse݁ՌΛ٧Ίͨ΋ͷʢ[]feeditem.FeedItemʣΛ౉ͯ͠ɺ ಛఆετϨʔδʹΞοϓϩʔυ͢Δ

Slide 34

Slide 34 text

UploaderɿॳظԽॲཧ(ྫ: S3) S3ΞοϓϩʔμͷॳظԽॲཧɻ γʔΫϨοτͳ৘ใΛ౉ͯ͠ɺS3ΞΫηε༻ͷΦϒδΣΫτΛ࡞੒

Slide 35

Slide 35 text

Uploaderɿ࣮૷Օॴ ͜͜Ͱ΋appengine/urlfetch Ͱੜ੒ͨ͠HTTPClientΛ ઃఆ͠ͳ͍ͱɺϦΫΤετ͕௨Βͳ͍ ॗʑͱbody΍Β credentialsΛઃఆͯ͠ɺ S3ClientΛ࡞੒ ͜ΕҎ߱͸requestΛ send()͢Δ͚ͩͳͷͰ লུ

Slide 36

Slide 36 text

Workerɿϋϯυϥ GAEͷcronͰୟ͘ΤϯυϙΠϯτͷϋϯυϥΛ༻ҙ͢Δ workerΛ࡞ͬͯɺCrawlϝιουΛݺͿ͚ͩɻ

Slide 37

Slide 37 text

Workerɿϋϯυϥ GAEͷcronͰୟ͘ΤϯυϙΠϯτͷϋϯυϥΛ༻ҙ͢Δ workerΛ࡞ͬͯɺCrawlϝιουΛݺͿ͚ͩɻ

Slide 38

Slide 38 text

Workerɿ࣮૷Օॴ fetch -> parse·Ͱɻ ΈΜͳେ޷͖goroutineɻ sync/errgroupΛ࢖͑͹ɺ ΤϥʔϋϯυϦϯά͠΍ͯ͘͢ ͓͢͢ΊͰ͢ɻ

Slide 39

Slide 39 text

Workerɿ࣮૷Օॴ context.ContextΛड͚औͬͯΔͷͰɺ δϣϒ࣮ߦதʹλΠϜΞ΢τͨ͠Β ΤϥʔΛฦ͢ɻ Ξοϓϩʔυॲཧ Ξοϓϩʔυ͕  ׬ྃͨ͠Βɺ ࠷ऴऩू࣌ࠁΛߋ৽

Slide 40

Slide 40 text

cron cron.yamlͱ͍͏ͷΛ༻ҙ͢Δͱɺ ಛఆͷΤϯυϙΠϯτʹܾ·ͬͨස౓ͰGETϦΫΤετΛૹͬͯ͘ΕΔͷͰɺ ͜ΕΛ࢖ͬͯworkerͷΤϯυϙΠϯτΛୟ͘

Slide 41

Slide 41 text

Go/GAEͰ٧·ͬͨϙΠϯτ

Slide 42

Slide 42 text

σΟϨΫτϦߏ੒ͷݟ௚͠ • ͍ܰؾ࣋ͪͰGAEࢼͯ͠ΈΑ͏ͱɺ  ॳظσϓϩΠΛ͠ͳ͍··ਐΊͯͨΒɺ  ͋ͱ͋ͱมߋ͕͍ͬͺ͍ೖΔɻ • ಛʹvendoringπʔϧΛ࢖͏ͱɺ  package໊িಥͱ͍͏ΫϦςΟΧϧͳॴͰ  Ϗϧυ͕௨Βͳ͍ͷͰɺGAE্ʹσϓϩΠ͢ΔͳΒ  ࠷ॳ͔Β༷ࣜʹ߹ΘͤΔɻ

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

ൿີ৘ใͷ؅ཧ • access tokenͱ͔ΛͲ͜ʹஔ͔͘ɻ • ࠷ॳtomlͰ؅ཧͯͨ͠Μ͚ͩͲɺgoapp deployͨ͠Β  srcҎԼͷtoml͕ফ໓͢ΔɻࠔΔɻ • app.yamlͷenvʹઃఆ͢Δͱ͍͏ํ๏΋͋Δ͕ɺ  gitignoreͰ͖ͳ͍ͷͰौ͍ɻ • ConﬁgurationRepositoryΛ࡞ͬͯɺDataStore্ʹ  อଘ͢Δͱ͍͏ํ਑Λͱͬͨɻɹ

Slide 45

Slide 45 text

ൿີ৘ใͷ؅ཧ

Slide 46

Slide 46 text

CloudSQL or DataStore • ࠷ॳCloudSQL(gormܦ༝)Λར༻͍͕ͯͨ͠ɺ  ͳʹ΍Β ͕͔͔͍ͬͯΔɻ  ͜ͷAPIࣗମ΁ͷϦΫΤετ͸ଟ͘ͳ͍͸ͣͳͷͰɺ  খن໛Ͱ͔͔ۚΔͷ͸ौ͍ɻ • ConﬁgurationRepositoryΛ࡞ΔλΠϛϯάͰɺ  શ෦ετϨʔδΛDataStore΁Ҡߦͨ͠ɻ • RepositoryΛ෼཭͍ͯ͠Ε͹Ҡߦ΋ָͩ͠ɺ  ίϯιʔϧ͔ΒΤϯςΟςΟ৘ใݟΕΔͷͰ  DataStoreͷํ͕Αͦ͞͏ɻ

Slide 47

Slide 47 text

before after

Slide 48

Slide 48 text

·ͱΊ • ΫϩʔϥʔΛ࡞Γ·ͨ͠ɻ • DDD͸ݟ௨͕͠Α͘ɺpackage໊ͷিಥ΋ආ͚ΒΕͯΑ͍ɻ • ಛʹɺͪΌΜͱinterfaceΛఆٛ͢Ε͹ɺ  υϝΠϯ͝ͱͷίʔυͷՄಡੑ͕ඈ༂తʹ্͕ΔͷͰɺ  Goͱ૬ੑ͕͍͍ͱࢥ͏ɻ • GAE͸σΟϨΫτϦߏ੒౳ͷ໘ͰΫη΋͋Γ·͕͢ɺ  cron΍ΒDataStore΍ΒͰԸܙ͕͋ΔͷͰɺҰ୴؀ڥΛ  ߏங͢Δͱେมศརɻඪ४ύοέʔδͷcontext͕  ར༻Ͱ͖ΔΑ͏ʹͳΔͱ͍͍ͳɻ

Slide 49

Slide 49 text

͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ twitter: @__timakin__ github: timakin