Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
実践!Go/GAE+DDDでのクローラー構築
Search
Seiji Takahashi
March 25, 2017
Technology
6
31k
実践!Go/GAE+DDDでのクローラー構築
Go Conference 2017 Springでの講演スライドです。
Seiji Takahashi
March 25, 2017
Tweet
Share
More Decks by Seiji Takahashi
See All by Seiji Takahashi
Go Backends for frontends with GraphQL and gRPC
timakin
6
4k
Design Pattern for Image and Text Composition in Go
timakin
5
6.7k
Golang API Testing the HARD way
timakin
13
6.7k
Head First Golang Image Package
timakin
2
10k
React Native Beyond Prototype
timakin
2
1.6k
Performance Optimization on Google AppEngine
timakin
5
6.3k
testcache.pdf
timakin
1
150
How Go cache
timakin
1
89
How Go cache tests
timakin
1
3.1k
Other Decks in Technology
See All in Technology
プロダクトエンジニアリング組織への歩み、その現在地 / Our journey to becoming a product engineering organization
hiro_torii
0
130
Observability infrastructure behind the trillion-messages scale Kafka platform
lycorptech_jp
PRO
0
140
Delegating the chores of authenticating users to Keycloak
ahus1
0
120
Claude Code Actionを使ったコード品質改善の取り組み
potix2
PRO
6
2.3k
低レイヤを知りたいPHPerのためのCコンパイラ作成入門 完全版 / Building a C Compiler for PHPers Who Want to Dive into Low-Level Programming - Expanded
tomzoh
4
3.3k
AIの最新技術&テーマをつまんで紹介&フリートークするシリーズ #1 量子機械学習の入門
tkhresk
0
140
変化する開発、進化する体系時代に適応するソフトウェアエンジニアの知識と考え方(JaSST'25 Kansai)
mizunori
1
230
250627 関西Ruby会議08 前夜祭 RejectKaigi「DJ on Ruby Ver.0.1」
msykd
PRO
2
320
米国国防総省のDevSecOpsライフサイクルをAWSのセキュリティサービスとOSSで実現
syoshie
2
1.1k
Wasm元年
askua
0
150
エンジニア向け技術スタック情報
kauche
1
270
Clineを含めたAIエージェントを 大規模組織に導入し、投資対効果を考える / Introducing AI agents into your organization
i35_267
4
1.6k
Featured
See All Featured
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
31
1.2k
Adopting Sorbet at Scale
ufuk
77
9.4k
Designing for humans not robots
tammielis
253
25k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
800
Bash Introduction
62gerente
614
210k
Become a Pro
speakerdeck
PRO
28
5.4k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
46
9.6k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
124
52k
Code Review Best Practice
trishagee
68
18k
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
How STYLIGHT went responsive
nonsquared
100
5.6k
Transcript
࣮ફ!Go/GAE+DDD ͰͷΫϩʔϥʔߏங @__timakin__ GoConference 2017 Spring
ࣗݾհ
ࣗݾհ • twitter: @__timakin__ • github: timakin • גࣜձࣾGunosy ৽نࣄۀ։ൃࣨ
← New! • ओͳGoϥΠϒϥϦ։ൃ • gopli (DBϨϓϦέʔγϣϯπʔϧ) • gonvert (จࣈίʔυมϥΠϒϥϦ) • octop (githubͷissue, PRϏϡʔ༻CLIπʔϧ) • ts (ٕज़ɾϏδωεܥχϡʔε८ճCLIπʔϧ)
Copyright© Gunosy Inc. All Rights Reserved 4 Go / Python
ΤϯδχΞืूத ▶https://gunosy.co.jp/recruit/ Gunosyɺ౦ژେֶʹ௨͏3ਓͷֶੜͷ ʮใΛੈքதͷਓʹ࠷దʹಧ͚͍ͨʯͱ͍͏͍͔Β࢝·Γ·ͨ͠ɻ ౦ূϚβʔζ্ɺຊώϧζͷΦϑΟεҠసΛܦͯɺ େ͖͍ͯ͘͠ΔձࣾͰ׆༂͍ͨ͠ϝϯόʔΛืू͍ͯ͠·͢ɻ
ΞδΣϯμ
ΞδΣϯμ • Go/GAE + DDDͰ࡞ΔΫϩʔϥʔ • APIͷ֓ཁ • σΟϨΫτϦߏ •
ڞ௨ॲཧɺݸผͷυϝΠϯͷৄࡉ • Go/GAEͰ٧·ͬͨϙΠϯτ • ·ͱΊ
ࠓճͷ։ൃϓϩηε
APIͷ֓ཁ
APIͷ֓ཁ • ओͳ༻ٕज़ • GAE SDK: go version go1.6.3 (appengine-1.9.48)
darwin/amd64 • Web Framework: echo v3.0.3 • Vendoring: dep • ػೳཁ݅ • Facebook, TwitterͷAPI & Σϒϖʔδ͔ΒίϯςϯπΛऔಘ • ͋Β͔͡ΊऔಘઌͷީิΛDBʹొ͓͖ͯ͠ɺcronδϣϒͰదٓΫϩʔϧ • λΠτϧɺbodyɺαϜωΠϧͷڞ௨ϓϩύςΟʹ֨ೲ • ҎલΫϩʔϧͨ͠λΠϛϯά͔Βߋ৽͕ͳ͔ͬͨΒಡΈࠐ·ͳ͍ • Ϋϩʔϧ݁ՌJSONΦϒδΣΫτʹ·ͱΊͯ࠷ޙʹS3ʹΞοϓϩʔυ͢Δ • ը૾ͷՃͱ͔ͳ͠ɻ͍͔ͭإೝࣝͱ͔γϡοͱΓ͍ͨɻ
ίϯςΩετϚοϓ(DDD) ఆظऩू Ϋϩʔϧδϣϒͷ ൃՐ ίϯςϯπऔಘ औಘઌͷཧ ίϯςϯπऔಘ ʢੜͷϨεϙϯεʣ ༰ͷՃ औಘͨ͠ใΛՃͯ͠+40/༻ͷ
ύϥϝʔλʔʹม͢Δ อଘ ετϨʔδ 4 ʹΞοϓϩʔυ A B calls Worker Fetcher Parser Uploader
σΟϨΫτϦߏ
σΟϨΫτϦߏ GAEͷίϯςΩετੜ ϧʔςΟϯά GAEͷδϣϒఆٛ ίΞػೳͷ࣮෦ ґଘύοέʔδ GAEϏϧυͷ߹্ىಈεΫϦϓτͱ࣮ɺvendorσΟϨΫτϦผʹ͚Δ (package໊ͷিಥɺunsafeͷඇਪػೳͱ͔ͰΞϥʔτ͕ग़Δ)
σΟϨΫτϦߏ ΞϓϦͷڞ௨ઃఆ Λѻ͏Օॴʢޙड़ʣ ֤υϝΠϯͷ࣮ DBΞΫηεΛ୲͏ ϦϙδτϦ܈ Contextઃఆ
DDDߏʹͯ͠Α͔ͬͨͱ͜Ζ • ࣮ݱ͍ͨ͠ϏδωεϩδοΫ୯ҐͰ package໊͕౷Ұ͞ΕΔͷͰɺ package໊ͷিಥ͕ى͜Γʹ͘͘ͳͬͨɻ • Ϧιʔεͱ͍͏ʮϞϊʯͰͳ͘ɺ υϝΠϯͱ͍͏ʮߦҝʯʹͯ͠ ίʔυΛॻ͘͜ͱͰɺʮ͍ͭ͜ԿΛͯ͘͠ΕΔͭͳͷʁʯ ͱ͍͏ٙͷղ͕͙͢ʹಘΒΕͯɺ୯७ʹಡΈਐΊ͘͢ͳͬͨɻ
• repositoryΛΓ͢ͱɺ࣮ͱσʔλΞΫηεͷ͕ؔ ૄʹͳΓɺϝϯςφϯε͔ͬͨ͢͠ɻ (CloudSQL͔ΒDataStoreͷҠߦͱ͔͔ͬͨ͢͠)
ڞ௨ॲཧ ݸผͷυϝΠϯͷৄࡉ
࣮ɿڞ௨ॲཧ GAEϏϧυʹඞཁͳechoͷίʔυҎԼΛࢀߟʹ͍ͯͩ͘͠͞ɻ https://echo.labstack.com/cookbook/google-app-engine
࣮ɿڞ௨ॲཧ GAEϏϧυʹඞཁͳechoͷίʔυҎԼΛࢀߟʹ͍ͯͩ͘͠͞ɻ https://echo.labstack.com/cookbook/google-app-engine ʁ
࣮ɿڞ௨ॲཧ GAEͷίϯςΩετΛɺ ɾtimeout limitΛઃఆ ɾίϯςΩετʹrepositoryͷΠϯελϯεΛ࣋ͨͤΔ ͱ͍͏͜ͱΛ্ͨ͠ͰɺechoͷΧελϜίϯςΩετͱͯ͠ઃఆ͢Δ
࣮ɿڞ௨ॲཧ echo.Contextɺඪ४ύοέʔδͷcontext.Context͡Όͳ͍… echo.ContextΛར༻ͨ͠GrawlerCtxͱ͍͏ΧελϜίϯςΩετΛ࡞ͯ͠ɺ Contextͷ1มͱͯ͠AppEngineCtxΛ࣋ͭɻ
Fetcherɿίϯςϯπऔಘઌొ ϦΫΤετύϥϝʔλʔΛͱʹɺ”Entity”Λ࡞ͬͯɺ ίϯςΩετ෦ͷrepository͔ΒɺίϯςϯπऔಘઌΛొ͢Δ
ิɿrepositoryͷར༻ํ๏ ΧελϜίϯςΫετͷத͔ΒAppEngineͷContextΛऔಘ͠ɺ ͦͷContext͕࣋ͭFetcher(·ͨଞͷυϝΠϯͷ) DBͷΞΫηαϝιουΛݺͼग़͢
ิɿrepositoryͷ෦ Entity(ࣝผࢠΛ࣋ͬͨσʔλߏ)Λࡐྉʹͯ͠σʔλΞΫηε͢Δɻ ݸผͷϦϙδτϦΛ ೖΕࢠͰࢀরͯ͠ɺ ୯ҰίϯςΩετ ͔ΒશϦϙδτϦʹ ΞΫηεͰ͖Δ Α͏ʹ͢Δɻ
ิɿrepositoryͷ෦ Entity(ࣝผࢠΛ࣋ͬͨσʔλߏ)Λࡐྉʹͯ͠σʔλΞΫηε͢Δɻ
FetcherɿΠϯλʔϑΣʔε
FetcherɿΠϯλʔϑΣʔε
FetcherɿΠϯλʔϑΣʔε ϨεϙϯεΛ ͦͷ··interface{} ͱͯ͠ฦ͢ ֎෦ͷϦΫΤετ appengine/urlfetchΛ ͏
ParserɿΠϯλʔϑΣʔε HTMLύʔαʔͱ͔ɺαΠτʹΑͬͯେ෯ʹparseͷํ͕ࣜҟͳΔɻ ͦͷͨΊɺParserFactory.CreateΛܦ༝ͯ͠ɺParseServiceΛ࡞͢Δ fetcherInstanceϖʔδϯά࣌ͷϦΫΤετ༻
ParserɿHTMLύʔαͷఆٛ FactoryͰͲͷ ύʔαʔΛ࡞͢Δ͔ ಛఆ͢Δͱ͖ʹ͏ ʢεϥΠυͰ ߹্໊લΛมߋͯ͠·͢ʣ appengineͷόʔδϣϯ ͷ߹্ɺcontext.Context x/netͷͷΛ͏ɻ
ParserɿParserFactory ࢦఆ͞ΕͨparserKeyʹԠͯ͡ParseServiceΛ࡞ɻ FactoryTypeߏମΛ࡞Δ্Ͱக͠ํͳ͘༻ҙ͍ͯͯ͠ɺ ”html”ͱ͍͏จࣈྻΛೖΕͯΔ͚ͩɻ
ParserɿParse࣮Օॴ ίϯςΫετΛड͚औͬͯFetch͠ɺͦͷ݁ՌΛparse͢Δɻ ) ຊParserʹFetcherΛ͢ͷݏͩͬͨͷͰ͕͢ɺ ɹ ϖʔδϯάͰͲ͏ͯ͠ඞཁͩͬͨͷͰɺFetcherΛ ɹ ࣋ͨͤΔ͜ͱʹ͠·ͨ͠ɻͭΒ͍ʂ FacebookParserͱ͔ϖʔδϯά͍Βͳ͍ͷͰɺ શʹFetcherͱParser͕͍ͯ͠·͢ɻ
ParserɿParse࣮Օॴ yhat/scrapeΛͬͯ ཁૉΛऔΓग़͢ɻ goqueryΈ͍ͨʹ ίʔϧόοΫͷॻ͖ํ ͠ͳ͍͠γϯϓϧͰ͢ɻ ྫ) aλά͕ΘΕͯͯɺ ͕h1λάɺͦͯ͠ ֘NodeͷΫϥε͕
“skin-entryTitle”Ͱ͋Δ ͷɺ”title”ͱͯ͠ ύʔε͢Δ
ParserɿParse࣮Օॴ ϒϩάهࣄҰཡ͔Βɺ هࣄৄࡉϖʔδͷ༰Λ ࠶Fetch ͦͷதͷTitleཁૉͱ͔Λ औಘ͢Δॲཧ
UploaderɿΠϯλʔϑΣʔε JSONΞοϓϩʔυ༻ͷڞ௨σʔλߏʹ parse݁ՌΛ٧Ίͨͷʢ[]feeditem.FeedItemʣΛͯ͠ɺ ಛఆετϨʔδʹΞοϓϩʔυ͢Δ
UploaderɿॳظԽॲཧ(ྫ: S3) S3ΞοϓϩʔμͷॳظԽॲཧɻ γʔΫϨοτͳใΛͯ͠ɺS3ΞΫηε༻ͷΦϒδΣΫτΛ࡞
Uploaderɿ࣮Օॴ ͜͜Ͱappengine/urlfetch Ͱੜͨ͠HTTPClientΛ ઃఆ͠ͳ͍ͱɺϦΫΤετ͕௨Βͳ͍ ॗʑͱbodyΒ credentialsΛઃఆͯ͠ɺ S3ClientΛ࡞ ͜ΕҎ߱requestΛ send()͢Δ͚ͩͳͷͰ লུ
Workerɿϋϯυϥ GAEͷcronͰୟ͘ΤϯυϙΠϯτͷϋϯυϥΛ༻ҙ͢Δ workerΛ࡞ͬͯɺCrawlϝιουΛݺͿ͚ͩɻ
Workerɿϋϯυϥ GAEͷcronͰୟ͘ΤϯυϙΠϯτͷϋϯυϥΛ༻ҙ͢Δ workerΛ࡞ͬͯɺCrawlϝιουΛݺͿ͚ͩɻ
Workerɿ࣮Օॴ fetch -> parse·Ͱɻ ΈΜͳେ͖goroutineɻ sync/errgroupΛ͑ɺ ΤϥʔϋϯυϦϯάͯ͘͢͠ ͓͢͢ΊͰ͢ɻ
Workerɿ࣮Օॴ context.ContextΛड͚औͬͯΔͷͰɺ δϣϒ࣮ߦதʹλΠϜΞτͨ͠Β ΤϥʔΛฦ͢ɻ Ξοϓϩʔυॲཧ Ξοϓϩʔυ͕ ྃͨ͠Βɺ ࠷ऴऩू࣌ࠁΛߋ৽
cron cron.yamlͱ͍͏ͷΛ༻ҙ͢Δͱɺ ಛఆͷΤϯυϙΠϯτʹܾ·ͬͨසͰGETϦΫΤετΛૹͬͯ͘ΕΔͷͰɺ ͜ΕΛͬͯworkerͷΤϯυϙΠϯτΛୟ͘
Go/GAEͰ٧·ͬͨϙΠϯτ
σΟϨΫτϦߏͷݟ͠ • ͍ܰؾ࣋ͪͰGAEࢼͯ͠ΈΑ͏ͱɺ ॳظσϓϩΠΛ͠ͳ͍··ਐΊͯͨΒɺ ͋ͱ͋ͱมߋ͕͍ͬͺ͍ೖΔɻ • ಛʹvendoringπʔϧΛ͏ͱɺ package໊িಥͱ͍͏ΫϦςΟΧϧͳॴͰ Ϗϧυ͕௨Βͳ͍ͷͰɺGAE্ʹσϓϩΠ͢ΔͳΒ ࠷ॳ͔Β༷ࣜʹ߹ΘͤΔɻ
None
ൿີใͷཧ • access tokenͱ͔ΛͲ͜ʹஔ͔͘ɻ • ࠷ॳtomlͰཧͯͨ͠Μ͚ͩͲɺgoapp deployͨ͠Β srcҎԼͷtoml͕ফ໓͢ΔɻࠔΔɻ • app.yamlͷenvʹઃఆ͢Δͱ͍͏ํ๏͋Δ͕ɺ
gitignoreͰ͖ͳ͍ͷͰौ͍ɻ • ConfigurationRepositoryΛ࡞ͬͯɺDataStore্ʹ อଘ͢Δͱ͍͏ํΛͱͬͨɻɹ
ൿີใͷཧ
CloudSQL or DataStore • ࠷ॳCloudSQL(gormܦ༝)Λར༻͍͕ͯͨ͠ɺ ͳʹΒ ͕͔͔͍ͬͯΔɻ ͜ͷAPIࣗମͷϦΫΤετଟ͘ͳ͍ͣͳͷͰɺ খنͰ͔͔ۚΔͷौ͍ɻ •
ConfigurationRepositoryΛ࡞ΔλΠϛϯάͰɺ શ෦ετϨʔδΛDataStoreҠߦͨ͠ɻ • RepositoryΛ͍ͯ͠ΕҠߦָͩ͠ɺ ίϯιʔϧ͔ΒΤϯςΟςΟใݟΕΔͷͰ DataStoreͷํ͕Αͦ͞͏ɻ
before after
·ͱΊ • ΫϩʔϥʔΛ࡞Γ·ͨ͠ɻ • DDDݟ௨͕͠Α͘ɺpackage໊ͷিಥආ͚ΒΕͯΑ͍ɻ • ಛʹɺͪΌΜͱinterfaceΛఆٛ͢Εɺ υϝΠϯ͝ͱͷίʔυͷՄಡੑ͕ඈ༂తʹ্͕ΔͷͰɺ Goͱ૬ੑ͕͍͍ͱࢥ͏ɻ •
GAEσΟϨΫτϦߏͷ໘ͰΫη͋Γ·͕͢ɺ cronΒDataStoreΒͰԸܙ͕͋ΔͷͰɺҰ୴ڥΛ ߏங͢Δͱେมศརɻඪ४ύοέʔδͷcontext͕ ར༻Ͱ͖ΔΑ͏ʹͳΔͱ͍͍ͳɻ
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ twitter: @__timakin__ github: timakin