Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
実践!Go/GAE+DDDでのクローラー構築
Search
Seiji Takahashi
March 25, 2017
Technology
6
31k
実践!Go/GAE+DDDでのクローラー構築
Go Conference 2017 Springでの講演スライドです。
Seiji Takahashi
March 25, 2017
Tweet
Share
More Decks by Seiji Takahashi
See All by Seiji Takahashi
Go Backends for frontends with GraphQL and gRPC
timakin
6
3.8k
Design Pattern for Image and Text Composition in Go
timakin
5
6.6k
Golang API Testing the HARD way
timakin
13
6.5k
Head First Golang Image Package
timakin
2
10k
React Native Beyond Prototype
timakin
2
1.6k
Performance Optimization on Google AppEngine
timakin
5
6.2k
testcache.pdf
timakin
1
140
How Go cache
timakin
1
75
How Go cache tests
timakin
1
3k
Other Decks in Technology
See All in Technology
【Findy】「正しく」失敗できる チームの作り方 〜リアルな事例から紐解く失敗を恐れない組織とは〜 / A team that can fail correctly by findy
i35_267
5
900
Goで作って学ぶWebSocket
ryuichi1208
3
2.8k
AIエージェント元年@日本生成AIユーザ会
shukob
1
210
AWSではじめる Web APIテスト実践ガイド / A practical guide to testing Web APIs on AWS
yokawasa
8
710
OPENLOGI Company Profile
hr01
0
60k
Cracking the Coding Interview 6th Edition
gdplabs
14
28k
Amazon Q Developerの無料利用枠を使い倒してHello worldを表示させよう!
nrinetcom
PRO
2
120
【内製開発Summit 2025】イオンスマートテクノロジーの内製化組織の作り方/In-house-development-summit-AST
aeonpeople
2
670
LINEギフトにおけるバックエンド開発
lycorptech_jp
PRO
0
290
JavaにおけるNull非許容性
skrb
2
2.6k
Ruby on Railsで持続可能な開発を行うために取り組んでいること
am1157154
3
160
クラウド関連のインシデントケースを収集して見えてきたもの
lhazy
8
1.1k
Featured
See All Featured
Building a Modern Day E-commerce SEO Strategy
aleyda
38
7.1k
How GitHub (no longer) Works
holman
314
140k
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
The Cost Of JavaScript in 2023
addyosmani
47
7.4k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
193
16k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
133
33k
A Modern Web Designer's Workflow
chriscoyier
693
190k
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3k
KATA
mclloyd
29
14k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Producing Creativity
orderedlist
PRO
344
40k
Faster Mobile Websites
deanohume
306
31k
Transcript
࣮ફ!Go/GAE+DDD ͰͷΫϩʔϥʔߏங @__timakin__ GoConference 2017 Spring
ࣗݾհ
ࣗݾհ • twitter: @__timakin__ • github: timakin • גࣜձࣾGunosy ৽نࣄۀ։ൃࣨ
← New! • ओͳGoϥΠϒϥϦ։ൃ • gopli (DBϨϓϦέʔγϣϯπʔϧ) • gonvert (จࣈίʔυมϥΠϒϥϦ) • octop (githubͷissue, PRϏϡʔ༻CLIπʔϧ) • ts (ٕज़ɾϏδωεܥχϡʔε८ճCLIπʔϧ)
Copyright© Gunosy Inc. All Rights Reserved 4 Go / Python
ΤϯδχΞืूத ▶https://gunosy.co.jp/recruit/ Gunosyɺ౦ژେֶʹ௨͏3ਓͷֶੜͷ ʮใΛੈքதͷਓʹ࠷దʹಧ͚͍ͨʯͱ͍͏͍͔Β࢝·Γ·ͨ͠ɻ ౦ূϚβʔζ্ɺຊώϧζͷΦϑΟεҠసΛܦͯɺ େ͖͍ͯ͘͠ΔձࣾͰ׆༂͍ͨ͠ϝϯόʔΛืू͍ͯ͠·͢ɻ
ΞδΣϯμ
ΞδΣϯμ • Go/GAE + DDDͰ࡞ΔΫϩʔϥʔ • APIͷ֓ཁ • σΟϨΫτϦߏ •
ڞ௨ॲཧɺݸผͷυϝΠϯͷৄࡉ • Go/GAEͰ٧·ͬͨϙΠϯτ • ·ͱΊ
ࠓճͷ։ൃϓϩηε
APIͷ֓ཁ
APIͷ֓ཁ • ओͳ༻ٕज़ • GAE SDK: go version go1.6.3 (appengine-1.9.48)
darwin/amd64 • Web Framework: echo v3.0.3 • Vendoring: dep • ػೳཁ݅ • Facebook, TwitterͷAPI & Σϒϖʔδ͔ΒίϯςϯπΛऔಘ • ͋Β͔͡ΊऔಘઌͷީิΛDBʹొ͓͖ͯ͠ɺcronδϣϒͰదٓΫϩʔϧ • λΠτϧɺbodyɺαϜωΠϧͷڞ௨ϓϩύςΟʹ֨ೲ • ҎલΫϩʔϧͨ͠λΠϛϯά͔Βߋ৽͕ͳ͔ͬͨΒಡΈࠐ·ͳ͍ • Ϋϩʔϧ݁ՌJSONΦϒδΣΫτʹ·ͱΊͯ࠷ޙʹS3ʹΞοϓϩʔυ͢Δ • ը૾ͷՃͱ͔ͳ͠ɻ͍͔ͭإೝࣝͱ͔γϡοͱΓ͍ͨɻ
ίϯςΩετϚοϓ(DDD) ఆظऩू Ϋϩʔϧδϣϒͷ ൃՐ ίϯςϯπऔಘ औಘઌͷཧ ίϯςϯπऔಘ ʢੜͷϨεϙϯεʣ ༰ͷՃ औಘͨ͠ใΛՃͯ͠+40/༻ͷ
ύϥϝʔλʔʹม͢Δ อଘ ετϨʔδ 4 ʹΞοϓϩʔυ A B calls Worker Fetcher Parser Uploader
σΟϨΫτϦߏ
σΟϨΫτϦߏ GAEͷίϯςΩετੜ ϧʔςΟϯά GAEͷδϣϒఆٛ ίΞػೳͷ࣮෦ ґଘύοέʔδ GAEϏϧυͷ߹্ىಈεΫϦϓτͱ࣮ɺvendorσΟϨΫτϦผʹ͚Δ (package໊ͷিಥɺunsafeͷඇਪػೳͱ͔ͰΞϥʔτ͕ग़Δ)
σΟϨΫτϦߏ ΞϓϦͷڞ௨ઃఆ Λѻ͏Օॴʢޙड़ʣ ֤υϝΠϯͷ࣮ DBΞΫηεΛ୲͏ ϦϙδτϦ܈ Contextઃఆ
DDDߏʹͯ͠Α͔ͬͨͱ͜Ζ • ࣮ݱ͍ͨ͠ϏδωεϩδοΫ୯ҐͰ package໊͕౷Ұ͞ΕΔͷͰɺ package໊ͷিಥ͕ى͜Γʹ͘͘ͳͬͨɻ • Ϧιʔεͱ͍͏ʮϞϊʯͰͳ͘ɺ υϝΠϯͱ͍͏ʮߦҝʯʹͯ͠ ίʔυΛॻ͘͜ͱͰɺʮ͍ͭ͜ԿΛͯ͘͠ΕΔͭͳͷʁʯ ͱ͍͏ٙͷղ͕͙͢ʹಘΒΕͯɺ୯७ʹಡΈਐΊ͘͢ͳͬͨɻ
• repositoryΛΓ͢ͱɺ࣮ͱσʔλΞΫηεͷ͕ؔ ૄʹͳΓɺϝϯςφϯε͔ͬͨ͢͠ɻ (CloudSQL͔ΒDataStoreͷҠߦͱ͔͔ͬͨ͢͠)
ڞ௨ॲཧ ݸผͷυϝΠϯͷৄࡉ
࣮ɿڞ௨ॲཧ GAEϏϧυʹඞཁͳechoͷίʔυҎԼΛࢀߟʹ͍ͯͩ͘͠͞ɻ https://echo.labstack.com/cookbook/google-app-engine
࣮ɿڞ௨ॲཧ GAEϏϧυʹඞཁͳechoͷίʔυҎԼΛࢀߟʹ͍ͯͩ͘͠͞ɻ https://echo.labstack.com/cookbook/google-app-engine ʁ
࣮ɿڞ௨ॲཧ GAEͷίϯςΩετΛɺ ɾtimeout limitΛઃఆ ɾίϯςΩετʹrepositoryͷΠϯελϯεΛ࣋ͨͤΔ ͱ͍͏͜ͱΛ্ͨ͠ͰɺechoͷΧελϜίϯςΩετͱͯ͠ઃఆ͢Δ
࣮ɿڞ௨ॲཧ echo.Contextɺඪ४ύοέʔδͷcontext.Context͡Όͳ͍… echo.ContextΛར༻ͨ͠GrawlerCtxͱ͍͏ΧελϜίϯςΩετΛ࡞ͯ͠ɺ Contextͷ1มͱͯ͠AppEngineCtxΛ࣋ͭɻ
Fetcherɿίϯςϯπऔಘઌొ ϦΫΤετύϥϝʔλʔΛͱʹɺ”Entity”Λ࡞ͬͯɺ ίϯςΩετ෦ͷrepository͔ΒɺίϯςϯπऔಘઌΛొ͢Δ
ิɿrepositoryͷར༻ํ๏ ΧελϜίϯςΫετͷத͔ΒAppEngineͷContextΛऔಘ͠ɺ ͦͷContext͕࣋ͭFetcher(·ͨଞͷυϝΠϯͷ) DBͷΞΫηαϝιουΛݺͼग़͢
ิɿrepositoryͷ෦ Entity(ࣝผࢠΛ࣋ͬͨσʔλߏ)Λࡐྉʹͯ͠σʔλΞΫηε͢Δɻ ݸผͷϦϙδτϦΛ ೖΕࢠͰࢀরͯ͠ɺ ୯ҰίϯςΩετ ͔ΒશϦϙδτϦʹ ΞΫηεͰ͖Δ Α͏ʹ͢Δɻ
ิɿrepositoryͷ෦ Entity(ࣝผࢠΛ࣋ͬͨσʔλߏ)Λࡐྉʹͯ͠σʔλΞΫηε͢Δɻ
FetcherɿΠϯλʔϑΣʔε
FetcherɿΠϯλʔϑΣʔε
FetcherɿΠϯλʔϑΣʔε ϨεϙϯεΛ ͦͷ··interface{} ͱͯ͠ฦ͢ ֎෦ͷϦΫΤετ appengine/urlfetchΛ ͏
ParserɿΠϯλʔϑΣʔε HTMLύʔαʔͱ͔ɺαΠτʹΑͬͯେ෯ʹparseͷํ͕ࣜҟͳΔɻ ͦͷͨΊɺParserFactory.CreateΛܦ༝ͯ͠ɺParseServiceΛ࡞͢Δ fetcherInstanceϖʔδϯά࣌ͷϦΫΤετ༻
ParserɿHTMLύʔαͷఆٛ FactoryͰͲͷ ύʔαʔΛ࡞͢Δ͔ ಛఆ͢Δͱ͖ʹ͏ ʢεϥΠυͰ ߹্໊લΛมߋͯ͠·͢ʣ appengineͷόʔδϣϯ ͷ߹্ɺcontext.Context x/netͷͷΛ͏ɻ
ParserɿParserFactory ࢦఆ͞ΕͨparserKeyʹԠͯ͡ParseServiceΛ࡞ɻ FactoryTypeߏମΛ࡞Δ্Ͱக͠ํͳ͘༻ҙ͍ͯͯ͠ɺ ”html”ͱ͍͏จࣈྻΛೖΕͯΔ͚ͩɻ
ParserɿParse࣮Օॴ ίϯςΫετΛड͚औͬͯFetch͠ɺͦͷ݁ՌΛparse͢Δɻ ) ຊParserʹFetcherΛ͢ͷݏͩͬͨͷͰ͕͢ɺ ɹ ϖʔδϯάͰͲ͏ͯ͠ඞཁͩͬͨͷͰɺFetcherΛ ɹ ࣋ͨͤΔ͜ͱʹ͠·ͨ͠ɻͭΒ͍ʂ FacebookParserͱ͔ϖʔδϯά͍Βͳ͍ͷͰɺ શʹFetcherͱParser͕͍ͯ͠·͢ɻ
ParserɿParse࣮Օॴ yhat/scrapeΛͬͯ ཁૉΛऔΓग़͢ɻ goqueryΈ͍ͨʹ ίʔϧόοΫͷॻ͖ํ ͠ͳ͍͠γϯϓϧͰ͢ɻ ྫ) aλά͕ΘΕͯͯɺ ͕h1λάɺͦͯ͠ ֘NodeͷΫϥε͕
“skin-entryTitle”Ͱ͋Δ ͷɺ”title”ͱͯ͠ ύʔε͢Δ
ParserɿParse࣮Օॴ ϒϩάهࣄҰཡ͔Βɺ هࣄৄࡉϖʔδͷ༰Λ ࠶Fetch ͦͷதͷTitleཁૉͱ͔Λ औಘ͢Δॲཧ
UploaderɿΠϯλʔϑΣʔε JSONΞοϓϩʔυ༻ͷڞ௨σʔλߏʹ parse݁ՌΛ٧Ίͨͷʢ[]feeditem.FeedItemʣΛͯ͠ɺ ಛఆετϨʔδʹΞοϓϩʔυ͢Δ
UploaderɿॳظԽॲཧ(ྫ: S3) S3ΞοϓϩʔμͷॳظԽॲཧɻ γʔΫϨοτͳใΛͯ͠ɺS3ΞΫηε༻ͷΦϒδΣΫτΛ࡞
Uploaderɿ࣮Օॴ ͜͜Ͱappengine/urlfetch Ͱੜͨ͠HTTPClientΛ ઃఆ͠ͳ͍ͱɺϦΫΤετ͕௨Βͳ͍ ॗʑͱbodyΒ credentialsΛઃఆͯ͠ɺ S3ClientΛ࡞ ͜ΕҎ߱requestΛ send()͢Δ͚ͩͳͷͰ লུ
Workerɿϋϯυϥ GAEͷcronͰୟ͘ΤϯυϙΠϯτͷϋϯυϥΛ༻ҙ͢Δ workerΛ࡞ͬͯɺCrawlϝιουΛݺͿ͚ͩɻ
Workerɿϋϯυϥ GAEͷcronͰୟ͘ΤϯυϙΠϯτͷϋϯυϥΛ༻ҙ͢Δ workerΛ࡞ͬͯɺCrawlϝιουΛݺͿ͚ͩɻ
Workerɿ࣮Օॴ fetch -> parse·Ͱɻ ΈΜͳେ͖goroutineɻ sync/errgroupΛ͑ɺ ΤϥʔϋϯυϦϯάͯ͘͢͠ ͓͢͢ΊͰ͢ɻ
Workerɿ࣮Օॴ context.ContextΛड͚औͬͯΔͷͰɺ δϣϒ࣮ߦதʹλΠϜΞτͨ͠Β ΤϥʔΛฦ͢ɻ Ξοϓϩʔυॲཧ Ξοϓϩʔυ͕ ྃͨ͠Βɺ ࠷ऴऩू࣌ࠁΛߋ৽
cron cron.yamlͱ͍͏ͷΛ༻ҙ͢Δͱɺ ಛఆͷΤϯυϙΠϯτʹܾ·ͬͨසͰGETϦΫΤετΛૹͬͯ͘ΕΔͷͰɺ ͜ΕΛͬͯworkerͷΤϯυϙΠϯτΛୟ͘
Go/GAEͰ٧·ͬͨϙΠϯτ
σΟϨΫτϦߏͷݟ͠ • ͍ܰؾ࣋ͪͰGAEࢼͯ͠ΈΑ͏ͱɺ ॳظσϓϩΠΛ͠ͳ͍··ਐΊͯͨΒɺ ͋ͱ͋ͱมߋ͕͍ͬͺ͍ೖΔɻ • ಛʹvendoringπʔϧΛ͏ͱɺ package໊িಥͱ͍͏ΫϦςΟΧϧͳॴͰ Ϗϧυ͕௨Βͳ͍ͷͰɺGAE্ʹσϓϩΠ͢ΔͳΒ ࠷ॳ͔Β༷ࣜʹ߹ΘͤΔɻ
None
ൿີใͷཧ • access tokenͱ͔ΛͲ͜ʹஔ͔͘ɻ • ࠷ॳtomlͰཧͯͨ͠Μ͚ͩͲɺgoapp deployͨ͠Β srcҎԼͷtoml͕ফ໓͢ΔɻࠔΔɻ • app.yamlͷenvʹઃఆ͢Δͱ͍͏ํ๏͋Δ͕ɺ
gitignoreͰ͖ͳ͍ͷͰौ͍ɻ • ConfigurationRepositoryΛ࡞ͬͯɺDataStore্ʹ อଘ͢Δͱ͍͏ํΛͱͬͨɻɹ
ൿີใͷཧ
CloudSQL or DataStore • ࠷ॳCloudSQL(gormܦ༝)Λར༻͍͕ͯͨ͠ɺ ͳʹΒ ͕͔͔͍ͬͯΔɻ ͜ͷAPIࣗମͷϦΫΤετଟ͘ͳ͍ͣͳͷͰɺ খنͰ͔͔ۚΔͷौ͍ɻ •
ConfigurationRepositoryΛ࡞ΔλΠϛϯάͰɺ શ෦ετϨʔδΛDataStoreҠߦͨ͠ɻ • RepositoryΛ͍ͯ͠ΕҠߦָͩ͠ɺ ίϯιʔϧ͔ΒΤϯςΟςΟใݟΕΔͷͰ DataStoreͷํ͕Αͦ͞͏ɻ
before after
·ͱΊ • ΫϩʔϥʔΛ࡞Γ·ͨ͠ɻ • DDDݟ௨͕͠Α͘ɺpackage໊ͷিಥආ͚ΒΕͯΑ͍ɻ • ಛʹɺͪΌΜͱinterfaceΛఆٛ͢Εɺ υϝΠϯ͝ͱͷίʔυͷՄಡੑ͕ඈ༂తʹ্͕ΔͷͰɺ Goͱ૬ੑ͕͍͍ͱࢥ͏ɻ •
GAEσΟϨΫτϦߏͷ໘ͰΫη͋Γ·͕͢ɺ cronΒDataStoreΒͰԸܙ͕͋ΔͷͰɺҰ୴ڥΛ ߏங͢Δͱେมศརɻඪ४ύοέʔδͷcontext͕ ར༻Ͱ͖ΔΑ͏ʹͳΔͱ͍͍ͳɻ
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ twitter: @__timakin__ github: timakin