Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
実践!Go/GAE+DDDでのクローラー構築
Search
Seiji Takahashi
March 25, 2017
Technology
6
31k
実践!Go/GAE+DDDでのクローラー構築
Go Conference 2017 Springでの講演スライドです。
Seiji Takahashi
March 25, 2017
Tweet
Share
More Decks by Seiji Takahashi
See All by Seiji Takahashi
Go Backends for frontends with GraphQL and gRPC
timakin
6
4k
Design Pattern for Image and Text Composition in Go
timakin
5
6.7k
Golang API Testing the HARD way
timakin
13
6.7k
Head First Golang Image Package
timakin
2
10k
React Native Beyond Prototype
timakin
2
1.6k
Performance Optimization on Google AppEngine
timakin
5
6.3k
testcache.pdf
timakin
1
150
How Go cache
timakin
1
90
How Go cache tests
timakin
1
3.1k
Other Decks in Technology
See All in Technology
対話型音声AIアプリケーションの信頼性向上の取り組み
ivry_presentationmaterials
1
430
AI エージェントと考え直すデータ基盤
na0
17
5.9k
「クラウドコスト絶対削減」を支える技術—FinOpsを超えた徹底的なクラウドコスト削減の実践論
delta_tech
4
180
CDKコード品質UP!ナイスな自作コンストラクタを作るための便利インターフェース
harukasakihara
2
140
FOSS4G 2025 KANSAI QGISで点群データをいろいろしてみた
kou_kita
0
410
ゼロからはじめる採用広報
yutadayo
3
1k
Enhancing SaaS Product Reliability and Release Velocity through Optimized Testing Approach
ropqa
1
240
開発生産性を測る前にやるべきこと - 組織改善の実践 / Before Measuring Dev Productivity
kaonavi
14
6.6k
ポストコロナ時代の SaaS におけるコスト削減の意義
izzii
1
130
PO初心者が考えた ”POらしさ”
nb_rady
0
220
第4回Snowflake 金融ユーザー会 Snowflake summit recap
tamaoki
1
300
60以上のプロダクトを持つ組織における開発者体験向上への取り組み - チームAPIとBackstageで構築する組織の可視化基盤 - / sre next 2025 Efforts to Improve Developer Experience in an Organization with Over 60 Products
vtryo
2
520
Featured
See All Featured
Measuring & Analyzing Core Web Vitals
bluesmoon
7
510
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Agile that works and the tools we love
rasmusluckow
329
21k
How to Think Like a Performance Engineer
csswizardry
25
1.7k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
A designer walks into a library…
pauljervisheath
207
24k
Why Our Code Smells
bkeepers
PRO
336
57k
How to Ace a Technical Interview
jacobian
278
23k
RailsConf 2023
tenderlove
30
1.1k
Side Projects
sachag
455
42k
Faster Mobile Websites
deanohume
307
31k
Transcript
࣮ફ!Go/GAE+DDD ͰͷΫϩʔϥʔߏங @__timakin__ GoConference 2017 Spring
ࣗݾհ
ࣗݾհ • twitter: @__timakin__ • github: timakin • גࣜձࣾGunosy ৽نࣄۀ։ൃࣨ
← New! • ओͳGoϥΠϒϥϦ։ൃ • gopli (DBϨϓϦέʔγϣϯπʔϧ) • gonvert (จࣈίʔυมϥΠϒϥϦ) • octop (githubͷissue, PRϏϡʔ༻CLIπʔϧ) • ts (ٕज़ɾϏδωεܥχϡʔε८ճCLIπʔϧ)
Copyright© Gunosy Inc. All Rights Reserved 4 Go / Python
ΤϯδχΞืूத ▶https://gunosy.co.jp/recruit/ Gunosyɺ౦ژେֶʹ௨͏3ਓͷֶੜͷ ʮใΛੈքதͷਓʹ࠷దʹಧ͚͍ͨʯͱ͍͏͍͔Β࢝·Γ·ͨ͠ɻ ౦ূϚβʔζ্ɺຊώϧζͷΦϑΟεҠసΛܦͯɺ େ͖͍ͯ͘͠ΔձࣾͰ׆༂͍ͨ͠ϝϯόʔΛืू͍ͯ͠·͢ɻ
ΞδΣϯμ
ΞδΣϯμ • Go/GAE + DDDͰ࡞ΔΫϩʔϥʔ • APIͷ֓ཁ • σΟϨΫτϦߏ •
ڞ௨ॲཧɺݸผͷυϝΠϯͷৄࡉ • Go/GAEͰ٧·ͬͨϙΠϯτ • ·ͱΊ
ࠓճͷ։ൃϓϩηε
APIͷ֓ཁ
APIͷ֓ཁ • ओͳ༻ٕज़ • GAE SDK: go version go1.6.3 (appengine-1.9.48)
darwin/amd64 • Web Framework: echo v3.0.3 • Vendoring: dep • ػೳཁ݅ • Facebook, TwitterͷAPI & Σϒϖʔδ͔ΒίϯςϯπΛऔಘ • ͋Β͔͡ΊऔಘઌͷީิΛDBʹొ͓͖ͯ͠ɺcronδϣϒͰదٓΫϩʔϧ • λΠτϧɺbodyɺαϜωΠϧͷڞ௨ϓϩύςΟʹ֨ೲ • ҎલΫϩʔϧͨ͠λΠϛϯά͔Βߋ৽͕ͳ͔ͬͨΒಡΈࠐ·ͳ͍ • Ϋϩʔϧ݁ՌJSONΦϒδΣΫτʹ·ͱΊͯ࠷ޙʹS3ʹΞοϓϩʔυ͢Δ • ը૾ͷՃͱ͔ͳ͠ɻ͍͔ͭإೝࣝͱ͔γϡοͱΓ͍ͨɻ
ίϯςΩετϚοϓ(DDD) ఆظऩू Ϋϩʔϧδϣϒͷ ൃՐ ίϯςϯπऔಘ औಘઌͷཧ ίϯςϯπऔಘ ʢੜͷϨεϙϯεʣ ༰ͷՃ औಘͨ͠ใΛՃͯ͠+40/༻ͷ
ύϥϝʔλʔʹม͢Δ อଘ ετϨʔδ 4 ʹΞοϓϩʔυ A B calls Worker Fetcher Parser Uploader
σΟϨΫτϦߏ
σΟϨΫτϦߏ GAEͷίϯςΩετੜ ϧʔςΟϯά GAEͷδϣϒఆٛ ίΞػೳͷ࣮෦ ґଘύοέʔδ GAEϏϧυͷ߹্ىಈεΫϦϓτͱ࣮ɺvendorσΟϨΫτϦผʹ͚Δ (package໊ͷিಥɺunsafeͷඇਪػೳͱ͔ͰΞϥʔτ͕ग़Δ)
σΟϨΫτϦߏ ΞϓϦͷڞ௨ઃఆ Λѻ͏Օॴʢޙड़ʣ ֤υϝΠϯͷ࣮ DBΞΫηεΛ୲͏ ϦϙδτϦ܈ Contextઃఆ
DDDߏʹͯ͠Α͔ͬͨͱ͜Ζ • ࣮ݱ͍ͨ͠ϏδωεϩδοΫ୯ҐͰ package໊͕౷Ұ͞ΕΔͷͰɺ package໊ͷিಥ͕ى͜Γʹ͘͘ͳͬͨɻ • Ϧιʔεͱ͍͏ʮϞϊʯͰͳ͘ɺ υϝΠϯͱ͍͏ʮߦҝʯʹͯ͠ ίʔυΛॻ͘͜ͱͰɺʮ͍ͭ͜ԿΛͯ͘͠ΕΔͭͳͷʁʯ ͱ͍͏ٙͷղ͕͙͢ʹಘΒΕͯɺ୯७ʹಡΈਐΊ͘͢ͳͬͨɻ
• repositoryΛΓ͢ͱɺ࣮ͱσʔλΞΫηεͷ͕ؔ ૄʹͳΓɺϝϯςφϯε͔ͬͨ͢͠ɻ (CloudSQL͔ΒDataStoreͷҠߦͱ͔͔ͬͨ͢͠)
ڞ௨ॲཧ ݸผͷυϝΠϯͷৄࡉ
࣮ɿڞ௨ॲཧ GAEϏϧυʹඞཁͳechoͷίʔυҎԼΛࢀߟʹ͍ͯͩ͘͠͞ɻ https://echo.labstack.com/cookbook/google-app-engine
࣮ɿڞ௨ॲཧ GAEϏϧυʹඞཁͳechoͷίʔυҎԼΛࢀߟʹ͍ͯͩ͘͠͞ɻ https://echo.labstack.com/cookbook/google-app-engine ʁ
࣮ɿڞ௨ॲཧ GAEͷίϯςΩετΛɺ ɾtimeout limitΛઃఆ ɾίϯςΩετʹrepositoryͷΠϯελϯεΛ࣋ͨͤΔ ͱ͍͏͜ͱΛ্ͨ͠ͰɺechoͷΧελϜίϯςΩετͱͯ͠ઃఆ͢Δ
࣮ɿڞ௨ॲཧ echo.Contextɺඪ४ύοέʔδͷcontext.Context͡Όͳ͍… echo.ContextΛར༻ͨ͠GrawlerCtxͱ͍͏ΧελϜίϯςΩετΛ࡞ͯ͠ɺ Contextͷ1มͱͯ͠AppEngineCtxΛ࣋ͭɻ
Fetcherɿίϯςϯπऔಘઌొ ϦΫΤετύϥϝʔλʔΛͱʹɺ”Entity”Λ࡞ͬͯɺ ίϯςΩετ෦ͷrepository͔ΒɺίϯςϯπऔಘઌΛొ͢Δ
ิɿrepositoryͷར༻ํ๏ ΧελϜίϯςΫετͷத͔ΒAppEngineͷContextΛऔಘ͠ɺ ͦͷContext͕࣋ͭFetcher(·ͨଞͷυϝΠϯͷ) DBͷΞΫηαϝιουΛݺͼग़͢
ิɿrepositoryͷ෦ Entity(ࣝผࢠΛ࣋ͬͨσʔλߏ)Λࡐྉʹͯ͠σʔλΞΫηε͢Δɻ ݸผͷϦϙδτϦΛ ೖΕࢠͰࢀরͯ͠ɺ ୯ҰίϯςΩετ ͔ΒશϦϙδτϦʹ ΞΫηεͰ͖Δ Α͏ʹ͢Δɻ
ิɿrepositoryͷ෦ Entity(ࣝผࢠΛ࣋ͬͨσʔλߏ)Λࡐྉʹͯ͠σʔλΞΫηε͢Δɻ
FetcherɿΠϯλʔϑΣʔε
FetcherɿΠϯλʔϑΣʔε
FetcherɿΠϯλʔϑΣʔε ϨεϙϯεΛ ͦͷ··interface{} ͱͯ͠ฦ͢ ֎෦ͷϦΫΤετ appengine/urlfetchΛ ͏
ParserɿΠϯλʔϑΣʔε HTMLύʔαʔͱ͔ɺαΠτʹΑͬͯେ෯ʹparseͷํ͕ࣜҟͳΔɻ ͦͷͨΊɺParserFactory.CreateΛܦ༝ͯ͠ɺParseServiceΛ࡞͢Δ fetcherInstanceϖʔδϯά࣌ͷϦΫΤετ༻
ParserɿHTMLύʔαͷఆٛ FactoryͰͲͷ ύʔαʔΛ࡞͢Δ͔ ಛఆ͢Δͱ͖ʹ͏ ʢεϥΠυͰ ߹্໊લΛมߋͯ͠·͢ʣ appengineͷόʔδϣϯ ͷ߹্ɺcontext.Context x/netͷͷΛ͏ɻ
ParserɿParserFactory ࢦఆ͞ΕͨparserKeyʹԠͯ͡ParseServiceΛ࡞ɻ FactoryTypeߏମΛ࡞Δ্Ͱக͠ํͳ͘༻ҙ͍ͯͯ͠ɺ ”html”ͱ͍͏จࣈྻΛೖΕͯΔ͚ͩɻ
ParserɿParse࣮Օॴ ίϯςΫετΛड͚औͬͯFetch͠ɺͦͷ݁ՌΛparse͢Δɻ ) ຊParserʹFetcherΛ͢ͷݏͩͬͨͷͰ͕͢ɺ ɹ ϖʔδϯάͰͲ͏ͯ͠ඞཁͩͬͨͷͰɺFetcherΛ ɹ ࣋ͨͤΔ͜ͱʹ͠·ͨ͠ɻͭΒ͍ʂ FacebookParserͱ͔ϖʔδϯά͍Βͳ͍ͷͰɺ શʹFetcherͱParser͕͍ͯ͠·͢ɻ
ParserɿParse࣮Օॴ yhat/scrapeΛͬͯ ཁૉΛऔΓग़͢ɻ goqueryΈ͍ͨʹ ίʔϧόοΫͷॻ͖ํ ͠ͳ͍͠γϯϓϧͰ͢ɻ ྫ) aλά͕ΘΕͯͯɺ ͕h1λάɺͦͯ͠ ֘NodeͷΫϥε͕
“skin-entryTitle”Ͱ͋Δ ͷɺ”title”ͱͯ͠ ύʔε͢Δ
ParserɿParse࣮Օॴ ϒϩάهࣄҰཡ͔Βɺ هࣄৄࡉϖʔδͷ༰Λ ࠶Fetch ͦͷதͷTitleཁૉͱ͔Λ औಘ͢Δॲཧ
UploaderɿΠϯλʔϑΣʔε JSONΞοϓϩʔυ༻ͷڞ௨σʔλߏʹ parse݁ՌΛ٧Ίͨͷʢ[]feeditem.FeedItemʣΛͯ͠ɺ ಛఆετϨʔδʹΞοϓϩʔυ͢Δ
UploaderɿॳظԽॲཧ(ྫ: S3) S3ΞοϓϩʔμͷॳظԽॲཧɻ γʔΫϨοτͳใΛͯ͠ɺS3ΞΫηε༻ͷΦϒδΣΫτΛ࡞
Uploaderɿ࣮Օॴ ͜͜Ͱappengine/urlfetch Ͱੜͨ͠HTTPClientΛ ઃఆ͠ͳ͍ͱɺϦΫΤετ͕௨Βͳ͍ ॗʑͱbodyΒ credentialsΛઃఆͯ͠ɺ S3ClientΛ࡞ ͜ΕҎ߱requestΛ send()͢Δ͚ͩͳͷͰ লུ
Workerɿϋϯυϥ GAEͷcronͰୟ͘ΤϯυϙΠϯτͷϋϯυϥΛ༻ҙ͢Δ workerΛ࡞ͬͯɺCrawlϝιουΛݺͿ͚ͩɻ
Workerɿϋϯυϥ GAEͷcronͰୟ͘ΤϯυϙΠϯτͷϋϯυϥΛ༻ҙ͢Δ workerΛ࡞ͬͯɺCrawlϝιουΛݺͿ͚ͩɻ
Workerɿ࣮Օॴ fetch -> parse·Ͱɻ ΈΜͳେ͖goroutineɻ sync/errgroupΛ͑ɺ ΤϥʔϋϯυϦϯάͯ͘͢͠ ͓͢͢ΊͰ͢ɻ
Workerɿ࣮Օॴ context.ContextΛड͚औͬͯΔͷͰɺ δϣϒ࣮ߦதʹλΠϜΞτͨ͠Β ΤϥʔΛฦ͢ɻ Ξοϓϩʔυॲཧ Ξοϓϩʔυ͕ ྃͨ͠Βɺ ࠷ऴऩू࣌ࠁΛߋ৽
cron cron.yamlͱ͍͏ͷΛ༻ҙ͢Δͱɺ ಛఆͷΤϯυϙΠϯτʹܾ·ͬͨසͰGETϦΫΤετΛૹͬͯ͘ΕΔͷͰɺ ͜ΕΛͬͯworkerͷΤϯυϙΠϯτΛୟ͘
Go/GAEͰ٧·ͬͨϙΠϯτ
σΟϨΫτϦߏͷݟ͠ • ͍ܰؾ࣋ͪͰGAEࢼͯ͠ΈΑ͏ͱɺ ॳظσϓϩΠΛ͠ͳ͍··ਐΊͯͨΒɺ ͋ͱ͋ͱมߋ͕͍ͬͺ͍ೖΔɻ • ಛʹvendoringπʔϧΛ͏ͱɺ package໊িಥͱ͍͏ΫϦςΟΧϧͳॴͰ Ϗϧυ͕௨Βͳ͍ͷͰɺGAE্ʹσϓϩΠ͢ΔͳΒ ࠷ॳ͔Β༷ࣜʹ߹ΘͤΔɻ
None
ൿີใͷཧ • access tokenͱ͔ΛͲ͜ʹஔ͔͘ɻ • ࠷ॳtomlͰཧͯͨ͠Μ͚ͩͲɺgoapp deployͨ͠Β srcҎԼͷtoml͕ফ໓͢ΔɻࠔΔɻ • app.yamlͷenvʹઃఆ͢Δͱ͍͏ํ๏͋Δ͕ɺ
gitignoreͰ͖ͳ͍ͷͰौ͍ɻ • ConfigurationRepositoryΛ࡞ͬͯɺDataStore্ʹ อଘ͢Δͱ͍͏ํΛͱͬͨɻɹ
ൿີใͷཧ
CloudSQL or DataStore • ࠷ॳCloudSQL(gormܦ༝)Λར༻͍͕ͯͨ͠ɺ ͳʹΒ ͕͔͔͍ͬͯΔɻ ͜ͷAPIࣗମͷϦΫΤετଟ͘ͳ͍ͣͳͷͰɺ খنͰ͔͔ۚΔͷौ͍ɻ •
ConfigurationRepositoryΛ࡞ΔλΠϛϯάͰɺ શ෦ετϨʔδΛDataStoreҠߦͨ͠ɻ • RepositoryΛ͍ͯ͠ΕҠߦָͩ͠ɺ ίϯιʔϧ͔ΒΤϯςΟςΟใݟΕΔͷͰ DataStoreͷํ͕Αͦ͞͏ɻ
before after
·ͱΊ • ΫϩʔϥʔΛ࡞Γ·ͨ͠ɻ • DDDݟ௨͕͠Α͘ɺpackage໊ͷিಥආ͚ΒΕͯΑ͍ɻ • ಛʹɺͪΌΜͱinterfaceΛఆٛ͢Εɺ υϝΠϯ͝ͱͷίʔυͷՄಡੑ͕ඈ༂తʹ্͕ΔͷͰɺ Goͱ૬ੑ͕͍͍ͱࢥ͏ɻ •
GAEσΟϨΫτϦߏͷ໘ͰΫη͋Γ·͕͢ɺ cronΒDataStoreΒͰԸܙ͕͋ΔͷͰɺҰ୴ڥΛ ߏங͢Δͱେมศརɻඪ४ύοέʔδͷcontext͕ ར༻Ͱ͖ΔΑ͏ʹͳΔͱ͍͍ͳɻ
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ twitter: @__timakin__ github: timakin