Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
scraping system blueprint using scrapinghub and fargate for collecting POI data
Search
wataru
March 07, 2019
Technology
0
590
scraping system blueprint using scrapinghub and fargate for collecting POI data
This presentation explains the blueprint of scraping system.
wataru
March 07, 2019
Tweet
Share
Other Decks in Technology
See All in Technology
require(ESM)とECMAScript仕様
uhyo
4
980
よく聞くけど使ったことないソフトウェアNo.1 KafkaとSnowflake
foursue
4
510
Babylon.jsと色々なものを組み合わせる:ブラウザのAPIやガジェットや2D描画ライブラリなど / Babylon.js 勉強会 vol.3
you
PRO
0
160
生成AIの変革の時代に、直近1年で直面した課題とその解決策
ktc_wada
0
670
VSCodeの拡張機能を作っている話
ebarakazuhiro
1
810
Google Cloud Next '24 Recap(Cloud Run/k8s)
mokocm
0
330
KubeConにproposalを送りたい人へのアドバイス
sat
PRO
3
270
地理空間データ可視化・解析・活用ソリューション Pacific Spatial Solutions (PSS)
pacificspatialsolutions
0
330
Documentação de Produtos: Artefatos essenciais na prática
rigolon
1
110
【NW X Security JAWS#3】L3-4:AWS環境のIPv6移行に向けて知っておきたいこと
shotashiratori
1
650
止まらないLinuxシステムを構築する_高信頼性クラスタ入門
koedoyoshida
2
440
Rustで「プリズモイダル法」を利用して「土量計算」をガチでやる
nokonoko1203
1
310
Featured
See All Featured
Unsuck your backbone
ammeep
663
57k
VelocityConf: Rendering Performance Case Studies
addyosmani
321
23k
Fantastic passwords and where to find them - at NoRuKo
philnash
39
2.5k
The MySQL Ecosystem @ GitHub 2015
samlambert
244
12k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
126
32k
The Invisible Customer
myddelton
114
12k
A better future with KSS
kneath
231
16k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
221
21k
The Pragmatic Product Professional
lauravandoore
26
5.8k
The Brand Is Dead. Long Live the Brand.
mthomps
49
29k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
13
8.3k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
34
8.9k
Transcript
4DSBQJOH10* ECS with Fargate 使 定期収集 設計
2 ొஃͷత "84ͷࣝͷ͋ΔํʑͷલͰΔ ΞυόΠεΛ͍͖ͨͩɺ ΑΓྑ͍ͷΛ࡞
3 ຊͷ͋Β͢͡ ఆظऩूγεςϜ࡞ͷഎܠ ఆظऩूγεςϜ࡞ͷཁ݅ γεςϜͷઃܭ ·ͱΊ
4 ຊͷ͋Β͢͡ ఆظऩूγεςϜ࡞ͷഎܠ ఆظऩूγεςϜ࡞ͷཁ݅ γεςϜͷઃܭ ·ͱΊ
5 ༷ʑͳछྨͷ10*Λऩूͯ͠·͢ γϣοϐϯάϞʔϧ ؍ޫ Πϕϯτܥࢪઃ νΣʔϯళ
6 ༷ʑͳछྨͷ10*Λऩूͯ͠·͢ γϣοϐϯάϞʔϧ ؍ޫ Πϕϯτܥࢪઃ νΣʔϯళ ࠓͷϝΠϯ εΫϨΠϐϯάͰऔಘ͍ͯ͠·͢
ఆظऩू͢Δత 7 νΣʔϯళΑ͘ঢ়ଶมԽ͢Δ ดళฮ৽نళฮͷѲՄೳ σʔλͷΛอͭ σʔλऩू࡞ۀͷࣗಈԽ σʔλऩूΛࣗಈԽ͢Δ͜ͱͰɺ ظతͳݮΛ࣮ݱ͢Δ ޮՌੑ ޮੑ
8 ຊͷ͋Β͢͡ ఆظऩूγεςϜ࡞ͷഎܠ ఆظऩूγεςϜ࡞ͷཁ݅ γεςϜͷઃܭ ·ͱΊ
10*σʔλϕʔεߏங·Ͱͷ̏εςοϓ ళฮ63-Ϧετ Λऔಘ ళฮৄࡉใ Λऔಘ औಘͨ͠ σʔλͷՃ 9 45&1
45&1 45&1
ݱঢ় 10 ɾ൚༻εύΠμʔ ɾ४൚༻εύΠμʔ ɾ֎εύΠμʔ ɾTDSBQJOHIVC &$্ʹଘࡏ εύΠμʔ͕ෳݸଘࡏ εύΠμʔʹઃఆΛૹΔ͜ͱͰɺ εύΠμʔىಈ
ઃఆͰεύΠμʔΛ࣮ߦ ۀҕୗ͞Μʹ࡞ґཔ͍ͯ͠Δ ઃఆ%SPQCPY্ʹ֨ೲ ESPQCPYͷར༻ ༻εύΠμʔͷࢦఆػೳ ىಈεύΠμʔͱઃఆͷ࿈ܞ ESPQCPYͱ"84ͱͷ࿈ܞ
γεςϜʹ.VTUͰΈࠐΈ͍ͨػೳɾઃܭ 11 ɾ࢛ظʹҰճ࣮ߦ ɾδϣϒ੍ݶ͔Βδϣϒௐ εέδϡʔϦϯά ɾਐḿ௨ ɾྃ௨ ɾࣦഊ௨ ϝʔϧɾεϥοΫ௨ػೳ ɾʑXFCٕज़͕৽͘͠ͳ͍ͬͯΔ
ɾ൚༻εύΠμʔ·ͩվम͢Δඞཁ͋Γ ɾίʔυͷมߋΛͦͷ··өͰ͖Δঢ়ଶ ɾςετͱ͔͘͢͠ ɾ։ൃͱຊ൪ͷҠಈΛ؆୯ʹ ίʔυɾڥͷߋ৽͕༰қ
γεςϜʹ.VTUͰΈࠐΈ͍ͨػೳɾઃܭ 12 ɾ࢛ظʹҰճ࣮ߦ ɾδϣϒ੍ݶ͔Βδϣϒௐ εέδϡʔϦϯά ɾਐḿ௨ ɾྃ௨ ɾࣦഊ௨ ϝʔϧɾεϥοΫ௨ػೳ ɾʑXFCٕज़͕৽͘͠ͳ͍ͬͯΔ
ɾ൚༻εύΠμʔ·ͩվम͢Δඞཁ͋Γ ɾίʔυͷมߋΛͦͷ··өͰ͖Δঢ়ଶ ɾςετͱ͔͘͢͠ ɾ։ൃͱຊ൪ͷҠಈΛ؆୯ʹ ίʔυɾڥͷߋ৽͕༰қ DSPO 4&4
13 EPDLFSͷಛ *OGSBTUSVDUVSFBT$PEF *NNVUBCMF*OGSBTUSVDUVSF %PDLFSGJMFͰόʔδϣϯཧ͕Մೳ ڥͷઃఆཧ͕ෆཁ
γεςϜʹ.VTUͰΈࠐΈ͍ͨػೳɾઃܭ 14 ɾ࢛ظʹҰճ࣮ߦ ɾδϣϒ੍ݶ͔Βδϣϒௐ εέδϡʔϦϯά ɾਐḿ௨ ɾྃ௨ ɾࣦഊ௨ ϝʔϧɾεϥοΫ௨ػೳ ɾʑXFCٕज़͕৽͘͠ͳ͍ͬͯΔ
ɾ൚༻εύΠμʔ·ͩվम͢Δඞཁ͋Γ ɾίʔυͷมߋΛͦͷ··өͰ͖Δঢ়ଶ ɾςετͱ͔͍͢͠ ίʔυɾڥͷߋ৽͕༰қ DSPO 4&4 ίϯςφӡ༻
γεςϜʹ.VTUͰΈࠐΈ͍ͨػೳɾઃܭ 15 ɾ࢛ظʹҰճ࣮ߦ ɾδϣϒ੍ݶ͔Βδϣϒௐ εέδϡʔϦϯά ɾਐḿ௨ ɾྃ௨ ɾࣦഊ௨ ϝʔϧɾεϥοΫ௨ػೳ ɾʑXFCٕज़͕৽͘͠ͳ͍ͬͯΔ
ɾ൚༻εύΠμʔ·ͩվम͢Δඞཁ͋Γ ɾίʔυͷมߋΛͦͷ··өͰ͖Δঢ়ଶ ɾςετͱ͔͍͢͠ ίʔυɾڥͷߋ৽͕༰қ DSPO 4&4 ίϯςφӡ༻ ͜ΕΛຬͨ͢Α͏ʹઃܭͯ͠ΈΑ͏
16 ຊͷ͋Β͢͡ ఆظऩूγεςϜ࡞ͷഎܠ ఆظऩूγεςϜ࡞ͷཁ݅ γεςϜͷઃܭ ·ͱΊ
17 ͜ͷཁ݅ͳΒɾɾɾ &$ container 1 container 2 container 3 EPDLFSDPNQPTFͳͲΛར༻͠ɺ
՝Ͱར༻͍ͯ͠Δ&$্Ͱ࣮Ͱ͖ͦ͏
ઃܭͷશମ૾ ઃఆͷ ΓͱΓ ఆظಉظ &$ 63-Ϧετ֫ಘ ৄࡉใऔಘ σʔλՃ औಘ݁Ռ Ճޙ࠷ऴσʔλ
.BTUFS 4503& 4503& ΦϯΦϑͷ੍ޚ ϦΫΤετͷ੍ޚ ઃఆDB 18 ͋ΔఔճͬͨΒɺ ෳϒϥϯυΛ ·ͱΊͯॲཧ͢Δ ઃఆϑΝΠϧΛ શͯ֨ೲ
19 ͪΐͬͱ͍ͬͯͩ͘͞
20 ͜ͷ··ͷઃܭͩͱɾɾɾ &$ͷఀࢭ γεςϜͷఀࢭ
21 ͜Εͩͱσʔλ͕ߋ৽͞ΕͣʹࠔΔ
22 ͜ͷ··ͷઃܭͩͱɾɾɾ &$ͷఀࢭ γεςϜͷఀࢭ αʔόʔ֎Ͱ࣮ߦͰ͖ΔΑ͏ʹ͍ͨ͠ʂʂ ˠ&$4XJUI'BSHBUFΛͬͯΈΑ͏ʂ
-BNCEBWT&$WT&$4XJUI'BSHBUF 23 αʔόʔؔͰͷ ཧ͕ෆཁ ఏڙ͞ΕͨڥͰ࣮ߦ ࣗͰఆٛͨ͠ڥͰ࣮ߦ ଞʹ༷ʑͳ੍͋Γ -BNCEB &$4XJUI'BSHBUF &$
αʔόʔؔͰͷ ཧ͕ඞཁ ΑΓॊೈʹڥߏங͕Մೳ
24 'BSHBUFͷѱ͔ͬͨ෦ ࢀߟURLɿhttps://aws.amazon.com/jp/about-aws/whats-new/2019/01/announcing-aws-fargate-price-reduction-by-up-to-50-/ େ෯ʹίετ͕Լ͕ͬͨ ˠ&$4XJUI'BSHBUFͰߦͬͨΔʂ
ઃܭͷશମ૾ ઃఆͷ ΓͱΓ ఆظಉظ '"3 ("5& 63-Ϧετ֫ಘ ৄࡉใऔಘ σʔλՃ औಘ݁Ռ
Ճޙ࠷ऴσʔλ .BTUFS 4503& 4503& ΦϯΦϑͷ੍ޚɾϦΫΤετͷ ੍ޚ ઃఆDB 25 ͋ΔఔճͬͨΒɺ ෳϒϥϯυΛ ·ͱΊͯॲཧ͢Δ ઃఆϑΝΠϧΛ શͯ֨ೲ
.BTUFSίϯςφ εΫϨΠϐϯάॲཧͷ੍ޚ ઃఆ%#ʹج͍ͮͨ εέδϡʔϦϯά ਐḿ௨ػೳ 26 .BTUFS ֤ఔίϯςφ
63-औಘίϯςφ 27 .BTUFS 4औಘσʔλͷసૹ .BTUFS͔Βͷઃఆͷ సૹ ਐḿɾΤϥʔ௨ػೳ 63-औಘ ίϯςφ ֎εύΠμʔͷىಈ
औಘσʔλͷఆػೳ
ৄࡉใ֫ಘίϯςφ 28 .BTUFS ళฮใऔಘ ίϯςφ 4औಘσʔλͷసૹ .BTUFS͔Βͷઃఆͷ సૹ ਐḿɾΤϥʔ௨ػೳ εύΠμʔͷબ
औಘσʔλͷఆػೳ
σʔλՃίϯςφ 29 4औಘσʔλͷసૹ .BTUFS͔Βͷઃఆͷ సૹ ਐḿɾΤϥʔ௨ػೳ ཷ·ͬͨσʔλΛ·ͱΊΔ ػೳ ϑΥʔϚοτܗ .BTUFS
σʔλՃ ίϯςφ
30 ຊͷ͋Β͢͡ ఆظऩूγεςϜ࡞ͷഎܠ ఆظऩूγεςϜ࡞ͷཁ݅ γεςϜͷઃܭ ·ͱΊ
31 ·ͱΊ λεΫͷ࿈݁Ͱઃܭ͢Δ͜ͱ͕Ͱ͖Δ ɹ؆୯ͰΘ͔Γ͍͢ʂ 4&4ઃఆ͕؆୯͔ͩΒɺศརʂʂ ༨ஊ -BNCEBΛ͏ΑΓ੍ͳ͘ॊೈʹ ઃܭɾߏஙͰ͖Δ͔Βݸਓతʹخ͍͠ʂʂ αʔόʔͷཧΛߦ͏ඞཁͳ͍͔ΒϥΫʂ
2019/2/22 32 Thank you ΞυόΠε͓͍ͪͯ͠·͢