scraping system blueprint using scrapinghub and fargate for collecting POI data

by wataru

Slide 1

Slide 1 text

4DSBQJOH10*  ECS with Fargate 使   定期収集設計

Slide 2

Slide 2 text

2 ొஃͷ໨త "84ͷ஌ࣝͷ͋ΔํʑͷલͰ஻Δ ΞυόΠεΛ͍͖ͨͩɺ ΑΓྑ͍΋ͷΛ࡞੒

Slide 3

Slide 3 text

3 ຊ೔ͷ͋Β͢͡ ఆظऩूγεςϜ࡞੒ͷഎܠ ఆظऩूγεςϜ࡞੒ͷཁ݅ γεςϜͷઃܭ ·ͱΊ

Slide 4

Slide 4 text

4 ຊ೔ͷ͋Β͢͡ ఆظऩूγεςϜ࡞੒ͷഎܠ ఆظऩूγεςϜ࡞੒ͷཁ݅ γεςϜͷઃܭ ·ͱΊ

Slide 5

Slide 5 text

5 ༷ʑͳछྨͷ10*Λऩूͯ͠·͢ γϣοϐϯάϞʔϧ ؍ޫ஍ Πϕϯτܥࢪઃ νΣʔϯళ

Slide 6

Slide 6 text

6 ༷ʑͳछྨͷ10*Λऩूͯ͠·͢ γϣοϐϯάϞʔϧ ؍ޫ஍ Πϕϯτܥࢪઃ νΣʔϯళ ࠓ೔ͷϝΠϯ εΫϨΠϐϯάͰऔಘ͍ͯ͠·͢

Slide 7

Slide 7 text

ఆظऩू͢Δ໨త 7 νΣʔϯళ͸Α͘ঢ়ଶมԽ͢Δ ด࠯ళฮ΍৽نళฮͷ೺Ѳ΋Մೳ σʔλͷ઱౓Λอͭ σʔλऩू࡞ۀͷࣗಈԽ σʔλऩूΛࣗಈԽ͢Δ͜ͱͰɺ ௕ظతͳ޻਺࡟ݮΛ࣮ݱ͢Δ ޮՌੑ ޮ཰ੑ

Slide 8

Slide 8 text

8 ຊ೔ͷ͋Β͢͡ ఆظऩूγεςϜ࡞੒ͷഎܠ ఆظऩूγεςϜ࡞੒ͷཁ݅ γεςϜͷઃܭ ·ͱΊ

Slide 9

Slide 9 text

10*σʔλϕʔεߏங·Ͱͷ̏εςοϓ ళฮ63-Ϧετ Λऔಘ ళฮৄࡉ৘ใ  Λऔಘ औಘͨ͠  σʔλͷՃ޻ ׬੒ 9 45&1 45&1 45&1

Slide 10

Slide 10 text

ݱঢ় 10 ɾ൚༻εύΠμʔ  ɾ४൚༻εύΠμʔ  ɾ֎஫εύΠμʔ  ɾTDSBQJOHIVC&$্ʹଘࡏ εύΠμʔ͕ෳ਺ݸଘࡏ εύΠμʔʹઃఆ஋ΛૹΔ͜ͱͰɺ  εύΠμʔىಈ ઃఆ஋ͰεύΠμʔΛ࣮ߦ ۀ຿ҕୗ͞Μʹ࡞੒ґཔ͍ͯ͠Δ  ઃఆ஋͸%SPQCPY্ʹ֨ೲ ESPQCPYͷར༻ ࢖༻εύΠμʔͷࢦఆػೳ ىಈεύΠμʔͱઃఆ஋ͷ࿈ܞ ESPQCPYͱ"84ͱͷ࿈ܞ

Slide 11

Slide 11 text

γεςϜʹ.VTUͰ૊ΈࠐΈ͍ͨػೳɾઃܭ 11 ɾ࢛൒ظʹҰճ࣮ߦ  ɾδϣϒ੍ݶ਺͔Βδϣϒௐ੔ εέδϡʔϦϯά ɾਐḿ௨஌  ɾ׬ྃ௨஌  ɾࣦഊ௨஌ ϝʔϧɾεϥοΫ௨஌ػೳ ɾ೔ʑXFCٕज़͕৽͘͠ͳ͍ͬͯΔ  ɾ൚༻εύΠμʔ͸·ͩվम͢Δඞཁ͋Γ  ɾίʔυͷมߋΛͦͷ··൓өͰ͖Δঢ়ଶ  ɾςετͱ͔΋͠΍͘͢  ɾ։ൃͱຊ൪ͷҠಈΛ؆୯ʹ ίʔυɾ؀ڥͷߋ৽͕༰қ

Slide 12

Slide 12 text

γεςϜʹ.VTUͰ૊ΈࠐΈ͍ͨػೳɾઃܭ 12 ɾ࢛൒ظʹҰճ࣮ߦ  ɾδϣϒ੍ݶ਺͔Βδϣϒௐ੔ εέδϡʔϦϯά ɾਐḿ௨஌  ɾ׬ྃ௨஌  ɾࣦഊ௨஌ ϝʔϧɾεϥοΫ௨஌ػೳ ɾ೔ʑXFCٕज़͕৽͘͠ͳ͍ͬͯΔ  ɾ൚༻εύΠμʔ͸·ͩվम͢Δඞཁ͋Γ  ɾίʔυͷมߋΛͦͷ··൓өͰ͖Δঢ়ଶ  ɾςετͱ͔΋͠΍͘͢  ɾ։ൃͱຊ൪ͷҠಈΛ؆୯ʹ ίʔυɾ؀ڥͷߋ৽͕༰қ DSPO౳ 4&4

Slide 13

Slide 13 text

13 EPDLFSͷಛ௃ *OGSBTUSVDUVSFBT$PEF *NNVUBCMF*OGSBTUSVDUVSF %PDLFSGJMFͰόʔδϣϯ؅ཧ͕Մೳ ؀ڥͷઃఆ؅ཧ͕ෆཁ

Slide 14

Slide 14 text

γεςϜʹ.VTUͰ૊ΈࠐΈ͍ͨػೳɾઃܭ 14 ɾ࢛൒ظʹҰճ࣮ߦ  ɾδϣϒ੍ݶ਺͔Βδϣϒௐ੔ εέδϡʔϦϯά ɾਐḿ௨஌  ɾ׬ྃ௨஌  ɾࣦഊ௨஌ ϝʔϧɾεϥοΫ௨஌ػೳ ɾ೔ʑXFCٕज़͕৽͘͠ͳ͍ͬͯΔ  ɾ൚༻εύΠμʔ͸·ͩվम͢Δඞཁ͋Γ  ɾίʔυͷมߋΛͦͷ··൓өͰ͖Δঢ়ଶ  ɾςετͱ͔΋͠΍͍͢ ίʔυɾ؀ڥͷߋ৽͕༰қ DSPO౳ 4&4 ίϯςφӡ༻

Slide 15

Slide 15 text

γεςϜʹ.VTUͰ૊ΈࠐΈ͍ͨػೳɾઃܭ 15 ɾ࢛൒ظʹҰճ࣮ߦ  ɾδϣϒ੍ݶ਺͔Βδϣϒௐ੔ εέδϡʔϦϯά ɾਐḿ௨஌  ɾ׬ྃ௨஌  ɾࣦഊ௨஌ ϝʔϧɾεϥοΫ௨஌ػೳ ɾ೔ʑXFCٕज़͕৽͘͠ͳ͍ͬͯΔ  ɾ൚༻εύΠμʔ͸·ͩվम͢Δඞཁ͋Γ  ɾίʔυͷมߋΛͦͷ··൓өͰ͖Δঢ়ଶ  ɾςετͱ͔΋͠΍͍͢ ίʔυɾ؀ڥͷߋ৽͕༰қ DSPO౳ 4&4 ίϯςφӡ༻ ͜ΕΛຬͨ͢Α͏ʹઃܭͯ͠ΈΑ͏

Slide 16

Slide 16 text

16 ຊ೔ͷ͋Β͢͡ ఆظऩूγεςϜ࡞੒ͷഎܠ ఆظऩूγεςϜ࡞੒ͷཁ݅ γεςϜͷઃܭ ·ͱΊ

Slide 17

Slide 17 text

17 ͜ͷཁ݅ͳΒɾɾɾ &$ container 1 container 2 container 3 EPDLFSDPNQPTFͳͲΛར༻͠ɺ ՝Ͱར༻͍ͯ͠Δ&$্Ͱ࣮૷Ͱ͖ͦ͏

Slide 18

Slide 18 text

ઃܭͷશମ૾ ઃఆ஋ͷ ΍ΓͱΓ ఆظಉظ &$ 63-Ϧετ֫ಘ ৄࡉ৘ใऔಘ σʔλՃ޻ औಘ݁Ռ Ճ޻ޙ࠷ऴσʔλ .BTUFS 4503& 4503& ΦϯΦϑͷ੍ޚ  ϦΫΤετͷ੍ޚ ઃఆ஋DB 18 ͋Δఔ౓ճͬͨΒɺ ෳ਺ϒϥϯυΛ  ·ͱΊͯॲཧ͢Δ ઃఆ஋ϑΝΠϧΛ  શͯ֨ೲ

Slide 19

Slide 19 text

19 ͪΐͬͱ଴͍ͬͯͩ͘͞

Slide 20

Slide 20 text

20 ͜ͷ··ͷઃܭͩͱɾɾɾ &$ͷఀࢭ γεςϜͷఀࢭ

Slide 21

Slide 21 text

21 ͜Εͩͱσʔλ͕ߋ৽͞ΕͣʹࠔΔ

Slide 22

Slide 22 text

22 ͜ͷ··ͷઃܭͩͱɾɾɾ &$ͷఀࢭ γεςϜͷఀࢭ αʔόʔ֎Ͱ࣮ߦͰ͖ΔΑ͏ʹ͍ͨ͠ʂʂ ˠ&$4XJUI'BSHBUFΛ࢖ͬͯΈΑ͏ʂ

Slide 23

Slide 23 text

-BNCEBWT&$WT&$4XJUI'BSHBUF 23 αʔόʔؔ܎Ͱͷ  ؅ཧ͕ෆཁ ఏڙ͞Εͨ؀ڥͰ࣮ߦ ࣗ෼Ͱఆٛͨ͠؀ڥͰ࣮ߦ ଞʹ΋༷ʑͳ੍໿͋Γ -BNCEB &$4XJUI'BSHBUF &$ αʔόʔؔ܎Ͱͷ ؅ཧ͕ඞཁ ΑΓॊೈʹ؀ڥߏங͕Մೳ

Slide 24

Slide 24 text

24 'BSHBUFͷѱ͔ͬͨ෦෼ ࢀߟURLɿhttps://aws.amazon.com/jp/about-aws/whats-new/2019/01/announcing-aws-fargate-price-reduction-by-up-to-50-/ େ෯ʹίετ͕Լ͕ͬͨ ˠ&$4XJUI'BSHBUFͰߦͬͨΔʂ

Slide 25

Slide 25 text

ઃܭͷશମ૾ ઃఆ஋ͷ ΍ΓͱΓ ఆظಉظ '"3 ("5& 63-Ϧετ֫ಘ ৄࡉ৘ใऔಘ σʔλՃ޻ औಘ݁Ռ Ճ޻ޙ࠷ऴσʔλ .BTUFS 4503& 4503& ΦϯΦϑͷ੍ޚɾϦΫΤετͷ ੍ޚ ઃఆ஋DB 25 ͋Δఔ౓ճͬͨΒɺ ෳ਺ϒϥϯυΛ  ·ͱΊͯॲཧ͢Δ ઃఆ஋ϑΝΠϧΛ  શͯ֨ೲ

Slide 26

Slide 26 text

.BTUFSίϯςφ εΫϨΠϐϯάॲཧͷ੍ޚ ઃఆ஋%#ʹج͍ͮͨ εέδϡʔϦϯά ਐḿ௨஌ػೳ 26 .BTUFS ֤޻ఔίϯςφ

Slide 27

Slide 27 text

63-औಘίϯςφ 27 .BTUFS 4΁औಘσʔλͷసૹ .BTUFS͔Βͷઃఆ஋ͷ సૹ ਐḿɾΤϥʔ௨஌ػೳ 63-औಘ  ίϯςφ ֎஫εύΠμʔͷىಈ औಘσʔλͷ൑ఆػೳ

Slide 28

Slide 28 text

ৄࡉ৘ใ֫ಘίϯςφ 28 .BTUFS ళฮ৘ใऔಘ  ίϯςφ 4΁औಘσʔλͷసૹ .BTUFS͔Βͷઃఆ஋ͷ సૹ ਐḿɾΤϥʔ௨஌ػೳ εύΠμʔͷબ୒ औಘσʔλͷ൑ఆػೳ

Slide 29

Slide 29 text

σʔλՃ޻ίϯςφ 29 4΁औಘσʔλͷసૹ .BTUFS͔Βͷઃఆ஋ͷ సૹ ਐḿɾΤϥʔ௨஌ػೳ ཷ·ͬͨσʔλΛ·ͱΊΔ ػೳ ϑΥʔϚοτ੔ܗ .BTUFS σʔλՃ޻  ίϯςφ

Slide 30

Slide 30 text

30 ຊ೔ͷ͋Β͢͡ ఆظऩूγεςϜ࡞੒ͷഎܠ ఆظऩूγεςϜ࡞੒ͷཁ݅ γεςϜͷઃܭ ·ͱΊ

Slide 31

Slide 31 text

31 ·ͱΊ λεΫͷ࿈݁Ͱઃܭ͢Δ͜ͱ͕Ͱ͖Δ ɹ؆୯ͰΘ͔Γ΍͍͢ʂ 4&4͸ઃఆ͕؆୯͔ͩΒɺศརʂʂ ༨ஊ -BNCEBΛ࢖͏ΑΓ΋੍໿ͳ͘ॊೈʹ ઃܭɾߏஙͰ͖Δ͔Βݸਓతʹ͸خ͍͠ʂʂ αʔόʔͷ؅ཧΛߦ͏ඞཁͳ͍͔ΒϥΫʂ

Slide 32

Slide 32 text

2019/2/22 32 Thank you ΞυόΠε͓଴͍ͪͯ͠·͢