Tweet
Tweet

Slide 1

Slide 1 text

Yuji Isobe Զ͕࠷ॳʹϔουϨε ChromeͰΫϩʔϥ࡞ͬͨ ࣄʹͳΜͶʔ͔ͳ NodeֶԂ29࣌ݶ໨

Slide 2

Slide 2 text

min e ϓϩδΣΫτϚωʔδϟʔ at
 
 @yujiosaka https://speakerdeck.com/yujiosaka/hitasurale-sitedeipuraningu

Slide 3

Slide 3 text

✓ Կނ͍·͞ΒΫϩʔϥͳͷ͔ ✓ ԿΛ໨ࢦͯ͠࡞͔ͬͨ ✓ ԿΛߟ͑ͳ͕Β࡞͔ͬͨ ✓ ͜Ε͔ΒͷΫϩʔϥ ࠓճ͸ΫϩʔϥΛ࡞ͬͨ࿩

Slide 4

Slide 4 text

ڈ೥͸৭Μͳ͜ͱΛ΍ͬͨ…

Slide 5

Slide 5 text

ECZine࿈ࡌ http://eczine.jp/article/detail/4869

Slide 6

Slide 6 text

ECઐ໳ՈσϏϡʔ http://amzn.asia/aOkwFjH

Slide 7

Slide 7 text

ࠔͬͨ(´ɾωɾʆ)

Slide 8

Slide 8 text

ձࣾͰΤϯδχΞͩͱ
 ࢥΘΕͳ͘ͳ͖ͬͯͨorz

Slide 9

Slide 9 text

ݸࣾຖʹνϡʔχϯάΛߦ͏ Ӧۀಉߦʹग़͔͚Δ ৽نϓϩμΫτͷఏҊ ӦۀࢿྉΛॻ͖࢝ΊΔ ϓϨεϦϦʔεΛॻ͖࢝ΊΔ ௒͑ͯ͸͍͚ͳ͍Ұઢ ←AIΤϯδχΞͰ͢͠ ← ٕज़Ӧۀ͔ͳ ← BizDevͩΑͶ ← ͓ɺ͓͏… ←͍͋ͭ΋͏
 ɹΤϯδχΞ͡ΌͶʔΘ

Slide 10

Slide 10 text

Ͱ͖Ε͹ΤϯδχΞͱͯ͠
 Ұੜ൧Λ৯͍͖͍ͬͯͨ

Slide 11

Slide 11 text

ձࣾͰΤϯδχΞͱͯ͠ͷ ଚݫΛ࠶ͼऔΓ໭͢

Slide 12

Slide 12 text

ͦΜͳ͋Δ࣌…

Slide 13

Slide 13 text

ϔουϨεChromeΛ஌Δ https://developers.google.com/web/updates/2017/04/headless-chrome?hl=ja

Slide 14

Slide 14 text

✓ Chrome͕ϔουϨεϞʔυͰىಈͰ͖Δ ✓ ChromeͷىಈΦϓγϣϯʹʮ--headessʯΛՃ͑Δ͚ͩ ✓ ୅දతͳϔουϨεϒϥ΢βͱ͍͑͹PhantomJS ✓ ߴ଎Ͱ҆ఆͯ͠ಈ࡞͢Δ ✓ ඪ४΁ͷରԠ͕ૣ͍ʢES2017΍Async-Await͕࢖͑Δʣ ✓ ओͳ༻్͸ςετࣗಈԽͱಈతΫϩʔϥ ϔουϨεChromeͱ͸

Slide 15

Slide 15 text

✓ ੩తΫϩʔϥʢwget΍curl౳ʣ ✓ υΩϡϝϯτʢHTMLϑΝΠϧ౳ʣ΁ͷϦΫΤετͷΈ ✓ ϑΝΠϧΛύʔε͢Δ͚ͩͳͷͰߴ଎ʹಈ࡞͢Δ ✓ AngularJSɺReactɺVue.jsͰ࡞ΒΕͨSPAαΠτͰ͸ಈ࡞͠ͳ͍ ✓ ಈతΫϩʔϥʢPhantomJS΍ϔουϨεChrome౳ʣ ✓ ը૾΍JavaScript͓ΑͼCSSΛಡΈࠐΜͰඳը·Ͱߦ͏ ✓ JavaScriptͷ࣮ߦ·Ͱߦ͏ͷͰҰൠతʹ௿଎ ✓ SPAαΠτͰ΋ैདྷͷαΠτͱಉ͡Α͏ʹಈ࡞͢Δ ੩తΫϩʔϥ vs. ಈతΫϩʔϥ ※ উखͳ໋໊Ͱ͢

Slide 16

Slide 16 text

Chrome DevTools Protocol https://chromedevtools.github.io/devtools-protocol/ ✓ ࠷৽ͷ࢓༷͸ Chromiumίʔυ্ͷ
 JSONϑΝΠϧ ✓ 1࣌ؒʹ1ճGitHubͷ
 ϨϙδτϦʹίϐʔ
 ͞Ε͍ͯΔ

Slide 17

Slide 17 text

ϕϯνϚʔΫ https://hackernoon.com/benchmark-headless-chrome-vs-phantomjs-e7f44c6956c

Slide 18

Slide 18 text

RIP PhantomJS https://groups.google.com/forum/#!topic/phantomjs/9aI5d-LDuNE

Slide 19

Slide 19 text

͜Ε͔Β࢝ΊΔͳΒ
 ϔουϨεChrome

Slide 20

Slide 20 text

✓ API͕௿Ϩϕϧա͗ͯѻ͍͕೉͍͠ ✓ ࢓༷͕·ͩෆ҆ఆͰ௥͍͔͚Δͷ͕େม ✓ ηΩϡϦςΟͷϒϩοΫʹҾ͔͔ͬΔ ✓ Content Security PolicyͳͲɺϢʔβʔͷอޢ͕࡞ಈͯ͠͠·͏ ✓ ΧδϡΞϧʹόάΛ౿Ή ✓ setRequestInterceptionͷ࣮૷͕·࣮ͩݧஈ֊ ͔͠͠໰୊΋ࢁੵΈ

Slide 21

Slide 21 text

✓ Google ChromeνʔϜ͕
 ϝϯςφϯε ✓ ߴϨϕϧͷAPIͰϔουϨε
 Chrome͕ѻ͑Δϥούʔ ✓ 1݄ʹv1.0.0͕ϦϦʔε͞Εͨ ✓ Slackάϧʔϓ΋࡞ΒΕ
 ରԠ΋ஸೡͰૣ͍ GoogleChrome / puppeteer https://github.com/GoogleChrome/puppeteer

Slide 22

Slide 22 text

No content

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

ϔουϨε ChromeͰ Ϋϩʔϥ

Slide 25

Slide 25 text

ͬͯ΍ͭ ϝονϟ ྲྀߦͬͯΔ ʙʙʙ

Slide 26

Slide 26 text

Զ͕ ࠷ॳʹ ࡞ͬͨ ͜ͱʹ ͳΜͶ ʔ͔ͳ

Slide 27

Slide 27 text

ؾ͍ͮͨ

Slide 28

Slide 28 text

puppeteer / examples https://github.com/GoogleChrome/puppeteer/tree/master/examples

Slide 29

Slide 29 text

ʮ࢖ͬͯΈͨʯͱʮղઆʯ
 ͹͔ΓͰ࣮༻తͳ΋ͷ͸গͳ͍

Slide 30

Slide 30 text

ϔουϨεChromeͰ࠷ॳͷ ࣮༻తͳΫϩʔϥΛ࡞Ζ͏

Slide 31

Slide 31 text

✓ طଘͷΫϩʔϥ͕PromiseʹରԠ͍ͯ͠ͳ͍ ✓ ෼ࢄ؀ڥͰಈ࡞͢ΔNode.jsͷΫϩʔϥ͕ͳ͔ͬͨ ͦͷଞͷཧ༝

Slide 32

Slide 32 text

✓ ࣮༻తͳΫϩʔϥͱͯ͠ඞཁͳػೳΛຬ͍ͨͯ͠Δ ✓ υΩϡϝϯτ͕ӳޠͰॻ͔Ε͍ͯΔ ✓ ςετ͕े෼Χόʔ͞Ε͍ͯΔ ✓ ෼ࢄ؀ڥͰಈ࡞͢Δ ✓ API͸γϯϓϧʹอͭ ✓ puppeteer / examples ʹࡌͤͯ΋Β͏ ΰʔϧΛܾΊΔ

Slide 33

Slide 33 text

͜ΕͰΤϯδχΞͱͯ͠ͷ
 ଚݫΛऔΓ໭͢

Slide 34

Slide 34 text

…

Slide 35

Slide 35 text

Ͱ͖ͨ https://github.com/yujiosaka/headless-chrome-crawler

Slide 36

Slide 36 text

ΰʔϧୡ੒ https://github.com/GoogleChrome/puppeteer/tree/master/examples

Slide 37

Slide 37 text

Google Developersʹసࡌ https://developers.google.com/web/tools/puppeteer/examples

Slide 38

Slide 38 text

ΞΫηε͕૿͑ͯϏϏΔ

Slide 39

Slide 39 text

׬

Slide 40

Slide 40 text

)$$SBXMFSMBVODI \ NBY%FQUI ୳ࡧ͢Δ࠷େͷਂ͞ NBY$PODVSSFODZ ࠷େฒྻ਺ BMMPXFE%PNBJOT ڐՄ͞Ε͍ͯΔυϝΠϯ FWBMVBUF1BHF bUJUMF UFYU ϖʔδ্ͰධՁ͞ΕΔؔ਺ PO4VDDFTT SFTVMU\੒ޭ࣌ʹධՁ͞ΕΔؔ਺ DPOTPMFMPH A\SFTVMUPQUJPOTVSM^aU\SFTVMUSFTVMU^A ^ ^ UIFO BTZODDSBXMFS\ DSBXMFSRVFVF IUUQTXXXFNJODPKQ BXBJUDSBXMFSPO*EMF BXBJUDSBXMFSDMPTF ^ σϞ

Slide 41

Slide 41 text

Ϋϩʔϥ͕Ͱ͖Δ·Ͱ

Slide 42

Slide 42 text

✓ ʮΫϩʔϦϯάʯͱʮεΫϨΠϐϯάʯ͸ҧ͏ ✓ ΫϩʔϦϯάɿHTML͔ΒϦϯΫΛݟ͚ͭΔ ✓ εΫϨΠϐϯάɿHTML͔Βཉ͍͠৘ใΛݟ͚ͭΔ ✓ ͦΕͧΕ୯ମͰଘࡏͯ͠΋ҙຯ͕ͳ͍ ࠷΋ϛχϚϧͳΫϩʔϥ

Slide 43

Slide 43 text

ೋͭͷڞ௨఺͸Կ͔

Slide 44

Slide 44 text

HTML͔ΒɹɹɹΛݟ͚ͭΔ

Slide 45

Slide 45 text

ͦΕͬͯjQueryͰΑ͘Ͷʁ

Slide 46

Slide 46 text

jQuery: true, ϖʔδʹK2VFSZΛࣗಈૠೖ v1.0.0ϦϦʔε

Slide 47

Slide 47 text

)$$SBXMFSMBVODI \ K2VFSZUSVF FWBMVBUF1BHF bUJUMF UFYU PO4VDDFTT SFTVMU\ DPOTPMFMPH A\SFTVMUPQUJPOTVSM^aU\SFTVMUSFTVMU^A ^ ^ UIFO BTZODDSBXMFS\ DSBXMFSRVFVF IUUQTXXXFNJODPKQ BXBJUDSBXMFSPO*EMF BXBJUDSBXMFSDMPTF ^ example

Slide 48

Slide 48 text

✓ ੩తΫϩʔϥʹ׳Ε͍ͯΔͱɺ͛͢ʔ஗͘ײ͡Δ ✓ ͻͬͦΓΤϥʔͰࢭ·ͬͯͨΓ͢ΔͱϚδͰԜΉ ΠϥΠϥ͠ͳ͍Ϋϩʔϥ

Slide 49

Slide 49 text

✓ λεΫΩϡʔͱΩϟογϡʹRedisΛ༻͍Δ ✓ ෳ਺ͷαʔόͰRedisΛڞ༗ ෼ࢄ؀ڥͰಈ࡞ͤ͞Δ

Slide 50

Slide 50 text

cache: new RedisCache(), ΩϟογϡετϨʔδʹ3FEJTΛࢦఆ v1.3.0ϦϦʔε

Slide 51

Slide 51 text

)$$SBXMFSMBVODI \ DBDIFOFX3FEJT$BDIF \IPTU QPSU^ FWBMVBUF1BHF bUJUMF UFYU PO4VDDFTT SFTVMU\ DPOTPMFMPH A\SFTVMUPQUJPOTVSM^aU\SFTVMUSFTVMU^A ^ ^ UIFO BTZODDSBXMFS\ DSBXMFSRVFVF IUUQTXXXBNB[PODPKQ BXBJUDSBXMFSPO*EMF BXBJUDSBXMFSDMPTF ^ example )$$SBXMFSMBVODI \ DBDIFOFX3FEJT$BDIF \IPTU QPSU^ FWBMVBUF1BHF bUJUMF UFYU PO4VDDFTT SFTVMU\ DPOTPMFMPH A\SFTVMUPQUJPOTVSM^aU\SFTVMUSFTVMU^A ^ ^ UIFO BTZODDSBXMFS\ DSBXMFSRVFVF IUUQTXXXBNB[PODPKQ BXBJUDSBXMFSPO*EMF BXBJUDSBXMFSDMPTF ^ )$$SBXMFSMBVODI \ DBDIFOFX3FEJT$BDIF \IPTU QPSU^ FWBMVBUF1BHF bUJUMF UFYU PO4VDDFTT SFTVMU\ DPOTPMFMPH A\SFTVMUPQUJPOTVSM^aU\SFTVMUSFTVMU^A ^ ^ UIFO BTZODDSBXMFS\ DSBXMFSRVFVF IUUQTXXXBNB[PODPKQ BXBJUDSBXMFSPO*EMF BXBJUDSBXMFSDMPTF ^

Slide 52

Slide 52 text

✓ ෯༏ઌ୳ࡧʢBFSʣˍਂ͞༏ઌ୳ࡧʢDFSʣ ✓ robots.txtʹै͏ ✓ XMLαΠτϚοϓ୳ࡧ ✓ σόΠεͷΤϛϡϨʔγϣϯ ✓ ϖʔδͷεΫϦʔϯγϣοτ ✓ JSON/CSVग़ྗ ͦͷଞͷػೳ

Slide 53

Slide 53 text

͜Ε͔ΒͷΫϩʔϥ

Slide 54

Slide 54 text

✓ ͜ͷΫϩʔϥͷͨΊʹαʔόʔ100୆ฒ΂ͯ
 ΫϩʔϦϯά͢ΔౕͳΜ͍ͯͳ͍͠ΊΜͲ͍͘͞ ✓ ίϚϯυҰൃͰ෼ࢄ؀ڥʹσϓϩΠͯ͠ཉ͍͠ ݱࡏͷ՝୊

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

✓ ߏ੒؅ཧʰπʔϧʱʹ͍ۙ ✓ AWS LambdaɺAzure Functionsɺ
 Google CloudFunctionsΛ༰қʹσϓϩΠɾ࣮ߦ ✓ Node.js, Python, Java, Scala, C#, F#, Go, Groovy, Kotlin, PHP & SwiftΛαϙʔτ ✓ ศརͳϓϥάΠϯ΋ͨ͘͞Μ Serverless Frameworkͱ͸

Slide 57

Slide 57 text

yarn (npm run) deploy yarn (npm run) start v2.0.0 will be… "84-BNCEBʹσϓϩΠ ฒྻͰΫϩʔϦϯά։࢝

Slide 58

Slide 58 text

Զ͕࠷ॳʹϔουϨε ChromeͰΫϩʔϥ ࡞ͬͨࣄʹͳΜͶʔ͔ͳ

Slide 59

Slide 59 text

Զ͕࠷ॳʹϔουϨε ChromeͰ࣮༻తͳΫϩʔϥ ࡞ͬͨࣄʹͳΜͶʔ͔ͳ

Slide 60

Slide 60 text

͚ͩͲຊ౰͸ɺ࢓ࣄͰ
 ΋ͬͱίʔυΛॻ͖͍ͨ

Slide 61

Slide 61 text

WE ARE HIRING https://www.emin.co.jp/blog/news/1527/ ηʔϧε΋