俺が最初にヘッドレスChromeでクローラ作った 事になんねーかな

Transcript

1. Yuji Isobe Զ͕࠷ॳʹϔουϨε ChromeͰΫϩʔϥ࡞ͬͨ ࣄʹͳΜͶʔ͔ͳ NodeֶԂ29࣌ݶ໨

2. min e ϓϩδΣΫτϚωʔδϟʔ at
 
 @yujiosaka https://speakerdeck.com/yujiosaka/hitasurale-sitedeipuraningu

3. ✓ Կނ͍·͞ΒΫϩʔϥͳͷ͔ ✓ ԿΛ໨ࢦͯ͠࡞͔ͬͨ ✓ ԿΛߟ͑ͳ͕Β࡞͔ͬͨ ✓ ͜Ε͔ΒͷΫϩʔϥ ࠓճ͸ΫϩʔϥΛ࡞ͬͨ࿩

4. ڈ೥͸৭Μͳ͜ͱΛ΍ͬͨ…

5. ECZine࿈ࡌ http://eczine.jp/article/detail/4869

6. ECઐ໳ՈσϏϡʔ http://amzn.asia/aOkwFjH

7. ࠔͬͨ(´ɾωɾʆ)

8. ձࣾͰΤϯδχΞͩͱ
 ࢥΘΕͳ͘ͳ͖ͬͯͨorz

9. ݸࣾຖʹνϡʔχϯάΛߦ͏ Ӧۀಉߦʹग़͔͚Δ ৽نϓϩμΫτͷఏҊ ӦۀࢿྉΛॻ͖࢝ΊΔ ϓϨεϦϦʔεΛॻ͖࢝ΊΔ ௒͑ͯ͸͍͚ͳ͍Ұઢ ←AIΤϯδχΞͰ͢͠ ← ٕज़Ӧۀ͔ͳ ←

   BizDevͩΑͶ ← ͓ɺ͓͏… ←͍͋ͭ΋͏
 ɹΤϯδχΞ͡ΌͶʔΘ

10. Ͱ͖Ε͹ΤϯδχΞͱͯ͠
 Ұੜ൧Λ৯͍͖͍ͬͯͨ

11. ձࣾͰΤϯδχΞͱͯ͠ͷ ଚݫΛ࠶ͼऔΓ໭͢

12. ͦΜͳ͋Δ࣌…

13. ϔουϨεChromeΛ஌Δ https://developers.google.com/web/updates/2017/04/headless-chrome?hl=ja

14. ✓ Chrome͕ϔουϨεϞʔυͰىಈͰ͖Δ ✓ ChromeͷىಈΦϓγϣϯʹʮ--headessʯΛՃ͑Δ͚ͩ ✓ ୅දతͳϔουϨεϒϥ΢βͱ͍͑͹PhantomJS ✓ ߴ଎Ͱ҆ఆͯ͠ಈ࡞͢Δ ✓ ඪ४΁ͷରԠ͕ૣ͍ʢES2017΍Async-Await͕࢖͑Δʣ

    ✓ ओͳ༻్͸ςετࣗಈԽͱಈతΫϩʔϥ ϔουϨεChromeͱ͸

15. ✓ ੩తΫϩʔϥʢwget΍curl౳ʣ ✓ υΩϡϝϯτʢHTMLϑΝΠϧ౳ʣ΁ͷϦΫΤετͷΈ ✓ ϑΝΠϧΛύʔε͢Δ͚ͩͳͷͰߴ଎ʹಈ࡞͢Δ ✓ AngularJSɺReactɺVue.jsͰ࡞ΒΕͨSPAαΠτͰ͸ಈ࡞͠ͳ͍ ✓ ಈతΫϩʔϥʢPhantomJS΍ϔουϨεChrome౳ʣ

    ✓ ը૾΍JavaScript͓ΑͼCSSΛಡΈࠐΜͰඳը·Ͱߦ͏ ✓ JavaScriptͷ࣮ߦ·Ͱߦ͏ͷͰҰൠతʹ௿଎ ✓ SPAαΠτͰ΋ैདྷͷαΠτͱಉ͡Α͏ʹಈ࡞͢Δ ੩తΫϩʔϥ vs. ಈతΫϩʔϥ ※ উखͳ໋໊Ͱ͢

16. Chrome DevTools Protocol https://chromedevtools.github.io/devtools-protocol/ ✓ ࠷৽ͷ࢓༷͸ Chromiumίʔυ্ͷ
 JSONϑΝΠϧ ✓ 1࣌ؒʹ1ճGitHubͷ


    ϨϙδτϦʹίϐʔ
 ͞Ε͍ͯΔ

17. ϕϯνϚʔΫ https://hackernoon.com/benchmark-headless-chrome-vs-phantomjs-e7f44c6956c

18. RIP PhantomJS https://groups.google.com/forum/#!topic/phantomjs/9aI5d-LDuNE

19. ͜Ε͔Β࢝ΊΔͳΒ
 ϔουϨεChrome

20. ✓ API͕௿Ϩϕϧա͗ͯѻ͍͕೉͍͠ ✓ ࢓༷͕·ͩෆ҆ఆͰ௥͍͔͚Δͷ͕େม ✓ ηΩϡϦςΟͷϒϩοΫʹҾ͔͔ͬΔ ✓ Content Security PolicyͳͲɺϢʔβʔͷอޢ͕࡞ಈͯ͠͠·͏

    ✓ ΧδϡΞϧʹόάΛ౿Ή ✓ setRequestInterceptionͷ࣮૷͕·࣮ͩݧஈ֊ ͔͠͠໰୊΋ࢁੵΈ

21. ✓ Google ChromeνʔϜ͕
 ϝϯςφϯε ✓ ߴϨϕϧͷAPIͰϔουϨε
 Chrome͕ѻ͑Δϥούʔ ✓ 1݄ʹv1.0.0͕ϦϦʔε͞Εͨ ✓

    Slackάϧʔϓ΋࡞ΒΕ
 ରԠ΋ஸೡͰૣ͍ GoogleChrome / puppeteer https://github.com/GoogleChrome/puppeteer
22. None
23. None

24. ϔουϨε ChromeͰ Ϋϩʔϥ

25. ͬͯ΍ͭ ϝονϟ ྲྀߦͬͯΔ ʙʙʙ

26. Զ͕ ࠷ॳʹ ࡞ͬͨ ͜ͱʹ ͳΜͶ ʔ͔ͳ

27. ؾ͍ͮͨ

28. puppeteer / examples https://github.com/GoogleChrome/puppeteer/tree/master/examples

29. ʮ࢖ͬͯΈͨʯͱʮղઆʯ
 ͹͔ΓͰ࣮༻తͳ΋ͷ͸গͳ͍

30. ϔουϨεChromeͰ࠷ॳͷ ࣮༻తͳΫϩʔϥΛ࡞Ζ͏

31. ✓ طଘͷΫϩʔϥ͕PromiseʹରԠ͍ͯ͠ͳ͍ ✓ ෼ࢄ؀ڥͰಈ࡞͢ΔNode.jsͷΫϩʔϥ͕ͳ͔ͬͨ ͦͷଞͷཧ༝

32. ✓ ࣮༻తͳΫϩʔϥͱͯ͠ඞཁͳػೳΛຬ͍ͨͯ͠Δ ✓ υΩϡϝϯτ͕ӳޠͰॻ͔Ε͍ͯΔ ✓ ςετ͕े෼Χόʔ͞Ε͍ͯΔ ✓ ෼ࢄ؀ڥͰಈ࡞͢Δ ✓ API͸γϯϓϧʹอͭ

    ✓ puppeteer / examples ʹࡌͤͯ΋Β͏ ΰʔϧΛܾΊΔ

33. ͜ΕͰΤϯδχΞͱͯ͠ͷ
 ଚݫΛऔΓ໭͢

34. …

35. Ͱ͖ͨ https://github.com/yujiosaka/headless-chrome-crawler

36. ΰʔϧୡ੒ https://github.com/GoogleChrome/puppeteer/tree/master/examples

37. Google Developersʹసࡌ https://developers.google.com/web/tools/puppeteer/examples

38. ΞΫηε͕૿͑ͯϏϏΔ

39. ׬

40. )$$SBXMFSMBVODI \
    
    
    
    
    NBY%FQUI
    
    
    ୳ࡧ͢Δ࠷େͷਂ͞
    
    
    
    
    NBY$PODVSSFODZ
    
    
    ࠷େฒྻ਺
    
    
    
    
    BMMPXFE%PNBJOT
    <bXXXFNJODPKQ>
    
    ڐՄ͞Ε͍ͯΔυϝΠϯ
    
    
    
    
    FWBMVBUF1BHF
    
    
    

    bUJUMF UFYU
    
    ϖʔδ্ͰධՁ͞ΕΔؔ਺
    
    
    
    
    PO4VDDFTT
    SFTVMU
    
    \
    
    ੒ޭ࣌ʹධՁ͞ΕΔؔ਺
    
    
    
    
    
    
    
    
    DPOTPMFMPH A\SFTVMUPQUJPOTVSM^aU\SFTVMUSFTVMU^A 
    
    
    
    
    ^
    ^
    UIFO BTZOD
    DSBXMFS
    
    \
    
    
    
    
    DSBXMFSRVFVF IUUQTXXXFNJODPKQ 
    
    
    
    
    BXBJU
    DSBXMFSPO*EMF 
    
    
    
    
    BXBJU
    DSBXMFSDMPTF 
    ^  σϞ

41. Ϋϩʔϥ͕Ͱ͖Δ·Ͱ

42. ✓ ʮΫϩʔϦϯάʯͱʮεΫϨΠϐϯάʯ͸ҧ͏ ✓ ΫϩʔϦϯάɿHTML͔ΒϦϯΫΛݟ͚ͭΔ ✓ εΫϨΠϐϯάɿHTML͔Βཉ͍͠৘ใΛݟ͚ͭΔ ✓ ͦΕͧΕ୯ମͰଘࡏͯ͠΋ҙຯ͕ͳ͍ ࠷΋ϛχϚϧͳΫϩʔϥ

43. ೋͭͷڞ௨఺͸Կ͔

44. HTML͔ΒɹɹɹΛݟ͚ͭΔ

45. ͦΕͬͯjQueryͰΑ͘Ͷʁ

46. jQuery: true, ϖʔδʹK2VFSZΛࣗಈૠೖ v1.0.0ϦϦʔε

47. )$$SBXMFSMBVODI \
    
    
    
    
    K2VFSZ
    USVF
    
    
    
    
    FWBMVBUF1BHF
    
    
     bUJUMF UFYU
    
    
    
    
    PO4VDDFTT
    

    SFTVMU
    
    \
    
    
    
    
    
    
    
    
    DPOTPMFMPH A\SFTVMUPQUJPOTVSM^aU\SFTVMUSFTVMU^A 
    
    
    
    
    ^
    ^
    UIFO BTZOD
    DSBXMFS
    
    \
    
    
    
    
    DSBXMFSRVFVF IUUQTXXXFNJODPKQ 
    
    
    
    
    BXBJU
    DSBXMFSPO*EMF 
    
    
    
    
    BXBJU
    DSBXMFSDMPTF 
    ^  example

48. ✓ ੩తΫϩʔϥʹ׳Ε͍ͯΔͱɺ͛͢ʔ஗͘ײ͡Δ ✓ ͻͬͦΓΤϥʔͰࢭ·ͬͯͨΓ͢ΔͱϚδͰԜΉ ΠϥΠϥ͠ͳ͍Ϋϩʔϥ

49. ✓ λεΫΩϡʔͱΩϟογϡʹRedisΛ༻͍Δ ✓ ෳ਺ͷαʔόͰRedisΛڞ༗ ෼ࢄ؀ڥͰಈ࡞ͤ͞Δ

50. cache: new RedisCache(), ΩϟογϡετϨʔδʹ3FEJTΛࢦఆ v1.3.0ϦϦʔε

51. )$$SBXMFSMBVODI \
    
    
    
    
    DBDIF
    OFX
    3FEJT$BDIF \
    IPTU
    
    QPSU
    
    ^
    
    
    
    
    FWBMVBUF1BHF
    
    
     bUJUMF UFYU

    
    
    
    
    
    PO4VDDFTT
    SFTVMU
    
    \
    
    
    
    
    
    
    
    
    DPOTPMFMPH A\SFTVMUPQUJPOTVSM^aU\SFTVMUSFTVMU^A 
    
    
    
    
    ^
    ^
    UIFO BTZOD
    DSBXMFS
    
    \
    
    
    
    
    DSBXMFSRVFVF IUUQTXXXBNB[PODPKQ 
    
    
    
    
    BXBJU
    DSBXMFSPO*EMF 
    
    
    
    
    BXBJU
    DSBXMFSDMPTF 
    ^  example )$$SBXMFSMBVODI \
    
    
    
    
    DBDIF
    OFX
    3FEJT$BDIF \
    IPTU
    
    QPSU
    
    ^
    
    
    
    
    FWBMVBUF1BHF
    
    
     bUJUMF UFYU
    
    
    
    
    PO4VDDFTT
    SFTVMU
    
    \
    
    
    
    
    
    
    
    
    DPOTPMFMPH A\SFTVMUPQUJPOTVSM^aU\SFTVMUSFTVMU^A 
    
    
    
    
    ^
    ^
    UIFO BTZOD
    DSBXMFS
    
    \
    
    
    
    
    DSBXMFSRVFVF IUUQTXXXBNB[PODPKQ 
    
    
    
    
    BXBJU
    DSBXMFSPO*EMF 
    
    
    
    
    BXBJU
    DSBXMFSDMPTF 
    ^  )$$SBXMFSMBVODI \
    
    
    
    
    DBDIF
    OFX
    3FEJT$BDIF \
    IPTU
    
    QPSU
    
    ^
    
    
    
    
    FWBMVBUF1BHF
    
    
     bUJUMF UFYU
    
    
    
    
    PO4VDDFTT
    SFTVMU
    
    \
    
    
    
    
    
    
    
    
    DPOTPMFMPH A\SFTVMUPQUJPOTVSM^aU\SFTVMUSFTVMU^A 
    
    
    
    
    ^
    ^
    UIFO BTZOD
    DSBXMFS
    
    \
    
    
    
    
    DSBXMFSRVFVF IUUQTXXXBNB[PODPKQ 
    
    
    
    
    BXBJU
    DSBXMFSPO*EMF 
    
    
    
    
    BXBJU
    DSBXMFSDMPTF 
    ^ 

52. ✓ ෯༏ઌ୳ࡧʢBFSʣˍਂ͞༏ઌ୳ࡧʢDFSʣ ✓ robots.txtʹै͏ ✓ XMLαΠτϚοϓ୳ࡧ ✓ σόΠεͷΤϛϡϨʔγϣϯ ✓ ϖʔδͷεΫϦʔϯγϣοτ

    ✓ JSON/CSVग़ྗ ͦͷଞͷػೳ

53. ͜Ε͔ΒͷΫϩʔϥ

54. ✓ ͜ͷΫϩʔϥͷͨΊʹαʔόʔ100୆ฒ΂ͯ
 ΫϩʔϦϯά͢ΔౕͳΜ͍ͯͳ͍͠ΊΜͲ͍͘͞ ✓ ίϚϯυҰൃͰ෼ࢄ؀ڥʹσϓϩΠͯ͠ཉ͍͠ ݱࡏͷ՝୊

55. None

56. ✓ ߏ੒؅ཧʰπʔϧʱʹ͍ۙ ✓ AWS LambdaɺAzure Functionsɺ
 Google CloudFunctionsΛ༰қʹσϓϩΠɾ࣮ߦ ✓ Node.js,

    Python, Java, Scala, C#, F#, Go, Groovy, Kotlin, PHP & SwiftΛαϙʔτ ✓ ศརͳϓϥάΠϯ΋ͨ͘͞Μ Serverless Frameworkͱ͸

57. yarn (npm run) deploy yarn (npm run) start v2.0.0 will

    be… "84
    -BNCEBʹσϓϩΠ  ฒྻͰΫϩʔϦϯά։࢝

58. Զ͕࠷ॳʹϔουϨε ChromeͰΫϩʔϥ ࡞ͬͨࣄʹͳΜͶʔ͔ͳ

59. Զ͕࠷ॳʹϔουϨε ChromeͰ࣮༻తͳΫϩʔϥ ࡞ͬͨࣄʹͳΜͶʔ͔ͳ

60. ͚ͩͲຊ౰͸ɺ࢓ࣄͰ
 ΋ͬͱίʔυΛॻ͖͍ͨ

61. WE ARE HIRING https://www.emin.co.jp/blog/news/1527/ ηʔϧε΋