F#でスクレイピングをしてみた！（F# SCRAPING）

2016/07/03開催のF#談話室(23)の発表資料です。

Transcript

1. F# SCRAPING
   F#ͰεΫϨΠϐϯάΛͯ͠Έͨʂ
   callmekohei
   2016/07/03 Sun
   

   View Slide

2. Overview
   • ࣗݾ঺հ
   • εΫϨΠϐϯάͷ͖͔͚ͬ
   • HTMLͷ2ͭͷऔಘํ๏
   • HTML͔ΒσʔλʔΛͱΓͩͯ͠ΈΔ
   

   View Slide

3. callmkohei
   VBA
   
   3 years
   
   F#
   
   9 months
   ๻ͷ໊લ͸
   ϨΦɻ
   ΑΖ͘͠Ͷʂ
   

   View Slide

4. εΫϨΠϐϯάͷ͖͔͚ͬ
   

   View Slide

5. ͜Μͳײ͡Ͱ
   ϩτ̓ͷ༧ଌΛ
   ͍ͨ͠
   

   View Slide

6. ϩτ̓ͷ౰બ൪߸Λ
   ϗʔϜϖʔδΑΓ
   ೖख͍ͨ͠
   

   View Slide

7. View Slide

8. ϗʔϜϖʔδ͸
   HTMLιʔεͰ
   Ͱ͖͍ͯΔ
   

   View Slide

9. View Slide

10. ϗʔϜϖʔδͷσʔλʔΛ
    औಘ͢Δʹ͸
    HTMLιʔε͕ඞཁ
    

    View Slide

11. Ͳ͏΍ͬͯ
    HTMLιʔεΛ
    औಘ͢Δ͔
    

    View Slide

12. HTMLιʔεΛऔಘ͢Δ̎ͭͷํ๏
    

    View Slide

13. ͦͷ̍
    System.NetΛ͔ͭ͏
    

    View Slide

14. ͦͷ̎
    PhantomJS Λ͔ͭ͏
    

    View Slide

15. جຊతʹ͸
    System.NetͷΈͰ
    େৎ෉ʢͩͱࢥ;ɻɻɻʣ
    ͜͜͸
    ߟ͑Ͳ͜Ζ
    

    View Slide

16. Ͳ͏ͯ͠΋্ख͘
    औಘͰ͖ͳ͍৔߹͸
    PhantomJSΛ࢖ͬͯΈΔ
    

    View Slide

17. ͨͩ
    PhantomJS͸
    ஗͍ͷͰ͢
    

    View Slide

18. αʔόʔ΁ͷ઀ଓൺֱ
    System.Net: 2s
    PhantomJS: 7s
    ࣮ߦ࣌ؒ͸ࢀߟ஋
    

    View Slide

19. ͜͜Ͱ
    ࣮ࡍʹαʔόʔʹ
    ͭͳ͛ͯΈΔ
    demo
    

    View Slide

20. HTMLιʔε͔Β
    σʔλʔΛ
    ͱΓͩͯ͠ΈΔ
    

    View Slide

21. ͜͜Ͱ
    ศརͳ
    ϥΠϒϥϦ
    FSharp.Data
    

    View Slide

22. FSharp Data ͱ͸ʁ
    The F# Data library implements
    everything you need to access data
    in your F# applications and scripts.
    CSV, HTML, JSON and XML
    ʹରͯ͠ͷศརϥΠϒϥϦ
    http://fsharp.github.io/FSharp.Data/
    

    View Slide

23. a tag
    

    View Slide

24. ͜͜Ͱ
    a tag Λ
    ͱΓͩͯ͠ΈΔ
    demo
    

    View Slide

25. table tag
    

    View Slide

26. ͜͜Ͱ
    table tag Λ
    ͱΓͩͯ͠ΈΔ
    demo
    

    View Slide

27. εΫϨΠϐϯά͢Δͱ͖ʹ
    ศརͩͱࢥ͏ؔ਺Λ
    ·ͱΊͯΈͨ
    

    View Slide

28. SCRAPINGfs
    https://github.com/callmekohei/SCRAPINGfs
    

    View Slide

29. ͞Βʹɺɺ
    

    View Slide

30. FSharp.Data
    HtmlDocument.Load
    ͍͚ͯͳ͍ͱࢥͬͯΔͱ͜Ζ
    

    View Slide

31. ϒϥ΢βʔ͡Όͳ͍ͱ͸͔͡ΕΔ
    จࣈίʔυ͕͏·͘ॲཧ͞Εͳ͍
    Http Utilities Λ࢖͏͜ͱͰղܾʂ
    

    View Slide

32. Formॲཧ͕Ͱ͖ͳ͍
    

    View Slide

33. Formॲཧ͕Ͱ͖ͳ͍
    Http Utilities Λ࢖͏͜ͱͰͪΐͬͱղܾʂ
    

    View Slide

34. Formॲཧ͕Ͱ͖ͳ͍
    System.Net Λ࢖͏͜ͱͰղܾʂ
    

    View Slide

35. ؆୯ͳൺֱද
    

    View Slide

36. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠
    

    View Slide