Rubyで機械学習が出来る 未来を目指すRed Data Toolsの 現状と今後について

Transcript

1. RubyͰػցֶश͕ग़དྷΔ ະདྷΛ໨ࢦ͢Red Data Toolsͷ ݱঢ়ͱࠓޙʹ͍ͭͯ by Yusaku Hatanaka @RubyWorld Conference

   2017 1

2. self.intoroduction { name: "Yusaku Hatanaka", twitter: "@hatappi", github: "hatappi", hatena:

   "hatappi1225", company: "Speee, Inc." languages: %w(ruby go python), icon: "ɹ " } 2

3. ࣗݾ঺հ w ാத
   ༔࡞
   w 5XJUUFSɺ(JUIVC
   
   IBUBQQJ
 ͸ͯͳϒϩά
   
   
   
   
   
   
   IBUBQQJ
   w גࣜձࣾ4QFFF
   w σδλϧίϯαϧςΟϯάࣄۀຊ෦


   ΞυςΫࣄۀ෦
   6;06ࣄۀ
   w 6;06ࣄۀΤϯδχΞ

4. Agenda • Rubyʹ͓͚Δػցֶशͷݱঢ়ʹ͍ͭͯ • Red Data Toolsʹ͍ͭͯ • ݱঢ়ͷऔΓ૊Έ •

   ະདྷʹ͍ͭͯ • ·ͱΊ 4

5. Agenda • Rubyʹ͓͚Δػցֶशͷݱঢ়ʹ͍ͭͯ • Red Data Toolsʹ͍ͭͯ • ݱঢ়ͷऔΓ૊Έ •

   ະདྷʹ͍ͭͯ • ·ͱΊ 5

6. 6 RubyͰ ػցֶश͍ͨ͠ʂ

7. 7 ػցֶशʹݶΒͣ σʔλ෼ੳʹ͓͍ͯ σʔλ࿈ܞ͸ඞཁͱͳΔ

8. 8 σʔλ࿈ܞͱ͍͏ͱ JSON΍CSV ??

9. JSON΍CSV ϑΝΠϧͰͷ΍ΓͱΓ͚ͩͰͳ͘APIͰͷϦΫΤε τɾϨεϙϯεͳͲͰ࢖༻͞ΕΔ 9 PythonͰग़ྗͨ͠CSVΛRubyͰಡΈࠐΉྫ write: 0.8009s read: 0.1227s

10. CSV΍JSONͰ
 େ͖ͳσʔλΛॲཧ ಛఆͷྻͰूܭॲཧΛ͍ͨ͠৔߹Ͱ΋
 Ұ౓͢΂ͯΛಡΈࠐΉඞཁ͕͋ΔͷͰ
 େ͖ͳσʔλΛॲཧ͢Δʹ͸ݶք͕͋Δ 10 ػցֶशͰ͸਺ඦສͷେ͖ͳσʔληοτΛ࢖͏͜ͱ΋
 ͋ΔͷͰେ͖ͳσʔλΛѻ͑ΔΑ͏ʹͳΓ͍ͨ

11. 11 Apache Parquet

12. Apache Parquetͱ͸ • ΧϥϜܕͷσʔλϑΥʔϚοτ • શ෦ΛಡΈࠐ·ͣͱ΋σʔλͷҰ෦ΛऔΓग़ͤΔ • ྫ͑͹ಛఆͷྻΛऔΓग़ͯ͠ूܭ͢Δ • αΠζ͕খ͍͞

    • ΧϥϜͰσʔλΛ֨ೲ͢ΔͨΊಉ͡ܕͷσʔλΛѹॖ͢ Δ͜ͱ͕ग़དྷΔ 12 େ͖ͳσʔλΛॲཧͯ͠อଘ͢Δ͜ͱʹ޲͍͍ͯΔ

13. Apache Parquet 13 PythonͰग़ྗͨ͠ParquetΛRubyͰಡΈࠐΉྫ write: 0.1390s read: 0.0422s

14. 14 େྔͷσʔλΛ ѻ͑ΔΑ͏ʹͳͬͨ

15. ෳ਺ͷγεςϜؒͰͷ σʔλΛ࿈ܞ Parquet͸σʔλΛѹॖ͢ΔͨΊσʔλॲཧ͢Δաఔ Ͱෳ਺γεςϜؒͰσʔλ࿈ܞΛߦ͏ࡍ͸ͦΕΛݩʹ ໭͢Φʔόʔϔου͕ੜ͡Δ 15 γʔϜϨεʹγεςϜؒͰσʔλ࿈ܞ͍ͨ͠!

16. 16 Apache Arrow

17. Apache Arrowͱ͸ • ΠϯϝϞϦͰΧϥϜܕσʔλΛѻ͏ͨΊͷϑΥʔϚοτͱΞϧΰ ϦζϜ • γϦΞϥΠζɾσγϦΞϥΠζίετ͕΄΅0 • σʔλΛ౉͢ଆ΋ड͚औΔଆ΋CPUΛ࢖Θͳ͍ •

    θϩίϐʔ • σʔλΛड͚औΔଆ͸ϝϞϦίϐʔʹ࣌ؒΛ࢖Θͳͯ͘΋Α͍ 17 σʔλަ׵ʹΉ͍͍ͯΔ

18. Apache Arrow 18 PythonͰग़ྗͨ͠ParquetΛRubyͰಡΈࠐΉྫ write: 0.0775s read: 0.0074s

19. 19 1ZUIPOͰͷॻ͖ࠐΈ
    T 3VCZͰͷಡΈࠐΈ
    T $47   "QBDIF
    1BSRVFU 

     "QBDIF
    "SSPX   5000Ϩίʔυʹର͢Δ ಡΈॻ͖ൺֱ·ͱΊ

20. Apache Arrow͕ͳ͍࣌ 20

21. Apache Arrow͕͋Δ࣌ 21

22. RubyͰParquetΛѻ͑Δͱ 22 ෳ਺ͷγεςϜؒ͸"QBDIF
    "SSPXͰσʔλΛ࿈ܞΛߦ͍ σʔλ෼ੳΛߦ͏ ParquetΛѻ͏͜ͱͰσʔλͷ࿈ܞ͸ग़དྷͨ
 ͔͠͠RubyͰσʔλ෼ੳ͕͍ͨ͠ʂʂʹ͸ಧ͔ͳ͍

23. RubyͰArrowΛѻ͑Δͱ 23 ෳ਺ͷγεςϜؒ͸"QBDIF
    "SSPXͰσʔλΛ࿈ܞΛߦ͍ σʔλ෼ੳΛߦ͏ RubyͰσʔλॲཧͷҰ෦Λ୲͏͜ͱ͕ग़དྷΔ!!

24. 3VCZͰ"SSPXΛѻ͑Δͱ 24 "QBDIF
    "SSPXʹରԠ͢Δ͜ͱͰඞཁͳ෦෼͔Β 3VCZΛ࢖ͬͨσʔλ෼ੳΛ͸͡ΊΔ͜ͱ͕ग़དྷΔ w 3VCZͰूΊͨσʔλΛ"QBDIF
    "SSPXʹରԠͯ͠ ͍Δ1BOEBT΍4QBSLʹ࿈ܞ͠෼ੳͨ݁͠ՌΛ3VCZ Ͱड͚औͬͯ3BJMTΛ࢖ͬͨXFCΞϓϦͰՄࢹԽ
    w ޙʹ1BOEBT΍4QBSL෦෼Λঃʑʹ3VCZ΁Ҡߦ͢

    Δ͜ͱ΋ग़དྷΔ ྫ

25. 25

26. Agenda • Rubyʹ͓͚Δػցֶशͷݱঢ়ʹ͍ͭͯ • Red Data Toolsʹ͍ͭͯ • ݱঢ়ͷऔΓ૊Έ •

    ະདྷʹ͍ͭͯ • ·ͱΊ 26

27. Red Data Tools • גࣜձࣾΫϦΞίʔυͷਢ౻͞Μ͕2017೥2݄ʹ ϓϩδΣΫτΛઃཱ • Ruby༻ͷσʔλॲཧπʔϧΛఏڙ͢Δ͜ͱΛ໨త ͱͨ͠ϓϩδΣΫτ •

    ଟ͘ͷݴޠ͕ڞ௨ͯ͠࢖༻Ͱ͖ΔApache ArrowΛ ࢖༻͢Δ͜ͱͰRubyίϛϡχςΟʔΛ௒͑ͯڠྗ ͢Δ 27

28. Agenda • Rubyʹ͓͚Δػցֶशͷݱঢ়ʹ͍ͭͯ • Red Data Toolsʹ͍ͭͯ • ݱঢ়ͷऔΓ૊Έ •

    ະདྷʹ͍ͭͯ • ·ͱΊ 28

29. ݱঢ়ͷऔΓ૊Έ • Red Arrow • طଘgemͷArrowରԠ • ৽͍͠πʔϧΛఏڙ 29

30. ݱঢ়ͷऔΓ૊Έ • Red Arrow • طଘgemͷArrowରԠ • ৽͍͠πʔϧΛఏڙ 30

31. Red Arrow w "QBDIF
    "SSPXͷ3VCZόΠϯσΟϯά
    w (JU)VC
    SFEEBUBUPPMTSFEBSSPX
    w "SSPX
    (-JCͱHPCKFDUJOUSPTQFDUJPOΛ࢖ͬͯ "QBDIF
    "SSPXͷόΠϯσΟϯάΛ࣮ݱ͍ͯ͠Δ 31

32. ݱঢ়ͷऔΓ૊Έ • Red Arrow • طଘgemͷArrowରԠ • ৽͍͠πʔϧΛఏڙ 32

33. SciRubyɾRuby Numo • Պֶٕज़ܭࢉɺσʔλՄࢹԽ༻్ͷGem܈Λ։ൃ ͍ͯ͠ΔϓϩδΣΫτ • σʔλϑϨʔϜΛѻ͑Δdaru΍ߦྻܭࢉ͕ग़དྷΔ numo-narryͳͲ͕͋Δ • ֤༻్Ͱݸผͷgem͸ଘࡏ͢Δ͔Β


    ͦΕΒ͕࿈ܞͰ͖ΔΑ͏ʹͳΔͱ
 ΋ͬͱΑ͍ʂ 33

34. PyCall • @mrkn͞Μ͕࡞੒͞Ε͍ͯΔRubyͱPythonͷϒ ϦοδϥΠϒϥϦ • PythonͰ࡞੒͞Εͨطଘͷࢿ࢈Λ࢖ͬͯ
 PythonͷΦϒδΣΫτΛ
 RubyͰ࢖༻͢Δ͜ͱ͕ग़དྷΔ 34

35. طଘgemͷArrowରԠ 35 4DJ3VCZ 1Z$BMM "QBDIF
    "SSPXʹରԠͤ͞Δ͜ͱͰ
    طଘͷHFNΛ࢖༻ͯ͠σʔλ෼ੳΛ͸͡ΊΒΕΔ 3VCZ
    /VNP

36. ݱঢ়ͷऔΓ૊Έ • Red Arrow • طଘgemͷArrowରԠ • ৽͍͠πʔϧΛఏڙ 36

37. Red Chainer • ChainerΛRuby΁ϙʔςΟϯάͨ͠΋ͷ • ChainerͷΫϥε΍ύϥϝʔλͷ࣋ͪํΛࢀߟʹ RubyͰॻ͘͜ͱͰ0͔Β࡞ΔͷͰ͸ͳ͘طଘͷࢿ࢈ Λ׆͔ͯ͠࡞Δ͜ͱ͕ग़དྷΔ • ಺෦ͷ഑ྻʹ͸Apache

    ArrowʹରԠͨ͠numo- narrayͤ͞Δ͜ͱͰApache Arrowܗࣜʹม׵͢Δ ͜ͱ΋Մೳ 37

38. 38 MNIST

39. Agenda • Rubyʹ͓͚Δػցֶशͷݱঢ়ʹ͍ͭͯ • Red Data Toolsʹ͍ͭͯ • ݱঢ়ͷऔΓ૊Έ •

    ະདྷʹ͍ͭͯ • ·ͱΊ 39

40. Red Data Toolsͷࠓޙ • Ҿ͖ଓ͖طଘͷgemͷAapche Arrow΁ͷରԠ͸ߦ ͏ • Red ChainerͷΑ͏ͳ৽͍͠πʔϧͷ࡞੒

    40 3VCZؒͰσʔλ෼ੳ͕
    ग़དྷΔΑ͏ʹͳΓ͍ͨ

41. Red Data Toolsͷࠓޙ Apache Arrowຊମ΁ͷ։ൃ΁ͷࢀՃ 41 "QBDIF
    "SSPX
    $ FUD

42. Agenda • Rubyʹ͓͚Δػցֶशͷݱঢ়ʹ͍ͭͯ • Red Data Toolsʹ͍ͭͯ • ݱঢ়ͷऔΓ૊Έ •

    ະདྷʹ͍ͭͯ • ·ͱΊ 42

43. ·ͱΊ w "QBDIF
    "SSPXʹରԠ͢Δ͜ͱͰඞཁͳ෦෼͔Β 3VCZͰσʔλ෼ੳΛ͸͡ΊΔ͜ͱ͕ग़དྷΔ
    w গͮͭ͠Ͱ΋3VCZͰσʔλ෼ੳ͕ग़དྷΔະདྷ΁ʂ 43

44. 44 ࠂ஌

45. ։ൃΠϕϯτ΍ͬͯ·͢ 45 w ຖ݄ճͷఆظ։࠵
    w ৔ॴ͸4QFFFͰߦͬͯ·͢
    !౦ژ
    
    ࿡ຊ໦
    w σʔλ෼ੳ͜Ε͔ΒͷਓͰ΋ͲͳͨͰ΋0,Ͱ͢ʂ

46. Gitter 46 w ೔ຊޠɹ
    
    IUUQTHJUUFSJNSFEEBUBUPPMTKB
    w &OHMJTI
    
    IUUQTHJUUFSJNSFEEBUBUPPMTFO