Rubyで機械学習が出来る 未来を目指すRed Data Toolsの 現状と今後について

RubyWorld Conferencee #rubyworld

Transcript

1. RubyͰػցֶश͕ग़དྷΔ ະདྷΛ໨ࢦ͢Red Data Toolsͷ ݱঢ়ͱࠓޙʹ͍ͭͯ by Yusaku Hatanaka @RubyWorld Conference 2017 1
2. self.intoroduction { name: "Yusaku Hatanaka", twitter: "@hatappi", github: "hatappi", hatena: "hatappi1225", company: "Speee, Inc." languages: %w(ruby go python), icon: "ɹ " } 2
3. ࣗݾ঺հ w ാத
   ༔࡞
   w 5XJUUFSɺ(JUIVC
   
   IBUBQQJ
 ͸ͯͳϒϩά
   
   
   
   
   
   
   IBUBQQJ
   w גࣜձࣾ4QFFF
   w σδλϧίϯαϧςΟϯάࣄۀຊ෦
 ΞυςΫࣄۀ෦
   6;06ࣄۀ
   w 6;06ࣄۀΤϯδχΞ
4. Agenda • Rubyʹ͓͚Δػցֶशͷݱঢ়ʹ͍ͭͯ • Red Data Toolsʹ͍ͭͯ • ݱঢ়ͷऔΓ૊Έ • ະདྷʹ͍ͭͯ • ·ͱΊ 4
5. Agenda • Rubyʹ͓͚Δػցֶशͷݱঢ়ʹ͍ͭͯ • Red Data Toolsʹ͍ͭͯ • ݱঢ়ͷऔΓ૊Έ • ະདྷʹ͍ͭͯ • ·ͱΊ 5
6. 6 RubyͰ ػցֶश͍ͨ͠ʂ
7. 7 ػցֶशʹݶΒͣ σʔλ෼ੳʹ͓͍ͯ σʔλ࿈ܞ͸ඞཁͱͳΔ
8. 8 σʔλ࿈ܞͱ͍͏ͱ JSON΍CSV ??
9. JSON΍CSV ϑΝΠϧͰͷ΍ΓͱΓ͚ͩͰͳ͘APIͰͷϦΫΤε τɾϨεϙϯεͳͲͰ࢖༻͞ΕΔ 9 PythonͰग़ྗͨ͠CSVΛRubyͰಡΈࠐΉྫ write: 0.8009s read: 0.1227s
10. CSV΍JSONͰ
 େ͖ͳσʔλΛॲཧ ಛఆͷྻͰूܭॲཧΛ͍ͨ͠৔߹Ͱ΋
 Ұ౓͢΂ͯΛಡΈࠐΉඞཁ͕͋ΔͷͰ
 େ͖ͳσʔλΛॲཧ͢Δʹ͸ݶք͕͋Δ 10 ػցֶशͰ͸਺ඦສͷେ͖ͳσʔληοτΛ࢖͏͜ͱ΋
 ͋ΔͷͰେ͖ͳσʔλΛѻ͑ΔΑ͏ʹͳΓ͍ͨ
11. 11 Apache Parquet
12. Apache Parquetͱ͸ • ΧϥϜܕͷσʔλϑΥʔϚοτ • શ෦ΛಡΈࠐ·ͣͱ΋σʔλͷҰ෦ΛऔΓग़ͤΔ • ྫ͑͹ಛఆͷྻΛऔΓग़ͯ͠ूܭ͢Δ • αΠζ͕খ͍͞ • ΧϥϜͰσʔλΛ֨ೲ͢ΔͨΊಉ͡ܕͷσʔλΛѹॖ͢ Δ͜ͱ͕ग़དྷΔ 12 େ͖ͳσʔλΛॲཧͯ͠อଘ͢Δ͜ͱʹ޲͍͍ͯΔ
13. Apache Parquet 13 PythonͰग़ྗͨ͠ParquetΛRubyͰಡΈࠐΉྫ write: 0.1390s read: 0.0422s
14. 14 େྔͷσʔλΛ ѻ͑ΔΑ͏ʹͳͬͨ
15. ෳ਺ͷγεςϜؒͰͷ σʔλΛ࿈ܞ Parquet͸σʔλΛѹॖ͢ΔͨΊσʔλॲཧ͢Δաఔ Ͱෳ਺γεςϜؒͰσʔλ࿈ܞΛߦ͏ࡍ͸ͦΕΛݩʹ ໭͢Φʔόʔϔου͕ੜ͡Δ 15 γʔϜϨεʹγεςϜؒͰσʔλ࿈ܞ͍ͨ͠!
16. 16 Apache Arrow
17. Apache Arrowͱ͸ • ΠϯϝϞϦͰΧϥϜܕσʔλΛѻ͏ͨΊͷϑΥʔϚοτͱΞϧΰ ϦζϜ • γϦΞϥΠζɾσγϦΞϥΠζίετ͕΄΅0 • σʔλΛ౉͢ଆ΋ड͚औΔଆ΋CPUΛ࢖Θͳ͍ • θϩίϐʔ • σʔλΛड͚औΔଆ͸ϝϞϦίϐʔʹ࣌ؒΛ࢖Θͳͯ͘΋Α͍ 17 σʔλަ׵ʹΉ͍͍ͯΔ
18. Apache Arrow 18 PythonͰग़ྗͨ͠ParquetΛRubyͰಡΈࠐΉྫ write: 0.0775s read: 0.0074s
19. 19 1ZUIPOͰͷॻ͖ࠐΈ
    T 3VCZͰͷಡΈࠐΈ
    T $47   "QBDIF
    1BSRVFU   "QBDIF
    "SSPX   5000Ϩίʔυʹର͢Δ ಡΈॻ͖ൺֱ·ͱΊ
20. Apache Arrow͕ͳ͍࣌ 20
21. Apache Arrow͕͋Δ࣌ 21
22. RubyͰParquetΛѻ͑Δͱ 22 ෳ਺ͷγεςϜؒ͸"QBDIF
    "SSPXͰσʔλΛ࿈ܞΛߦ͍ σʔλ෼ੳΛߦ͏ ParquetΛѻ͏͜ͱͰσʔλͷ࿈ܞ͸ग़དྷͨ
 ͔͠͠RubyͰσʔλ෼ੳ͕͍ͨ͠ʂʂʹ͸ಧ͔ͳ͍
23. RubyͰArrowΛѻ͑Δͱ 23 ෳ਺ͷγεςϜؒ͸"QBDIF
    "SSPXͰσʔλΛ࿈ܞΛߦ͍ σʔλ෼ੳΛߦ͏ RubyͰσʔλॲཧͷҰ෦Λ୲͏͜ͱ͕ग़དྷΔ!!
24. 3VCZͰ"SSPXΛѻ͑Δͱ 24 "QBDIF
    "SSPXʹରԠ͢Δ͜ͱͰඞཁͳ෦෼͔Β 3VCZΛ࢖ͬͨσʔλ෼ੳΛ͸͡ΊΔ͜ͱ͕ग़དྷΔ w 3VCZͰूΊͨσʔλΛ"QBDIF
    "SSPXʹରԠͯ͠ ͍Δ1BOEBT΍4QBSLʹ࿈ܞ͠෼ੳͨ݁͠ՌΛ3VCZ Ͱड͚औͬͯ3BJMTΛ࢖ͬͨXFCΞϓϦͰՄࢹԽ
    w ޙʹ1BOEBT΍4QBSL෦෼Λঃʑʹ3VCZ΁Ҡߦ͢ Δ͜ͱ΋ग़དྷΔ ྫ
25. 25
26. Agenda • Rubyʹ͓͚Δػցֶशͷݱঢ়ʹ͍ͭͯ • Red Data Toolsʹ͍ͭͯ • ݱঢ়ͷऔΓ૊Έ • ະདྷʹ͍ͭͯ • ·ͱΊ 26
27. Red Data Tools • גࣜձࣾΫϦΞίʔυͷਢ౻͞Μ͕2017೥2݄ʹ ϓϩδΣΫτΛઃཱ • Ruby༻ͷσʔλॲཧπʔϧΛఏڙ͢Δ͜ͱΛ໨త ͱͨ͠ϓϩδΣΫτ • ଟ͘ͷݴޠ͕ڞ௨ͯ͠࢖༻Ͱ͖ΔApache ArrowΛ ࢖༻͢Δ͜ͱͰRubyίϛϡχςΟʔΛ௒͑ͯڠྗ ͢Δ 27
28. Agenda • Rubyʹ͓͚Δػցֶशͷݱঢ়ʹ͍ͭͯ • Red Data Toolsʹ͍ͭͯ • ݱঢ়ͷऔΓ૊Έ • ະདྷʹ͍ͭͯ • ·ͱΊ 28
29. ݱঢ়ͷऔΓ૊Έ • Red Arrow • طଘgemͷArrowରԠ • ৽͍͠πʔϧΛఏڙ 29
30. ݱঢ়ͷऔΓ૊Έ • Red Arrow • طଘgemͷArrowରԠ • ৽͍͠πʔϧΛఏڙ 30
31. Red Arrow w "QBDIF
    "SSPXͷ3VCZόΠϯσΟϯά
    w (JU)VC
    SFEEBUBUPPMTSFEBSSPX
    w "SSPX
    (-JCͱHPCKFDUJOUSPTQFDUJPOΛ࢖ͬͯ "QBDIF
    "SSPXͷόΠϯσΟϯάΛ࣮ݱ͍ͯ͠Δ 31
32. ݱঢ়ͷऔΓ૊Έ • Red Arrow • طଘgemͷArrowରԠ • ৽͍͠πʔϧΛఏڙ 32
33. SciRubyɾRuby Numo • Պֶٕज़ܭࢉɺσʔλՄࢹԽ༻్ͷGem܈Λ։ൃ ͍ͯ͠ΔϓϩδΣΫτ • σʔλϑϨʔϜΛѻ͑Δdaru΍ߦྻܭࢉ͕ग़དྷΔ numo-narryͳͲ͕͋Δ • ֤༻్Ͱݸผͷgem͸ଘࡏ͢Δ͔Β
 ͦΕΒ͕࿈ܞͰ͖ΔΑ͏ʹͳΔͱ
 ΋ͬͱΑ͍ʂ 33
34. PyCall • @mrkn͞Μ͕࡞੒͞Ε͍ͯΔRubyͱPythonͷϒ ϦοδϥΠϒϥϦ • PythonͰ࡞੒͞Εͨطଘͷࢿ࢈Λ࢖ͬͯ
 PythonͷΦϒδΣΫτΛ
 RubyͰ࢖༻͢Δ͜ͱ͕ग़དྷΔ 34
35. طଘgemͷArrowରԠ 35 4DJ3VCZ 1Z$BMM "QBDIF
    "SSPXʹରԠͤ͞Δ͜ͱͰ
    طଘͷHFNΛ࢖༻ͯ͠σʔλ෼ੳΛ͸͡ΊΒΕΔ 3VCZ
    /VNP
36. ݱঢ়ͷऔΓ૊Έ • Red Arrow • طଘgemͷArrowରԠ • ৽͍͠πʔϧΛఏڙ 36
37. Red Chainer • ChainerΛRuby΁ϙʔςΟϯάͨ͠΋ͷ • ChainerͷΫϥε΍ύϥϝʔλͷ࣋ͪํΛࢀߟʹ RubyͰॻ͘͜ͱͰ0͔Β࡞ΔͷͰ͸ͳ͘طଘͷࢿ࢈ Λ׆͔ͯ͠࡞Δ͜ͱ͕ग़དྷΔ • ಺෦ͷ഑ྻʹ͸Apache ArrowʹରԠͨ͠numo- narrayͤ͞Δ͜ͱͰApache Arrowܗࣜʹม׵͢Δ ͜ͱ΋Մೳ 37
38. 38 MNIST
39. Agenda • Rubyʹ͓͚Δػցֶशͷݱঢ়ʹ͍ͭͯ • Red Data Toolsʹ͍ͭͯ • ݱঢ়ͷऔΓ૊Έ • ະདྷʹ͍ͭͯ • ·ͱΊ 39
40. Red Data Toolsͷࠓޙ • Ҿ͖ଓ͖طଘͷgemͷAapche Arrow΁ͷରԠ͸ߦ ͏ • Red ChainerͷΑ͏ͳ৽͍͠πʔϧͷ࡞੒ 40 3VCZؒͰσʔλ෼ੳ͕
    ग़དྷΔΑ͏ʹͳΓ͍ͨ
41. Red Data Toolsͷࠓޙ Apache Arrowຊମ΁ͷ։ൃ΁ͷࢀՃ 41 "QBDIF
    "SSPX
    $ FUD
42. Agenda • Rubyʹ͓͚Δػցֶशͷݱঢ়ʹ͍ͭͯ • Red Data Toolsʹ͍ͭͯ • ݱঢ়ͷऔΓ૊Έ • ະདྷʹ͍ͭͯ • ·ͱΊ 42
43. ·ͱΊ w "QBDIF
    "SSPXʹରԠ͢Δ͜ͱͰඞཁͳ෦෼͔Β 3VCZͰσʔλ෼ੳΛ͸͡ΊΔ͜ͱ͕ग़དྷΔ
    w গͮͭ͠Ͱ΋3VCZͰσʔλ෼ੳ͕ग़དྷΔະདྷ΁ʂ 43
44. 44 ࠂ஌
45. ։ൃΠϕϯτ΍ͬͯ·͢ 45 w ຖ݄ճͷఆظ։࠵
    w ৔ॴ͸4QFFFͰߦͬͯ·͢
    !౦ژ
    
    ࿡ຊ໦
    w σʔλ෼ੳ͜Ε͔ΒͷਓͰ΋ͲͳͨͰ΋0,Ͱ͢ʂ
46. Gitter 46 w ೔ຊޠɹ
    
    IUUQTHJUUFSJNSFEEBUBUPPMTKB
    w &OHMJTI
    
    IUUQTHJUUFSJNSFEEBUBUPPMTFO