Rubyで機械学習が出来る_未来を目指すRed_Data_Toolsの_現状と今後について.pdf

C582b722e015633f7900083f8ea75732?s=47 hatappi
November 01, 2017
27

 Rubyで機械学習が出来る_未来を目指すRed_Data_Toolsの_現状と今後について.pdf

C582b722e015633f7900083f8ea75732?s=128

hatappi

November 01, 2017
Tweet

Transcript

  1. RubyͰػցֶश͕ग़དྷΔ ະདྷΛ໨ࢦ͢Red Data Toolsͷ ݱঢ়ͱࠓޙʹ͍ͭͯ by Yusaku Hatanaka @RubyWorld Conference

    2017 1
  2. self.intoroduction { name: "Yusaku Hatanaka", twitter: "@hatappi", github: "hatappi", hatena:

    "hatappi1225", company: "Speee, Inc." languages: %w(ruby go python), icon: "ɹ " } 2
  3. ࣗݾ঺հ w ാத༔࡞ w 5XJUUFSɺ(JUIVCIBUBQQJ
 ͸ͯͳϒϩάIBUBQQJ w גࣜձࣾ4QFFF w σδλϧίϯαϧςΟϯάࣄۀຊ෦


    ΞυςΫࣄۀ෦6;06ࣄۀ w 6;06ࣄۀΤϯδχΞ
  4. Agenda • Rubyʹ͓͚Δػցֶशͷݱঢ়ʹ͍ͭͯ • Red Data Toolsʹ͍ͭͯ • ݱঢ়ͷऔΓ૊Έ •

    ະདྷʹ͍ͭͯ • ·ͱΊ 4
  5. Agenda • Rubyʹ͓͚Δػցֶशͷݱঢ়ʹ͍ͭͯ • Red Data Toolsʹ͍ͭͯ • ݱঢ়ͷऔΓ૊Έ •

    ະདྷʹ͍ͭͯ • ·ͱΊ 5
  6. 6 RubyͰ ػցֶश͍ͨ͠ʂ

  7. 7 ػցֶशʹݶΒͣ σʔλ෼ੳʹ͓͍ͯ σʔλ࿈ܞ͸ඞཁͱͳΔ

  8. 8 σʔλ࿈ܞͱ͍͏ͱ JSON΍CSV ??

  9. JSON΍CSV ϑΝΠϧͰͷ΍ΓͱΓ͚ͩͰͳ͘APIͰͷϦΫΤε τɾϨεϙϯεͳͲͰ࢖༻͞ΕΔ 9 PythonͰग़ྗͨ͠CSVΛRubyͰಡΈࠐΉྫ write: 0.8009s read: 0.1227s

  10. CSV΍JSONͰ
 େ͖ͳσʔλΛॲཧ ಛఆͷྻͰूܭॲཧΛ͍ͨ͠৔߹Ͱ΋
 Ұ౓͢΂ͯΛಡΈࠐΉඞཁ͕͋ΔͷͰ
 େ͖ͳσʔλΛॲཧ͢Δʹ͸ݶք͕͋Δ 10 ػցֶशͰ͸਺ඦສͷେ͖ͳσʔληοτΛ࢖͏͜ͱ΋
 ͋ΔͷͰେ͖ͳσʔλΛѻ͑ΔΑ͏ʹͳΓ͍ͨ

  11. 11 Apache Parquet

  12. Apache Parquetͱ͸ • ΧϥϜܕͷσʔλϑΥʔϚοτ • શ෦ΛಡΈࠐ·ͣͱ΋σʔλͷҰ෦ΛऔΓग़ͤΔ • ྫ͑͹ಛఆͷྻΛऔΓग़ͯ͠ूܭ͢Δ • αΠζ͕খ͍͞

    • ΧϥϜͰσʔλΛ֨ೲ͢ΔͨΊಉ͡ܕͷσʔλΛѹॖ͢ Δ͜ͱ͕ग़དྷΔ 12 େ͖ͳσʔλΛॲཧͯ͠อଘ͢Δ͜ͱʹ޲͍͍ͯΔ
  13. Apache Parquet 13 PythonͰग़ྗͨ͠ParquetΛRubyͰಡΈࠐΉྫ write: 0.1390s read: 0.0422s

  14. 14 େྔͷσʔλΛ ѻ͑ΔΑ͏ʹͳͬͨ

  15. ෳ਺ͷγεςϜؒͰͷ σʔλΛ࿈ܞ Parquet͸σʔλΛѹॖ͢ΔͨΊσʔλॲཧ͢Δաఔ Ͱෳ਺γεςϜؒͰσʔλ࿈ܞΛߦ͏ࡍ͸ͦΕΛݩʹ ໭͢Φʔόʔϔου͕ੜ͡Δ 15 γʔϜϨεʹγεςϜؒͰσʔλ࿈ܞ͍ͨ͠!

  16. 16 Apache Arrow

  17. Apache Arrowͱ͸ • ΠϯϝϞϦͰΧϥϜܕσʔλΛѻ͏ͨΊͷϑΥʔϚοτͱΞϧΰ ϦζϜ • γϦΞϥΠζɾσγϦΞϥΠζίετ͕΄΅0 • σʔλΛ౉͢ଆ΋ड͚औΔଆ΋CPUΛ࢖Θͳ͍ •

    θϩίϐʔ • σʔλΛड͚औΔଆ͸ϝϞϦίϐʔʹ࣌ؒΛ࢖Θͳͯ͘΋Α͍ 17 σʔλަ׵ʹΉ͍͍ͯΔ
  18. Apache Arrow 18 PythonͰग़ྗͨ͠ParquetΛRubyͰಡΈࠐΉྫ write: 0.0775s read: 0.0074s

  19. 19 1ZUIPOͰͷॻ͖ࠐΈ T 3VCZͰͷಡΈࠐΈ T $47   "QBDIF1BSRVFU 

     "QBDIF"SSPX   5000Ϩίʔυʹର͢Δ ಡΈॻ͖ൺֱ·ͱΊ
  20. Apache Arrow͕ͳ͍࣌ 20

  21. Apache Arrow͕͋Δ࣌ 21

  22. RubyͰParquetΛѻ͑Δͱ 22 ෳ਺ͷγεςϜؒ͸"QBDIF "SSPXͰσʔλΛ࿈ܞΛߦ͍ σʔλ෼ੳΛߦ͏ ParquetΛѻ͏͜ͱͰσʔλͷ࿈ܞ͸ग़དྷͨ
 ͔͠͠RubyͰσʔλ෼ੳ͕͍ͨ͠ʂʂʹ͸ಧ͔ͳ͍

  23. RubyͰArrowΛѻ͑Δͱ 23 ෳ਺ͷγεςϜؒ͸"QBDIF "SSPXͰσʔλΛ࿈ܞΛߦ͍ σʔλ෼ੳΛߦ͏ RubyͰσʔλॲཧͷҰ෦Λ୲͏͜ͱ͕ग़དྷΔ!!

  24. 3VCZͰ"SSPXΛѻ͑Δͱ 24 "QBDIF"SSPXʹରԠ͢Δ͜ͱͰඞཁͳ෦෼͔Β 3VCZΛ࢖ͬͨσʔλ෼ੳΛ͸͡ΊΔ͜ͱ͕ग़དྷΔ w 3VCZͰूΊͨσʔλΛ"QBDIF"SSPXʹରԠͯ͠ ͍Δ1BOEBT΍4QBSLʹ࿈ܞ͠෼ੳͨ݁͠ՌΛ3VCZ Ͱड͚औͬͯ3BJMTΛ࢖ͬͨXFCΞϓϦͰՄࢹԽ w ޙʹ1BOEBT΍4QBSL෦෼Λঃʑʹ3VCZ΁Ҡߦ͢

    Δ͜ͱ΋ग़དྷΔ ྫ
  25. 25

  26. Agenda • Rubyʹ͓͚Δػցֶशͷݱঢ়ʹ͍ͭͯ • Red Data Toolsʹ͍ͭͯ • ݱঢ়ͷऔΓ૊Έ •

    ະདྷʹ͍ͭͯ • ·ͱΊ 26
  27. Red Data Tools • גࣜձࣾΫϦΞίʔυͷਢ౻͞Μ͕2017೥2݄ʹ ϓϩδΣΫτΛઃཱ • Ruby༻ͷσʔλॲཧπʔϧΛఏڙ͢Δ͜ͱΛ໨త ͱͨ͠ϓϩδΣΫτ •

    ଟ͘ͷݴޠ͕ڞ௨ͯ͠࢖༻Ͱ͖ΔApache ArrowΛ ࢖༻͢Δ͜ͱͰRubyίϛϡχςΟʔΛ௒͑ͯڠྗ ͢Δ 27
  28. Agenda • Rubyʹ͓͚Δػցֶशͷݱঢ়ʹ͍ͭͯ • Red Data Toolsʹ͍ͭͯ • ݱঢ়ͷऔΓ૊Έ •

    ະདྷʹ͍ͭͯ • ·ͱΊ 28
  29. ݱঢ়ͷऔΓ૊Έ • Red Arrow • طଘgemͷArrowԽ • ৽͍͠πʔϧΛఏڙ 29

  30. ݱঢ়ͷऔΓ૊Έ • Red Arrow • طଘgemͷArrowԽ • ৽͍͠πʔϧΛఏڙ 30

  31. Red Arrow w "QBDIF"SSPXͷ3VCZόΠϯσΟϯά w (JU)VCSFEEBUBUPPMTSFEBSSPX w "SSPX(-JCͱHPCKFDUJOUSPTQFDUJPOΛ࢖ͬͯ "QBDIF"SSPXͷόΠϯσΟϯάΛ࣮ݱ͍ͯ͠Δ 31

  32. ݱঢ়ͷऔΓ૊Έ • Red Arrow • طଘgemͷArrowԽ • ৽͍͠πʔϧΛఏڙ 32

  33. SciRubyɾRuby Numo • Պֶٕज़ܭࢉɺσʔλՄࢹԽ༻్ͷGem܈Λ։ൃ ͍ͯ͠ΔϓϩδΣΫτ • σʔλϑϨʔϜΛѻ͑Δdaru΍ߦྻܭࢉ͕ग़དྷΔ numo-narryͳͲ͕͋Δ • ֤༻్Ͱݸผͷgem͸ଘࡏ͢Δ͔Β


    ͦΕΒ͕࿈ܞͰ͖ΔΑ͏ʹͳΔͱ
 ΋ͬͱΑ͍ʂ 33
  34. PyCall • @mrkn͞Μ͕࡞੒͞Ε͍ͯΔRubyͱPythonͷϒ ϦοδϥΠϒϥϦ • PythonͰ࡞੒͞Εͨطଘͷࢿ࢈Λ࢖ͬͯ
 PythonͷΦϒδΣΫτΛ
 RubyͰ࢖༻͢Δ͜ͱ͕ग़དྷΔ 34

  35. طଘgemͷArrowԽ 35 4DJ3VCZ 1Z$BMM "QBDIF"SSPXʹରԠͤ͞Δ͜ͱͰ طଘͷHFNΛ࢖༻ͯ͠σʔλ෼ੳΛ͸͡ΊΒΕΔ 3VCZ/VNP

  36. ݱঢ়ͷऔΓ૊Έ • Red Arrow • طଘgemͷArrowԽ • ৽͍͠πʔϧΛఏڙ 36

  37. Red Chainer • ChainerΛRuby΁ϙʔςΟϯάͨ͠΋ͷ • ChainerͷΫϥε΍ύϥϝʔλͷ࣋ͪํΛࢀߟʹ RubyͰॻ͘͜ͱͰ0͔Β࡞ΔͷͰ͸ͳ͘طଘͷࢿ࢈ Λ׆͔ͯ͠࡞Δ͜ͱ͕ग़དྷΔ • ಺෦ͷ഑ྻʹ͸Apache

    ArrowʹରԠͨ͠numo- narrayͤ͞Δ͜ͱͰApache Arrowܗࣜʹม׵͢Δ ͜ͱ΋Մೳ 37
  38. 38 MNIST

  39. Agenda • Rubyʹ͓͚Δػցֶशͷݱঢ়ʹ͍ͭͯ • Red Data Toolsʹ͍ͭͯ • ݱঢ়ͷऔΓ૊Έ •

    ະདྷʹ͍ͭͯ • ·ͱΊ 39
  40. Red Data Toolsͷࠓޙ • Ҿ͖ଓ͖طଘͷgemͷAapche Arrow΁ͷରԠ͸ߦ ͏ • Red ChainerͷΑ͏ͳ৽͍͠πʔϧͷ࡞੒

    40 3VCZؒͰσʔλ෼ੳ͕ ग़དྷΔΑ͏ʹͳΓ͍ͨ
  41. Red Data Toolsͷࠓޙ Apache Arrowຊମ΁ͷ։ൃ΁ͷࢀՃ 41 "QBDIF"SSPX $ FUD

  42. Agenda • Rubyʹ͓͚Δػցֶशͷݱঢ়ʹ͍ͭͯ • Red Data Toolsʹ͍ͭͯ • ݱঢ়ͷऔΓ૊Έ •

    ະདྷʹ͍ͭͯ • ·ͱΊ 42
  43. ·ͱΊ w "QBDIF"SSPXʹରԠ͢Δ͜ͱͰඞཁͳ෦෼͔Β 3VCZͰσʔλ෼ੳΛ͸͡ΊΔ͜ͱ͕ग़དྷΔ w গͮͭ͠Ͱ΋3VCZͰσʔλ෼ੳ͕ग़དྷΔະདྷ΁ʂ 43

  44. 44 ࠂ஌

  45. ։ൃΠϕϯτ΍ͬͯ·͢ 45 w ຖ݄ճͷఆظ։࠵ w ৔ॴ͸4QFFFͰߦͬͯ·͢!౦ژ࿡ຊ໦ w σʔλ෼ੳ͜Ε͔ΒͷਓͰ΋ͲͳͨͰ΋0,Ͱ͢ʂ

  46. Gitter 46 w ೔ຊޠɹIUUQTHJUUFSJNSFEEBUBUPPMTKB w &OHMJTIIUUQTHJUUFSJNSFEEBUBUPPMTFO