Tweet
Tweet

Slide 1

Slide 1 text

RubyͰػցֶश͕ग़དྷΔ ະདྷΛ໨ࢦ͢Red Data Toolsͷ ݱঢ়ͱࠓޙʹ͍ͭͯ by Yusaku Hatanaka @RubyWorld Conference 2017 1

Slide 2

Slide 2 text

self.intoroduction { name: "Yusaku Hatanaka", twitter: "@hatappi", github: "hatappi", hatena: "hatappi1225", company: "Speee, Inc." languages: %w(ruby go python), icon: "ɹ " } 2

Slide 3

Slide 3 text

ࣗݾ঺հ w ാத༔࡞ w 5XJUUFSɺ(JUIVCIBUBQQJ
 ͸ͯͳϒϩάIBUBQQJ w גࣜձࣾ4QFFF w σδλϧίϯαϧςΟϯάࣄۀຊ෦
 ΞυςΫࣄۀ෦6;06ࣄۀ w 6;06ࣄۀΤϯδχΞ

Slide 4

Slide 4 text

Agenda • Rubyʹ͓͚Δػցֶशͷݱঢ়ʹ͍ͭͯ • Red Data Toolsʹ͍ͭͯ • ݱঢ়ͷऔΓ૊Έ • ະདྷʹ͍ͭͯ • ·ͱΊ 4

Slide 5

Slide 5 text

Agenda • Rubyʹ͓͚Δػցֶशͷݱঢ়ʹ͍ͭͯ • Red Data Toolsʹ͍ͭͯ • ݱঢ়ͷऔΓ૊Έ • ະདྷʹ͍ͭͯ • ·ͱΊ 5

Slide 6

Slide 6 text

6 RubyͰ ػցֶश͍ͨ͠ʂ

Slide 7

Slide 7 text

7 ػցֶशʹݶΒͣ σʔλ෼ੳʹ͓͍ͯ σʔλ࿈ܞ͸ඞཁͱͳΔ

Slide 8

Slide 8 text

8 σʔλ࿈ܞͱ͍͏ͱ JSON΍CSV ??

Slide 9

Slide 9 text

JSON΍CSV ϑΝΠϧͰͷ΍ΓͱΓ͚ͩͰͳ͘APIͰͷϦΫΤε τɾϨεϙϯεͳͲͰ࢖༻͞ΕΔ 9 PythonͰग़ྗͨ͠CSVΛRubyͰಡΈࠐΉྫ write: 0.8009s read: 0.1227s

Slide 10

Slide 10 text

CSV΍JSONͰ
 େ͖ͳσʔλΛॲཧ ಛఆͷྻͰूܭॲཧΛ͍ͨ͠৔߹Ͱ΋
 Ұ౓͢΂ͯΛಡΈࠐΉඞཁ͕͋ΔͷͰ
 େ͖ͳσʔλΛॲཧ͢Δʹ͸ݶք͕͋Δ 10 ػցֶशͰ͸਺ඦສͷେ͖ͳσʔληοτΛ࢖͏͜ͱ΋
 ͋ΔͷͰେ͖ͳσʔλΛѻ͑ΔΑ͏ʹͳΓ͍ͨ

Slide 11

Slide 11 text

11 Apache Parquet

Slide 12

Slide 12 text

Apache Parquetͱ͸ • ΧϥϜܕͷσʔλϑΥʔϚοτ • શ෦ΛಡΈࠐ·ͣͱ΋σʔλͷҰ෦ΛऔΓग़ͤΔ • ྫ͑͹ಛఆͷྻΛऔΓग़ͯ͠ूܭ͢Δ • αΠζ͕খ͍͞ • ΧϥϜͰσʔλΛ֨ೲ͢ΔͨΊಉ͡ܕͷσʔλΛѹॖ͢ Δ͜ͱ͕ग़དྷΔ 12 େ͖ͳσʔλΛॲཧͯ͠อଘ͢Δ͜ͱʹ޲͍͍ͯΔ

Slide 13

Slide 13 text

Apache Parquet 13 PythonͰग़ྗͨ͠ParquetΛRubyͰಡΈࠐΉྫ write: 0.1390s read: 0.0422s

Slide 14

Slide 14 text

14 େྔͷσʔλΛ ѻ͑ΔΑ͏ʹͳͬͨ

Slide 15

Slide 15 text

ෳ਺ͷγεςϜؒͰͷ σʔλΛ࿈ܞ Parquet͸σʔλΛѹॖ͢ΔͨΊσʔλॲཧ͢Δաఔ Ͱෳ਺γεςϜؒͰσʔλ࿈ܞΛߦ͏ࡍ͸ͦΕΛݩʹ ໭͢Φʔόʔϔου͕ੜ͡Δ 15 γʔϜϨεʹγεςϜؒͰσʔλ࿈ܞ͍ͨ͠!

Slide 16

Slide 16 text

16 Apache Arrow

Slide 17

Slide 17 text

Apache Arrowͱ͸ • ΠϯϝϞϦͰΧϥϜܕσʔλΛѻ͏ͨΊͷϑΥʔϚοτͱΞϧΰ ϦζϜ • γϦΞϥΠζɾσγϦΞϥΠζίετ͕΄΅0 • σʔλΛ౉͢ଆ΋ड͚औΔଆ΋CPUΛ࢖Θͳ͍ • θϩίϐʔ • σʔλΛड͚औΔଆ͸ϝϞϦίϐʔʹ࣌ؒΛ࢖Θͳͯ͘΋Α͍ 17 σʔλަ׵ʹΉ͍͍ͯΔ

Slide 18

Slide 18 text

Apache Arrow 18 PythonͰग़ྗͨ͠ParquetΛRubyͰಡΈࠐΉྫ write: 0.0775s read: 0.0074s

Slide 19

Slide 19 text

19 1ZUIPOͰͷॻ͖ࠐΈ T 3VCZͰͷಡΈࠐΈ T $47 "QBDIF1BSRVFU "QBDIF"SSPX 5000Ϩίʔυʹର͢Δ ಡΈॻ͖ൺֱ·ͱΊ

Slide 20

Slide 20 text

Apache Arrow͕ͳ͍࣌ 20

Slide 21

Slide 21 text

Apache Arrow͕͋Δ࣌ 21

Slide 22

Slide 22 text

RubyͰParquetΛѻ͑Δͱ 22 ෳ਺ͷγεςϜؒ͸"QBDIF "SSPXͰσʔλΛ࿈ܞΛߦ͍ σʔλ෼ੳΛߦ͏ ParquetΛѻ͏͜ͱͰσʔλͷ࿈ܞ͸ग़དྷͨ
 ͔͠͠RubyͰσʔλ෼ੳ͕͍ͨ͠ʂʂʹ͸ಧ͔ͳ͍

Slide 23

Slide 23 text

RubyͰArrowΛѻ͑Δͱ 23 ෳ਺ͷγεςϜؒ͸"QBDIF "SSPXͰσʔλΛ࿈ܞΛߦ͍ σʔλ෼ੳΛߦ͏ RubyͰσʔλॲཧͷҰ෦Λ୲͏͜ͱ͕ग़དྷΔ!!

Slide 24

Slide 24 text

3VCZͰ"SSPXΛѻ͑Δͱ 24 "QBDIF"SSPXʹରԠ͢Δ͜ͱͰඞཁͳ෦෼͔Β 3VCZΛ࢖ͬͨσʔλ෼ੳΛ͸͡ΊΔ͜ͱ͕ग़དྷΔ w 3VCZͰूΊͨσʔλΛ"QBDIF"SSPXʹରԠͯ͠ ͍Δ1BOEBT΍4QBSLʹ࿈ܞ͠෼ੳͨ݁͠ՌΛ3VCZ Ͱड͚औͬͯ3BJMTΛ࢖ͬͨXFCΞϓϦͰՄࢹԽ w ޙʹ1BOEBT΍4QBSL෦෼Λঃʑʹ3VCZ΁Ҡߦ͢ Δ͜ͱ΋ग़དྷΔ ྫ

Slide 25

Slide 25 text

25

Slide 26

Slide 26 text

Agenda • Rubyʹ͓͚Δػցֶशͷݱঢ়ʹ͍ͭͯ • Red Data Toolsʹ͍ͭͯ • ݱঢ়ͷऔΓ૊Έ • ະདྷʹ͍ͭͯ • ·ͱΊ 26

Slide 27

Slide 27 text

Red Data Tools • גࣜձࣾΫϦΞίʔυͷਢ౻͞Μ͕2017೥2݄ʹ ϓϩδΣΫτΛઃཱ • Ruby༻ͷσʔλॲཧπʔϧΛఏڙ͢Δ͜ͱΛ໨త ͱͨ͠ϓϩδΣΫτ • ଟ͘ͷݴޠ͕ڞ௨ͯ͠࢖༻Ͱ͖ΔApache ArrowΛ ࢖༻͢Δ͜ͱͰRubyίϛϡχςΟʔΛ௒͑ͯڠྗ ͢Δ 27

Slide 28

Slide 28 text

Agenda • Rubyʹ͓͚Δػցֶशͷݱঢ়ʹ͍ͭͯ • Red Data Toolsʹ͍ͭͯ • ݱঢ়ͷऔΓ૊Έ • ະདྷʹ͍ͭͯ • ·ͱΊ 28

Slide 29

Slide 29 text

ݱঢ়ͷऔΓ૊Έ • Red Arrow • طଘgemͷArrowԽ • ৽͍͠πʔϧΛఏڙ 29

Slide 30

Slide 30 text

ݱঢ়ͷऔΓ૊Έ • Red Arrow • طଘgemͷArrowԽ • ৽͍͠πʔϧΛఏڙ 30

Slide 31

Slide 31 text

Red Arrow w "QBDIF"SSPXͷ3VCZόΠϯσΟϯά w (JU)VCSFEEBUBUPPMTSFEBSSPX w "SSPX(-JCͱHPCKFDUJOUSPTQFDUJPOΛ࢖ͬͯ "QBDIF"SSPXͷόΠϯσΟϯάΛ࣮ݱ͍ͯ͠Δ 31

Slide 32

Slide 32 text

ݱঢ়ͷऔΓ૊Έ • Red Arrow • طଘgemͷArrowԽ • ৽͍͠πʔϧΛఏڙ 32

Slide 33

Slide 33 text

SciRubyɾRuby Numo • Պֶٕज़ܭࢉɺσʔλՄࢹԽ༻్ͷGem܈Λ։ൃ ͍ͯ͠ΔϓϩδΣΫτ • σʔλϑϨʔϜΛѻ͑Δdaru΍ߦྻܭࢉ͕ग़དྷΔ numo-narryͳͲ͕͋Δ • ֤༻్Ͱݸผͷgem͸ଘࡏ͢Δ͔Β
 ͦΕΒ͕࿈ܞͰ͖ΔΑ͏ʹͳΔͱ
 ΋ͬͱΑ͍ʂ 33

Slide 34

Slide 34 text

PyCall • @mrkn͞Μ͕࡞੒͞Ε͍ͯΔRubyͱPythonͷϒ ϦοδϥΠϒϥϦ • PythonͰ࡞੒͞Εͨطଘͷࢿ࢈Λ࢖ͬͯ
 PythonͷΦϒδΣΫτΛ
 RubyͰ࢖༻͢Δ͜ͱ͕ग़དྷΔ 34

Slide 35

Slide 35 text

طଘgemͷArrowԽ 35 4DJ3VCZ 1Z$BMM "QBDIF"SSPXʹରԠͤ͞Δ͜ͱͰ طଘͷHFNΛ࢖༻ͯ͠σʔλ෼ੳΛ͸͡ΊΒΕΔ 3VCZ/VNP

Slide 36

Slide 36 text

ݱঢ়ͷऔΓ૊Έ • Red Arrow • طଘgemͷArrowԽ • ৽͍͠πʔϧΛఏڙ 36

Slide 37

Slide 37 text

Red Chainer • ChainerΛRuby΁ϙʔςΟϯάͨ͠΋ͷ • ChainerͷΫϥε΍ύϥϝʔλͷ࣋ͪํΛࢀߟʹ RubyͰॻ͘͜ͱͰ0͔Β࡞ΔͷͰ͸ͳ͘طଘͷࢿ࢈ Λ׆͔ͯ͠࡞Δ͜ͱ͕ग़དྷΔ • ಺෦ͷ഑ྻʹ͸Apache ArrowʹରԠͨ͠numo- narrayͤ͞Δ͜ͱͰApache Arrowܗࣜʹม׵͢Δ ͜ͱ΋Մೳ 37

Slide 38

Slide 38 text

38 MNIST

Slide 39

Slide 39 text

Agenda • Rubyʹ͓͚Δػցֶशͷݱঢ়ʹ͍ͭͯ • Red Data Toolsʹ͍ͭͯ • ݱঢ়ͷऔΓ૊Έ • ະདྷʹ͍ͭͯ • ·ͱΊ 39

Slide 40

Slide 40 text

Red Data Toolsͷࠓޙ • Ҿ͖ଓ͖طଘͷgemͷAapche Arrow΁ͷରԠ͸ߦ ͏ • Red ChainerͷΑ͏ͳ৽͍͠πʔϧͷ࡞੒ 40 3VCZؒͰσʔλ෼ੳ͕ ग़དྷΔΑ͏ʹͳΓ͍ͨ

Slide 41

Slide 41 text

Red Data Toolsͷࠓޙ Apache Arrowຊମ΁ͷ։ൃ΁ͷࢀՃ 41 "QBDIF"SSPX $ FUD

Slide 42

Slide 42 text

Agenda • Rubyʹ͓͚Δػցֶशͷݱঢ়ʹ͍ͭͯ • Red Data Toolsʹ͍ͭͯ • ݱঢ়ͷऔΓ૊Έ • ະདྷʹ͍ͭͯ • ·ͱΊ 42

Slide 43

Slide 43 text

·ͱΊ w "QBDIF"SSPXʹରԠ͢Δ͜ͱͰඞཁͳ෦෼͔Β 3VCZͰσʔλ෼ੳΛ͸͡ΊΔ͜ͱ͕ग़དྷΔ w গͮͭ͠Ͱ΋3VCZͰσʔλ෼ੳ͕ग़དྷΔະདྷ΁ʂ 43

Slide 44

Slide 44 text

44 ࠂ஌

Slide 45

Slide 45 text

։ൃΠϕϯτ΍ͬͯ·͢ 45 w ຖ݄ճͷఆظ։࠵ w ৔ॴ͸4QFFFͰߦͬͯ·͢!౦ژ࿡ຊ໦ w σʔλ෼ੳ͜Ε͔ΒͷਓͰ΋ͲͳͨͰ΋0,Ͱ͢ʂ

Slide 46

Slide 46 text

Gitter 46 w ೔ຊޠɹIUUQTHJUUFSJNSFEEBUBUPPMTKB w &OHMJTIIUUQTHJUUFSJNSFEEBUBUPPMTFO