大きなデータと戦う技術 / fighting-large-data

Slide 1

Slide 1 text

େ͖ͳσʔλͱઓ͏ٕज़ yuuki takezawa asucon 2018ळ

Slide 2

Slide 2 text

Proﬁle • ஛ᖒ ༗و / ytake • גࣜձࣾΞΠελΠϧ CTO • PHP, Hack, Go, Scala • Apache Hadoop, Apache Spark, Apache Kafka  • twitter https://twitter.com/ex_takezawa • facebook https://www.facebook.com/yuuki.takezawa • github https://github.com/ytake

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Agenda • ΞϓϦέʔγϣϯͱσʔλઃܭ  • ղܾ͢ΔͨΊʹ

Slide 5

Slide 5 text

ΞϓϦέʔγϣϯͱσʔλઃܭ

Slide 6

Slide 6 text

ΞϓϦέʔγϣϯͷσʔλʹ͍ͭͯ • WebΞϓϦέʔγϣϯͳͲΛࢧ͑Δ  RDBMS • IoTͳͲʹ୅ද͞ΕΔେن໛ͳσʔλ

Slide 7

Slide 7 text

ΞϓϦέʔγϣϯͷ੒௕Λࢧ͖͑ΕΔʁ • ૝ఆ֎ͷ੒௕Λ਱͛Δ  WebΞϓϦέʔγϣϯ    ఆظతͳσʔλϕʔεϦϑΝΫλϦϯάɺ ΞϓϦέʔγϣϯͷϦϑΝΫλϦϯά  ͕࣮ࢪͰ͖Δ͔Ͳ͏͔

Slide 8

Slide 8 text

ΞϓϦέʔγϣϯͷ੒௕Λࢧ͖͑ΕΔʁ • ϋʔυ΢ΣΞɾΞϓϦέʔγϣϯো֐    ΞϓϦέʔγϣϯʹ߹ΘͤͯΫϥ΢υ ͔ɺΦϯϓϨΛબ୒͢Δ

Slide 9

Slide 9 text

খ͞ͳνʔϜͷ৔߹

Slide 10

Slide 10 text

࠷ॳͷΞϓϦέʔγϣϯ • σʔλϕʔεઃܭ + Active Record etc  ϑϨʔϜϫʔΫͰߏங͞ΕΔ  ΞϓϦέʔγϣϯ  • গਓ਺ͷ։ൃऀͰߏ੒͞ΕΔ։ൃ૊৫ 

Slide 11

Slide 11 text

ෳ਺νʔϜ΁ͷ੒௕

Slide 12

Slide 12 text

ΞϓϦέʔγϣϯͷ੒௕ • ૿͑ΔΞϓϦέʔγϣϯػೳ  • ։ൃνʔϜͷ૿һ  εΩϧ͸༷ʑ

Slide 13

Slide 13 text

ΞϓϦέʔγϣϯͷ੒௕ • Ϩίʔυ૿Ճɾ࣮૷ίʔυ૿ՃʹΑΔ  ύϑΥʔϚϯεͷ௿Լ    ϥΠϒϥϦͰൃߦ͞ΕΔSQLʹ͍ͭͯ  ཧղ͍ͯ͠Δ͔Ͳ͏͔  όΠφϦΛσʔλϕʔεʹ֨ೲʂʁ

Slide 14

Slide 14 text

ΞϓϦέʔγϣϯͱσʔλϕʔε • खܰʹ࢖͑Δ͔Β࢖͏ Ͱ͸ͳٙ͘໰Λ࣋ͭ  ൃߦ͞ΕΔSQL͸ݱࡏͷ  ΞϓϦέʔγϣϯن໛ʹ߹͍ͬͯΔ͔Ͳ͏͔ • ϋʔυ΢ΣΞ૿ڧͰ৐Γ੾Δ  ໰୊͕ޙճ͠ʹͳΔ͜ͱ΋

Slide 15

Slide 15 text

ΞϓϦέʔγϣϯͷ੒௕ͱσʔλϕʔε • σʔλऔಘ؆ུԽͷͨΊͷ  σʔλϕʔεઃܭ  • ΞΫηεϩάͳͲͷσʔλΛ֨ೲ  ཁ஫ҙ

Slide 16

Slide 16 text

େن໛νʔϜ΁ͷ੒௕

Slide 17

Slide 17 text

ߋͳΔΞϓϦέʔγϣϯͷ੒௕ • ૿͑ଓ͚ΔΞϓϦέʔγϣϯػೳ  • ։ൃνʔϜͷڊେԽ  ෳ਺ͷνʔϜߏ੒ͱ  ෳ਺ͷεςʔΫϗϧμ

Slide 18

Slide 18 text

ΞϓϦέʔγϣϯͷ੒௕ • Ϩίʔυ૿Ճɾ࣮૷ίʔυ૿ՃʹΑΔ  ͞ΒͳΔύϑΥʔϚϯε௿Լ • ͋ͪͪ͜Ͱى͜Γ࢝ΊΔো֐

Slide 19

Slide 19 text

ϦϦʔεΛ༏ઌͤ͞Αʂ

Slide 20

Slide 20 text

ฐ֐ • ϦϦʔε༏ઌͷͨΊɺ  ܧ͗଍͠ͷΞϓϦέʔγϣϯ  • εςʔΫϗϧμ૿Ճʹ൐͏  ΞϓϦέʔγϣϯͷෳࡶԽ • খதن໛ͷΞϓϦέʔγϣϯ࣌୅ͷ  ઃܭͱ࣮૷༝དྷͷෆ۩߹͕૿Ճ

Slide 21

Slide 21 text

σʔλઃܭ༝དྷͷ໰୊ • େྔσʔλͷϑϧεΩϟϯ  • INDEXෆ଍ͷͨΊͷύϑΥʔϚϯε௿Լ • γϯϓϧͳߏ੒ނͷػೳ௥Ճ࣌ͷ  ΫΤϦෳࡶԽ

Slide 22

Slide 22 text

ղܾ͢ΔͨΊʹ

Slide 23

Slide 23 text

ෳࡶ͞ͱͷઓ͍

Slide 24

Slide 24 text

ΞϓϦέʔγϣϯͷ෼ྨ • ॻ͖ࠐΈ͕ଟ਺ͷΞϓϦέʔγϣϯ • ಡΈࠐΈ͕ଟ਺ͷΞϓϦέʔγϣϯ    ඞͣͲͪΒ͔ʹ෼ྨ͞ΕΔ

Slide 25

Slide 25 text

྆ํ͋Γ·͚͢Ͳɾɾʁ

Slide 26

Slide 26 text

ΞϓϦέʔγϣϯͷ୯Ґ • ҰͭͷΞϓϦέʔγϣϯʹ  ͨ͘͞Μͷػೳ͕٧·͍ͬͯΔέʔε    ػೳҰͭͣͭΛ෼ղͯ͠ߟ͑Δ

Slide 27

Slide 27 text

ॻ͖ࠐΈଟ਺ͷΞϓϦέʔγϣϯ • ॻ͖ࠐΈʹڧ͘ɺ  εέʔϧ͕༰қͳσʔλϕʔε΁  Cassandra, Dynamodb, MongoDB  • ػೳ୯ҐͰߟ͑Δ  ܾࡁܥͳΒRDBMSซ༻ͳͲ

Slide 28

Slide 28 text

ಡΈࠐΈଟ਺ͷΞϓϦέʔγϣϯ • RDBMSͷΈͰ΋े෼  • LIKEݕࡧͳͲ͸Elasticsearch, Solr΁

Slide 29

Slide 29 text

ॻ͖ࠐΈͱಡΈࠐΈͷ౷߹ • ͲͪΒ͔͚ͩͰ΍Ζ͏ͱ͠ͳ͍ࣄ΋Ұͭ • Message Brokerซ༻ʹΑΔ෼ࢄॲཧ  Apache Kafka, RabbitMQ  Amazon SQS, Amazon Kinesis(Firehose)

Slide 30

Slide 30 text

CQRS "A few myths about CQRS". Ouarzy's Blog.   http://www.ouarzy.com/2016/10/02/a-few-myths-about-cqrs/ ࢀর

Slide 31

Slide 31 text

࣮ྫ

Slide 32

Slide 32 text

େྔσʔλ΁ͷΞϓϩʔν

Slide 33

Slide 33 text

େྔσʔλ΁ͷΞϓϩʔν Ϣʔβʔͷ࣌ܥྻߦಈϩά͕ QIQSELBGLBܦ༝ͰૹΒΕͯ͘Δ

Slide 34

Slide 34 text

େྔσʔλ΁ͷΞϓϩʔν "QBDIF,BGLB "QBDIF;PPLFFQFS QBSUJUJPO ݱࡏԯ͘Β͍  ΞϓϦέʔγϣϯ΁ͷো֐ɾऔΓ͜΅͠ͳ͠

Slide 35

Slide 35 text

େྔσʔλ΁ͷΞϓϩʔν σʔλϕʔεΛ݁߹ͯ͠ϏδωεϩδοΫٵऩ QVTI௨஌ࢦࣔͳͲΠϕϯτΛૹ৴ ଞαʔϏε͕SBCCJUNRΛ࢖͍ͬͯΔͨΊ

Slide 36

Slide 36 text

େྔσʔλ΁ͷΞϓϩʔν ,BGLB$POOFDUʹΑΔసૹΛซ༻

Slide 37

Slide 37 text

େྔσʔλ΁ͷΞϓϩʔν $BTTBOESB$MVTUFS ͪ͜Β΋ԯͪΐͬͱ͘Β͍ ো֐ͳ͠ɾίϯύΫγϣϯఆظ࣮ߦͰ τϥϒϧͳ͠

Slide 38

Slide 38 text

ूܭܥσʔλͱͷઓ͍

Slide 39

Slide 39 text

ΞΫηεϩάͳͲͷσʔλͷ׆༻ • ΞΫηεϩάͳͲͷղܾํ๏ • ΞϓϦέʔγϣϯͰఏڙ͞ΕΔػೳ  ϩάΛར༻͢ΔϨίϝϯσʔγϣϯ  ෼ੳػೳ

Slide 40

Slide 40 text

ϩάσʔλ΁ͷΞϓϩʔν • ΄ͱΜͲ͸աڈͷσʔλͷूܭͰ  ΄΅ෆม  • ूܭޙʹ  ଞͷσʔλͱֻ͚߹ΘͤΔͳͲ

Slide 41

Slide 41 text

ϩάσʔλ΁ͷΞϓϩʔν • RDBMSͰूܭ  ୯७ͳεϨʔϒͱ͸෼཭͓ͯ͘͠ࣄ  ਺ेԯҎ্ͷσʔλͰ͸ແཧ͠ͳ͍  • ूܭςʔϒϧͱΞϓϦέʔγϣϯ༻ͷ  ςʔϒϧ͸ซ༻͠ͳ͍

Slide 42

Slide 42 text

ϩάσʔλ΁ͷΞϓϩʔν • HDFSͰूܭ  RDBMS͔ΒApache Sqoopɺ  Apache SparkͳͲͰసૹ • ूܭॲཧ͸Apache SparkͳͲͰߦ͍ɺ  ଞͷσʔλϕʔεͱ݁߹͠ɺ֨ೲ

Slide 43

Slide 43 text

࣮ྫ

Slide 44

Slide 44 text

ϩάσʔλ΁ͷΞϓϩʔν

Slide 45

Slide 45 text

ϩάσʔλ΁ͷΞϓϩʔν ूܭର৅ͷ σʔλϕʔεɾςʔϒϧΛసૹ

Slide 46

Slide 46 text

ϩάσʔλ΁ͷΞϓϩʔν )%'4΁3%#.4ͷσʔλ Λอ؅

Slide 47

Slide 47 text

ϩάσʔλ΁ͷΞϓϩʔν )%'4ʹ͋Δσʔλɺ ଞͷ3%#.4্ͷσʔλΛ݁߹

Slide 48

Slide 48 text

ϩάσʔλ΁ͷΞϓϩʔν ूܭॲཧޙ࠶ͼ)%'4ͳͲʹ ֨ೲ͠௚͢FUD

Slide 49

Slide 49 text

ूܭσʔλͱϦΞϧλΠϜσʔλ΁ͷΞϓϩʔν • ूܭ݁ՌΛ֨ೲͨ͠σʔλετϨʔδʴ  ετϦʔϜॲཧͷ૊Έ߹Θͤ  • WebΞϓϦέʔγϣϯ૚Ͱ  ूܭ౳͸ߦΘͳ͍

Slide 50

Slide 50 text

KappaΞʔΩςΫνϟ

Slide 51

Slide 51 text

KappaΞʔΩςΫνϟ

Slide 52

Slide 52 text

࣮ྫ

Slide 53

Slide 53 text

ϩάσʔλ΁ͷΞϓϩʔν ͦͷ2

Slide 54

Slide 54 text

ϩάσʔλ΁ͷΞϓϩʔν ͦͷ2 ༷ʑͳΞϓϦέʔγϣϯ͔Β  σʔλૹ৴

Slide 55

Slide 55 text

ϩάσʔλ΁ͷΞϓϩʔν ͦͷ2 "QBDIF,BGLB͕  શͯͷσʔλΛड৴

Slide 56

Slide 56 text

ϩάσʔλ΁ͷΞϓϩʔν ͦͷ2 ,BGLB4QBSL4USFBNJOH ΞϓϦέʔγϣϯ͔Βૹ৴͞Εͨσʔλͱɺ 3%#.4ʹ֨ೲ͞ΕͨσʔλΛ݁߹͠ɺ ूܭɾू໿Λߦ͏

Slide 57

Slide 57 text

ϩάσʔλ΁ͷΞϓϩʔν ͦͷ2 ूܭɾू໿͞ΕͨσʔλΛɺ  ಡΈࠐΈͰར༻͢ΔΞϓϦέʔγϣϯʹ  ߹Θͤͯอ؅ $BTTBOESBͱ4QBSL4USFBNJOHͷΈͰ  ೖग़ྗΛߦ͏έʔε΋

Slide 58

Slide 58 text

ϩάσʔλ΁ͷΞϓϩʔν ͦͷ2 ूܭɾू໿͞Εͨσʔλͷ͏ͪ ༷ʑͳՕॴͰར༻͞ΕΔ΋ͷ͸ɺ)%'4΁ ࠶ܭࢉ΍ɺো֐ൃੜ࣌ʹ෮چͤ͞ΔͳͲ

Slide 59

Slide 59 text

ϩάσʔλ΁ͷΞϓϩʔν ͦͷ2 ΞϓϦέʔγϣϯଆ͔Β͸ $BTTBOESBͷΈʹ໰͍߹ΘͤΛߦ͏

Slide 60

Slide 60 text

·ͱΊ

Slide 61

Slide 61 text

·ͱΊ • ن໛ʹ߹Θͤͨσʔλઃܭ  ఆظతͳσʔλϕʔεϦϑΝΫλϦϯά • దࡐదॴΛݟۃΊΔٕज़ • ΞϓϦέʔγϣϯͱઓ͏৺