Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up
for free
大きなデータと戦う技術 / fighting-large-data
yuuki takezawa
October 13, 2018
Technology
3
270
大きなデータと戦う技術 / fighting-large-data
明日の開発カンファレンス 2018秋
yuuki takezawa
October 13, 2018
Tweet
Share
More Decks by yuuki takezawa
See All by yuuki takezawa
ytake
4
900
ytake
0
5.4k
ytake
3
6.9k
ytake
2
2.6k
ytake
7
10k
ytake
1
1.7k
ytake
4
1.7k
ytake
1
3.3k
ytake
0
270
Other Decks in Technology
See All in Technology
sakon310
4
4.2k
pinboro
1
1.4k
am7cinnamon
2
2.6k
satotakeshi
2
410
cygames
1
340
line_developers
PRO
3
470
awsbelaraby
4
1.1k
kenya888
1
110
ocise
1
910
nihonbuson
2
1.5k
clustervr
0
160
soachr
1
150
Featured
See All Featured
dougneiner
55
5.4k
paulrobertlloyd
71
3.6k
morganepeng
17
1.1k
pauljervisheath
196
15k
shpigford
368
42k
dotmariusz
94
5.1k
skipperchong
7
660
brianwarren
83
4.7k
rocio
155
11k
holman
461
280k
lara
16
2.6k
rasmusluckow
318
18k
Transcript
େ͖ͳσʔλͱઓ͏ٕज़ yuuki takezawa asucon 2018ळ
Profile • ᖒ ༗و / ytake • גࣜձࣾΞΠελΠϧ CTO •
PHP, Hack, Go, Scala • Apache Hadoop, Apache Spark, Apache Kafka • twitter https://twitter.com/ex_takezawa • facebook https://www.facebook.com/yuuki.takezawa • github https://github.com/ytake
None
Agenda • ΞϓϦέʔγϣϯͱσʔλઃܭ • ղܾ͢ΔͨΊʹ
ΞϓϦέʔγϣϯͱσʔλઃܭ
ΞϓϦέʔγϣϯͷσʔλʹ͍ͭͯ • WebΞϓϦέʔγϣϯͳͲΛࢧ͑Δ RDBMS • IoTͳͲʹද͞ΕΔେنͳσʔλ
ΞϓϦέʔγϣϯͷΛࢧ͖͑ΕΔʁ • ఆ֎ͷΛ͛Δ WebΞϓϦέʔγϣϯ ఆظతͳσʔλϕʔεϦϑΝΫλϦϯάɺ ΞϓϦέʔγϣϯͷϦϑΝΫλϦϯά ͕࣮ࢪͰ͖Δ͔Ͳ͏͔
ΞϓϦέʔγϣϯͷΛࢧ͖͑ΕΔʁ • ϋʔυΣΞɾΞϓϦέʔγϣϯো ΞϓϦέʔγϣϯʹ߹ΘͤͯΫϥυ ͔ɺΦϯϓϨΛબ͢Δ
খ͞ͳνʔϜͷ߹
࠷ॳͷΞϓϦέʔγϣϯ • σʔλϕʔεઃܭ + Active Record etc ϑϨʔϜϫʔΫͰߏங͞ΕΔ ΞϓϦέʔγϣϯ •
গਓͷ։ൃऀͰߏ͞ΕΔ։ൃ৫
ෳνʔϜͷ
ΞϓϦέʔγϣϯͷ • ૿͑ΔΞϓϦέʔγϣϯػೳ • ։ൃνʔϜͷ૿һ εΩϧ༷ʑ
ΞϓϦέʔγϣϯͷ • Ϩίʔυ૿Ճɾ࣮ίʔυ૿ՃʹΑΔ ύϑΥʔϚϯεͷԼ ϥΠϒϥϦͰൃߦ͞ΕΔSQLʹ͍ͭͯ ཧղ͍ͯ͠Δ͔Ͳ͏͔ όΠφϦΛσʔλϕʔεʹ֨ೲʂʁ
ΞϓϦέʔγϣϯͱσʔλϕʔε • खܰʹ͑Δ͔Β͏ Ͱͳٙ͘Λ࣋ͭ ൃߦ͞ΕΔSQLݱࡏͷ ΞϓϦέʔγϣϯنʹ߹͍ͬͯΔ͔Ͳ͏͔ • ϋʔυΣΞ૿ڧͰΓΔ ͕ޙճ͠ʹͳΔ͜ͱ
ΞϓϦέʔγϣϯͷͱσʔλϕʔε • σʔλऔಘ؆ུԽͷͨΊͷ σʔλϕʔεઃܭ • ΞΫηεϩάͳͲͷσʔλΛ֨ೲ ཁҙ
େنνʔϜͷ
ߋͳΔΞϓϦέʔγϣϯͷ • ૿͑ଓ͚ΔΞϓϦέʔγϣϯػೳ • ։ൃνʔϜͷڊେԽ ෳͷνʔϜߏͱ ෳͷεςʔΫϗϧμ
ΞϓϦέʔγϣϯͷ • Ϩίʔυ૿Ճɾ࣮ίʔυ૿ՃʹΑΔ ͞ΒͳΔύϑΥʔϚϯεԼ • ͋ͪͪ͜Ͱى͜Γ࢝ΊΔো
ϦϦʔεΛ༏ઌͤ͞Αʂ
ฐ • ϦϦʔε༏ઌͷͨΊɺ ܧ͗͠ͷΞϓϦέʔγϣϯ • εςʔΫϗϧμ૿Ճʹ͏ ΞϓϦέʔγϣϯͷෳࡶԽ • খதنͷΞϓϦέʔγϣϯ࣌ͷ ઃܭͱ࣮༝དྷͷෆ۩߹͕૿Ճ
σʔλઃܭ༝དྷͷ • େྔσʔλͷϑϧεΩϟϯ • INDEXෆͷͨΊͷύϑΥʔϚϯεԼ • γϯϓϧͳߏނͷػೳՃ࣌ͷ ΫΤϦෳࡶԽ
ղܾ͢ΔͨΊʹ
ෳࡶ͞ͱͷઓ͍
ΞϓϦέʔγϣϯͷྨ • ॻ͖ࠐΈ͕ଟͷΞϓϦέʔγϣϯ • ಡΈࠐΈ͕ଟͷΞϓϦέʔγϣϯ ඞͣͲͪΒ͔ʹྨ͞ΕΔ
྆ํ͋Γ·͚͢Ͳɾɾʁ
ΞϓϦέʔγϣϯͷ୯Ґ • ҰͭͷΞϓϦέʔγϣϯʹ ͨ͘͞Μͷػೳ͕٧·͍ͬͯΔέʔε ػೳҰͭͣͭΛղͯ͠ߟ͑Δ
ॻ͖ࠐΈଟͷΞϓϦέʔγϣϯ • ॻ͖ࠐΈʹڧ͘ɺ εέʔϧ͕༰қͳσʔλϕʔε Cassandra, Dynamodb, MongoDB • ػೳ୯ҐͰߟ͑Δ ܾࡁܥͳΒRDBMSซ༻ͳͲ
ಡΈࠐΈଟͷΞϓϦέʔγϣϯ • RDBMSͷΈͰे • LIKEݕࡧͳͲElasticsearch, Solr
ॻ͖ࠐΈͱಡΈࠐΈͷ౷߹ • ͲͪΒ͔͚ͩͰΖ͏ͱ͠ͳ͍ࣄҰͭ • Message Brokerซ༻ʹΑΔࢄॲཧ Apache Kafka, RabbitMQ Amazon
SQS, Amazon Kinesis(Firehose)
CQRS "A few myths about CQRS". Ouarzy's Blog. http://www.ouarzy.com/2016/10/02/a-few-myths-about-cqrs/
ࢀর
࣮ྫ
େྔσʔλͷΞϓϩʔν
େྔσʔλͷΞϓϩʔν Ϣʔβʔͷ࣌ܥྻߦಈϩά͕ QIQSELBGLBܦ༝ͰૹΒΕͯ͘Δ
େྔσʔλͷΞϓϩʔν "QBDIF,BGLB "QBDIF;PPLFFQFS QBSUJUJPO ݱࡏԯ͘Β͍ ΞϓϦέʔγϣϯͷোɾऔΓ͜΅͠ͳ͠
େྔσʔλͷΞϓϩʔν σʔλϕʔεΛ݁߹ͯ͠ϏδωεϩδοΫٵऩ QVTI௨ࢦࣔͳͲΠϕϯτΛૹ৴ ଞαʔϏε͕SBCCJUNRΛ͍ͬͯΔͨΊ
େྔσʔλͷΞϓϩʔν ,BGLB$POOFDUʹΑΔసૹΛซ༻
େྔσʔλͷΞϓϩʔν $BTTBOESB $MVTUFS ͪ͜Βԯͪΐͬͱ͘Β͍ োͳ͠ɾίϯύΫγϣϯఆظ࣮ߦͰ τϥϒϧͳ͠
ूܭܥσʔλͱͷઓ͍
ΞΫηεϩάͳͲͷσʔλͷ׆༻ • ΞΫηεϩάͳͲͷղܾํ๏ • ΞϓϦέʔγϣϯͰఏڙ͞ΕΔػೳ ϩάΛར༻͢ΔϨίϝϯσʔγϣϯ ੳػೳ
ϩάσʔλͷΞϓϩʔν • ΄ͱΜͲաڈͷσʔλͷूܭͰ ΄΅ෆม • ूܭޙʹ ଞͷσʔλͱֻ͚߹ΘͤΔͳͲ
ϩάσʔλͷΞϓϩʔν • RDBMSͰूܭ ୯७ͳεϨʔϒͱ͓ͯ͘͠ࣄ ेԯҎ্ͷσʔλͰແཧ͠ͳ͍ • ूܭςʔϒϧͱΞϓϦέʔγϣϯ༻ͷ ςʔϒϧซ༻͠ͳ͍
ϩάσʔλͷΞϓϩʔν • HDFSͰूܭ RDBMS͔ΒApache Sqoopɺ Apache SparkͳͲͰసૹ • ूܭॲཧApache SparkͳͲͰߦ͍ɺ
ଞͷσʔλϕʔεͱ݁߹͠ɺ֨ೲ
࣮ྫ
ϩάσʔλͷΞϓϩʔν
ϩάσʔλͷΞϓϩʔν ूܭରͷ σʔλϕʔεɾςʔϒϧΛసૹ
ϩάσʔλͷΞϓϩʔν )%'43%#.4ͷσʔλ Λอ
ϩάσʔλͷΞϓϩʔν )%'4ʹ͋Δσʔλɺ ଞͷ3%#.4্ͷσʔλΛ݁߹
ϩάσʔλͷΞϓϩʔν ूܭॲཧޙ࠶ͼ)%'4ͳͲʹ ֨ೲ͢͠FUD
ूܭσʔλͱϦΞϧλΠϜσʔλͷΞϓϩʔν • ूܭ݁ՌΛ֨ೲͨ͠σʔλετϨʔδʴ ετϦʔϜॲཧͷΈ߹Θͤ • WebΞϓϦέʔγϣϯͰ ूܭߦΘͳ͍
KappaΞʔΩςΫνϟ
KappaΞʔΩςΫνϟ
࣮ྫ
ϩάσʔλͷΞϓϩʔν ͦͷ2
ϩάσʔλͷΞϓϩʔν ͦͷ2 ༷ʑͳΞϓϦέʔγϣϯ͔Β σʔλૹ৴
ϩάσʔλͷΞϓϩʔν ͦͷ2 "QBDIF,BGLB͕ શͯͷσʔλΛड৴
ϩάσʔλͷΞϓϩʔν ͦͷ2 ,BGLB 4QBSL4USFBNJOH ΞϓϦέʔγϣϯ͔Βૹ৴͞Εͨσʔλͱɺ 3%#.4ʹ֨ೲ͞ΕͨσʔλΛ݁߹͠ɺ ूܭɾूΛߦ͏
ϩάσʔλͷΞϓϩʔν ͦͷ2 ूܭɾू͞ΕͨσʔλΛɺ ಡΈࠐΈͰར༻͢ΔΞϓϦέʔγϣϯʹ ߹Θͤͯอ $BTTBOESBͱ4QBSL4USFBNJOHͷΈͰ ೖग़ྗΛߦ͏έʔε
ϩάσʔλͷΞϓϩʔν ͦͷ2 ूܭɾू͞Εͨσʔλͷ͏ͪ ༷ʑͳՕॴͰར༻͞ΕΔͷɺ)%'4 ࠶ܭࢉɺোൃੜ࣌ʹ෮چͤ͞ΔͳͲ
ϩάσʔλͷΞϓϩʔν ͦͷ2 ΞϓϦέʔγϣϯଆ͔Β $BTTBOESBͷΈʹ͍߹ΘͤΛߦ͏
·ͱΊ
·ͱΊ • نʹ߹Θͤͨσʔλઃܭ ఆظతͳσʔλϕʔεϦϑΝΫλϦϯά • దࡐదॴΛݟۃΊΔٕज़ • ΞϓϦέʔγϣϯͱઓ͏৺