DynamoDBを利用したKPI保存システム

by gree_tech

Slide 1

Slide 1 text

DynamoDBΛར༻ͨ͠ KPIอଘγεςϜ άϦʔגࣜձࣾɹ௕୩઒و೭ 2016/01/25 VisualizationNight#1

Slide 2

Slide 2 text

ຊ೔ͷ಺༰ • ࣗݾ঺հ • άϦʔ಺੡ͷՄࢹԽγεςϜ • DynamoDBΛར༻ͨ͠KPIอଘγεςϜ

Slide 3

Slide 3 text

ࣗݾ঺հ • σʔλΤϯδχΞϦϯάνʔϜ • HadoopͳͲͷ෼ੳج൫ͷӡ༻ • πʔϧͷ։ൃɼར༻ऀͷαϙʔτ • ࠷ۙͰ͸ΦϯϓϨ͚ͩͰͳ͘AWS΋ར༻ • KinesisɼEMRͷݕূΛ͢Δϝϯόʔ΋

Slide 4

Slide 4 text

άϦʔ಺੡ͷՄࢹԽγεςϜ ஫) ͜Ε͔Βग़ͯ͘Δ஋͸αϯϓϧσʔλͰ͢

Slide 5

Slide 5 text

ՄࢹԽγεςϜ • NativeGame KPIϨϙʔτ • ܾΊΒΕͨ਺஋ࢦඪͷΈදࣔ • ϓϩμΫτؒͷൺֱ͕༰қ • ϢʔβʔςΩετμογϡϘʔυ • ςΩετӾཡʹಛԽͨ͠γεςϜ • ϢʔβʔͷΞϯέʔτɼ໰͍߹ΘͤͳͲ͕ݟΕΔ • BIπʔϧ(Macaron) • ϓϩμΫτӡӦऀ͕ࣗ༝ʹ਺஋ࢦඪΛ࡞੒Ͱ͖Δ • εέδϡʔϦϯάͷػೳΛඋ͑Δ (Ұ෦ͷγεςϜͷ঺հͰ͢)

Slide 6

Slide 6 text

NativeGame KPIϨϙʔτ

Slide 7

Slide 7 text

NativeGame KPIϨϙʔτ Rails + RDS(MySQL) + TD or Hadoop Apache®, Apache Hadoop, Hadoop®(http://hadoop.apache.org/), and the yellow elephant logo are either registered trademarks or trademarks of the Apache Software Foundation(http://www.apache.org/) in the United States and/or other countries. Treasure Data Inc.(https://www.treasuredata.com/)

Slide 8

Slide 8 text

ϢʔβʔςΩετμογϡϘʔυ

Slide 9

Slide 9 text

ϢʔβʔςΩετμογϡϘʔυ Django + Presto + Hadoop Django

Slide 10

Slide 10 text

BIπʔϧ(Macaron)

Slide 11

Slide 11 text

BIπʔϧ(چMacaron) Rails + HBase or Hive or Presto + Hadoop

Slide 12

Slide 12 text

BIπʔϧ(৽Macaron) Rails + HBase or Hive or Presto + Hadoop DynamoDB΁ʂ

Slide 13

Slide 13 text

͔͜͜Βͷ಺༰ • Macaronͷػೳ • طଘετϨʔδ(HBase, OpenTSDB) • ϦϓϨΠεͷഎܠ / ݕ౼ • ઃܭ / ࣮૷

Slide 14

Slide 14 text

Macaronͷػೳ • Hive΍PrestoʹΫΤϦͷൃߦ͢Δ • ΫΤϦͷ݁ՌΛ࣌ܥྻσʔλϕʔεʹอଘ • with εέδϡʔϦϯά • ηάϝϯτຖͷάϥϑඳը • ͦͷଞ

Slide 15

Slide 15 text

Macaronͷػೳ / ΫΤϦͷൃߦ

Slide 16

Slide 16 text

Macaronͷػೳ / σʔλͷอଘ ఆظ࣮ߦͨ݁͠ՌΛอଘ͢Δ

Slide 17

Slide 17 text

Macaronͷػೳ / άϥϑͷඳը

Slide 18

Slide 18 text

BIπʔϧ(Macaron) Rails + DynamoDB or Hive or Presto + Hadoop • HBase(OpenTSDB)͔ΒDynamoDB΁

Slide 19

Slide 19 text

ͦ΋ͦ΋OpenTSDBͬͯʁ • HBaseΛετϨʔδͱͨ࣌͠ܥྻσʔλϕʔε • Φʔϓϯιʔε • RRDToolͷΑ͏ʹ஋ΛؙΊͳ͍ • γεςϜͷϝτϦΫεͳͲେن໛σʔλ͕ಘҙ • ถYahoo!Ͱ͸15୆ͷαʔόʔͰຖඵ28ສ݅ͷWrite Λ͞͹͘ • http://www.slideshare.net/HBaseCon/ecosystem-session-6/6

Slide 20

Slide 20 text

OpenTSDBͷϝλσʔλ • DataPoints (time, value) • Metrics σόΠεผDAU • Tags device=fp, device=sp • Metrics + Tags = TimeSeries ref. OpenTSDB HBaseCon http://opentsdb.net/misc/opentsdb-hbasecon.pdf

Slide 21

Slide 21 text

ϦϓϨΠεͨ͠എܠ • αʔόʔͷҠઃͷඞཁ͕͋ͬͨ • HBaseΫϥελΛ࠶ߏஙʁ • ৽͘͠࡞Γ௚͢ʁ • ظؒ͸7݄຤ ~ 10݄ • ܾΊΒΕͨظݶɼٻΊΒΕΔඅ༻࡟ݮ

Slide 22

Slide 22 text

ϦϓϨΠεͨ͠എܠ • ݱঢ়ͷγεςϜ͕ߴػೳ͗͢Δ • ඳըػೳɼूܭؔ਺ͳͲ͸ར༻͍ͯ͠ͳ͍ • HBaseͰ࣋ͭ΄Ͳͷσʔλن໛Ͱ͸ͳ͍ • গ͠ΧελϚΠζ͢Δඞཁ͕͋ͬͨ • ೔ຊޠʹରԠͤ͞ΔͨΊʹύονΛ౰ͯͨ • ϝτϦοΫͷૢ࡞͕ҙ֎ͱ໘౗

Slide 23

Slide 23 text

AWSΛલఏͱͨ͠ϦϓϨΠεҊ • HBase on EMR • ৗ࣌Քಇ͓ͤͯ͘͞ͱίετ͕ߴ͍ • InﬂuxDB on EC2 • ݕূظ͕ؒͱΕͣӡ༻໘Ͱ৴པ͖͠Εͳ͔ͬͨ • MySQL(RDS) • ࣮੷͕͋Γɼ஋ஈ΋༧૝Ͱ͖Δ • DynamoDB • ৄ͘͠ͳ͍͕ɼσʔλ͸ιʔτͰ͖ΔΒ͍͠

Slide 24

Slide 24 text

DynamoDBʹؔ͢Δௐࠪ • εϧʔϓοτ՝ۚͰ͋Δ • Ұ౓ʹେ͖ͳσʔλΛऔಘɼอଘ͸ۤख • ςʔϒϧ͝ͱʹ՝ۚ͞ΕΔ • 10G͝ͱʹύʔςΟγϣϯ͕෼ׂ͞ΕΔ • อଘ͸͍͕҆εϧʔϓοτ͕෼ࢄ͢Δ

Slide 25

Slide 25 text

͋Εɺ޲͍ͯͳ͍…ʁ

Slide 26

Slide 26 text

Unitফඅʹ͍ͭͯௐࠪ͢Δ • GemItem, BatchGetItem • ͲΜͳʹখ͍͞recordͰ΋ɼ1 record = 1 unit • 1೔1recordͩͱ͢Δͱɼ3೥෼औಘͰ1095unit • Query, Scan • ྦྷܭαΠζΛ࢖༻ͯ͠εϧʔϓοτΛܭࢉ • 1record = 1unitͰ͸ͳ͍ • 1record͕1KBͱ͢Δͱɼ3೥෼औಘͰ273unit

Slide 27

Slide 27 text

Slide 28

Slide 28 text

ϢʔεέʔεΛ֬ೝ͢Δ • 1ϖʔδʹ͍͔ͭ͘ͷKPIࢦඪΛ഑ஔ • ظؒ͸ݻఆɼdailyͳΒ͹90೔ఔ౓ • ࢦඪͷߋ৽ස౓͸ߴ͘ͳ͍ • ͔͠͠ɼաڈ෼Λ1౓ʹೖΕΔ͜ͱ͸͋Δ • ࣌ؒଳʹΑͬͯݟΔݟͳ͍͕͸͖ͬΓ͢Δ • ໷ؒ΍౔೔͸΄ͱΜͲݟΒΕͳ͍

Slide 29

Slide 29 text

޻෉͢Δ༨஍͕͋Γͦ͏…ʂ

Slide 30

Slide 30 text

εϧʔϓοτΛԼ͛ΔऔΓ૊Έ • ಡΈࠐΈεϧʔϓοτͷ࡟ݮ • ݻఆ͞Εͨظؒͷσʔλ͸Ωϟογϡ΁ • ࣌ؒଳʹ߹Θͤͨಈతεϧʔϓοτมߋ • ॻ͖ࠐΈεϧʔϓοτͷ࡟ݮ • ٸͳॻ͖ࠐΈ૿ʹඋ͑ΩϡʔΛར༻

Slide 31

Slide 31 text

ΩϟογϡΛ׆༻͢Δ • ElastiCache(Redis)Λ׆༻͢Δ • ௚ۙͷσʔλ(ex. 90೔)ΛΩϟγϡ͢Δ • جຊతʹաڈͷࢪࡦΛৼΓฦΔ৔߹ͷΈ DynamoDBʹΞΫηε͕͍͘ • ElastiCache͸͍҆ͷͰ͓ಘ • ௚ۙͷσʔλͷΈͳΒt2.microͰ΋Մ

Slide 32

Slide 32 text

ΩϡʔΛ׆༻͢Δ • SQSΛར༻͢Δ • ॻ͖ࠐΈ͕૿ࣦ͑ͯഊͯ͠΋ϦτϥΠͰ͖Δ • ௿͍εϧʔϓοτΛࢦఆͯ͠΋σʔλͷϩετ Λ๷͙͜ͱ͕Ͱ͖Δ • ஋ஈ͸΄΅ແྉ

Slide 33

Slide 33 text

εϧʔϓοτΛมಈͤ͞Δ SumoLogicͰՄࢹԽͨ͠ΞΫηε਺ (Ұ෦) ໌Β͔ʹ໷ؒʹΞΫηε͕ͳ͘΋͍ͬͨͳ͍ʂ

Slide 34

Slide 34 text

ྫ) εϧʔϓοτΛมಈͤ͞Δ Ұఆʹอͬͨ৔߹ ऴ೔(read: 1000) • 0.00742 * 24 * 30 * 1000 / 50 = $107 ಡΈࠐΈεϧʔϓοτΛมಈͤͨ͞৔߹ ฏ೔&೔த(read: 1000)ɼ໷ؒor౔೔(read: 500) • 0.00742 * 24 * 22 * (1000+500) / 2 / 50 = $59 • 0.00742 * 24 * 8 * 500 / 50 = $14 • $59 + $14 = $73 ͜ͷྫͰ͸ɼεϧʔϓοτͷมಈͰ32%Φϑʹʂ

Slide 35

Slide 35 text

DynamoDBͰ͍͚ΔͷͰ͸…

Slide 36

Slide 36 text

αʔόʔߏ੒

Slide 37

Slide 37 text

࣮૷ • ScalaɼϑϨʔϜϫʔΫ͸SprayΛར༻ • AWSͷClient͸Java͕Ұ൪ॆ࣮͍ͯ͠Δ • AWSϩοΫΠϯ͞Εͯ΋ϩʔΧϧͰ։ൃՄೳ • DynamoDB → DynamoDBLocal • SQS → ElasticMQ

Slide 38

Slide 38 text

ӡ༻ • cronͰεϧʔϓοτΛมߋ • ே9࣌ ~ ໷9࣌ͰεϧʔϓοτΛ͋͛Δ • ϩά͸SumoLogicͰ؅ཧͯ͠ΞΫηε਺ͷ֬ೝ • σϓϩΠ͸CodeDeployͰ • ͱ͖Ͳ͖agentͷϓϩηε͕down͢Δ…!!

Slide 39

Slide 39 text

·ͱΊ • άϦʔͷՄࢹԽγεςϜʹ͍ͭͯ঺հͨ͠ • ໾ׂʹΑͬͯͦΕͧΕଘࡏ͢Δ • HBase(OpenTSDB)͔ΒDynamoDB΁ • εϧʔϓοτͷมಈͰ஋ஈΛ཈͑Δ • ElastiCache΍SQS΋ίετμ΢ϯʹߩݙ

Slide 40

Slide 40 text

Happy Hacking :)