Amazon Kinesis + socket.io + D3.jsを使った webブラウザで行うリアルタイム可視化の仕組み/inovationegg-8

by 堀家隆宏

Slide 1

Slide 1 text

Amazon Kinesis + socket.io + D3.jsΛ࢖ͬͨ webϒϥ΢βͰߦ͏ϦΞϧλΠϜՄࢹԽͷ࢓૊Έ @Amimoto_Ami amimoto-ami.com

Slide 2

Slide 2 text

ࣗݾ঺հ • ໊લɿງՈོ޺ • ॴଐɿAMIMOTO TEAM • Githubɿhttps://github.com/horike37 • Qiitaɿhttp://qiita.com/horike37 • ಘҙͳ͜ͱɿAWS, WordPress

Slide 3

Slide 3 text

https://ja.amimoto-ami.com/ AMIMOTO

Slide 4

Slide 4 text

Qiita http://qiita.com/horike37

Slide 5

Slide 5 text

https://github.com/horike37/ Github

Slide 6

Slide 6 text

ࠓ೔࿩͢͜ͱ

Slide 7

Slide 7 text

AWSͰՄࢹԽΞʔΩςΫνϟΛ࡞Δٕज़

Slide 8

Slide 8 text

ࠓ೔࿩͞ͳ͍͜ͱ

Slide 9

Slide 9 text

ՄࢹԽͷ໨తɺ֓೦ɺ఩ֶ

Slide 10

Slide 10 text

ΞδΣϯμ

Slide 11

Slide 11 text

• TweetՄࢹԽπʔϧʮSocial Stream MonitorʯͷσϞ • Social Stream MonitorΞʔΩςΫνϟͷ࿩ • ϦΞϧλΠϜՄࢹԽ+σʔλͷӬଓԽ

Slide 12

Slide 12 text

Social Straem Monitor

Slide 13

Slide 13 text

Social Straem Monitorͱ͸ TwitterͷετϦʔϛϯάσʔλΛ ϦΞϧλΠϜʹՄࢹԽ͢Δπʔϧ

Slide 14

Slide 14 text

https://github.com/megumiteam/social-stream-monitor Github

Slide 15

Slide 15 text

·ͣ͸ɺσϞ

Slide 16

Slide 16 text

ΞʔΩςΫνϟ

Slide 17

Slide 17 text

Amazon   Kinesis ϦΞϧλΠϜ ετϦʔϛϯάॲཧ

Slide 18

Slide 18 text

ΞʔΩςΫνϟ ετϦʔϛϯά σʔλΛॲཧ

Slide 19

Slide 19 text

Amazon Kinesisͱ͸ • ϑϧϚωʔδυͳϦΞϧλΠϜσʔλॲ ཧαʔϏε • େྔͷετϦʔϛϯάσʔλΛγϟʔ υʹอଘɻόοΫΤϯυͷΞϓϦέʔ γϣϯͰऔΓग़ͯ͠ॲཧ

Slide 20

Slide 20 text

Amazon Kinesis͕ແ͍ͱ͖ ετϦʔϛϯάσʔλ σʔλॲཧܥ

Slide 21

Slide 21 text

Amazon Kinesis͕͋Δ࣌ ετϦʔϛϯάσʔλ σʔλॲཧܥ ॲཧܥ͕ѻ͍΍͍͢Α͏ʹσʔλΛอଘ

Slide 22

Slide 22 text

Social Stream Metor্ͷ໾ׂ • TwitterͷετϦʔϛϯάσʔλΛҰ࣌ తʹอଘͯ͠ΞϓϦέʔγϣϯ͔Βσʔ λͱͯ͠ॲཧ͠΍͘͢͢Δ

Slide 23

Slide 23 text

Lambda ίʔυͷ࣮ߦ

Slide 24

Slide 24 text

ΞʔΩςΫνϟ Kinesis͔Βड͚ औͬͨσʔλΛ SNSʹ౉͢

Slide 25

Slide 25 text

Lambdaͱ͸ • AWS ΠϯϑϥετϥΫνϟΛ࢖༻ͯ͠ ίʔυͷ࣮ߦΛ୅ߦ͢ΔίϯϐϡʔςΟ ϯάαʔϏε • ΠϕϯτυϦϒϯͳॲཧ͕Մೳ

Slide 26

Slide 26 text

SNS http endpoint΁σʔλΛ౉͢ σʔλϨίʔυΛSNSʹpublish

Slide 27

Slide 27 text

Social Stream Metor্ͷ໾ׂ • webϒϥ΢βͰදࣔ͢ΔͨΊʹhttpϓϩ τίϧͰσʔλΛྲྀ͢ඞཁ͕͋Δ • ͦͷͨΊͷSNS http push௨஌΁σʔλ Λ౉͢

Slide 28

Slide 28 text

ΞʔΩςΫνϟ HTTPΤϯυϙΠϯ τ΁σʔλΛpush

Slide 29

Slide 29 text

SNS HTTP notiﬁcation HTTPΤϯυϙΠϯτ  PUSH௨஌

Slide 30

Slide 30 text

SNS HTTP notiﬁcationͱ͸ • Amazon SNS Λ࢖༻ͯ͠ɺ1 ͭҎ্ͷ HTTPΤϯυϙΠϯτʹ௨஌ϝοηʔδ Λૹ৴ • HTTP POST ϦΫΤετΛૹ৴͠ɺ௨ ஌ͷ಺༰Λड৴ొ࿥ͨ͠ΤϯυϙΠϯ τʹ഑৴

Slide 31

Slide 31 text

SNS HTTP notiﬁcationͱ͸ Topicʹड৴ొ࿥͞ΕͨΤϯυϙΠϯτʹσʔλΛPOST

Slide 32

Slide 32 text

Social Stream Metor্ͷ໾ׂ • webϒϥ΢βͰදࣔ͢ΔͨΊʹhttpϓϩτίϧͰ σʔλΛྲྀ͢ඞཁ͕͋Δ • websocketͰϒϥ΢βʹσʔλΛૹ৴͢ΔͨΊͷ socket.ioαʔόʹσʔλΛpush͢Δ

Slide 33

Slide 33 text

socket.io ϦΞϧλΠϜͳ WebΞϓϦ

Slide 34

Slide 34 text

ΞʔΩςΫνϟ WebSocketͰϒϥ ΢βʹσʔλૹ৴

Slide 35

Slide 35 text

socket.io • ͢΂ͯͷϒϥ΢βɾϞόΠϧσόΠεͰ ϦΞϧλΠϜ௨৴ΛՄೳͱ͢Δ͜ͱΛ໨ తʹ࡞ΒΕͨJavaScriptϥΠϒϥϦ • WebSocketʹΑΔϒϥ΢βͱαʔόͷ ૒ํ޲௨৴

Slide 36

Slide 36 text

Social Stream Metor্ͷ໾ׂ • SNS͔ΒૹΒΕ͖ͯͨTweetΛWebSocketΛ࢖͍ ϒϥ΢β΁ϦΞϧλΠϜʹૹ৴

Slide 37

Slide 37 text

SNS͔ΒૹΒΕ͖ͯͨσʔλΛड৴ https://github.com/megumiteam/social-stream-monitor/blob/master/app.js

Slide 38

Slide 38 text

ϒϥ΢β΁σʔλΛૹ৴ https://github.com/megumiteam/social-stream-monitor/blob/master/app.js

Slide 39

Slide 39 text

D3.js σʔλɾυϦϒϯɾ υΩϡϝϯτ

Slide 40

Slide 40 text

ΞʔΩςΫνϟ D3.jsʹΑΔඳը

Slide 41

Slide 41 text

D3.jsͱ͸ • javascriptͷσʔλϏδϡΞϥΠθʔ γϣϯΤϯδϯ • σʔλΛجʹDOMΛૢ࡞ • άϥϑඳըϥΠϒϥϦͰ͸ͳ͍ɻD3Λ ϕʔεʹͨ͠άϥϑඳըϥΠϒϥϦ͸ଟ ਺ଘࡏ

Slide 42

Slide 42 text

ϦΞϧλΠϜνϟʔτ

Slide 43

Slide 43 text

Real-time Charting Library http://epochjs.github.io/epoch/

Slide 44

Slide 44 text

εϐʔυϝʔλʔ

Slide 45

Slide 45 text

http://iop.io/iopctrl Speedometer

Slide 46

Slide 46 text

͜͜·ͰͷྲྀΕΛ੔ཧ͢Δͱ • KinesisʹΑΓTweetσʔλΛऩू • LambdaʹΑΓKinesis͔ΒσʔλΛऔಘ ͯ͠SNS Topicʹσʔλૹ৴ • SNS HTTP notiﬁcation pushʹΑΓ socket.ioαʔόʹσʔλૹ৴ • socket.ioαʔό͔Βϒϥ΢β͔Βૹ৴ ͞ΕͨσʔλΛD3.jsͰՄࢹԽ

Slide 47

Slide 47 text

σʔλͷӬଓԽ

Slide 48

Slide 48 text

՝୊ • ϦΞϧλΠϜͳՄࢹԽʹ߹ΘͤͯɺσʔλΛ஝ੵͯ͠෼ੳΛߦ͍͍ͨ

Slide 49

Slide 49 text

DynamoDB   Stream σʔλϕʔεͷ σʔλΩϟϓνϟ

Slide 50

Slide 50 text

DynamoDB Streamͱ͸ • DynamoDB ςʔϒϧʹอଘ͞Ε߲ͨ໨ͷมߋΛɺม ߋͷൃੜ࣌ʹΩϟϓνϟͰ͖Δ Ϣʔεέʔε • ৽͍͠ը૾ΛΞοϓϩʔυ͢Δͱ͙͢ʹɺάϧʔϓ಺ ͷ͢΂ͯͷ༑ਓͷϞόΠϧσόΠεʹ௨஌Λࣗಈૹ৴ • Ϣʔβొ࿥͕͋Ε͹ɺΑ͏ͦ͜ϝʔϧΛૹ৴͢Δ

Slide 51

Slide 51 text

ΞʔΩςΫνϟ

Slide 52

Slide 52 text

ΞʔΩςΫνϟ σʔλͷӬଓԽ+ ετϦʔϛϯάσʔλ഑৴

Slide 53

Slide 53 text

ΞʔΩςΫνϟ τϦΨʔΛ༗ޮԽͯ͠ DynamoDB Streamͷ σʔλॲཧ

Slide 54

Slide 54 text

σʔλͷӬଓԽ

Slide 55

Slide 55 text

LambdaϑΝϯΫγϣϯͷτϦΨʔ༗ޮԽ

Slide 56

Slide 56 text

τϦΨʔϑΝϯΫγϣϯ SNS Topicʹσʔλૹ৴ DynamoDBϨίʔυऔಘ

Slide 57

Slide 57 text

ӬଓԽ͞ΕͨσʔλΛޙ͔ΒऔΓग़͢ DymanoDB Lambda API Gateway

Slide 58

Slide 58 text

͜͜·ͰͷྲྀΕΛ੔ཧ͢Δͱ • DynamoDBΛ૊ΈࠐΉ͜ͱͰΓσʔλ ͷӬଓԽ͕Մೳ • ӬଓԽͤͨ͞σʔλ͸ɺϨϙʔτσʔ λͱͯ͠ޙ͔ΒऔΓग़͢͜ͱ͕ग़དྷΔ

Slide 59

Slide 59 text

CloudFormation Ͱߏ੒ΛݻΊΔ

Slide 60

Slide 60 text

՝୊ • ͜͜·Ͱෳࡶͳߏ੒Λຖճ࡞Δͷ͸ݱ࣮తͰ͸ͳ͍

Slide 61

Slide 61 text

CloudFormation ΞʔΩςΫνϟΛ ΞʔΧΠϒ

Slide 62

Slide 62 text

CloudFormation https://github.com/megumiteam/social-stream-monitor/blob/master/ social_stream_processing.template

Slide 63

Slide 63 text

CloudFormationϝϦοτ • ಉ͡ߏ੒Λ͍ͭ͘Ͱ΋্ཱͪ͛ΒΕΔ • ࣅͨΑ͏ͳՄࢹԽߏ੒Λͭ͘ΔͷͰ͋Ε͹ ϑΥʔΫͯ͠վม͢Ε͹ޮ཰Α͘ΞʔΩςΫ νϟ͕૊ΊΔ

Slide 64

Slide 64 text

·ͱΊ • ࠓճ঺հͨ͠ߏ੒Λϕʔεʹ͢Ε͹TwitterҎ֎ͷՄࢹԽ΋Մೳ • AWS IoTʹ͸MQTT over WebSocketͱ͍͏࢓૊Έ͕͋Δͷ Ͱηϯαʔσʔλͷϒϥ΢βՄࢹԽ͸΋ͬͱ΍Γ΍͍͢ • ࡞ͬͨߏ੒͸CloudFormationͰݻΊͯ࠶ར༻Մೳʹ͢Δͷ ͕Φεεϝ • ࠓճ঺հͨ͠ߏ੒ͰϦΞϧλΠϜՄࢹԽ+σʔλͷӬଓԽ͕Մೳ

Slide 65

Slide 65 text

·ͱΊ http://qiita.com/horike37 ࠓ೔঺հͨ͠ΞʔΩςΫνϟͷৄࡉ͸QiitaͷํʹιʔείʔυؚΊͯॻ͍͍ͯ·͢ ڵຯ͋Δํ͸ੋඇͷ͍ͧͯΈ͍ͯͩ͘͞

Slide 66

Slide 66 text

@Amimoto_Ami amimoto-ami.com THANK YOU! Amazon Kinesis + socket.io + D3.jsΛ࢖ͬͨ webϒϥ΢βͰߦ͏ϦΞϧλΠϜՄࢹԽͷ࢓૊Έ

Slide 67

Slide 67 text

No content