Amazon Kinesis + socket.io + D3.jsを使った webブラウザで行うリアルタイム可視化の仕組み/inovationegg-8

Transcript

1. Amazon Kinesis + socket.io + D3.jsΛ࢖ͬͨ webϒϥ΢βͰߦ͏ϦΞϧλΠϜՄࢹԽͷ࢓૊Έ @Amimoto_Ami amimoto-ami.com

2. ࣗݾ঺հ • ໊લɿງՈོ޺ • ॴଐɿAMIMOTO TEAM • Githubɿhttps://github.com/horike37 • Qiitaɿhttp://qiita.com/horike37

   • ಘҙͳ͜ͱɿAWS, WordPress

3. https://ja.amimoto-ami.com/ AMIMOTO

4. Qiita http://qiita.com/horike37

5. https://github.com/horike37/ Github

6. ࠓ೔࿩͢͜ͱ

7. AWSͰՄࢹԽΞʔΩςΫνϟΛ࡞Δٕज़

8. ࠓ೔࿩͞ͳ͍͜ͱ

9. ՄࢹԽͷ໨తɺ֓೦ɺ఩ֶ

10. ΞδΣϯμ

11. • TweetՄࢹԽπʔϧʮSocial Stream MonitorʯͷσϞ • Social Stream MonitorΞʔΩςΫνϟͷ࿩ • ϦΞϧλΠϜՄࢹԽ+σʔλͷӬଓԽ

12. Social Straem Monitor

13. Social Straem Monitorͱ͸ TwitterͷετϦʔϛϯάσʔλΛ ϦΞϧλΠϜʹՄࢹԽ͢Δπʔϧ

14. https://github.com/megumiteam/social-stream-monitor Github

15. ·ͣ͸ɺσϞ

16. ΞʔΩςΫνϟ

17. Amazon 
 Kinesis ϦΞϧλΠϜ ετϦʔϛϯάॲཧ

18. ΞʔΩςΫνϟ ετϦʔϛϯά σʔλΛॲཧ

19. Amazon Kinesisͱ͸ • ϑϧϚωʔδυͳϦΞϧλΠϜσʔλॲ ཧαʔϏε • େྔͷετϦʔϛϯάσʔλΛγϟʔ υʹอଘɻόοΫΤϯυͷΞϓϦέʔ γϣϯͰऔΓग़ͯ͠ॲཧ

20. Amazon Kinesis͕ແ͍ͱ͖ ετϦʔϛϯάσʔλ σʔλॲཧܥ

21. Amazon Kinesis͕͋Δ࣌ ετϦʔϛϯάσʔλ σʔλॲཧܥ ॲཧܥ͕ѻ͍΍͍͢Α͏ʹσʔλΛอଘ

22. Social Stream Metor্ͷ໾ׂ • TwitterͷετϦʔϛϯάσʔλΛҰ࣌ తʹอଘͯ͠ΞϓϦέʔγϣϯ͔Βσʔ λͱͯ͠ॲཧ͠΍͘͢͢Δ

23. Lambda ίʔυͷ࣮ߦ

24. ΞʔΩςΫνϟ Kinesis͔Βड͚ औͬͨσʔλΛ SNSʹ౉͢

25. Lambdaͱ͸ • AWS ΠϯϑϥετϥΫνϟΛ࢖༻ͯ͠ ίʔυͷ࣮ߦΛ୅ߦ͢ΔίϯϐϡʔςΟ ϯάαʔϏε • ΠϕϯτυϦϒϯͳॲཧ͕Մೳ

26. SNS http endpoint΁σʔλΛ౉͢ σʔλϨίʔυΛSNSʹpublish

27. Social Stream Metor্ͷ໾ׂ • webϒϥ΢βͰදࣔ͢ΔͨΊʹhttpϓϩ τίϧͰσʔλΛྲྀ͢ඞཁ͕͋Δ • ͦͷͨΊͷSNS http push௨஌΁σʔλ

    Λ౉͢

28. ΞʔΩςΫνϟ HTTPΤϯυϙΠϯ τ΁σʔλΛpush

29. SNS HTTP notification HTTPΤϯυϙΠϯτ
 PUSH௨஌

30. SNS HTTP notificationͱ͸ • Amazon SNS Λ࢖༻ͯ͠ɺ1 ͭҎ্ͷ HTTPΤϯυϙΠϯτʹ௨஌ϝοηʔδ Λૹ৴

    • HTTP POST ϦΫΤετΛૹ৴͠ɺ௨ ஌ͷ಺༰Λड৴ొ࿥ͨ͠ΤϯυϙΠϯ τʹ഑৴

31. SNS HTTP notificationͱ͸ Topicʹड৴ొ࿥͞ΕͨΤϯυϙΠϯτʹσʔλΛPOST

32. Social Stream Metor্ͷ໾ׂ • webϒϥ΢βͰදࣔ͢ΔͨΊʹhttpϓϩτίϧͰ σʔλΛྲྀ͢ඞཁ͕͋Δ • websocketͰϒϥ΢βʹσʔλΛૹ৴͢ΔͨΊͷ socket.ioαʔόʹσʔλΛpush͢Δ

33. socket.io ϦΞϧλΠϜͳ WebΞϓϦ

34. ΞʔΩςΫνϟ WebSocketͰϒϥ ΢βʹσʔλૹ৴

35. socket.io • ͢΂ͯͷϒϥ΢βɾϞόΠϧσόΠεͰ ϦΞϧλΠϜ௨৴ΛՄೳͱ͢Δ͜ͱΛ໨ తʹ࡞ΒΕͨJavaScriptϥΠϒϥϦ • WebSocketʹΑΔϒϥ΢βͱαʔόͷ ૒ํ޲௨৴

36. Social Stream Metor্ͷ໾ׂ • SNS͔ΒૹΒΕ͖ͯͨTweetΛWebSocketΛ࢖͍ ϒϥ΢β΁ϦΞϧλΠϜʹૹ৴

37. SNS͔ΒૹΒΕ͖ͯͨσʔλΛड৴ https://github.com/megumiteam/social-stream-monitor/blob/master/app.js

38. ϒϥ΢β΁σʔλΛૹ৴ https://github.com/megumiteam/social-stream-monitor/blob/master/app.js

39. D3.js σʔλɾυϦϒϯɾ υΩϡϝϯτ

40. ΞʔΩςΫνϟ D3.jsʹΑΔඳը

41. D3.jsͱ͸ • javascriptͷσʔλϏδϡΞϥΠθʔ γϣϯΤϯδϯ • σʔλΛجʹDOMΛૢ࡞ • άϥϑඳըϥΠϒϥϦͰ͸ͳ͍ɻD3Λ ϕʔεʹͨ͠άϥϑඳըϥΠϒϥϦ͸ଟ ਺ଘࡏ

42. ϦΞϧλΠϜνϟʔτ

43. Real-time Charting Library http://epochjs.github.io/epoch/

44. εϐʔυϝʔλʔ

45. http://iop.io/iopctrl Speedometer

46. ͜͜·ͰͷྲྀΕΛ੔ཧ͢Δͱ • KinesisʹΑΓTweetσʔλΛऩू • LambdaʹΑΓKinesis͔ΒσʔλΛऔಘ ͯ͠SNS Topicʹσʔλૹ৴ • SNS HTTP

    notification pushʹΑΓ socket.ioαʔόʹσʔλૹ৴ • socket.ioαʔό͔Βϒϥ΢β͔Βૹ৴ ͞ΕͨσʔλΛD3.jsͰՄࢹԽ

47. σʔλͷӬଓԽ

48. ՝୊ • ϦΞϧλΠϜͳՄࢹԽʹ߹ΘͤͯɺσʔλΛ஝ੵͯ͠෼ੳΛߦ͍͍ͨ

49. DynamoDB 
 Stream σʔλϕʔεͷ σʔλΩϟϓνϟ

50. DynamoDB Streamͱ͸ • DynamoDB ςʔϒϧʹอଘ͞Ε߲ͨ໨ͷมߋΛɺม ߋͷൃੜ࣌ʹΩϟϓνϟͰ͖Δ Ϣʔεέʔε • ৽͍͠ը૾ΛΞοϓϩʔυ͢Δͱ͙͢ʹɺάϧʔϓ಺ ͷ͢΂ͯͷ༑ਓͷϞόΠϧσόΠεʹ௨஌Λࣗಈૹ৴

    • Ϣʔβొ࿥͕͋Ε͹ɺΑ͏ͦ͜ϝʔϧΛૹ৴͢Δ

51. ΞʔΩςΫνϟ

52. ΞʔΩςΫνϟ σʔλͷӬଓԽ+ ετϦʔϛϯάσʔλ഑৴

53. ΞʔΩςΫνϟ τϦΨʔΛ༗ޮԽͯ͠ DynamoDB Streamͷ σʔλॲཧ

54. σʔλͷӬଓԽ

55. LambdaϑΝϯΫγϣϯͷτϦΨʔ༗ޮԽ

56. τϦΨʔϑΝϯΫγϣϯ SNS Topicʹσʔλૹ৴ DynamoDBϨίʔυऔಘ

57. ӬଓԽ͞ΕͨσʔλΛޙ͔ΒऔΓग़͢ DymanoDB Lambda API Gateway

58. ͜͜·ͰͷྲྀΕΛ੔ཧ͢Δͱ • DynamoDBΛ૊ΈࠐΉ͜ͱͰΓσʔλ ͷӬଓԽ͕Մೳ • ӬଓԽͤͨ͞σʔλ͸ɺϨϙʔτσʔ λͱͯ͠ޙ͔ΒऔΓग़͢͜ͱ͕ग़དྷΔ

59. CloudFormation Ͱߏ੒ΛݻΊΔ

60. ՝୊ • ͜͜·Ͱෳࡶͳߏ੒Λຖճ࡞Δͷ͸ݱ࣮తͰ͸ͳ͍

61. CloudFormation ΞʔΩςΫνϟΛ ΞʔΧΠϒ

62. CloudFormation https://github.com/megumiteam/social-stream-monitor/blob/master/ social_stream_processing.template

63. CloudFormationϝϦοτ • ಉ͡ߏ੒Λ͍ͭ͘Ͱ΋্ཱͪ͛ΒΕΔ • ࣅͨΑ͏ͳՄࢹԽߏ੒Λͭ͘ΔͷͰ͋Ε͹ ϑΥʔΫͯ͠վม͢Ε͹ޮ཰Α͘ΞʔΩςΫ νϟ͕૊ΊΔ

64. ·ͱΊ • ࠓճ঺հͨ͠ߏ੒Λϕʔεʹ͢Ε͹TwitterҎ֎ͷՄࢹԽ΋Մೳ • AWS IoTʹ͸MQTT over WebSocketͱ͍͏࢓૊Έ͕͋Δͷ Ͱηϯαʔσʔλͷϒϥ΢βՄࢹԽ͸΋ͬͱ΍Γ΍͍͢ •

    ࡞ͬͨߏ੒͸CloudFormationͰݻΊͯ࠶ར༻Մೳʹ͢Δͷ ͕Φεεϝ • ࠓճ঺հͨ͠ߏ੒ͰϦΞϧλΠϜՄࢹԽ+σʔλͷӬଓԽ͕Մೳ

65. ·ͱΊ http://qiita.com/horike37 ࠓ೔঺հͨ͠ΞʔΩςΫνϟͷৄࡉ͸QiitaͷํʹιʔείʔυؚΊͯॻ͍͍ͯ·͢ ڵຯ͋Δํ͸ੋඇͷ͍ͧͯΈ͍ͯͩ͘͞

66. @Amimoto_Ami amimoto-ami.com THANK YOU! Amazon Kinesis + socket.io + D3.jsΛ࢖ͬͨ

    webϒϥ΢βͰߦ͏ϦΞϧλΠϜՄࢹԽͷ࢓૊Έ
67. None